v1 of the same-model substrate-delta harness's non-jagged metric.
For one corpus title, surface-perturb its question (numeral / accent /
hyphen / honorific / amp / brit) preserving the referent, then ask
whether retrieval surfaces the SAME target for canonical vs perturbed
phrasing. J_norm = XOR disagreement rate @k (lower = less jagged);
graded mean |Δrank| catches rank instability the binary metric misses.
Pure query --dry-run: no LLM, no verifier, no judge, no n=3 noise, no
5pp floor — the recall_at_k discipline. Reuses recall_at_k.probe +
mine_questions._surface_variant. Feeds #000012 ForkScore
ΔJaggednessReduction. A-vs-C answer-quality arm already exists under
#000057 (control_ab/control_sweep) — not rebuilt. Curvature + LLM-arm
jaggedness delta remain open (ticket §8).
make bench-jaggedness JAGGED_LIMIT=40 JAGGED_K=8
Implements fox's 2026-05-20 architecture: don't ship shards to the
tight-on-disk GPU boxes; run the benchmark FROM the laptop (local
shards + retrieval + judge + workload loop, driving the worker's LLM
endpoint over the network) and have the worker boxes REPORT their own
power.
Adds a probe abstraction with two implementations behind one
start/stop/gpu_stats/cpu_stats interface:
- LocalProbe: wraps PowerSampler + CpuSampler (watt_bench runs ON the
GPU box — the original mode)
- RemoteProbe: orchestrates bench/watt_probe.py on a remote worker
over SSH (scp the stdlib probe once, launch --until-file detached,
touch the stop-file after the workload, fetch the JSON). The worker
needs no shards / arborist / venv.
--remote-gpu-host HOST selects RemoteProbe; --endpoint points at the
worker's endpoint. Idle baseline now records gpu_util_mean_pct and
flags ">5% ⇒ contaminated by live traffic" so the 3090-style
contamination (live hermes traffic polluting the baseline) is visible
in the output, not silent.
Verified end-to-end: laptop-driven n=2 solo qwen, power sampled on the
4090 over SSH — idle gpu 20.2W @ 0% util (clean) + cpu 86W; load
populated; report tagged mode=remote. The on-box LocalProbe path is
unchanged (omit --remote-gpu-host).
Report schema gains mode / remote_gpu_host / gpu_available /
cpu_energy_available. Per-cell records gpu+cpu+total joules/question.
Splits the power-sampling half out for the laptop-driver / worker-
reporter architecture (fox 2026-05-20): don't ship shards to the tight-
on-disk GPU boxes; instead drive the benchmark FROM the laptop (which
holds the shards + judge + workload loop) and have the worker boxes
just serve the model and REPORT their own power.
Power sampling must run on the box (nvidia-smi + RAPL are host-local),
but nothing else does. watt_probe.py is stdlib-only — python3 +
nvidia-smi + readable RAPL, no venv, no arborist, no shards. Copy it to
the worker and run; it samples GPU power (nvidia-smi, trapezoid-
integrated to joules) + CPU package energy (RAPL energy-diff) over a
window and emits the same energy schema watt_bench's local samplers
produce. Two window modes: --duration (fixed) or --until-file (laptop
touches a stop-file — the workload-correlated mode). Records
gpu_util_mean_pct so live-traffic contamination can be detected.
Takes only the top-level RAPL package (intel-rapl:N, single colon),
not the :N:M core/uncore subdomains, so no double-count.
Verified on the 4090: scp'd to /tmp, ran with no install, real readings
(GPU 20.2W idle @ 0% util, CPU 82W, joules computed, 6 samples / 3.2s).
Next: laptop-side orchestration in watt_bench (--remote-gpu-host) that
SSH-starts this probe for each cell's window while the driver runs
retrieval + judge locally and drives the worker's LLM endpoint.
Toward fox's next goal: score the full serving stack on quality AND
cost — {qwen, hermes} × {llama.cpp, vLLM} × {3090, 4090} × {solo,
arborist}, measuring CG% + GPU watts + CPU watts + joules/answer per
cell.
watt_bench.py — adds CpuSampler (Intel RAPL package energy via
/sys/class/powercap/intel-rapl:*/energy_uj). RAPL exposes a cumulative
microjoule counter, so energy-over-window is an end-minus-start diff
(handles wrap) — more accurate than integrating instantaneous power.
Sums multi-package. energy_uj is root-only by default (PLATYPUS /
CVE-2020-8694), so it degrades to available=False when locked;
--cpu-energy-cmd 'sudo cat {path}' supplies a privileged reader when a
sudo rule exists. Each cell now reports gpu/cpu/total joules-per-
question + gpu joules-per-token; the report records cpu_rapl_available.
Verified: graceful degradation when locked; RAPL diff math (1->4 MJ uJ
= 3.0 J, exact).
benchmark-matrix.md — expands the cost section to the full 16-cell
(model × engine × GPU × arm) design, the per-cell metric set (quality +
GPU + CPU energy), the serving-stack inventory from 2026-05-20 recon
(4090=qwen/llama.cpp, 3090=hermes/vLLM — each box has one engine + one
model today), and the buildout gap (vLLM+qwen, llama.cpp+hermes, cross-
GPU models). Notes idle-floor asymmetry (hermes/3090 ~127W vs
qwen/4090 ~20W) as a real optimizer input.
Harness is ready; the serving-config buildout + RAPL perm grant are the
remaining (ops, fox-directed) prerequisites to run the full matrix.
Two deliverables for the cost/energy axis of the constraint optimizer.
docs/benchmark-matrix.md — shareable spec of the control experiment:
the question, fixture (386 office-holder Qs with corpus-vintage gold),
the 3-model × 3-framing × 2-arm matrix (18 cells), the verdict
vocabulary + two reads (accuracy vs grounding-fidelity), the
deterministic code judge + its Opus calibration, the results-so-far
table, and the NEW cost dimension (tokens / latency / GPU watts /
joules-per-answer measured per GPU tier). Self-contained — readable
cold by David.
bench/watt_bench.py — GPU wattage harness. Samples nvidia-smi
power.draw on the inference GPU while driving a small representative
subset, reports mean/peak watts, trapezoid-integrated joules,
joules-per-question, and joules-per-token. Tags the GPU
(--gpu-label 3090|4090) so the optimizer can compare hardware tiers.
Idle-baseline sampling separates load draw from idle. Does NOT grade
(energy is independent of correctness); saves answers + per-question
timing to JSONL for a later quality-per-joule pass via
score_with_code_judge.
Designed to run ON the GPU box (the orchestrator has no GPU; the
3090/4090 live on the inference boxes). Degrades gracefully when
nvidia-smi is absent (energy fields null) so it is testable anywhere.
Verified: PowerSampler graceful degradation + trapezoid integration
(synthetic 100->200->200W over 2s = 350 J, exact).
The headline cost finding the optimizer must weight: qwen-think
reasoning = 1300-3300 tokens/answer vs qwen-nothink ~50-100 (20-50x),
for a workload where arborist+qwen-nothink already lands 82% CG. The
energy numbers will quantify whether reasoning's premium is ever
justified — grounding-fidelity per joule, not per answer.
fox was right ("or you have defects still"): the arborist+qwen-think
empties were NOT an inherent reasoning limitation, they were
max_tokens set too low. Diagnosed with finish_reason + token usage on
a realistic large arborist-style context:
max_tokens=1024 → 4/4 empty, ALL finish_reason='length', all hit
exactly 1024 tokens
max_tokens=4096 → 0/4 empty, ALL finish_reason='stop', used
1339-3295 completion tokens
qwen-think spends 1300-3300 tokens on its (internal) reasoning trace
BEFORE emitting the tiny claim-lattice JSON. The arborist arm's budget
comes from DEFAULT_QUERY_POLICY["max_tokens"]=512 (correct for non-
reasoning single-line JSON), so reasoning refs hit finish='length'
mid-trace and return EMPTY. Fix: arb_policy["max_tokens"]=8192 for
reasoning refs (generous headroom over the 3295 observed ceiling).
This also CORRECTS a wrong claim in commit aa9d9c8's message: the
json-schema grammar does NOT suppress the reasoning trace. The
reasoning happens (it burns those 1300-3300 completion tokens); the
grammar only shapes the final emitted answer into schema-valid JSON.
So arborist+qwen-think is a genuinely distinct config, not a clone of
arborist+qwen-nothink. The earlier "grammar suppresses reasoning"
read was an artefact of toy-prompt diagnostics where reasoning fit
under 1024 tokens.
The empty-retry from aa9d9c8 stays as belt-and-suspenders for any
residual model-side empties, but with an adequate budget it should
rarely fire.
Cost note (fox's standing point, now quantified): reasoning = 1300-
3300 tokens/answer vs qwen-nothink's ~50-100 = 20-50x token cost. For
a workload where arborist+qwen-nothink already lands 82% CG, that cost
multiplier is the open question — not whether the cell is measurable
(it now is), but whether the marginal lift justifies 20-50x spend.
Two defects blocked the arborist+qwen-think cell, both diagnosed
2026-05-20:
Defect 1 — stop sequence truncates to empty. The claim_lattice path
sets stop=['\n\n'] (runaway guard tuned for single-line Hermes JSON).
A reasoning model's output trips it immediately → 100% empty answers
→ 100% ABSTAINED (measured on the first 23 items of the killed run).
Fix: MODELS gains an explicit flag; the arborist arm
clears claim_lattice_json_stop_sequences for reasoning refs. Direct
A/B confirmed: stop=['\n\n'] → ''; stop=None → valid JSON.
Defect 2 — intermittent empty completions. Even with the stop cleared,
qwen-think under json-schema grammar emits an empty completion ~1/3 of
calls (a llama.cpp reasoning+grammar artefact; qwen-nothink phase 3
had ~0 spurious empties). Fix: empty-output self-heal — reasoning refs
retry up to 3 attempts, burning the cached empty each retry. Never
fabricates: a still-empty answer after retries is recorded as empty.
6-item smoke: 0/6 (broken) → 5/6 valid JSON; residual ~1/6 are
questions that reliably break (4 consecutive empties), a documented
artefact.
Structural finding (to fold into Addendum 8): json-schema grammar
enforcement SUPPRESSES the reasoning trace — output is pure single-
line JSON, no <think> block. So arborist+qwen-think is structurally
≈ arborist+qwen-nothink; the thinking lever that moved the solo arm
is neutralized by the claim_lattice grammar. The re-run will confirm
empirically.
Non-reasoning refs (hermes, qwen-nothink) unchanged: single pass,
stop sequence intact.
fox 2026-05-19: 'we don't need to redo anything'. The arborist+qwen-
nothink sweep would re-run qwen-nothink solo on 1158 records that
phase 1 already produced (control_sweep_2026-05-19T21-52-56Z.jsonl,
calibrated rescore on disk). Wasteful — qwen-nothink solo behaviour
doesn't depend on retrieval being on, so phase 1's numbers stand.
--skip-solo runs ONLY the arborist arm. Implementation: skip the
per-variant model loop in _process_item when the flag is set; the
arborist arm below it still runs if arborist_on. The spend banner
zeroes the solo-call count so the operator sees the actual LLM
budget for the arborist-only run.
Example: a full arborist+qwen-nothink sweep against the 386-item
fixture goes from 2316 LLM calls (1158 solo + 1158 arborist) to
1158 calls (arborist only). Halves wall-clock on single-worker
llama.cpp.
pytest 27/27, imports clean, --help shows the flag.
Reorder rule 3 (short-answer entity grounding) above rule 4 (NLI
contradiction) so positive lexical evidence cannot be overridden by
NLI clause-level noise. Surfaced by the 2026-05-19 arborist+qwen-
nothink smoke:
i=3 · who is the prime minister of Poland?
ans: 'Donald Tusk is listed as the Prime Minister of Poland.'
gold: ...lists Tusk + Marcinkiewicz + Belka + Kaczynski + Kopacz...
NLI contradiction p=0.892 (above 0.85 threshold)
NLI entailment p=0.744 (also high on the correct clause)
Tusk WAS PM in 2010 (served 2007-2014); answer is correct against
the corpus-vintage gold. The NLI contradiction signal came from
clause-level candidate selection picking a NON-Tusk PM the source
also mentions; entailment was high on the Tusk clause. Mixed signal
that the WRONG rule then over-confidently resolved.
The fix is a rule reorder, not a threshold change — the fast path's
positive-evidence combination (specifics-in-gold AND subject-in-gold)
is a strictly stronger signal than NLI's clause-level max
contradiction, so when it fires it should win. The combination
discriminates Poland-Tusk (Tusk ∈ gold, Poland ∈ gold → CG) from
Anthony-Albanese (Albanese ∉ gold → fast path declines → falls
through to UNGROUNDED-subject-in-gold → WRONG, unchanged).
Self-test 4/4 INSTRUMENT TRUSTWORTHY unchanged. pytest 27/27.
Poland-Tusk regression smoke: now CG via short_entity_grounded ✓.
No regression risk on the existing reconciliation cells:
- Iceland CG: short_entity_grounded was already winning (was rule
4, now rule 3 — same outcome, earlier exit)
- WWII-1812 WRONG: '1812' ∉ gold → fast path declines, NLI fires ✓
- Higgs-cafe FABRICATED: 'Higgs' ∉ gold → fast path declines ✓
- Anthony Albanese WRONG: 'Albanese' ∉ gold → fast path declines ✓
- Abstention phrases: rule 2 still fires first ✓
Two surgical fixes unblock 'arborist with synthesis LLM = Qwen-on-
llama.cpp' as a viable arm in the control sweep. Pre-existing
docstring said 'Arborist×Qwen needs proof-path guided_json+extra_body
surgery — coupled follow-up'; this is that follow-up.
Fix 1 — multi-engine structured-output extras
The runner / query JSON-mode paths previously sent only vLLM's
'guided_json' key for the claim_lattice schema. llama.cpp silently
drops it, leaving Qwen un-enforced (the parse-tolerant fallback did
all the work). Helper
claim_lattice_structured_output_extras() in arborist/qa/verify.py
now returns a dict carrying the schema under all three engine
conventions:
- guided_json (vLLM grammar-constrained sampling)
- json_schema (llama.cpp native shorthand)
- response_format (OpenAI-spec, honoured by llama.cpp and newer vLLM)
Each engine recognises its own key and silently drops the others.
Used at both inference call sites (runner.py:740, query.py:3324).
Hermes/vLLM path is unchanged — it picks up 'guided_json' and
ignores the other two.
Fix 2 — query() accepts user-supplied extra_body, merges with defaults
query() grew a keyword-only extra_body parameter (default None).
Per-model knobs (Qwen's {'chat_template_kwargs': {'enable_thinking':
False}} toggle, future template knobs) can flow from the caller to
the synthesis chat-completion call. Schema-enforcement extras are
added inside query() and merge under user keys — common case is
disjoint namespaces, but if a caller wants to override 'guided_json'
they can.
bench/control_sweep.py now passes MODELS[arborist_ref]['extra']
through to query() in the arborist branch, so --arborist-ref
qwen-nothink runs with reasoning disabled and --arborist-ref
qwen-think runs with reasoning enabled. Phase 1's arborist arm with
--arborist-ref=hermes is unaffected (MODELS['hermes']['extra'] is
None, merges to no-op).
Tests
+ 3 new in tests/test_verify_json.py covering helper default shape,
alternate-schema reuse, and query()'s new extra_body parameter
220 affected tests still green (verify / claim_lattice / judge /
runner suite)
pytest test_verify_json: 27/27
Next: small smoke run --arborist-ref qwen-nothink against 4-8 items
to confirm end-to-end before any full sweep. Phase 2 (qwen-think solo)
still running in background, unaffected — it doesn't touch the
arborist arm.
The Arborist arm runs answer_mode='claim_lattice' (per control_sweep.py
:179, control_ab.py:155) so its answers arrive as the JSON envelope
{"claims":[{"text":"...","evidence_ids":["E1"]},...]}.
_descaffold strips the [E1] evidence-pointer markup but the JSON
braces + key syntax remain. The verifier's strategy-2 (span) and
strategy-3 (proper-noun) extractors see brace noise instead of the
inner claim prose — every Arborist record degraded to UNGROUNDED.
The 2026-05-19T17-01-17Z sweep, re-graded with the freshly calibrated
judge (5a17f61), surfaced this: Arborist arm reported 0 CG across all
three variants in the live phase 1 output (the live run was pre-
calibration), and 29/120 CG (24%) under the calibrated rescore — clear
improvement just from theta_contra=0.85, but the JSON envelope was
still hobbling the verifier paths.
Fix: _unwrap_claim_lattice_json runs BEFORE all downstream rules.
Detection is conservative (three independent signals: starts-with-
brace AND "claims" key AND "text" key) so plain-prose answers
pass through unchanged. Multi-claim envelopes concatenate as discrete
sentences (extract_claim_spans treats each as its own span).
Malformed JSON falls back to the original answer — no silent
rewriting on broken input.
Smoke result on the Iceland Arborist case
ans: {"claims":[{"text":"The current president of Iceland is
Ólafur Ragnar Grímsson.","evidence_ids":["E1"]}]}
gold: {{Infobox Political post |post = President |body = Iceland
|incumbent = [[Ólafur Ragnar Grímsson]] ...}}
before: UNGROUNDED → FABRICATED (then WRONG after calibration)
after: short_entity_grounded → CORRECT_GROUNDED
pytest: 27/27 (added 7 unwrap-coverage tests covering single-claim
envelopes, multi-claim concatenation, plain-prose passthrough,
malformed-JSON tolerance, unrelated-JSON passthrough, and the
end-to-end Arborist-envelope CG flow). Self-test 4/4 unchanged.
Re-rescores of 17:01 sweep + phase 1 sweep run after this commit
to measure final Arborist scorecard improvement.
Opus is trusted; this commit closes the systematic gaps surfaced by
B's rescore on the 17:01 sweep (2289 records, of which 468 had real
Opus verdicts). Off-diagonal cells in descending size and the
root-cause fix for each:
cell n=114 code:WRONG · opus:CORRECT_GROUNDED
Root cause: NLI fires contradiction p in [0.5, 0.75] on factual
answers like 'Ólafur Ragnar Grímsson is president of Iceland'
against wikitext-shaped infobox gold — clause-level candidate
selection picks up co-mentioned earlier office-holders, reads
temporal-frame mismatch as contradiction. The TRUE contradiction
signal (WW2 1812 self-test fixture) measures p=0.985 — clean
margin above noise.
Fix: raise theta_contra 0.5 → 0.85 (code-judge override of the
NLI manifest's 0.5 contradiction_veto).
cell n=60 code:FABRICATED · opus:WRONG
Root cause: 'Anthony Albanese' answer vs Julia Gillard gold gets
FABRICATED (specifics-not-in-gold) but Opus correctly distinguishes
WRONG (source contradicts by naming someone else) from
FABRICATED (source silent on topic).
Fix: when verifier UNGROUNDED + specifics not in gold AND the
question's subject anchor IS in gold, demote FABRICATED → WRONG.
Subject anchor uses proper-noun-shaped terms from the question
(Iceland / Australia / Higgs) — not the last-content-token
heuristic, which mis-fires on coincidental matches like 'cafe'
appearing in a 'gold does not mention any cafe' denial.
cell n=18 code:WRONG · opus:ABSTAINED
Root cause: abstention patterns missed Hermes's most common
refusal phrasings — 'I do not have accurate information', 'I do
not have access to a reference knowledge base', 'I lack access
to'. Original patterns required determine/know/tell verbs right
after 'do not'.
Fix: three new patterns for the 'do not have / lack ...
information / access / knowledge' family.
cell n=13 code:ABSTAINED · opus:CORRECT_GROUNDED
Root cause: verifier's strategy-2 needs prose shape; terse-name
answers ('Pratibha Patil', 'Jalal Talabani') fall to
UNGROUNDED-no-specifics → ABSTAINED, missing valid CG.
Fix: short-answer entity-grounding fast path. When answer is
short (≤15 tokens) AND every specific asserted is present in
gold (no unsourced) AND at least one specific WAS asserted AND
the question's subject anchor is in gold → CG. Guards against
'wrong topic, right name' false-positives via the subject check.
Structural reorder: NLI contradiction now runs AFTER the
abstention check and short-answer fast path (instead of preempting
the verifier), so the verifier's STRICT/HYBRID positive signal
isn't overridden by NLI noise. NLI still leads the path on truly
unbounded answers — verifier UNGROUNDED + NLI ≥ 0.85 contradiction
keeps the WRONG label.
Self-test 4/4 INSTRUMENT TRUSTWORTHY. pytest contract 18/18.
v2 rescore on the same 17:01 sweep runs in the background to
measure agreement-matrix improvement empirically.
Added bench/analyze_judge_disagreement.py — the harness that drove
this calibration (reads B's rescore JSONL, bucketises off-diagonal
cells, dumps configurable samples per cell with question / answer /
gold / both rationales). Reusable for the next calibration round.
Reads an existing sweep JSONL (Opus verdicts already recorded), re-fetches
gold per record via _gold(), runs the code judge on (question_asked,
answer, gold), and emits:
- markdown scorecard: agreement matrix (code × original judge), per-arm
/ per-model / per-variant code-judge tallies, and a residue table of
the JUDGE_ERROR records (the natural input to a later LLM-batch
needle-haystack pass — Opus or Grok);
- JSONL with one row per sweep record (code_verdict + code_rationale
+ code_decision), joinable on (i, arm, model, variant) to the
source sweep.
Zero LLM calls. Reads sweep JSONL + shards read-only. Pairs with the
new --judge switch (a2e9b49): the switch decides what NEW data uses;
this script decides what the ALREADY-COLLECTED data looks like under
the deterministic judge.
Usage (parameter default matches control_sweep.py default fixture):
python -m bench.score_with_code_judge --in <sweep>.jsonl
Currently running against control_sweep_2026-05-19T17-01-17Z.jsonl
(the 2289-record sweep that ran on the prior huge-N pass before the
Opus quota burned out). Output will land at
bench/qa_results/control_sweep_2026-05-19T17-01-17Z_code_judge.{md,jsonl}.
Wire bench/judge_code.py into the sweep harnesses as the default judge.
Both control_sweep.py and control_ab.py grow a --judge {code,opus} CLI
arg; both share the same Verdict shape so the dispatch is a pointer
assignment + threading the judge_fn through _process_item.
Behaviour:
- DEFAULT = code: zero LLM, zero quota, deterministic. Self-test gate
is the code judge's 4-fixture contract.
- --judge opus: original gated Opus path; needs ARBORIST_JUDGE_ENABLE=1
set per 1cabfe6's fail-closed gate, otherwise every record returns
JUDGE_ERROR with rationale 'disabled — set ARBORIST_JUDGE_ENABLE=1'
and the sweep records that label honestly.
Reporting:
- Header line now records which judge ran ('Judge = code (...)' or
'Judge = opus (...)') so partial-reports & resumes don't lie about
provenance.
- Spend banner shows '0 LLM calls' for the code path so the no-burn
property is visible in the operator output.
Test surface: pytest sweep across tests/ still 136/136 (no regressions);
new --judge flag visible in --help on both harnesses.
Next: bench/score_with_code_judge.py to re-grade existing sweep JSONLs
(written under the gated-Opus run) with the code judge; agreement
matrix surfaces residue size for the eventual LLM-batch needle-haystack.
bench/judge_code.py — drop-in alternative to bench/judge.py with the
same Verdict shape & closed verdict vocabulary (CG/W/F/A/JE) but zero
quota cost: composes verifier + NLI + abstention + specificity into a
fixed-order pipeline. fox 2026-05-19: 'data first, judging later' —
this is the data-collection arm; LLM-based judging (Opus batched
needle-haystack, or Grok credit-card) is a separate downstream
concern that operates on the residue this judge cannot classify
deterministically.
Pipeline (first hit decides):
1. empty / no-gold guards
2. explicit abstention phrases (lexical regex)
3. NLI contradiction (arborist.qa.nli.shadow_check) — strongest
signal: gold contradicts the claim → WRONG
4. lexical verifier (arborist.qa.verify.verify_quotes) →
STRICT → CORRECT_GROUNDED
HYBRID + NLI entail >= 0.55 → CORRECT_GROUNDED
UNGROUNDED + specifics-not-in-gold → FABRICATED
UNGROUNDED + no specifics → ABSTAINED
HYBRID without NLI corroboration → JUDGE_ERROR (residue
for an LLM judge)
Threshold note: _CODE_JUDGE_THETA_ENTAIL_CORROBORATE=0.55 is distinct
from the NLI manifest's entailment_block_veto=0.9. The manifest's
threshold is calibrated for OVERRIDING a STRICT lexical signal with
negative evidence — high bar. The corroboration use here is the
opposite direction: additive positive evidence on an already-positive
anchor — moderate bar appropriate. Self-test case 1 measures NLI
entail=0.769 (clearly entailed, clear margin above 0.55).
Specificity for FABRICATED layers three scanners:
- verifier's multi-word proper-noun extractor (Higgs Boson, ...)
- local single-word capitalised-token scanner (Napoleon, Mars, ...)
deliberately separate because the verifier's gate is conservative
by design (multi-word only)
- numerics (years, dates, large counts, money)
Self-test: same 4 fixtures as bench/judge.py:self_test() so the two
instruments can be cross-checked when fox re-fires the Opus judge on
the residue later. Result: 4/4 INSTRUMENT TRUSTWORTHY.
tests/test_judge_code.py — pulls the contract into make test
(18 cases): module identifiers pinned, dataclass shape parity,
empty / no-gold guards, parametrised abstention phrases, specificity
layer behaviour, the canonical 4-case self-test, batch helper, and
graceful NLI-unavailable degradation. 18/18 pass.
Pre-existing known limitation, documented in the docstring: terse
correct answers ('In 1945.' against gold containing '1945') route to
ABSTAINED because the verifier's span extractor needs prose shape;
NLI sees no clause-level overlap at very short claims. The conservative
ABSTAINED label is correct deferral; tuning this is a calibration
question for real bench data, not the instrument's contract.
No callers touched yet — control_sweep.py & control_ab.py still
import the disabled Opus judge. Wiring this in is a separate ticket
move per fox's data-first sequencing.
2026-05-19: huge-N #000057 control sweep (f63b00d → 9dc02e4) burned
our Opus quota. Disable judge.py by default so a stray re-run can't
re-burn — every call short-circuits to JUDGE_ERROR with rationale
'disabled — set ARBORIST_JUDGE_ENABLE=1 ...' and zero subprocess
spawn (0ms in the disabled path, smoke-tested).
Why a gate, not a model swap:
- judge.py uses Opus deliberately as EXTERNAL SOTA outside both arms;
swapping the judge to Hermes/Qwen would corrupt the experiment
(Hermes is itself an arm under test). The hygiene comment at
judge.py:26-29 already names same-family-judging as the live threat
to validity at Opus level; downgrading further changes the science.
- Gating instead preserves the science when fox re-enables, and gives
us the data-first workflow he asked for: deterministic tool
pre-filters (verifier / NLI / recall@k) up front, judge only on
residue worth Opus tokens, with explicit go.
Behaviour:
- control_sweep.py + control_ab.py already treat JUDGE_ERROR
non-fatally (counted as JE in _bucket); disabled runs degrade to
100% JE in the tally and surface the disable reason in rationale —
the loudest possible 'judge did not run here' signal.
- Re-enable per-run: ARBORIST_JUDGE_ENABLE=1 python -m bench.control_sweep ...
- self_test() will report 4× JUDGE_ERROR when gated — intentional;
if the instrument is off, the self-test must NOT silently pass.
Smoke-test (without flag): label='JUDGE_ERROR' rationale='disabled — ...'
dt=0.0ms · no claude subprocess spawned.
Cross-referenced from CLAUDE.md '## Live endpoints' /
'Budget discipline' subsection added in 2365bd1.
fox: 'not 11 hours it shouldn't take that long'. Probed endpoints —
Qwen-27B absorbs 8 concurrent with 0 errors (0.5->2.1s); the
bottleneck is the serialized claude -p Opus judge, so more workers
≈ near-linear speedup. control_sweep.py gains --resume PATH: appends
to an existing JSONL, skips items already COMPLETE (full
models×variants for solo + variants for arborist if i<=arborist-n),
re-runs partial items; _aggregate now dedupes (i,arm,model,variant)
last-wins so a killed-mid-unit restart never double-counts, and
_load_recs tolerates a truncated trailing line from the kill.
Makefile control-sweep gains CONTROL_SWEEP_WORKERS / _RESUME / _ARB_N
so make stays the interface. 6-worker run killed cleanly (specific
pids, no pkill), relaunched resume @ 12 workers — 12 done items
preserved, 374 to run, ~5h -> ~2-2.5h.
fox: 'make the n huge huge, check in every ~7 turns'.
- stale fixture re-mined to the FULL pool: 386 questions (180 pres /
119 PM / 34 premier / 20 gov-gen / 17 chancellor / 15 CM / 1 FM),
deterministic.
- control_sweep.py rebuilt: ThreadPoolExecutor work-unit-per-item
(per-unit qa_db -> zero SQLite write contention; shards read-only
safe under concurrent readers), incremental flushed JSONL, and a
--report-only PATH mode that aggregates a PARTIAL file with ZERO
LLM/judge spend (the interim check-in path).
- huge N goes on the CONTROL (solo x3 models x3 framings) where the
open statistical question lives; Arborist A/B = fixed --arborist-n
(default 40), not re-measured 386x (power belongs on the control,
and query() over ~40GB shards is the heaviest call).
- bench-maxing doctrine applied: independent hermetic judge calls
fanned out, 'serial-by-caution is halting in disguise'.
N=3 (prior run) already shows the coherent, review-relevant story:
plain -> all 3 models confidently assert (0% abstain); source_relative
-> abstain jumps (hermes 33 / qwen-think 67 / qwen-nothink 100%);
as_of_corpus -> ~100% correct all models. The gap is largely
framing+snapshot, not a universal capability deficit.
fox ruling: 'we both do not know which framing is right, measure all
benchmarks and bring results forward for review'; 'we have qwen with
and without reasoning to use'.
bench/control_sweep.py: sweeps the CONTROL (solo) arm across
{Hermes-3-8B, Qwen3.6-27B reasoning, Qwen3.6-27B no-reasoning} ×
{plain, source-relative, as-of-corpus-era} questions, judged vs the
fixed corpus-vintage gold; Arborist-Hermes treatment reference run
alongside at the same N. Presents the SAME judge verdicts under both
the accuracy framing (the naive read fox flagged as unfair-as-truth,
shown for contrast) and the grounding-fidelity framing (the
defensible read), plus an explicit note on the faithfulness-ablation
framing + Arborist×Qwen — both deliberately NOT run (proof-path
surgery / different instrument, not a relabelling).
Answers fox's open question empirically: does a 27B *reasoning*
control honestly ABSTAIN where the 8B fabricates (gap = weak-small-
model artefact) or still confidently assert post-corpus (gap = real,
scale-independent)? as_of_corpus separates 'can't recall the era'
from 'won't constrain to a source'.
Qwen toggles probed live: reasoning answer in message.content (CoT in
separate reasoning_content, not surfaced — logged limitation);
no-reasoning via chat_template_kwargs{enable_thinking:false}. Gate =
in-script judge self_test (aborts on fail) — stronger than a make
edge, no double self-test spend.
Opt-in third NLI inference backend (ARBORIST_NLI_BACKEND=tinygrad) in
qa/nli/shadow.py, parallel to torch/onnx-int8, behind the #000049
cage: shadow-only, never an audit_mode input, never auto-preempts the
proven path (guarded so it cannot regress torch/onnx). Loads the ONNX
export through tinygrad's frontend wrapped to the existing
model(**enc).logits contract so _nli_batch is byte-unchanged.
bench/scripts/nli_backend_ab.py + make bench-nli-backends: deterministic
A/B, gate is numerical agreement with the torch reference first,
latency second (a divergent engine = a different shadow signal = a
different nli_policy_hash). Instrument is honest — reports a requested
backend as unavailable rather than relabelling a fallback's numbers.
First CPU-smoke run already quantified that the deployed §7 #22 int8
export diverges Δmax≈0.42 from torch — the immunity property made
measurable, not a defect. Real tinygrad numbers pending a producer-box
run (tinygrad not an arborist dep; frontend op-coverage for the large
MNLI checkpoints unverified by design).
docs/onnx-vendor-capture-immunity.md: why the model-in-proof-path cage
makes the inference engine an interchangeable sidecar, never a trust
dependency — public-domain positioning capital. Indexed in CLAUDE.md.
Full suite 2498 passed (identical to baseline); 24/24 NLI tests green.
control_ab.py called client.chat_completion(messages) but model is a
keyword-only REQUIRED arg → TypeError every call → solo answer was
always '[solo-error: TypeError]', which the hermetic judge correctly
classified ABSTAINED. Every prior 'Hermes-solo abstained' smoke
(incl. the n=2 the ticket cites as a case against the treatment) was
a broken-control-arm artefact, not a measurement. Fix: pass
model=a.model (symmetry with the Arborist arm). Error token now
carries the exception message so the next failure self-describes.
Also adds the #000057 'stale' fabrication-bait mining class
(time-varying executive offices; corpus = ~2010-2011 Wikipedia
VERIFIED from the artifact, NOT the 2003 dump CLAUDE.md names —
flagged) and restores the numeral fixture deterministically.
b573c59's accent-fold work removed 210 lines from
qa_questions_numeral_map.json (40 -> 5 entries) — collateral of an
earlier `mine_questions.py --cls numeral --limit 5` regression-check
that got committed. The control-ab run scored n=5 (not the --n 12
passed) because the fixture itself only had 5 rows. Re-mined at the
default --limit 40; deterministic mine yields a byte-identical
restoration of the a3ac653 original. Caught by reading the artifact,
not assuming.
bench/control_ab.py + `make control-ab` (gated on judge-self-test
as a make dependency — instrument gate cannot be skipped). Same
model both arms; gold = target-article text by target_root; Arborist
[E…] scaffolding stripped (blinding — format can't betray the arm);
Arborist UNGROUNDED credited as honest abstention; hermetic Opus
judge; deterministic aggregate; self-auditing JSONL; threats-to-
validity printed in the report.
N=2 smoke: clean end-to-end, 0 JUDGE_ERROR — and already surfaced a
case AGAINST the treatment (solo correctly ABSTAINED; Arborist
HYBRID-WRONG). The instrument can falsify the Arborist value claim;
that is the point. n=2 proves nothing (report says so) — verdict
needs a real N.
fox ruled judge = Opus via `claude -p`. bench/judge.py:
hermetic (`env -u CLAUDECODE claude -p`, fresh process, context =
only (Q, answer, gold) — no arm label, no Arborist context, no
session), blinded-by-caller, reference-grounded against the fixed
gold (ignore parametric knowledge), structured via FINAL_VERDICT=
sentinel parsed LAST-match.
Instrument-before-experiment gate worked: first cut parsed
first-match over the model's chain-of-thought → 0/3 self-test. The
judge REASONED correctly; the parser was the defect (+ two bad test
fixtures, my error). Hardened (sentinel contract + fixed fixtures),
re-verified: `make judge-self-test` = 4/4 on known-verdict triples
via real claude -p. The make target is the precondition gate; no
control run trusts the judge until it passes.
Threat to validity recorded, not hidden: same model family judging;
mitigated (blind + no-stake + reference-grounded) not eliminated —
different-family SOTA cross-check is the only full removal.
Next: bench/control_ab.py + `make control-ab` (Hermes-solo vs
Arborist, gold=target-article text, blinded, judged) — NOT yet
built; no broken make target shipped for it.
Second MEASURED fold-search win, and the instrument correcting my own
premature call. accent-fold ON vs OFF on the mined accent fixture:
recall@1 55% -> 85% (+30pp), rank-1 22/40 -> 34/40. recall@8 was
flat (95->98) — a too-lenient k nearly got a real lever wrongly
reverted; @1/@3 is the resolution that drives primary-source
selection. _accent_fold_variants: ASCII-fold then re-tokenise so a
diacritic title ("Béla Bartók", which _TITLE_TOKEN_RE otherwise
fragments to junk) matches the ASCII form a user types. Additive+
symmetric, no-op on pure-ASCII (zero effect on non-accent
queries/titles), mirrors _hyphen_fold_variants (#000007).
Also fixes a defect I shipped in a3ac653: an orphaned duplicate
body left as dead code after `return base` in _title_query_tokens
(unreachable — numeral-fold behaviour/measurement were valid — but
cruft; removed).
Fold-search factory, fanned out across the full survey backlog
(deterministic recall, no LLM, parallel — serial-by-caution was
halting in disguise):
- recall_at_k.py: returns rank -> recall@1/@3/@k from one retrieval
(verified offline). A coarse k hides rank-only lifts.
- mine_questions.py: numeral/accent/hyphen/honorific/amp/brit
ground-truth classes; fixtures committed.
- Measured @1 headroom verdicts: accent SHIP (this commit);
honorific 45% / brit 50% = real headroom (build next); hyphen
90% = existing #000007 already delivers, NOTHING to build (the
measure-the-unmeasured-thing check pays off); amp 82% = no fold
needed (prevalence-overranked, instrument kills it cheaply).
CLAUDE.md bench-maxing: two measured lessons codified — report
recall@1/@3/@k (a lenient k hides rank lifts; prevalence != miss-
rate), and fan out independent measurements (serial-by-caution is
halting). Full suite 2488 passed, 0 regressions (accent-fold is
hot-path in _title_query_tokens); real-path test (FakeSource->
ingest->query()->real _Hit).
The first MEASURED, above-noise retrieval win this thread. The 75-q
n=3 audit_mode bench couldn't resolve any single lever (every failure
class <=3-5 q, sub the 5pp floor — four hypotheses died there). Fix
the instrument, not just the lever:
- bench/mine_questions.py + bench/recall_at_k.py: mine questions from
corpus titles (ground-truth target known by construction), grade by
deterministic retrieval recall@k via `query --dry-run` — no LLM, no
verifier, no n=3 noise, scalable to the 22K-deep numeral pool. The
curated qa_questions.txt stays the separate verifier-honesty/trap
gate; mined fixtures measure the answerable long tail per class.
- _numeral_fold_variants in query.py: ordinal-word ("Alexander the
second") <-> multi-char Roman ("Alexander II"), additive+symmetric,
unioned into _title_query_tokens exactly like _hyphen_fold_variants
(#000007). Strict 2..40 Roman set → no English-word collision;
single-char Romans (I/V/X) intentionally out of scope (universal
len>1 token filter — stated before building, ~4 of 10 residual
misses).
Measured on the mined numeral fixture: recall@8 22/40 (55%) -> 30/40
(75%), +20pp; 20 hits now rank-1. Discipline applied end to end:
measured-first, mirrored precedent, full-suite regression run (2482
passed, 0 regressions — numeral-fold is hot-path in
_title_query_tokens), real-path test (FakeSource->ingest->query()->
real _Hit, not a hand-built object), measured-after on a noise-free
instrument. The ~6 multi-char residual misses are a different
downstream cause the instrument now exposes for future iteration.
Three workstreams, full suite 2482 passed, experimental paths default-OFF.
#000055 — Windows quickstart without make
tasks.py (pure-stdlib runner) + make.bat shim + .gitattributes;
README Windows section rewritten. Quickstart needs only Python
3.10+ (no make/bzip2/curl/bash). Mirrors the Makefile quickstart
subset; drift-pinned by tests/test_tasks_runner.py.
#000001 §7 Phase 0 — deterministic cross-language guard
arborist/qa/crosslang.py: non-English signal (¿/¡/non-ASCII) + an
es function-word stoppack. Fail-closed to UNGROUNDED before
retrieval/LLM (mirrors the quantifier reject-DAG) when no content
token survives, else strips es stopwords from the retrieval query
only. English path byte-identical by construction. Default OFF
(crosslang_guard_enabled). Measured: the anarcocapitalismo field
case 10.4s -> 1.6s.
#000056 — Operation Sandwich (cross-language grounding)
arborist/qa/mt/: opus-mt es/fr/ru<->en, lazy per-pair memoised
singleton (fixes the 88%-engine-error concurrency defect),
manifest-pinned, [mt] extra; entity_mask wrapper. Sandwich =
translate query in (retrieval + LLM prompt) -> English answer ->
UNTOUCHED verifier grounds English-vs-English -> translate the
verified answer out as display-only (banner-labelled, zero
grounding). question_hash + verifier_policy_hash invariant; MT
engine identity binds into RetrievalPlan, not governance. CLI
--crosslang-translate / make XLANG_MT=1. Default OFF; entity_mask
default OFF (measured net-negative at bench scale). Fan-out bench
(bench/*.py): Spanish ~0% -> 71% grounded vs the real no-support
baseline; the round-trip predictor was tried and refuted; the
entity-mask lever failed at scale (corpus-title anchoring untried).
CLAUDE.md: cross-language bright-line convention + module map.
Pre-existing modified diagram files are intentionally excluded.
Ran diagnose_deflection and diagnose_coherence on the 20+20 NEG
fixtures from step 3 parts A+B, plus all 808 STRICT.
Result: 0/40 NEG fire on diagnose_deflection (because the fixtures
are token-overlap-correct by construction — the question's subject
appears in the answer; that's exactly the failure mode #000052 was
built to catch beyond lexical). 0/40 on diagnose_coherence (the
fixtures are well-formed sentences).
So:
UNION (relevance OR deflection) = relevance alone (no lift)
INTERSECT (relevance AND deflection) = 0/40
Multi-signal combination doesn't help on these failure shapes.
But the architectural finding is positive in a different way: the
sidecars and the relevance reranker cover NON-OVERLAPPING failure
shapes cleanly:
- diagnose_coherence: structural breakage (word-salad, vacuous,
phrase-component-reuse). Owns the 'incoherent answer' slice.
- diagnose_deflection: token-overlap mismatch (subject anchor not
in answer). Owns the 'wholesale topic drift' slice.
- bge-large relevance: semantic aboutness mismatch despite shared
tokens. Owns the 'topic-collision / mis-cite / on-topic-but-
not-answering' slice — what §3.2 was built for.
Each signal owns its own slice; combining is redundant on these
cases. That's the architectural validation of the §3.1 + §3.2 +
existing-lexical-sidecars split as orthogonal, not overlapping.
Manifest runtime_viability.as_multi_signal_factor updated from
'viable' to 'TESTED — does not lift; sidecars are complementary
not combinatorial on these shapes'. Cleaned up stale
nli-shadow-grid-n1-minilm.json.
Also cleaned up a stale nli-shadow-grid JSON.
The §3.2 arc is now complete:
step 1 candidate-bench (overclaim, contrived data)
step 2 real STRICT FP (over-pessimistic 'NOT VIABLE')
step 3A real-context deflection (positive reversal, 55% at fp=0)
step 3B real-context mis-cite (100% at fp=0, motivating failure covered)
step 3C multi-signal combination (no lift; clean architectural split)
The recommended operating point holds: bge-reranker-large + cleaned
+ θ ≤ -2.42 → 100% mis-cite, 55% deflect, 0% STRICT FP.
Built bench/fixtures/5f/relevance-miscite-realcontext-v1.jsonl —
20 hand-crafted (claim, source) mis-cite pairs: claim about X, source
about Y, X≠Y but shared tokens. Each pair survives the lexical
title-relevance + verifier sidecars by construction. This is the
Zionist-entity failure mode (claim about a different entity than the
cited source, both lexically related).
Examples:
- 'Mercury is the smallest planet' / Roman-god Mercury source
- 'Java is a programming language' / Java-the-island source
- 'Apple Inc. was co-founded by Steve Jobs' / apple-the-fruit source
- 'The Eiffel Tower is in Paris' / Gustave-Eiffel-person source
- 'Mozart composed The Magic Flute' / Mozart-effect-theory source
Headline (bge-reranker-large + cleaned + θ ≤ -2.42):
Mis-cite catch: 20/20 = 100% ← FULL COVERAGE of motivating shape
Deflection catch: 11/20 = 55%
Combined NEG: 31/40 = 78%
Real-STRICT FP: 0/808 = 0% ← strictly safe
bge-large mis-cite scores: -9.37 to -3.81 (max). STRICT min: -2.42.
Mis-cite is STRICTLY SEPARABLE from real STRICT — there's a 1.4-pt
gap with no overlap. (Deflection harder; some overlap with weak STRICT.)
MiniLM-L-6 cost-pick (5× smaller, cleaned, θ=+3):
100% mis-cite + 65% deflect + 0.4% STRICT FP
Mis-cite is structurally MUCH easier than deflection — both models
hit 95-100% mis-cite catch at modest θ; deflection is harder because
'answer doesn't quite address question' can look like a weak STRICT.
That's appropriate: mis-cite is 'wrong topic entirely'; deflection
is 'right topic, not answering'.
Manifest:
- runtime_viability flipped (step 2 → step 3): NOT VIABLE → VIABLE
at the bge-large fp=0 operating point.
- demote_below_score still null pending fox+dav1d sign-off (setting
it folds relevance_policy_hash into governance_policy_hash per
#000049 §7 #2).
- PRIMARY RECOMMENDATION: bge-reranker-large + cleaned +
θ ≤ -2.42 → 100% mis-cite, 55% deflect, 0% STRICT FP.
Production verifier unchanged; still SHADOW. The §3.2 arc went:
candidate-bench (overclaim) → real STRICT step 2 (over-pessimistic
NOT VIABLE) → real-context deflection step 3 (positive reversal,
55% at fp=0) → real-context mis-cite step 3 part B (full closure
at fp=0 for the motivating failure mode). Eight meta-lesson
instances over the §000049 + §000052 arc, with the sharpest one
yet: clean candidate-bench can mis-predict in BOTH directions —
real-data fixtures on BOTH precision and recall axes are the
only load-bearing measurement.
Built bench/fixtures/5f/relevance-deflection-realcontext-v1.jsonl —
20 hand-crafted deflection answers against real bench-qa questions
(each: coherent well-formed answer using context-tokens, but NOT
addressing the question — the kind of failure §3.2 was built for).
Real-context deflection scores (cleaned, MiniLM-L-6):
min -6.71, median +1.56, max +6.34
Real-context deflection scores (cleaned, bge-reranker-large):
min -5.57, median -3.02, max +5.01
vs STRICT (cleaned):
MiniLM-L-6: min -5.29, p10 +6.90, median +9.32
bge-large: min -2.42, p10 +3.56, median +6.48
Distributions are CLEAN-separable on real data — 0% of MiniLM
deflections score above p10(STRICT); bge-large median is -3.02 vs
STRICT median +6.48.
Pareto frontier (MiniLM-L-6, cleaned):
θ=-5.29: catch 10% / fp 0% (fp=0 floor, low signal but safe)
θ=+3: catch 65% / fp 0.4% ← strong runtime soft-veto
θ=+5: catch 90% / fp 1.9% ← aggressive soft-veto
θ=+7: catch 100% / fp 10.4% (too FP for runtime)
bge-reranker-large:
θ=-2.42: catch 55% / fp 0% ← VIABLE runtime soft-veto at strict fp=0
The step-2 'NOT VIABLE' verdict was an artifact of using the
candidate-bench NEG distribution (tight contrived band, overlapped
real STRICT) as the recall denominator. Real-context deflections sit
in a much lower band (median around -3 for bge-large) than real
STRICT, so absolute thresholds DO separate them cleanly. The
relevance reranker IS a viable runtime soft-veto on this design —
the candidate-bench-only step 2 measurement misled us.
Manifest's runtime_viability.as_runtime_demotion_veto flipped from
'NOT VIABLE' to 'VIABLE at low-to-moderate FP', with both per-model
Pareto frontiers recorded. demote_below_score stays null pending
fox+dav1d sign-off; setting it folds relevance_policy_hash into
governance_policy_hash per #000049 §7 #2 discipline.
Still SHADOW; production verifier unchanged; no audit_mode effect.
Eighth instance of the meta-lesson, with the lesson sharpening AGAIN:
clean candidate-bench can MIS-PREDICT in BOTH directions — over-
optimistic on threshold (step 2) AND over-pessimistic on viability
(step 3 reversal). Real-data fixtures are the only load-bearing
denominators on either axis.
Hand-inspection of the bottom-15 STRICT-fires from the raw §3.2.2 step 2
sweep showed claim-lattice overlay markup ([E\d+ | title | hash: '…'])
depressing scores on correct concise answers (the 6× Henry-VIII case),
while true-positive deflections (broad-question / narrow-answer like
'winners of all major sports?' → just-one-sport) remained correctly
low-scored. So the noise FP class is the bracket metadata; cleaning it
should reduce FP without losing true-positive signal.
Built clean_for_relevance() in arborist/qa/relevance/shadow.py — strips
[E\d+ | ... ] blocks + trailing '...']' tails. Baked into
ShadowRelevance.check_question_answer / check_claim_source by default
(opt out with clean_input=False). relevance_shadow_sweep.py applies it
to inputs before _score_batch (opt out with --no-clean).
Re-ran the full 6-model sweep on the 808-cell pooled STRICT with
cleaning:
bge-reranker-large 21.8% → 18.6% (-3.2)
MiniLM-L-4-v2 15.0% → 9.4% (-5.6 pts, -37% rel)
MiniLM-L-6-v2 11.6% → 9.0% (-2.6)
MiniLM-L-12-v2 11.0% → 8.0% (-3.0)
bge-reranker-base 9.5% → 9.0% (-0.5)
MiniLM-L-2-v2 4.5% → 1.5% (-3.0 pts, -67% rel)
Universal improvement, every model better. Big surprise: MiniLM-L-2-v2
— the model that FAILED the candidate-bench separability (margin
-1.97, declared 'capacity floor') — has the LOWEST real-traffic FP
rate at its own cb θ (1.5%). Because L-2's compressed score range
gives it a low cb θ which few real STRICT pairs score below.
SEVENTH instance of 'candidate-bench doesn't predict real-traffic'.
Runtime-veto verdict UNCHANGED — still not viable; smallest fp=0 θ on
real STRICT is below the cb NEG max for every model, so at any
runtime-safe θ the catch on cb NEG is 0/12. But cleaning is now FREE
improvement for any soft-signal / advisory / contrastive use of the
relevance score. Hand-inspected bottom-10 post-cleaning confirms true-
positive deflection signal preserved.
All 6 swept rerankers (bge-large/base, MiniLM-L-2/L-4/L-6/L-12)
false-fire on 4.5–21.8% of real bench-qa STRICT at their
candidate-bench fp=0 θ. The smallest θ that yields fp=0 on real
STRICT is BELOW the candidate-bench NEG max for every model —
meaning at the runtime-safe θ, catch on the 12 candidate-bench NEG
= 0/12 across the board.
Structural reason: real bench-qa STRICT answers have a much wider
score distribution (bge-large STRICT: min -2.20, p10 +2.99, p50
+5.80, p90 +7.54) than the tight contrived candidate-bench POS
band. The bottom 10% of legitimate STRICT score below where the
candidate-bench NEG cases sat. Distributions overlap heavily; no
threshold separates them.
This is the §3.2 mirror of #000049 §7 #27's recall-side walk-back —
clean candidate-bench → fails real-pipeline gate. Same diagnosis:
lexical-candidate selection + cross-encoder scoring + hard threshold
doesn't survive real-pipeline heterogeneity.
Verdict: relevance reranker CANNOT be promoted to a runtime
demotion-only veto on this design. demote_below_score stays null;
manifest gains runtime_viability block documenting the negative
result + the still-viable advisory soft-signal uses (render-tail,
multi-signal advisory, contrastive Q→A vs claim→source delta).
bench/scripts/relevance_shadow_sweep.py + bench/results/
relevance-shadow-sweep-pooled808.json committed. Production
verifier unchanged; advisory only; no audit_mode effect.
All 4 manifest candidates clean-separate (12/12 NEG catch at 0/14 POS FP),
so 'reranker discriminates Zionist-entity-style mis-cites' is a property
of MS-MARCO-trained rerankers as a class — not the specific L-6 I picked
first. Ranked by separation margin (per §7 #18: separation beats raw):
ms-marco-electra-base +5.349 ← new primary
BAAI/bge-reranker-base +3.076
ms-marco-MiniLM-L-6-v2 +3.053 (previous primary, demoted)
ms-marco-MiniLM-L-12-v2 +2.663 (worst — deeper ≠ better)
Manifest primary moved to electra-base for the cushion. demote_below_score
STAYS null — clean candidate-bench thresholds don't predict real-pipeline
behavior (the §7 #18→#27 history is 6 verdict flips on the NLI side);
§3.2.2 step 2 (real-traffic shadow sweep on pooled bench-qa STRICT) is
what sets it. Expect a walk-back. Reranker still doesn't catch the
Kilimanjaro/Mount-Kenya recombination (aboutness ≠ truth-of-attribution;
that's #000049 territory). fox's bench-maxing correction applied.
Built bench/fixtures/5f/recombination-realcontext-v1.jsonl: 25 hand-built
recombination claims (deliberate re-attributions within real bench-qa
30 KB Wikipedia haystacks the pipeline produced — Mount Kenya into a
Kilimanjaro answer, Galileo onto all 63 Jupiter moons, Joshua Reynolds
onto the Mona Lisa, etc.). Ran the full 7-model × full-grid sweep vs
the 808 pooled STRICT. Result:
- bart-large-mnli / k=12 / max / θc=0.999 → 12/25 = 0.48 catch · 0/808 FP
(the real-haystack fp=0 leader)
- deberta-large-mnli / k=3 / margin / θc=0.995 → 6/25 = 0.24 (§7 #26's
'settled' config — 28/28 synthetic, 0.24 real-haystack: 4× over-estimate)
- roberta-large 0.12, MiniLM 0.08, deberta-base 0.04
So §7 #26's 'boundary closed' walks back to 'boundary PARTIALLY closed'
on real haystacks. The bottleneck is architectural: top-k by token
overlap misses the contradicting clause when it shares few subject-area
tokens with the answer (e.g. the Mount Kenya clause only shares 'Kenya'
with a Kilimanjaro claim — ranked low, NLI never sees it). Threshold
tuning doesn't lift the ceiling; a SEMANTIC candidate selector
(vec-driven, sibling of #000050/#000051's hybrid retrieval) does.
bart's pareto above fp=0: fp=0.011 catch=0.52, fp=0.057 catch=0.84,
fp=0.068 catch=0.92 — permissive operating points are on the menu if
fox+dav1d sign off. recommended_operating_point updated to
bart-large-mnli/k=12/max/θc=0.999; deberta-large/margin kept as the
synthetic-eval reference. Sixth meta-lesson instance: clean synthetic
eval doesn't predict bench-qa precision OR recall — neither contrived
dataset axis is load-bearing, only the real pipeline shape is.
Production verifier unchanged; falsification-hard stays 10/12. Still
SHADOW; runtime promotion fox+dav1d-decides.
(a) Mined the pooled n=1+3+5 bench-qa runs (808 distinct STRICT answers)
for natural recombinations at lowered θc≥0.7 → 37 would-fires, ALL
token-collision FPs on inspection (Mount Kenya pulled into a Kilimanjaro
answer, Dalí into da Vinci, Donovan into Superman). ZERO genuine
recombination errors — the boundary is theoretical-in-practice; the
failure mode is the candidate selector (top-k by token overlap) pulling
different-entity same-subject-area clauses.
(b) Re-ran the grid against the 808-cell pooled STRICT set: the §7 #25
'max@0.96' was itself a small-sample artifact — the n=5 444-cell set
lacked the high-confidence spurious hits the pooled set has. On 808
cells θc goes back to ~0.995, and at θc=0.995 only agg=margin still
catches 28/28 (max gets 27/28). microsoft/deberta-large-mnli / k=3 /
agg=margin / θc=0.995 → 28/28 synthetic recombinations · 0/808 pooled
real STRICT FP · 0/26 synthetic legit — the ONLY config in the
7-model×full-grid sweep that hits 1.0/0.0 on 808 cells, held at n=3
too. recommended_operating_point reverted to margin@0.995.
Realistic next check: ~20-30 hand-built synthetic-recombination-vs-
real-bench-qa-context fixtures (real haystack, deliberate re-attribution).
Still SHADOW; runtime promotion fox+dav1d-decides. Production verifier
unchanged; falsification-hard stays 10/12.
Meta-lesson instance five: a bigger sample can vindicate a config a
smaller one made look unnecessary — re-confirm the config choice (not
just the threshold) each time the denominator grows.
ARBORIST_NLI_SHADOW=1 make bench-qa BENCH_QA_N=5 → 1125 cells, 444 real
STRICT (5x n=1, 1.6x n=3). Re-ran the 7-model mega-grid: the lexical-
candidate NLI veto robustly clears the §7 #12 gate with
microsoft/deberta-large-mnli / k=2 / agg=max / guard=max_entail /
θc≈0.96 / θe=0.9 → 28/28 synthetic recombinations (incl. both fixtures)
· 0/444 real STRICT FP · 0/26 synthetic legit FP, ~4 pts θc headroom;
roberta-large-mnli equally good (k=2/max/θc=0.95). Resolved: agg=max +
max_entail guard is the robust score-shape (§7 #24's margin win was a
sample tie); the large checkpoints (~350-400M deberta-large-mnli /
roberta-large-mnli) hit 1.0/0.0, bart-large (similar size) only ~0.71,
deberta-base at the cliff (27/28 here, 11/28 at n=3), small models
(MiniLM-82M, deberta-v3-small) cap at ~0.82 — so §7 #18's 'MiniLM is
the cost-pick' is OVERTURNED by the proper-n evidence; k=2 consistent
winner; int8-ONNX costs ~1 catch. recommended_operating_point updated.
Remaining: a bench-qa-derived recombination set (the one check not
done); n=9 if dav1d wants more; still SHADOW; runtime promotion is
fox+dav1d-decides. Production verifier unchanged; falsification-hard
stays 10/12. This run is the worked example behind CLAUDE.md's new
bench-maxing line.
ARBORIST_NLI_SHADOW=1 make bench-qa BENCH_QA_N=3 → 275 real STRICT cells
(3x the n=1 sample). Re-ran the expanded grid (7 aggregations incl.
margin = max_clause(p_contra - p_entail), paired-entail guard variant,
θc to 0.999, --extra-models) over all manifest models + 4 extra
xsmall→large (microsoft/deberta-large-mnli, roberta-large-mnli,
nli-deberta-v3-small, deberta-v3-xsmall), synth-28 recombination vs the
275 STRICT cells. Result: the §7 #23 deberta-base/k=2/θc=0.99 config
does NOT survive — it catches only 11/28 at θc=0.995 (which the larger
STRICT sample forces). BUT the broader sweep found the config that does:
microsoft/deberta-large-mnli / k=3 / agg=margin / θc=0.995 → 28/28
synthetic recombinations (incl. both fixtures) + 8/12 falsification-hard,
0/275 real STRICT FP, 0/26 synthetic legit FP — a passing config at
proper n. Findings: margin is the right score-shape (single threshold,
folds the guard in); the specific checkpoint matters more than param
count (deberta-large-mnli wins clean, deberta-base collapses,
roberta/bart ~0.71-0.75 — no 'bigger is better' law). recommended_operating_point
updated. Still SHADOW; runtime promotion needs a bigger STRICT sample +
a bigger recombination set + fox/dav1d sign-off. Production verifier
unchanged; falsification-hard stays 10/12.
Meta-lesson sharpened twice: clean eval ≠ bench-qa precision (§7 #18→#20),
default config ≠ best config (§7 #22→#23), small FP sample ≠ large FP
rate (§7 #23→#24).
The §7 #22 'fails the gate' was the verdict for the DEFAULT config
(k=6/θc=0.5/θe=0.9, tuned on the clean synthetic set), not the
approach. A {model × candidate-cap k × aggregation × θc × θe} grid
sweep (bench/scripts/nli_shadow_grid.py — NLI runs once per
(model,record) over the top-12 candidate clauses, the k/agg/θ grid is
then arithmetic on cached scores; ~10s on the 4090 for 4 models) finds
clean passing configs: on the 89 real STRICT cells (n=1 bench-qa),
deberta-base-184M / k=2 / agg=max / θc=0.99 / θe=0.9 → 27/28 synthetic
recombinations caught (incl. both 5f-fal-hard fixtures), 0/89 STRICT
FP. MiniLM-82M passes too (24/28 · 0/89). Model science: 184M > 82M >
407M for fp=0 recombination recall; int8-ONNX costs ~1 catch vs fp32.
Caveats: FP side is n=1 (BENCH_QA_N=3 run in flight); recall is on the
synthetic set; flipping to a runtime demotion-only veto is fox-decides
(then nli_policy_hash folds into governance_policy_hash per §7 #2).
Manifest active defaults stay k=6/θc=0.5; recommended_operating_point
(deberta-base, k=2, agg=max, θc=0.99, θe=0.9) documented in the
manifest. 3 grid result JSONs committed. Standing lesson: neither the
clean synthetic eval NOR the default config predicts bench-qa precision
— you have to sweep. Production verifier unchanged; falsification-hard
stays 10/12.
First grid (MiniLM, n=1 bench-qa 89 STRICT cells): the §7 #22 'fails the
gate' verdict was config-specific (defaults k=6/θc=0.5/θe=0.9) — the grid
finds k=1/θc=0.93/θe=0.5 → catch 8/12 falsification-hard (incl. BOTH
recombination fixtures hard-003 Mercury + hard-005 Einstein), 0/89 STRICT
FP, 0/26 synthetic-legit FP; 18/28 synthetic recombination. So the
lexical-candidate approach is NOT a dead end. (k=1 = no haystack;
θe=0.5 tighter than the clean-set 0.9.) Full multi-model + n=3
characterization next.
Speedup (§3 plan): ShadowNLI._nli_batch batches forwards
(ARBORIST_NLI_BATCH=64); device auto-detect (ARBORIST_NLI_DEVICE, else
cuda-if-available); auto-prefer an ONNX export — bench/scripts/export_nli_onnx.py
/ make export-nli-onnx exports + int8-dynamic-quantizes the pinned
checkpoint into ~/.arborist/models/nli/<ver>/onnx/ (operator state, NOT
committed), _ensure_loaded loads model_quantized.onnx via
optimum.onnxruntime (backend onnx-int8), falls back to torch silently.
torch-cpu-batch1 ~120ms/pair → onnx-int8-cpu-batched ~32ms/pair (~4x);
seconds on a 4090. optimum[onnxruntime] added to the [nli] extra; 24
tests.
Gate-item-4 verdict at proper n: ARBORIST_NLI_SHADOW=1 make bench-qa
BENCH_QA_N=1 → 223 cells (89 STRICT / 90 HYBRID / 44 UNGROUNDED; also
surfaced + fixed a lone-surrogate bug). Shadow sweep over those: NLI-as-
runtime-veto on STRICT has ~26% FP at θc 0.5, ~8% at θc 0.90, ~0% only
at θc 0.99 — and θc 0.99 gives up most recombination recall (hard
synthetic recombinations bottom out ~0.76). FAILS the §7 #12 gate on
this design. Only untried path that might pass: a Phase-3 runtime hook
running NLI on the verifier's actual matched clauses (1-3), not
top-6-by-overlap. Until then: runtime NLI demotion stays off; the 2
fixtures stay permanent boundary markers; θc stays 0.5. Production
verifier unchanged; falsification-hard stays 10/12.
ARBORIST_NLI_SHADOW=1 carries the raw verifier-input text into bench
rows; real Wikipedia context occasionally has U+D800–U+DFFF code points
(mangled source encoding) that json.dumps(..., ensure_ascii=False) then
refuses to UTF-8-encode → the run died at row 224/225. Scrub via
encode('utf-8','replace').decode() — U+FFFD is fine for a measurement
field. Only the two new shadow fields are touched.
candidate_clauses() — NLI now runs only on the top-N source clauses by
content-token overlap with the answer claim (max_candidate_clauses=6),
not the whole context; records n_candidate_clauses / best_clause_overlap
/ recombination_risk. Synthetic sweep unchanged (28/28 recombination,
0/26 legit FP, mean 1.45 candidate clauses/record). Real-traffic smoke
re-run: STRICT would-demote 30% → 20%, overall 47% → 33% — better, not
fixed; recombination-risk split doesn't separate either. Residual STRICT
false-contras at ~0.83-0.92 → θc would need ≈ 0.90 (vs the clean-set
0.5); at θc=0.90 the data in hand gives 27/28 synthetic recall, 0/26
legit FP, 0/10 smoke STRICT FP — but n=10 is too small to set on.
Next: a fuller ARBORIST_NLI_SHADOW=1 bench-qa run → sweep θc on hundreds
of STRICT cells → confirm → set it. θc stays 0.5; runtime NLI demotion
stays off. Production verifier unchanged; falsification-hard stays 10/12.
Live hook: ARBORIST_NLI_SHADOW=1 makes query() surface the verifier-input
text (gated off-by-default, never a cache_key/governance/audit_mode
input); qa_sweep.py carries it + the answer into bench rows; the shadow
sweep reads them and buckets by audit_mode. ARBORIST_NLI_SHADOW=1 make
bench-qa-smoke (15 cells) → the naive 'NLI on every context clause'
scaffold has a ~30% would-demote rate on STRICT answers — a haystack /
multiple-comparisons artifact (real contexts segment into 100-336
clauses; max-over-all almost always finds a tangential clause the model
reads as contradicting; a paraphrased STRICT answer often isn't verbatim-
entailed by any single clause so the entailment guard doesn't rescue it).
Lesson: the §7 #5 'candidate source clauses' + recombination-risk gating
is load-bearing, not optional. Do NOT enable runtime NLI demotion on the
current scaffold; next step is the candidate-clause restriction, then
re-run, then gate item 4 is meaningful. Production verifier unchanged;
falsification-hard stays 10/12.
arborist/qa/nli/ — SHADOW ONLY (never an audit_mode input; manifest not
yet in governance_policy_hash per §7 #2). manifest.json pins
cross-encoder/nli-MiniLM2-L6-H768 @ a fixed HF revision + the
bench-validated θc 0.5/θe 0.9 + 2 alternates + the Phase-3 TODO;
shadow.py = ShadowNLI/shadow_check (lazy transformers+torch behind a new
[nli] extra, clauses() segmenter, the §7 #5 clause-level Demote()
decision, degrades to available=False when [nli] absent);
bench/scripts/nli_shadow_sweep.py + make bootstrap-nli / bench-nli-shadow
(the gate-item-4 instrument); 16 tests.
First sweep (116 records — 5f-falsification packs + the arborist-nli-bench
eval sets): 28/28 synth recombination demoted, 0/26 FP on legit summaries,
0/9 fires on already-STRICT_SPAN records, 25/50 on UNGROUNDED (the
contradiction half; quiet on non-sequiturs). Gate items 1/2/3/5/6 clear
on available data; item 4 — shadow FP rate on a real live-bench-qa
sample — remains the open measurement. Production verifier unchanged;
falsification-hard stays 10/12.
make bench-emergent EMERGENT_N=30 — 30 fresh 3-word-triangulation
cycles appended to bench/emergent_log.jsonl.
Aggregate: 29/30 UNGROUNDED, 1/30 HYBRID, 0 STRICT — the expected
healthy null result. The triangulator generates absurd questions
("How does the depth of one's music-making abilities on a kazoo …")
with no real corpus support; the verifier ladder honestly returns
UNGROUNDED rather than hallucinating-then-STRICTing. 0 false-positive
STRICT on garbage input.
The one HYBRID (perversely/succeeds/disagreeably) is a re-run of
failure shape E ("total deflection to unrelated trivia"): the common
words pulled Hirabah / Original-righteousness chunks via FTS5; the
model quoted them verbatim (3/4 claims verify) but the answer is
total topic-drift → DEFLECTION_DETECTED + CITATION_MISMATCH → capped
at HYBRID, not STRICT. The deflection sidecar + verbatim verifier
holding on adversarial-by-construction input — nothing new, logged as
a confirmation.
No new failure shape or correctness concern. #000006 amend records
it; the 30 entries join the --print-pending queue if a future
teacher-review pass wants a closer look. Doc + log only.
Closes the 8 mis-segments #000046 left in formulate-hard-v1.jsonl.
The parser was line/bullet-only — one line ⇒ one claim — so a line
that crammed several pointered claims onto one row ("Water is wet
[E1]; fire is hot [E2]", "X happened [E1]. Y followed [E2]") became
one monolithic claim with all the pointers, and a wrapped bullet
became two.
arborist/qa/parse_claims.py: _SEGMENT_SEP_RE splits a line on ';',
sentence boundaries ('. '/'! '/'? ' then a Capital), spaced dashes
(' - '/' — '/' – '), ' and '/' or '/' because '/' although '/' since
'/' while ', inline '(N)' enumeration markers, and commas — with
'(?![^\[]*\])' so a comma inside a [E1, E2] bracket never splits it.
_segment_line keeps the split ONLY IF every resulting non-empty
segment is a well-pointered claim — a legit single claim ("The cat
is black and white [E1].", "The cast: A, B, C [E1].") is never
broken because splitting it would manufacture pointer-less prose
fragments → guard rejects; a leading colon-terminated header with no
pointer ("Two facts:", "Key points:") is allowed and dropped. Plus a
wrapped-bullet join: a continuation line (leading whitespace then a
lowercase letter, no bullet glyph) folds its text + pointers into the
previous claim.
Effect: formulate-hard rate 4/12 → 12/12 (the pack is now at ceiling
— a harder Formulate tier would re-open below-ceiling headroom; a
#000046 follow-up). Remaining #000048 headroom: 2 STRICT_PARAPHRASE
recombinations in falsification-hard (Mercury, Einstein — step 2.2).
Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells; parse_pointer_claims
feeds the 450 claim_lattice_pointer + claim_lattice cells) after
(bench/qa_results/2026-05-11T20-26-37Z) vs the pre-step-2.4 baseline
(...T17-12-41Z = HEAD's parse_claims.py). STRICT-rate quote 0.54→0.55,
pointer 0.22→0.22, lattice 0.43→0.45 — all within the 5-pp noise
floor. Per-row diff: the segmenter changed the parsed-claim count on
the SAME answer text for 7 of the 450 lattice cells (0 in
claim_lattice, 7 in claim_lattice_pointer); of those, 2 caused an
audit_mode change — both correct: a wrap-join recovered an answer's
intended structure (4 claims, 2 pointer-less wrap-fragments → HYBRID)
into 2 well-pointered claims → STRICT; and a crammed-one-line blob (1
monolithic claim, all pointers → STRICT) split into 8 claims, some
not individually verifying → HYBRID (the honest verdict — false-
positive STRICT was the corruption). Every other lattice/quote delta
is LLM re-answer variance. No regression — the segmenter's only
visible effects on real traffic are honest improvements. Summarized
in qa-modes-bench.md Addendum 7 + ticket-000048 §5 step 2.4.
Tests: 8 new in test_claim_lattice.py (semicolon/sentence/conjunction
splits; pointerless-fragment + cast-list guards; leading-colon-header
drop; wrapped-bullet join; pointer-order/multi-pointer); existing
parse_pointer_claims tests pass untouched; test_5f_formulate_hard_pack
re-pinned 4/12 → 12/12. make test 2358 passed, 28 skipped.
#000048 → steps 2.1 + 2.4 landed; #000046 / #000012 §8 / TICKETS.md /
Makefile / fixture _meta + notes updated.
