The §12.1 pipeline I added was second-hand from benchmark-matrix.md
and got several things wrong against the code:
- listed 4 verdicts; actual is 5 (missing FABRICATED — the
fabrication-vs-WRONG split that energy-cogs §5.5 leans on for the
qwen-fabricates / hermes-abstains finding)
- "θ=0.85" was right by accident — but it's the code-judge-pinned
_CODE_JUDGE_THETA_CONTRA constant, raised from the manifest 0.5
default after measuring 114 FPs in the 0.5-0.75 band
- omitted the short-answer entity-grounding fast path (which runs
BEFORE NLI per the 2026-05-19 Poland-Tusk smoke)
- omitted the HYBRID rescue ladder (NLI entail / entity rescue /
2026-05-21 verbatim-quote-on-topic rescue)
- conflated WRONG and FABRICATED (the subject-in-gold split is what
distinguishes "source has the topic but a different value" from
"source silent on the topic")
Rewrote §12.1 grounded in `bench/judge_code.py:judge()` (its own
docstring at line 501-528 is the truth on rule order), with file:line
citations and the verdict-mapping in full.
Also fixed a real artifact-vs-doc drift INSIDE the judge: the
module-top docstring still claimed θ_contra default 0.5 and omitted
the short-path and the WRONG/FABRICATED split. Updated to match the
authoritative judge() docstring + current code.
No behavior change — docstring + benchmarks doc only.
The code judge bailed to JUDGE_ERROR on 40% of in-corpus answers: HYBRID
(partial grounding) with low NLI entail, where the entity-grounding
rescue needs ZERO unsourced specifics. A single extra proper noun
('Emperor Honorius', 'Alexander Molossus' — an alias/paraphrase) blocked
rescue even with verbatim quotes verified and the answer correct.
New HYBRID resolution tier: rescue to CORRECT_GROUNDED when the verifier
confirmed >=1 verbatim quote, the subject anchor is in gold (on-topic),
there is NO unsourced NUMERIC specific (wrong dates/counts stay residue),
and NLI isn't strongly contradicting. Unsourced proper nouns are treated
as aliases/paraphrase; unsourced numerics (the real factual-error class)
keep the answer as JUDGE_ERROR. Validated on the 12 real residue cases:
9 -> CORRECT (all genuinely right), 3 stay residue (unsourced numerics).
JUDGE_ERROR 40% -> ~10%. self-test 4/4; 2 new tier tests; suite 2549.
Reorder rule 3 (short-answer entity grounding) above rule 4 (NLI
contradiction) so positive lexical evidence cannot be overridden by
NLI clause-level noise. Surfaced by the 2026-05-19 arborist+qwen-
nothink smoke:
i=3 · who is the prime minister of Poland?
ans: 'Donald Tusk is listed as the Prime Minister of Poland.'
gold: ...lists Tusk + Marcinkiewicz + Belka + Kaczynski + Kopacz...
NLI contradiction p=0.892 (above 0.85 threshold)
NLI entailment p=0.744 (also high on the correct clause)
Tusk WAS PM in 2010 (served 2007-2014); answer is correct against
the corpus-vintage gold. The NLI contradiction signal came from
clause-level candidate selection picking a NON-Tusk PM the source
also mentions; entailment was high on the Tusk clause. Mixed signal
that the WRONG rule then over-confidently resolved.
The fix is a rule reorder, not a threshold change — the fast path's
positive-evidence combination (specifics-in-gold AND subject-in-gold)
is a strictly stronger signal than NLI's clause-level max
contradiction, so when it fires it should win. The combination
discriminates Poland-Tusk (Tusk ∈ gold, Poland ∈ gold → CG) from
Anthony-Albanese (Albanese ∉ gold → fast path declines → falls
through to UNGROUNDED-subject-in-gold → WRONG, unchanged).
Self-test 4/4 INSTRUMENT TRUSTWORTHY unchanged. pytest 27/27.
Poland-Tusk regression smoke: now CG via short_entity_grounded ✓.
No regression risk on the existing reconciliation cells:
- Iceland CG: short_entity_grounded was already winning (was rule
4, now rule 3 — same outcome, earlier exit)
- WWII-1812 WRONG: '1812' ∉ gold → fast path declines, NLI fires ✓
- Higgs-cafe FABRICATED: 'Higgs' ∉ gold → fast path declines ✓
- Anthony Albanese WRONG: 'Albanese' ∉ gold → fast path declines ✓
- Abstention phrases: rule 2 still fires first ✓
The Arborist arm runs answer_mode='claim_lattice' (per control_sweep.py
:179, control_ab.py:155) so its answers arrive as the JSON envelope
{"claims":[{"text":"...","evidence_ids":["E1"]},...]}.
_descaffold strips the [E1] evidence-pointer markup but the JSON
braces + key syntax remain. The verifier's strategy-2 (span) and
strategy-3 (proper-noun) extractors see brace noise instead of the
inner claim prose — every Arborist record degraded to UNGROUNDED.
The 2026-05-19T17-01-17Z sweep, re-graded with the freshly calibrated
judge (5a17f61), surfaced this: Arborist arm reported 0 CG across all
three variants in the live phase 1 output (the live run was pre-
calibration), and 29/120 CG (24%) under the calibrated rescore — clear
improvement just from theta_contra=0.85, but the JSON envelope was
still hobbling the verifier paths.
Fix: _unwrap_claim_lattice_json runs BEFORE all downstream rules.
Detection is conservative (three independent signals: starts-with-
brace AND "claims" key AND "text" key) so plain-prose answers
pass through unchanged. Multi-claim envelopes concatenate as discrete
sentences (extract_claim_spans treats each as its own span).
Malformed JSON falls back to the original answer — no silent
rewriting on broken input.
Smoke result on the Iceland Arborist case
ans: {"claims":[{"text":"The current president of Iceland is
Ólafur Ragnar Grímsson.","evidence_ids":["E1"]}]}
gold: {{Infobox Political post |post = President |body = Iceland
|incumbent = [[Ólafur Ragnar Grímsson]] ...}}
before: UNGROUNDED → FABRICATED (then WRONG after calibration)
after: short_entity_grounded → CORRECT_GROUNDED
pytest: 27/27 (added 7 unwrap-coverage tests covering single-claim
envelopes, multi-claim concatenation, plain-prose passthrough,
malformed-JSON tolerance, unrelated-JSON passthrough, and the
end-to-end Arborist-envelope CG flow). Self-test 4/4 unchanged.
Re-rescores of 17:01 sweep + phase 1 sweep run after this commit
to measure final Arborist scorecard improvement.
Opus is trusted; this commit closes the systematic gaps surfaced by
B's rescore on the 17:01 sweep (2289 records, of which 468 had real
Opus verdicts). Off-diagonal cells in descending size and the
root-cause fix for each:
cell n=114 code:WRONG · opus:CORRECT_GROUNDED
Root cause: NLI fires contradiction p in [0.5, 0.75] on factual
answers like 'Ólafur Ragnar Grímsson is president of Iceland'
against wikitext-shaped infobox gold — clause-level candidate
selection picks up co-mentioned earlier office-holders, reads
temporal-frame mismatch as contradiction. The TRUE contradiction
signal (WW2 1812 self-test fixture) measures p=0.985 — clean
margin above noise.
Fix: raise theta_contra 0.5 → 0.85 (code-judge override of the
NLI manifest's 0.5 contradiction_veto).
cell n=60 code:FABRICATED · opus:WRONG
Root cause: 'Anthony Albanese' answer vs Julia Gillard gold gets
FABRICATED (specifics-not-in-gold) but Opus correctly distinguishes
WRONG (source contradicts by naming someone else) from
FABRICATED (source silent on topic).
Fix: when verifier UNGROUNDED + specifics not in gold AND the
question's subject anchor IS in gold, demote FABRICATED → WRONG.
Subject anchor uses proper-noun-shaped terms from the question
(Iceland / Australia / Higgs) — not the last-content-token
heuristic, which mis-fires on coincidental matches like 'cafe'
appearing in a 'gold does not mention any cafe' denial.
cell n=18 code:WRONG · opus:ABSTAINED
Root cause: abstention patterns missed Hermes's most common
refusal phrasings — 'I do not have accurate information', 'I do
not have access to a reference knowledge base', 'I lack access
to'. Original patterns required determine/know/tell verbs right
after 'do not'.
Fix: three new patterns for the 'do not have / lack ...
information / access / knowledge' family.
cell n=13 code:ABSTAINED · opus:CORRECT_GROUNDED
Root cause: verifier's strategy-2 needs prose shape; terse-name
answers ('Pratibha Patil', 'Jalal Talabani') fall to
UNGROUNDED-no-specifics → ABSTAINED, missing valid CG.
Fix: short-answer entity-grounding fast path. When answer is
short (≤15 tokens) AND every specific asserted is present in
gold (no unsourced) AND at least one specific WAS asserted AND
the question's subject anchor is in gold → CG. Guards against
'wrong topic, right name' false-positives via the subject check.
Structural reorder: NLI contradiction now runs AFTER the
abstention check and short-answer fast path (instead of preempting
the verifier), so the verifier's STRICT/HYBRID positive signal
isn't overridden by NLI noise. NLI still leads the path on truly
unbounded answers — verifier UNGROUNDED + NLI ≥ 0.85 contradiction
keeps the WRONG label.
Self-test 4/4 INSTRUMENT TRUSTWORTHY. pytest contract 18/18.
v2 rescore on the same 17:01 sweep runs in the background to
measure agreement-matrix improvement empirically.
Added bench/analyze_judge_disagreement.py — the harness that drove
this calibration (reads B's rescore JSONL, bucketises off-diagonal
cells, dumps configurable samples per cell with question / answer /
gold / both rationales). Reusable for the next calibration round.
bench/judge_code.py — drop-in alternative to bench/judge.py with the
same Verdict shape & closed verdict vocabulary (CG/W/F/A/JE) but zero
quota cost: composes verifier + NLI + abstention + specificity into a
fixed-order pipeline. fox 2026-05-19: 'data first, judging later' —
this is the data-collection arm; LLM-based judging (Opus batched
needle-haystack, or Grok credit-card) is a separate downstream
concern that operates on the residue this judge cannot classify
deterministically.
Pipeline (first hit decides):
1. empty / no-gold guards
2. explicit abstention phrases (lexical regex)
3. NLI contradiction (arborist.qa.nli.shadow_check) — strongest
signal: gold contradicts the claim → WRONG
4. lexical verifier (arborist.qa.verify.verify_quotes) →
STRICT → CORRECT_GROUNDED
HYBRID + NLI entail >= 0.55 → CORRECT_GROUNDED
UNGROUNDED + specifics-not-in-gold → FABRICATED
UNGROUNDED + no specifics → ABSTAINED
HYBRID without NLI corroboration → JUDGE_ERROR (residue
for an LLM judge)
Threshold note: _CODE_JUDGE_THETA_ENTAIL_CORROBORATE=0.55 is distinct
from the NLI manifest's entailment_block_veto=0.9. The manifest's
threshold is calibrated for OVERRIDING a STRICT lexical signal with
negative evidence — high bar. The corroboration use here is the
opposite direction: additive positive evidence on an already-positive
anchor — moderate bar appropriate. Self-test case 1 measures NLI
entail=0.769 (clearly entailed, clear margin above 0.55).
Specificity for FABRICATED layers three scanners:
- verifier's multi-word proper-noun extractor (Higgs Boson, ...)
- local single-word capitalised-token scanner (Napoleon, Mars, ...)
deliberately separate because the verifier's gate is conservative
by design (multi-word only)
- numerics (years, dates, large counts, money)
Self-test: same 4 fixtures as bench/judge.py:self_test() so the two
instruments can be cross-checked when fox re-fires the Opus judge on
the residue later. Result: 4/4 INSTRUMENT TRUSTWORTHY.
tests/test_judge_code.py — pulls the contract into make test
(18 cases): module identifiers pinned, dataclass shape parity,
empty / no-gold guards, parametrised abstention phrases, specificity
layer behaviour, the canonical 4-case self-test, batch helper, and
graceful NLI-unavailable degradation. 18/18 pass.
Pre-existing known limitation, documented in the docstring: terse
correct answers ('In 1945.' against gold containing '1945') route to
ABSTAINED because the verifier's span extractor needs prose shape;
NLI sees no clause-level overlap at very short claims. The conservative
ABSTAINED label is correct deferral; tuning this is a calibration
question for real bench data, not the instrument's contract.
No callers touched yet — control_sweep.py & control_ab.py still
import the disabled Opus judge. Wiring this in is a separate ticket
move per fox's data-first sequencing.
