Opus is trusted; this commit closes the systematic gaps surfaced by
B's rescore on the 17:01 sweep (2289 records, of which 468 had real
Opus verdicts). Off-diagonal cells in descending size and the
root-cause fix for each:
cell n=114 code:WRONG · opus:CORRECT_GROUNDED
Root cause: NLI fires contradiction p in [0.5, 0.75] on factual
answers like 'Ólafur Ragnar Grímsson is president of Iceland'
against wikitext-shaped infobox gold — clause-level candidate
selection picks up co-mentioned earlier office-holders, reads
temporal-frame mismatch as contradiction. The TRUE contradiction
signal (WW2 1812 self-test fixture) measures p=0.985 — clean
margin above noise.
Fix: raise theta_contra 0.5 → 0.85 (code-judge override of the
NLI manifest's 0.5 contradiction_veto).
cell n=60 code:FABRICATED · opus:WRONG
Root cause: 'Anthony Albanese' answer vs Julia Gillard gold gets
FABRICATED (specifics-not-in-gold) but Opus correctly distinguishes
WRONG (source contradicts by naming someone else) from
FABRICATED (source silent on topic).
Fix: when verifier UNGROUNDED + specifics not in gold AND the
question's subject anchor IS in gold, demote FABRICATED → WRONG.
Subject anchor uses proper-noun-shaped terms from the question
(Iceland / Australia / Higgs) — not the last-content-token
heuristic, which mis-fires on coincidental matches like 'cafe'
appearing in a 'gold does not mention any cafe' denial.
cell n=18 code:WRONG · opus:ABSTAINED
Root cause: abstention patterns missed Hermes's most common
refusal phrasings — 'I do not have accurate information', 'I do
not have access to a reference knowledge base', 'I lack access
to'. Original patterns required determine/know/tell verbs right
after 'do not'.
Fix: three new patterns for the 'do not have / lack ...
information / access / knowledge' family.
cell n=13 code:ABSTAINED · opus:CORRECT_GROUNDED
Root cause: verifier's strategy-2 needs prose shape; terse-name
answers ('Pratibha Patil', 'Jalal Talabani') fall to
UNGROUNDED-no-specifics → ABSTAINED, missing valid CG.
Fix: short-answer entity-grounding fast path. When answer is
short (≤15 tokens) AND every specific asserted is present in
gold (no unsourced) AND at least one specific WAS asserted AND
the question's subject anchor is in gold → CG. Guards against
'wrong topic, right name' false-positives via the subject check.
Structural reorder: NLI contradiction now runs AFTER the
abstention check and short-answer fast path (instead of preempting
the verifier), so the verifier's STRICT/HYBRID positive signal
isn't overridden by NLI noise. NLI still leads the path on truly
unbounded answers — verifier UNGROUNDED + NLI ≥ 0.85 contradiction
keeps the WRONG label.
Self-test 4/4 INSTRUMENT TRUSTWORTHY. pytest contract 18/18.
v2 rescore on the same 17:01 sweep runs in the background to
measure agreement-matrix improvement empirically.
Added bench/analyze_judge_disagreement.py — the harness that drove
this calibration (reads B's rescore JSONL, bucketises off-diagonal
cells, dumps configurable samples per cell with question / answer /
gold / both rationales). Reusable for the next calibration round.
russell@unturf.com
5a17f617e2