docs+code: ground §12 judge pipeline in the actual judge_code.py

The §12.1 pipeline I added was second-hand from benchmark-matrix.md
and got several things wrong against the code:
- listed 4 verdicts; actual is 5 (missing FABRICATED — the
  fabrication-vs-WRONG split that energy-cogs §5.5 leans on for the
  qwen-fabricates / hermes-abstains finding)
- "θ=0.85" was right by accident — but it's the code-judge-pinned
  _CODE_JUDGE_THETA_CONTRA constant, raised from the manifest 0.5
  default after measuring 114 FPs in the 0.5-0.75 band
- omitted the short-answer entity-grounding fast path (which runs
  BEFORE NLI per the 2026-05-19 Poland-Tusk smoke)
- omitted the HYBRID rescue ladder (NLI entail / entity rescue /
  2026-05-21 verbatim-quote-on-topic rescue)
- conflated WRONG and FABRICATED (the subject-in-gold split is what
  distinguishes "source has the topic but a different value" from
  "source silent on the topic")

Rewrote §12.1 grounded in `bench/judge_code.py:judge()` (its own
docstring at line 501-528 is the truth on rule order), with file:line
citations and the verdict-mapping in full.

Also fixed a real artifact-vs-doc drift INSIDE the judge: the
module-top docstring still claimed θ_contra default 0.5 and omitted
the short-path and the WRONG/FABRICATED split. Updated to match the
authoritative judge() docstring + current code.

No behavior change — docstring + benchmarks doc only.
This commit is contained in:
russell@unturf.com 2026-05-22 19:17:28 -04:00
parent 7f5ef5c140
commit b6bb31a836
No known key found for this signature in database
2 changed files with 97 additions and 27 deletions

View file

@ -439,18 +439,63 @@ LLM-judge running on this path.
### 12.1 Code judge — `bench/judge_code.py`
Deterministic, **no LLM, no quota**. First hit decides:
empty/no-gold guard → explicit-abstention regex → short-answer
entity-grounding fast path → **NLI contradiction** (θ=0.85, runs on the
GPU box) → lexical verifier (quote/span/entity/paraphrase) →
WRONG-vs-FABRICATED tie-break on subject-in-gold. Claim-lattice JSON
envelopes are unwrapped to prose first so the substrate arm grades on
equal terms with solo. It is a *proxy* for grounding (lexical + NLI), not
SOTA reading. The `JUDGE_ERROR` residue (~10 % after the `2d31866`
HYBRID-resolution strengthening; see energy-cogs §5.5) is the residue
worth handing to a stronger judge if one is ever wired in.
**Full pipeline:** `docs/benchmark-matrix.md` "## The judge"; used in
context in `docs/energy-cogs-benchmark.md` §5.5.
Deterministic, no LLM, no quota. **Five verdicts:**
`CORRECT_GROUNDED` / `WRONG` / `FABRICATED` / `ABSTAINED` /
`JUDGE_ERROR`. The `WRONG`-vs-`FABRICATED` split is what powers the
bare-model failure-style finding in energy-cogs §5.5 (bare qwen
fabricates, bare hermes abstains; the substrate fixes both).
Authoritative source: `bench/judge_code.py` `judge()` at line 493 — its
own docstring (line 501-528) is the truth on rule order. Pipeline, first
hit decides:
1. **Empty / no-gold guards.** Empty answer → `ABSTAINED`; empty gold →
`JUDGE_ERROR`.
2. **Explicit abstention phrases** — broad lexical regex including the
"I do not have access to information / context" Hermes family that
the original narrow patterns missed. Match → `ABSTAINED`.
3. **Short-answer entity-grounding fast path.** When the answer is
short, asserts at least one specific, every specific is in gold, AND
the question's subject anchor is in gold → `CORRECT_GROUNDED`. Runs
**before** NLI (since the 2026-05-19 Poland-Tusk smoke): NLI's
clause-level aggregation can mis-fire as contradiction on multi-PM
lists even when the answer is right; positive lexical evidence wins.
4. **Strong NLI contradiction**`arborist.qa.nli.shadow_check`,
`θ_contra = 0.85` (code-judge-pinned `_CODE_JUDGE_THETA_CONTRA`,
**raised** from the manifest's 0.5 default after measuring 114 FPs
in the 0.5-0.75 band). `max_contradiction ≥ 0.85``WRONG`.
5. **Lexical verifier vs gold**`arborist.qa.verify.verify_quotes`
(the existing quote → span → entity → paraphrase ladder), then map:
- `STRICT``CORRECT_GROUNDED`
- `HYBRID` + NLI entail ≥ 0.55 (`_CODE_JUDGE_THETA_ENTAIL_CORROBORATE`) → `CORRECT_GROUNDED`
- `HYBRID` + subject-anchor-in-gold + every specific in gold → `CORRECT_GROUNDED` (entity-grounding rescue)
- `HYBRID` + ≥1 verbatim quote verified + on-topic + **no unsourced
numeric** specific + not NLI-contradicted → `CORRECT_GROUNDED`
(2026-05-21 relaxed rescue, `2d31866`: unsourced proper-nouns
treated as aliases / paraphrase, but a wrong date/count is held
back)
- `HYBRID` otherwise → `JUDGE_ERROR` (`"code judge ambiguous … residue
for LLM judge"`)
- `UNGROUNDED` + specifics-not-in-gold + subject-**in**-gold →
`WRONG` (source has the topic but a different value)
- `UNGROUNDED` + specifics-not-in-gold + subject-**not**-in-gold →
`FABRICATED` (source silent on the topic)
- `UNGROUNDED` + no asserted specifics → `ABSTAINED` (thin /
non-answer, not a fabrication)
Claim-lattice JSON envelopes are unwrapped to prose
(`_unwrap_claim_lattice_json`) before any rule runs, so substrate-CL
grades on equal terms with solo-prose. Each `Verdict` carries a
`decision` trace (rules_fired, NLI probabilities, subject anchor,
specifics, audit_mode) — fully replayable. `JUDGE_ERROR` is reserved
*only* for the `HYBRID`-without-corroboration class (the ~10 % residue
after the `2d31866` HYBRID-rescue strengthening; energy-cogs §5.5);
everything else lands on a concrete verdict.
Self-test: `bench/judge_code.py:self_test()` (`make judge-self-test`),
gating before any control-ab run. The pipeline cross-reference (older,
shorter) is `docs/benchmark-matrix.md` "## The judge"; used in context
in `docs/energy-cogs-benchmark.md` §5.5.
### 12.2 Why there is no LLM judge in active use — `bench/judge.py` (abandoned)