docs+code: ground §12 judge pipeline in the actual judge_code.py

The §12.1 pipeline I added was second-hand from benchmark-matrix.md
and got several things wrong against the code:
- listed 4 verdicts; actual is 5 (missing FABRICATED — the
  fabrication-vs-WRONG split that energy-cogs §5.5 leans on for the
  qwen-fabricates / hermes-abstains finding)
- "θ=0.85" was right by accident — but it's the code-judge-pinned
  _CODE_JUDGE_THETA_CONTRA constant, raised from the manifest 0.5
  default after measuring 114 FPs in the 0.5-0.75 band
- omitted the short-answer entity-grounding fast path (which runs
  BEFORE NLI per the 2026-05-19 Poland-Tusk smoke)
- omitted the HYBRID rescue ladder (NLI entail / entity rescue /
  2026-05-21 verbatim-quote-on-topic rescue)
- conflated WRONG and FABRICATED (the subject-in-gold split is what
  distinguishes "source has the topic but a different value" from
  "source silent on the topic")

Rewrote §12.1 grounded in `bench/judge_code.py:judge()` (its own
docstring at line 501-528 is the truth on rule order), with file:line
citations and the verdict-mapping in full.

Also fixed a real artifact-vs-doc drift INSIDE the judge: the
module-top docstring still claimed θ_contra default 0.5 and omitted
the short-path and the WRONG/FABRICATED split. Updated to match the
authoritative judge() docstring + current code.

No behavior change — docstring + benchmarks doc only.
This commit is contained in:
russell@unturf.com 2026-05-22 19:17:28 -04:00
parent 7f5ef5c140
commit b6bb31a836
No known key found for this signature in database
2 changed files with 97 additions and 27 deletions

View file

@ -18,32 +18,57 @@ judging (Opus batched needle-haystack, or Grok credit-card) is a
later, separate concern; it operates on the residue this judge can
not classify deterministically.
The pipeline, in fixed order — first hit decides:
The pipeline, in fixed order — first hit decides (the authoritative
listing is the ``judge()`` docstring at line 501; this is the same
sequence in summary form):
1. **Empty / no-gold guard.** Whitespace answer → ABSTAINED.
Empty gold → JUDGE_ERROR ("no gold source").
2. **Abstention phrases.** Lexical scan for explicit refusals
("I cannot determine", "the source does not state", …). Hit →
ABSTAINED. ABSTAINED is NOT failure — it's honest about not
answering and matches the Opus-judge prompt's definition.
3. **NLI contradiction.** ``arborist.qa.nli.shadow_check`` against
the gold source. If ``max_contradiction >= theta_contra`` (pinned
manifest threshold, default 0.5) → WRONG. Strongest possible
signal: the source *contradicts* the answer.
4. **Lexical verifier vs gold.** ``arborist.qa.verify.verify_quotes``
("I cannot determine", "the source does not state", the
Hermes "I do not have access to information / context" family).
Hit → ABSTAINED. ABSTAINED is NOT failure — it's honest about
not answering and matches the Opus-judge prompt's definition.
3. **Short-answer entity-grounding fast path.** Runs BEFORE NLI
(since the 2026-05-19 Poland-Tusk smoke): short answer + every
asserted specific in gold + question subject anchor in gold →
CORRECT_GROUNDED. Catches "Pratibha Patil" against gold
"Pratibha Devisingh Patil …" and prevents NLI clause-level
aggregation from over-ruling positive lexical evidence.
4. **NLI contradiction.** ``arborist.qa.nli.shadow_check`` against
the gold source. ``max_contradiction >= 0.85`` (the code-judge
constant ``_CODE_JUDGE_THETA_CONTRA``, **raised** from the manifest
0.5 default after measuring 114 false-positives in the 0.5-0.75
band) → WRONG.
5. **Lexical verifier vs gold.** ``arborist.qa.verify.verify_quotes``
classifies the answer's grounding against the gold source as
STRICT / HYBRID / UNGROUNDED (the existing four-strategy ladder:
quote → span → entity → paraphrase). Maps to:
* STRICT → CORRECT_GROUNDED
* HYBRID + NLI entailment ≥ theta_entail → CORRECT_GROUNDED
* UNGROUNDED + specifics (proper-nouns or
dates not in gold) → FABRICATED
* HYBRID + NLI entail ≥ 0.55 → CORRECT_GROUNDED
* HYBRID + subj-in-gold + all specs in
gold → CORRECT_GROUNDED
(entity-grounding rescue)
* HYBRID + ≥1 verbatim quote + on-topic +
no unsourced numeric + not NLI-contra → CORRECT_GROUNDED
(2026-05-21 relaxed
rescue: unsourced
proper-nouns treated
as aliases / paraphrase,
wrong-numbers held back)
* HYBRID otherwise → JUDGE_ERROR
(ambiguous; residue
for LLM judge)
* UNGROUNDED + specifics + subj-in-gold → WRONG
(source has the topic
but a different value —
Opus's WRONG vs
FABRICATED distinction)
* UNGROUNDED + specifics, no subj → FABRICATED
(source silent on topic)
* UNGROUNDED + no specifics → ABSTAINED
(thin, unsupported,
but not a fabrication)
* HYBRID without NLI corroboration → JUDGE_ERROR
(ambiguous; residue
for LLM judge)
The JUDGE_ERROR class is a feature: ``rationale`` carries
``"code judge ambiguous: <reason>"`` so the residue is greppable for