docs+code: ground §12 judge pipeline in the actual judge_code.py
The §12.1 pipeline I added was second-hand from benchmark-matrix.md and got several things wrong against the code: - listed 4 verdicts; actual is 5 (missing FABRICATED — the fabrication-vs-WRONG split that energy-cogs §5.5 leans on for the qwen-fabricates / hermes-abstains finding) - "θ=0.85" was right by accident — but it's the code-judge-pinned _CODE_JUDGE_THETA_CONTRA constant, raised from the manifest 0.5 default after measuring 114 FPs in the 0.5-0.75 band - omitted the short-answer entity-grounding fast path (which runs BEFORE NLI per the 2026-05-19 Poland-Tusk smoke) - omitted the HYBRID rescue ladder (NLI entail / entity rescue / 2026-05-21 verbatim-quote-on-topic rescue) - conflated WRONG and FABRICATED (the subject-in-gold split is what distinguishes "source has the topic but a different value" from "source silent on the topic") Rewrote §12.1 grounded in `bench/judge_code.py:judge()` (its own docstring at line 501-528 is the truth on rule order), with file:line citations and the verdict-mapping in full. Also fixed a real artifact-vs-doc drift INSIDE the judge: the module-top docstring still claimed θ_contra default 0.5 and omitted the short-path and the WRONG/FABRICATED split. Updated to match the authoritative judge() docstring + current code. No behavior change — docstring + benchmarks doc only.
This commit is contained in:
parent
7f5ef5c140
commit
b6bb31a836
2 changed files with 97 additions and 27 deletions
|
|
@ -18,32 +18,57 @@ judging (Opus batched needle-haystack, or Grok credit-card) is a
|
|||
later, separate concern; it operates on the residue this judge can
|
||||
not classify deterministically.
|
||||
|
||||
The pipeline, in fixed order — first hit decides:
|
||||
The pipeline, in fixed order — first hit decides (the authoritative
|
||||
listing is the ``judge()`` docstring at line 501; this is the same
|
||||
sequence in summary form):
|
||||
|
||||
1. **Empty / no-gold guard.** Whitespace answer → ABSTAINED.
|
||||
Empty gold → JUDGE_ERROR ("no gold source").
|
||||
2. **Abstention phrases.** Lexical scan for explicit refusals
|
||||
("I cannot determine", "the source does not state", …). Hit →
|
||||
ABSTAINED. ABSTAINED is NOT failure — it's honest about not
|
||||
answering and matches the Opus-judge prompt's definition.
|
||||
3. **NLI contradiction.** ``arborist.qa.nli.shadow_check`` against
|
||||
the gold source. If ``max_contradiction >= theta_contra`` (pinned
|
||||
manifest threshold, default 0.5) → WRONG. Strongest possible
|
||||
signal: the source *contradicts* the answer.
|
||||
4. **Lexical verifier vs gold.** ``arborist.qa.verify.verify_quotes``
|
||||
("I cannot determine", "the source does not state", the
|
||||
Hermes "I do not have access to information / context" family).
|
||||
Hit → ABSTAINED. ABSTAINED is NOT failure — it's honest about
|
||||
not answering and matches the Opus-judge prompt's definition.
|
||||
3. **Short-answer entity-grounding fast path.** Runs BEFORE NLI
|
||||
(since the 2026-05-19 Poland-Tusk smoke): short answer + every
|
||||
asserted specific in gold + question subject anchor in gold →
|
||||
CORRECT_GROUNDED. Catches "Pratibha Patil" against gold
|
||||
"Pratibha Devisingh Patil …" and prevents NLI clause-level
|
||||
aggregation from over-ruling positive lexical evidence.
|
||||
4. **NLI contradiction.** ``arborist.qa.nli.shadow_check`` against
|
||||
the gold source. ``max_contradiction >= 0.85`` (the code-judge
|
||||
constant ``_CODE_JUDGE_THETA_CONTRA``, **raised** from the manifest
|
||||
0.5 default after measuring 114 false-positives in the 0.5-0.75
|
||||
band) → WRONG.
|
||||
5. **Lexical verifier vs gold.** ``arborist.qa.verify.verify_quotes``
|
||||
classifies the answer's grounding against the gold source as
|
||||
STRICT / HYBRID / UNGROUNDED (the existing four-strategy ladder:
|
||||
quote → span → entity → paraphrase). Maps to:
|
||||
* STRICT → CORRECT_GROUNDED
|
||||
* HYBRID + NLI entailment ≥ theta_entail → CORRECT_GROUNDED
|
||||
* UNGROUNDED + specifics (proper-nouns or
|
||||
dates not in gold) → FABRICATED
|
||||
* HYBRID + NLI entail ≥ 0.55 → CORRECT_GROUNDED
|
||||
* HYBRID + subj-in-gold + all specs in
|
||||
gold → CORRECT_GROUNDED
|
||||
(entity-grounding rescue)
|
||||
* HYBRID + ≥1 verbatim quote + on-topic +
|
||||
no unsourced numeric + not NLI-contra → CORRECT_GROUNDED
|
||||
(2026-05-21 relaxed
|
||||
rescue: unsourced
|
||||
proper-nouns treated
|
||||
as aliases / paraphrase,
|
||||
wrong-numbers held back)
|
||||
* HYBRID otherwise → JUDGE_ERROR
|
||||
(ambiguous; residue
|
||||
for LLM judge)
|
||||
* UNGROUNDED + specifics + subj-in-gold → WRONG
|
||||
(source has the topic
|
||||
but a different value —
|
||||
Opus's WRONG vs
|
||||
FABRICATED distinction)
|
||||
* UNGROUNDED + specifics, no subj → FABRICATED
|
||||
(source silent on topic)
|
||||
* UNGROUNDED + no specifics → ABSTAINED
|
||||
(thin, unsupported,
|
||||
but not a fabrication)
|
||||
* HYBRID without NLI corroboration → JUDGE_ERROR
|
||||
(ambiguous; residue
|
||||
for LLM judge)
|
||||
|
||||
The JUDGE_ERROR class is a feature: ``rationale`` carries
|
||||
``"code judge ambiguous: <reason>"`` so the residue is greppable for
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue