docs+code: ground §12 judge pipeline in the actual judge_code.py
The §12.1 pipeline I added was second-hand from benchmark-matrix.md and got several things wrong against the code: - listed 4 verdicts; actual is 5 (missing FABRICATED — the fabrication-vs-WRONG split that energy-cogs §5.5 leans on for the qwen-fabricates / hermes-abstains finding) - "θ=0.85" was right by accident — but it's the code-judge-pinned _CODE_JUDGE_THETA_CONTRA constant, raised from the manifest 0.5 default after measuring 114 FPs in the 0.5-0.75 band - omitted the short-answer entity-grounding fast path (which runs BEFORE NLI per the 2026-05-19 Poland-Tusk smoke) - omitted the HYBRID rescue ladder (NLI entail / entity rescue / 2026-05-21 verbatim-quote-on-topic rescue) - conflated WRONG and FABRICATED (the subject-in-gold split is what distinguishes "source has the topic but a different value" from "source silent on the topic") Rewrote §12.1 grounded in `bench/judge_code.py:judge()` (its own docstring at line 501-528 is the truth on rule order), with file:line citations and the verdict-mapping in full. Also fixed a real artifact-vs-doc drift INSIDE the judge: the module-top docstring still claimed θ_contra default 0.5 and omitted the short-path and the WRONG/FABRICATED split. Updated to match the authoritative judge() docstring + current code. No behavior change — docstring + benchmarks doc only.
This commit is contained in:
parent
7f5ef5c140
commit
b6bb31a836
2 changed files with 97 additions and 27 deletions
|
|
@ -18,32 +18,57 @@ judging (Opus batched needle-haystack, or Grok credit-card) is a
|
|||
later, separate concern; it operates on the residue this judge can
|
||||
not classify deterministically.
|
||||
|
||||
The pipeline, in fixed order — first hit decides:
|
||||
The pipeline, in fixed order — first hit decides (the authoritative
|
||||
listing is the ``judge()`` docstring at line 501; this is the same
|
||||
sequence in summary form):
|
||||
|
||||
1. **Empty / no-gold guard.** Whitespace answer → ABSTAINED.
|
||||
Empty gold → JUDGE_ERROR ("no gold source").
|
||||
2. **Abstention phrases.** Lexical scan for explicit refusals
|
||||
("I cannot determine", "the source does not state", …). Hit →
|
||||
ABSTAINED. ABSTAINED is NOT failure — it's honest about not
|
||||
answering and matches the Opus-judge prompt's definition.
|
||||
3. **NLI contradiction.** ``arborist.qa.nli.shadow_check`` against
|
||||
the gold source. If ``max_contradiction >= theta_contra`` (pinned
|
||||
manifest threshold, default 0.5) → WRONG. Strongest possible
|
||||
signal: the source *contradicts* the answer.
|
||||
4. **Lexical verifier vs gold.** ``arborist.qa.verify.verify_quotes``
|
||||
("I cannot determine", "the source does not state", the
|
||||
Hermes "I do not have access to information / context" family).
|
||||
Hit → ABSTAINED. ABSTAINED is NOT failure — it's honest about
|
||||
not answering and matches the Opus-judge prompt's definition.
|
||||
3. **Short-answer entity-grounding fast path.** Runs BEFORE NLI
|
||||
(since the 2026-05-19 Poland-Tusk smoke): short answer + every
|
||||
asserted specific in gold + question subject anchor in gold →
|
||||
CORRECT_GROUNDED. Catches "Pratibha Patil" against gold
|
||||
"Pratibha Devisingh Patil …" and prevents NLI clause-level
|
||||
aggregation from over-ruling positive lexical evidence.
|
||||
4. **NLI contradiction.** ``arborist.qa.nli.shadow_check`` against
|
||||
the gold source. ``max_contradiction >= 0.85`` (the code-judge
|
||||
constant ``_CODE_JUDGE_THETA_CONTRA``, **raised** from the manifest
|
||||
0.5 default after measuring 114 false-positives in the 0.5-0.75
|
||||
band) → WRONG.
|
||||
5. **Lexical verifier vs gold.** ``arborist.qa.verify.verify_quotes``
|
||||
classifies the answer's grounding against the gold source as
|
||||
STRICT / HYBRID / UNGROUNDED (the existing four-strategy ladder:
|
||||
quote → span → entity → paraphrase). Maps to:
|
||||
* STRICT → CORRECT_GROUNDED
|
||||
* HYBRID + NLI entailment ≥ theta_entail → CORRECT_GROUNDED
|
||||
* UNGROUNDED + specifics (proper-nouns or
|
||||
dates not in gold) → FABRICATED
|
||||
* HYBRID + NLI entail ≥ 0.55 → CORRECT_GROUNDED
|
||||
* HYBRID + subj-in-gold + all specs in
|
||||
gold → CORRECT_GROUNDED
|
||||
(entity-grounding rescue)
|
||||
* HYBRID + ≥1 verbatim quote + on-topic +
|
||||
no unsourced numeric + not NLI-contra → CORRECT_GROUNDED
|
||||
(2026-05-21 relaxed
|
||||
rescue: unsourced
|
||||
proper-nouns treated
|
||||
as aliases / paraphrase,
|
||||
wrong-numbers held back)
|
||||
* HYBRID otherwise → JUDGE_ERROR
|
||||
(ambiguous; residue
|
||||
for LLM judge)
|
||||
* UNGROUNDED + specifics + subj-in-gold → WRONG
|
||||
(source has the topic
|
||||
but a different value —
|
||||
Opus's WRONG vs
|
||||
FABRICATED distinction)
|
||||
* UNGROUNDED + specifics, no subj → FABRICATED
|
||||
(source silent on topic)
|
||||
* UNGROUNDED + no specifics → ABSTAINED
|
||||
(thin, unsupported,
|
||||
but not a fabrication)
|
||||
* HYBRID without NLI corroboration → JUDGE_ERROR
|
||||
(ambiguous; residue
|
||||
for LLM judge)
|
||||
|
||||
The JUDGE_ERROR class is a feature: ``rationale`` carries
|
||||
``"code judge ambiguous: <reason>"`` so the residue is greppable for
|
||||
|
|
|
|||
|
|
@ -439,18 +439,63 @@ LLM-judge running on this path.
|
|||
|
||||
### 12.1 Code judge — `bench/judge_code.py`
|
||||
|
||||
Deterministic, **no LLM, no quota**. First hit decides:
|
||||
empty/no-gold guard → explicit-abstention regex → short-answer
|
||||
entity-grounding fast path → **NLI contradiction** (θ=0.85, runs on the
|
||||
GPU box) → lexical verifier (quote/span/entity/paraphrase) →
|
||||
WRONG-vs-FABRICATED tie-break on subject-in-gold. Claim-lattice JSON
|
||||
envelopes are unwrapped to prose first so the substrate arm grades on
|
||||
equal terms with solo. It is a *proxy* for grounding (lexical + NLI), not
|
||||
SOTA reading. The `JUDGE_ERROR` residue (~10 % after the `2d31866`
|
||||
HYBRID-resolution strengthening; see energy-cogs §5.5) is the residue
|
||||
worth handing to a stronger judge if one is ever wired in.
|
||||
**Full pipeline:** `docs/benchmark-matrix.md` "## The judge"; used in
|
||||
context in `docs/energy-cogs-benchmark.md` §5.5.
|
||||
Deterministic, no LLM, no quota. **Five verdicts:**
|
||||
`CORRECT_GROUNDED` / `WRONG` / `FABRICATED` / `ABSTAINED` /
|
||||
`JUDGE_ERROR`. The `WRONG`-vs-`FABRICATED` split is what powers the
|
||||
bare-model failure-style finding in energy-cogs §5.5 (bare qwen
|
||||
fabricates, bare hermes abstains; the substrate fixes both).
|
||||
|
||||
Authoritative source: `bench/judge_code.py` `judge()` at line 493 — its
|
||||
own docstring (line 501-528) is the truth on rule order. Pipeline, first
|
||||
hit decides:
|
||||
|
||||
1. **Empty / no-gold guards.** Empty answer → `ABSTAINED`; empty gold →
|
||||
`JUDGE_ERROR`.
|
||||
2. **Explicit abstention phrases** — broad lexical regex including the
|
||||
"I do not have access to information / context" Hermes family that
|
||||
the original narrow patterns missed. Match → `ABSTAINED`.
|
||||
3. **Short-answer entity-grounding fast path.** When the answer is
|
||||
short, asserts at least one specific, every specific is in gold, AND
|
||||
the question's subject anchor is in gold → `CORRECT_GROUNDED`. Runs
|
||||
**before** NLI (since the 2026-05-19 Poland-Tusk smoke): NLI's
|
||||
clause-level aggregation can mis-fire as contradiction on multi-PM
|
||||
lists even when the answer is right; positive lexical evidence wins.
|
||||
4. **Strong NLI contradiction** — `arborist.qa.nli.shadow_check`,
|
||||
`θ_contra = 0.85` (code-judge-pinned `_CODE_JUDGE_THETA_CONTRA`,
|
||||
**raised** from the manifest's 0.5 default after measuring 114 FPs
|
||||
in the 0.5-0.75 band). `max_contradiction ≥ 0.85` → `WRONG`.
|
||||
5. **Lexical verifier vs gold** — `arborist.qa.verify.verify_quotes`
|
||||
(the existing quote → span → entity → paraphrase ladder), then map:
|
||||
- `STRICT` → `CORRECT_GROUNDED`
|
||||
- `HYBRID` + NLI entail ≥ 0.55 (`_CODE_JUDGE_THETA_ENTAIL_CORROBORATE`) → `CORRECT_GROUNDED`
|
||||
- `HYBRID` + subject-anchor-in-gold + every specific in gold → `CORRECT_GROUNDED` (entity-grounding rescue)
|
||||
- `HYBRID` + ≥1 verbatim quote verified + on-topic + **no unsourced
|
||||
numeric** specific + not NLI-contradicted → `CORRECT_GROUNDED`
|
||||
(2026-05-21 relaxed rescue, `2d31866`: unsourced proper-nouns
|
||||
treated as aliases / paraphrase, but a wrong date/count is held
|
||||
back)
|
||||
- `HYBRID` otherwise → `JUDGE_ERROR` (`"code judge ambiguous … residue
|
||||
for LLM judge"`)
|
||||
- `UNGROUNDED` + specifics-not-in-gold + subject-**in**-gold →
|
||||
`WRONG` (source has the topic but a different value)
|
||||
- `UNGROUNDED` + specifics-not-in-gold + subject-**not**-in-gold →
|
||||
`FABRICATED` (source silent on the topic)
|
||||
- `UNGROUNDED` + no asserted specifics → `ABSTAINED` (thin /
|
||||
non-answer, not a fabrication)
|
||||
|
||||
Claim-lattice JSON envelopes are unwrapped to prose
|
||||
(`_unwrap_claim_lattice_json`) before any rule runs, so substrate-CL
|
||||
grades on equal terms with solo-prose. Each `Verdict` carries a
|
||||
`decision` trace (rules_fired, NLI probabilities, subject anchor,
|
||||
specifics, audit_mode) — fully replayable. `JUDGE_ERROR` is reserved
|
||||
*only* for the `HYBRID`-without-corroboration class (the ~10 % residue
|
||||
after the `2d31866` HYBRID-rescue strengthening; energy-cogs §5.5);
|
||||
everything else lands on a concrete verdict.
|
||||
|
||||
Self-test: `bench/judge_code.py:self_test()` (`make judge-self-test`),
|
||||
gating before any control-ab run. The pipeline cross-reference (older,
|
||||
shorter) is `docs/benchmark-matrix.md` "## The judge"; used in context
|
||||
in `docs/energy-cogs-benchmark.md` §5.5.
|
||||
|
||||
### 12.2 Why there is no LLM judge in active use — `bench/judge.py` (abandoned)
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue