From b6bb31a836740e065a9d8ef0210dbd2511c89221 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Fri, 22 May 2026 19:17:28 -0400 Subject: [PATCH] =?UTF-8?q?docs+code:=20ground=20=C2=A712=20judge=20pipeli?= =?UTF-8?q?ne=20in=20the=20actual=20judge=5Fcode.py?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The §12.1 pipeline I added was second-hand from benchmark-matrix.md and got several things wrong against the code: - listed 4 verdicts; actual is 5 (missing FABRICATED — the fabrication-vs-WRONG split that energy-cogs §5.5 leans on for the qwen-fabricates / hermes-abstains finding) - "θ=0.85" was right by accident — but it's the code-judge-pinned _CODE_JUDGE_THETA_CONTRA constant, raised from the manifest 0.5 default after measuring 114 FPs in the 0.5-0.75 band - omitted the short-answer entity-grounding fast path (which runs BEFORE NLI per the 2026-05-19 Poland-Tusk smoke) - omitted the HYBRID rescue ladder (NLI entail / entity rescue / 2026-05-21 verbatim-quote-on-topic rescue) - conflated WRONG and FABRICATED (the subject-in-gold split is what distinguishes "source has the topic but a different value" from "source silent on the topic") Rewrote §12.1 grounded in `bench/judge_code.py:judge()` (its own docstring at line 501-528 is the truth on rule order), with file:line citations and the verdict-mapping in full. Also fixed a real artifact-vs-doc drift INSIDE the judge: the module-top docstring still claimed θ_contra default 0.5 and omitted the short-path and the WRONG/FABRICATED split. Updated to match the authoritative judge() docstring + current code. No behavior change — docstring + benchmarks doc only. --- bench/judge_code.py | 55 ++++++++++++++++++++++++++---------- docs/benchmarks.md | 69 +++++++++++++++++++++++++++++++++++++-------- 2 files changed, 97 insertions(+), 27 deletions(-) diff --git a/bench/judge_code.py b/bench/judge_code.py index b94c52d..b9c5f88 100644 --- a/bench/judge_code.py +++ b/bench/judge_code.py @@ -18,32 +18,57 @@ judging (Opus batched needle-haystack, or Grok credit-card) is a later, separate concern; it operates on the residue this judge can not classify deterministically. -The pipeline, in fixed order — first hit decides: +The pipeline, in fixed order — first hit decides (the authoritative +listing is the ``judge()`` docstring at line 501; this is the same +sequence in summary form): 1. **Empty / no-gold guard.** Whitespace answer → ABSTAINED. Empty gold → JUDGE_ERROR ("no gold source"). 2. **Abstention phrases.** Lexical scan for explicit refusals - ("I cannot determine", "the source does not state", …). Hit → - ABSTAINED. ABSTAINED is NOT failure — it's honest about not - answering and matches the Opus-judge prompt's definition. - 3. **NLI contradiction.** ``arborist.qa.nli.shadow_check`` against - the gold source. If ``max_contradiction >= theta_contra`` (pinned - manifest threshold, default 0.5) → WRONG. Strongest possible - signal: the source *contradicts* the answer. - 4. **Lexical verifier vs gold.** ``arborist.qa.verify.verify_quotes`` + ("I cannot determine", "the source does not state", the + Hermes "I do not have access to information / context" family). + Hit → ABSTAINED. ABSTAINED is NOT failure — it's honest about + not answering and matches the Opus-judge prompt's definition. + 3. **Short-answer entity-grounding fast path.** Runs BEFORE NLI + (since the 2026-05-19 Poland-Tusk smoke): short answer + every + asserted specific in gold + question subject anchor in gold → + CORRECT_GROUNDED. Catches "Pratibha Patil" against gold + "Pratibha Devisingh Patil …" and prevents NLI clause-level + aggregation from over-ruling positive lexical evidence. + 4. **NLI contradiction.** ``arborist.qa.nli.shadow_check`` against + the gold source. ``max_contradiction >= 0.85`` (the code-judge + constant ``_CODE_JUDGE_THETA_CONTRA``, **raised** from the manifest + 0.5 default after measuring 114 false-positives in the 0.5-0.75 + band) → WRONG. + 5. **Lexical verifier vs gold.** ``arborist.qa.verify.verify_quotes`` classifies the answer's grounding against the gold source as STRICT / HYBRID / UNGROUNDED (the existing four-strategy ladder: quote → span → entity → paraphrase). Maps to: * STRICT → CORRECT_GROUNDED - * HYBRID + NLI entailment ≥ theta_entail → CORRECT_GROUNDED - * UNGROUNDED + specifics (proper-nouns or - dates not in gold) → FABRICATED + * HYBRID + NLI entail ≥ 0.55 → CORRECT_GROUNDED + * HYBRID + subj-in-gold + all specs in + gold → CORRECT_GROUNDED + (entity-grounding rescue) + * HYBRID + ≥1 verbatim quote + on-topic + + no unsourced numeric + not NLI-contra → CORRECT_GROUNDED + (2026-05-21 relaxed + rescue: unsourced + proper-nouns treated + as aliases / paraphrase, + wrong-numbers held back) + * HYBRID otherwise → JUDGE_ERROR + (ambiguous; residue + for LLM judge) + * UNGROUNDED + specifics + subj-in-gold → WRONG + (source has the topic + but a different value — + Opus's WRONG vs + FABRICATED distinction) + * UNGROUNDED + specifics, no subj → FABRICATED + (source silent on topic) * UNGROUNDED + no specifics → ABSTAINED (thin, unsupported, but not a fabrication) - * HYBRID without NLI corroboration → JUDGE_ERROR - (ambiguous; residue - for LLM judge) The JUDGE_ERROR class is a feature: ``rationale`` carries ``"code judge ambiguous: "`` so the residue is greppable for diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 06ce8b4..434a42e 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -439,18 +439,63 @@ LLM-judge running on this path. ### 12.1 Code judge — `bench/judge_code.py` -Deterministic, **no LLM, no quota**. First hit decides: -empty/no-gold guard → explicit-abstention regex → short-answer -entity-grounding fast path → **NLI contradiction** (θ=0.85, runs on the -GPU box) → lexical verifier (quote/span/entity/paraphrase) → -WRONG-vs-FABRICATED tie-break on subject-in-gold. Claim-lattice JSON -envelopes are unwrapped to prose first so the substrate arm grades on -equal terms with solo. It is a *proxy* for grounding (lexical + NLI), not -SOTA reading. The `JUDGE_ERROR` residue (~10 % after the `2d31866` -HYBRID-resolution strengthening; see energy-cogs §5.5) is the residue -worth handing to a stronger judge if one is ever wired in. -**Full pipeline:** `docs/benchmark-matrix.md` "## The judge"; used in -context in `docs/energy-cogs-benchmark.md` §5.5. +Deterministic, no LLM, no quota. **Five verdicts:** +`CORRECT_GROUNDED` / `WRONG` / `FABRICATED` / `ABSTAINED` / +`JUDGE_ERROR`. The `WRONG`-vs-`FABRICATED` split is what powers the +bare-model failure-style finding in energy-cogs §5.5 (bare qwen +fabricates, bare hermes abstains; the substrate fixes both). + +Authoritative source: `bench/judge_code.py` `judge()` at line 493 — its +own docstring (line 501-528) is the truth on rule order. Pipeline, first +hit decides: + +1. **Empty / no-gold guards.** Empty answer → `ABSTAINED`; empty gold → + `JUDGE_ERROR`. +2. **Explicit abstention phrases** — broad lexical regex including the + "I do not have access to information / context" Hermes family that + the original narrow patterns missed. Match → `ABSTAINED`. +3. **Short-answer entity-grounding fast path.** When the answer is + short, asserts at least one specific, every specific is in gold, AND + the question's subject anchor is in gold → `CORRECT_GROUNDED`. Runs + **before** NLI (since the 2026-05-19 Poland-Tusk smoke): NLI's + clause-level aggregation can mis-fire as contradiction on multi-PM + lists even when the answer is right; positive lexical evidence wins. +4. **Strong NLI contradiction** — `arborist.qa.nli.shadow_check`, + `θ_contra = 0.85` (code-judge-pinned `_CODE_JUDGE_THETA_CONTRA`, + **raised** from the manifest's 0.5 default after measuring 114 FPs + in the 0.5-0.75 band). `max_contradiction ≥ 0.85` → `WRONG`. +5. **Lexical verifier vs gold** — `arborist.qa.verify.verify_quotes` + (the existing quote → span → entity → paraphrase ladder), then map: + - `STRICT` → `CORRECT_GROUNDED` + - `HYBRID` + NLI entail ≥ 0.55 (`_CODE_JUDGE_THETA_ENTAIL_CORROBORATE`) → `CORRECT_GROUNDED` + - `HYBRID` + subject-anchor-in-gold + every specific in gold → `CORRECT_GROUNDED` (entity-grounding rescue) + - `HYBRID` + ≥1 verbatim quote verified + on-topic + **no unsourced + numeric** specific + not NLI-contradicted → `CORRECT_GROUNDED` + (2026-05-21 relaxed rescue, `2d31866`: unsourced proper-nouns + treated as aliases / paraphrase, but a wrong date/count is held + back) + - `HYBRID` otherwise → `JUDGE_ERROR` (`"code judge ambiguous … residue + for LLM judge"`) + - `UNGROUNDED` + specifics-not-in-gold + subject-**in**-gold → + `WRONG` (source has the topic but a different value) + - `UNGROUNDED` + specifics-not-in-gold + subject-**not**-in-gold → + `FABRICATED` (source silent on the topic) + - `UNGROUNDED` + no asserted specifics → `ABSTAINED` (thin / + non-answer, not a fabrication) + +Claim-lattice JSON envelopes are unwrapped to prose +(`_unwrap_claim_lattice_json`) before any rule runs, so substrate-CL +grades on equal terms with solo-prose. Each `Verdict` carries a +`decision` trace (rules_fired, NLI probabilities, subject anchor, +specifics, audit_mode) — fully replayable. `JUDGE_ERROR` is reserved +*only* for the `HYBRID`-without-corroboration class (the ~10 % residue +after the `2d31866` HYBRID-rescue strengthening; energy-cogs §5.5); +everything else lands on a concrete verdict. + +Self-test: `bench/judge_code.py:self_test()` (`make judge-self-test`), +gating before any control-ab run. The pipeline cross-reference (older, +shorter) is `docs/benchmark-matrix.md` "## The judge"; used in context +in `docs/energy-cogs-benchmark.md` §5.5. ### 12.2 Why there is no LLM judge in active use — `bench/judge.py` (abandoned)