#!/usr/bin/env python3 """Code-only judge for the #000057 control experiment — no LLM, no quota. Drop-in alternative to ``bench/judge.py`` (Opus headless via ``claude -p``). Same :class:`bench.judge.Verdict`-shaped dataclass, same closed verdict vocabulary (``CORRECT_GROUNDED`` / ``WRONG`` / ``FABRICATED`` / ``ABSTAINED`` / ``JUDGE_ERROR``), same ``judge(question, answer, gold_source)`` entry point — so a caller can swap one for the other without changing its scoring/aggregation code. **Why a code judge first** (fox 2026-05-19): the Opus judge burned our Anthropic quota on the huge-N sweep. A deterministic code judge gives us the data-collection arm we need RIGHT NOW (zero quota, free, replayable, no model-family hygiene threat — same-family judging is the Opus-judge's live limitation per ``judge.py:26-29``). LLM-based judging (Opus batched needle-haystack, or Grok credit-card) is a later, separate concern; it operates on the residue this judge can not classify deterministically. The pipeline, in fixed order — first hit decides (the authoritative listing is the ``judge()`` docstring at line 501; this is the same sequence in summary form): 1. **Empty / no-gold guard.** Whitespace answer → ABSTAINED. Empty gold → JUDGE_ERROR ("no gold source"). 2. **Abstention phrases.** Lexical scan for explicit refusals ("I cannot determine", "the source does not state", the Hermes "I do not have access to information / context" family). Hit → ABSTAINED. ABSTAINED is NOT failure — it's honest about not answering and matches the Opus-judge prompt's definition. 3. **Short-answer entity-grounding fast path.** Runs BEFORE NLI (since the 2026-05-19 Poland-Tusk smoke): short answer + every asserted specific in gold + question subject anchor in gold → CORRECT_GROUNDED. Catches "Pratibha Patil" against gold "Pratibha Devisingh Patil …" and prevents NLI clause-level aggregation from over-ruling positive lexical evidence. 4. **NLI contradiction.** ``arborist.qa.nli.shadow_check`` against the gold source. ``max_contradiction >= 0.85`` (the code-judge constant ``_CODE_JUDGE_THETA_CONTRA``, **raised** from the manifest 0.5 default after measuring 114 false-positives in the 0.5-0.75 band) → WRONG. 5. **Lexical verifier vs gold.** ``arborist.qa.verify.verify_quotes`` classifies the answer's grounding against the gold source as STRICT / HYBRID / UNGROUNDED (the existing four-strategy ladder: quote → span → entity → paraphrase). Maps to: * STRICT → CORRECT_GROUNDED * HYBRID + NLI entail ≥ 0.55 → CORRECT_GROUNDED * HYBRID + subj-in-gold + all specs in gold → CORRECT_GROUNDED (entity-grounding rescue) * HYBRID + ≥1 verbatim quote + on-topic + no unsourced numeric + not NLI-contra → CORRECT_GROUNDED (2026-05-21 relaxed rescue: unsourced proper-nouns treated as aliases / paraphrase, wrong-numbers held back) * HYBRID otherwise → JUDGE_ERROR (ambiguous; residue for LLM judge) * UNGROUNDED + specifics + subj-in-gold → WRONG (source has the topic but a different value — Opus's WRONG vs FABRICATED distinction) * UNGROUNDED + specifics, no subj → FABRICATED (source silent on topic) * UNGROUNDED + no specifics → ABSTAINED (thin, unsupported, but not a fabrication) The JUDGE_ERROR class is a feature: ``rationale`` carries ``"code judge ambiguous: "`` so the residue is greppable for later promotion to an LLM judge. Existing callers (``control_sweep.py``, ``control_ab.py``) already treat JUDGE_ERROR non-fatally and count it in the JE bucket. Threats to validity, stated: * Lexical grounding is a *proxy* for "supported by source" — the Opus judge can read & reason. Code judge can not. STRICT means "verbatim/span present in gold"; an answer can be STRICT but still subtly wrong if the gold's claim differs in interpretation. NLI guards against that (an answer whose proposition gold contradicts will trip rule 3 BEFORE rule 4 sees STRICT). * NLI is optional (``[nli]`` extra). When the model is unavailable we fall back to rules 1+2+4 only and label the rationale so the operator sees the degradation. * Specificity heuristic for FABRICATED has false-negatives on answers that fabricate WITHOUT proper nouns or numbers ("the answer to that is yes" with no grounding) — those will land in ABSTAINED, the conservative class. False-positives on STRICT answers cannot happen (STRICT pre-empts the FABRICATED branch). Self-test (``python -m bench.judge_code --self-test``) covers the same 4 fixtures as ``judge.py:self_test()`` so the two instruments can be cross-checked when fox does eventually re-fire the Opus judge on the residue. """ from __future__ import annotations import json import os import re import sys from dataclasses import dataclass, field from arborist.qa.verify import verify_quotes, extract_proper_nouns JUDGE_RULESET_ID = "code-judge-v1" JUDGE_MODEL = "code (deterministic, no LLM)" # Opus-judge parity: same closed vocabulary so a Verdict from this # module aggregates against the same buckets without translation. _VERDICTS = ("CORRECT_GROUNDED", "WRONG", "FABRICATED", "ABSTAINED") # Code-judge NLI corroboration threshold (rule 4 — HYBRID lexical + # NLI entailment → CORRECT_GROUNDED). Distinct from the NLI manifest's # ``entailment_block_veto=0.9``, which is calibrated for the OPPOSITE # use case (high bar to override a STRICT lexical signal — additive # negative evidence needs strong confidence). Corroborating a HYBRID # lexical signal is additive POSITIVE evidence on an already-positive # anchor; a moderate threshold is appropriate. Self-test fixture #1 # ("World War II ended in 1945." vs gold mentioning "it ended in 1945") # measures NLI entailment=0.769 — clearly entailed, well clear of the # 0.55 floor used here. Anything below 0.55 defers to JUDGE_ERROR # (residue for an LLM judge later) rather than risk a false promote. _CODE_JUDGE_THETA_ENTAIL_CORROBORATE = 0.55 # Code-judge NLI contradiction threshold (rule 3 — gold contradicts # the answer → WRONG). The 2026-05-19 reconciliation against the # trusted Opus judge measured **114 records** where the code judge # fired WRONG via NLI contradiction p ∈ [0.51, 0.74] on answers Opus # correctly graded CG — clear factual claims like "Ólafur Ragnar # Grímsson is the president of Iceland" against the corpus infobox # that DOES list him as incumbent. NLI is noisy in the 0.5–0.8 band # on wikitext-shaped gold (clause-level candidate selection picks up # co-mentioned earlier office-holders & reads temporal-frame mismatch # as contradiction). For TRUE contradictions the signal is far higher # (the WW2 self-test fixture #2 measures p=0.985). Threshold raised # from the manifest's 0.5 → 0.85 to keep the strong contradictions # (cleanly above the floor) while dropping the noise band. _CODE_JUDGE_THETA_CONTRA = 0.85 # Short-answer threshold (rule 3.5 — terse correct-noun answers). # The verifier's strategy 2 needs prose-shaped spans; very short # answers ("Pratibha Patil", "Jalal Talabani", "In 1945.") fall # through to UNGROUNDED-no-specifics → ABSTAINED, missing valid CG. # The reconciliation against Opus identified 13 such records. # Compensate by accepting a short answer as grounded when every # specific (proper noun + numeric) it asserts is present in gold AND # the question's subject anchor is present in gold (catches "wrong # topic, right name" false-positives). _CODE_JUDGE_SHORT_ANSWER_TOKENS = 15 @dataclass class Verdict: label: str # one of _VERDICTS, or "JUDGE_ERROR" rationale: str raw: str # serialised decision trace (logged for replay) prompt_id: str = JUDGE_RULESET_ID model: str = JUDGE_MODEL # Surface the decision trace separately for programmatic callers # who want to bucket by which rule fired. `raw` is the same data # JSON-serialised — both are kept so the human-facing ``judge.py`` # interface is preserved. decision: dict = field(default_factory=dict) # ----------------------------------------------------------- preprocess # Claim-lattice JSON envelope detector. Arborist's `claim_lattice` # answer_mode emits {"claims":[{"text":"...","evidence_ids":["E1"]},...]}. # Without unwrapping, the verifier sees brace + key syntax noise # instead of the actual claim prose — the proper-noun & span # extractors give weak signal, every Arborist record falls to # UNGROUNDED. Unwrapping to concatenated sentences lets the existing # lexical paths handle these uniformly with plain-prose answers, # preserving the judge's per-record cost (no special-case grader). def _unwrap_claim_lattice_json(answer: str) -> str: """Extract claim text(s) from claim-lattice JSON envelopes. Returns the original answer when it isn't claim-lattice shaped, so plain-prose answers pass through unchanged. The detection is conservative: must start with ``{`` AND contain ``"claims"`` AND contain ``"text"`` — three independent signals that this is the Arborist envelope rather than coincidental JSON. Parse failures fall back to the original answer (let the downstream verifier see what it gets — no silent rewriting on malformed input).""" a = (answer or "").strip() if not (a.startswith("{") and '"claims"' in a and '"text"' in a): return answer try: obj = json.loads(a) except Exception: # noqa: BLE001 — malformed JSON, let it through return answer if not isinstance(obj, dict): return answer claims = obj.get("claims") if not isinstance(claims, list): return answer texts: list[str] = [] for c in claims: if isinstance(c, dict): t = c.get("text") if isinstance(t, str) and t.strip(): texts.append(t.strip()) if not texts: return answer # Concatenate as discrete sentences so verify.extract_claim_spans # can pick each one up as its own span. Append a period only when # the claim text doesn't already end in sentence punctuation. return " ".join( t if t[-1:] in ".!?:;" else (t + ".") for t in texts ) # ----------------------------------------------------------- rule 2 # Lexical abstention phrases. Kept conservative: only fires when the # answer EXPLICITLY says it can not / will not answer or that the # source does not contain the information. A model that answers # "I think it is X" is NOT abstaining — it is making a claim. # # Pattern set expanded 2026-05-19 from the Opus-reconciliation # residue: Hermes's most common abstention phrasings under # `--variant source_relative` are "I do not have accurate information # about", "I do not have access to real-time / a reference knowledge # base to", "I do not have enough information to". The original # patterns missed all three (they require a verb like determine/know # right after "do not"). Eighteen records mis-classified as WRONG # trace to this gap. _ABSTAIN_PATTERNS = ( re.compile(r"\b(?:i\s+(?:cannot|can[' ]?t|do\s+not|don[' ]?t)\s+" r"(?:determine|know|tell|say|find|locate|verify))\b", re.I), # NEW: "I do not have information / access / data ..." re.compile(r"\bi\s+(?:do\s+not|don[' ]?t|cannot|can[' ]?t)\s+" r"have\s+(?:\w+\s+){0,4}" r"(?:information|access|data|knowledge|details)\b", re.I), # NEW: "I do not have access to ..." stand-alone re.compile(r"\bi\s+(?:do\s+not|don[' ]?t)\s+have\s+access\b", re.I), # NEW: "I lack (access / information / knowledge)" re.compile(r"\bi\s+lack\s+(?:\w+\s+){0,3}" r"(?:access|information|knowledge|data|details)\b", re.I), re.compile(r"\b(?:not\s+(?:enough|sufficient)\s+information)\b", re.I), re.compile(r"\b(?:no\s+information\s+(?:is\s+)?(?:available|provided|given))\b", re.I), re.compile(r"\b(?:the\s+(?:source|article|text|context|passage)\s+" r"(?:does\s+not|doesn[' ]?t|fails\s+to)\s+" r"(?:contain|mention|state|specify|provide|include))\b", re.I), re.compile(r"\b(?:unable\s+to\s+(?:determine|answer|verify|locate))\b", re.I), re.compile(r"\b(?:no\s+(?:answer|response)\s+(?:can\s+be|is)\s+(?:given|provided|made))\b", re.I), re.compile(r"\b(?:cannot\s+be\s+(?:determined|answered|found|verified)\s+from)\b", re.I), ) def _is_abstention(answer: str) -> tuple[bool, str]: """Hit on any of the explicit-refusal patterns above. Returns ``(is_abstention, matched_pattern_excerpt)``.""" for pat in _ABSTAIN_PATTERNS: m = pat.search(answer) if m: return True, m.group(0) return False, "" # ----------------------------------------------------------- rule 3 def _nli_check(answer: str, gold: str): """Run NLI shadow on (gold, answer). Returns the :class:`arborist.qa.nli.ShadowResult` or ``None`` when the ``[nli]`` extra is unavailable. Note: ``shadow_check`` itself returns an unavailable-result object rather than raising, so the actual ``None`` branch here is only the import-failure path.""" try: from arborist.qa.nli import shadow_check except Exception: # noqa: BLE001 — fail-open to lexical-only judging return None try: return shadow_check(claim=answer, source=gold) except Exception: # noqa: BLE001 return None # ----------------------------------------------------------- rule 4 # Tokens that are evidence the answer asserts SPECIFICS without # grounding — proper nouns (multi-word via verify.extract_proper_nouns # AND single-word via the local scanner below) and bare numerics that # look like dates / counts / measurements. _NUMERIC_SPECIFIC_RE = re.compile( r"\b(?:\d{2,4}(?:st|nd|rd|th)?|\d+\.\d+|\d{3,}|" # 4-digit year, ordinal year, decimal, large count r"\d{1,2}/\d{1,2}(?:/\d{2,4})?|" # date 1/2 or 1/2/34 r"\$\d[\d,]*(?:\.\d+)?)\b" ) # Single-word capitalised tokens. The verifier's # ``extract_proper_nouns`` is deliberately multi-word-only (to keep # the verifier's lexical contract conservative — single capitalised # tokens are noisy as evidence of grounding). For the FABRICATION # check we go the other direction: we want to catch "Napoleon" alone # when it appears in an UNGROUNDED answer with no source backing. # Sentence-leading tokens get filtered as a soft-noise reduction. _SINGLE_CAP_TOKEN_RE = re.compile(r"\b[A-Z][a-z]{2,}\b") # Common English words that frequently appear sentence-leading and # would otherwise produce noise (the FABRICATED class is about asserted # proper-noun-shaped specifics, not "The"/"This"/"It"). Lower-case # comparison so the filter is shape-agnostic. _CAP_TOKEN_STOPWORDS = frozenset({ "the", "this", "that", "these", "those", "it", "its", "they", "their", "there", "then", "thus", "such", "some", "all", "any", "many", "most", "much", "more", "less", "few", "several", "each", "every", "and", "but", "or", "nor", "yet", "so", "for", "however", "moreover", "also", "though", "although", "while", "since", "because", "if", "when", "where", "what", "which", "who", "whom", "whose", "why", "how", "is", "are", "was", "were", "be", "been", "being", "has", "have", "had", "do", "does", "did", "can", "could", "should", "would", "may", "might", "must", "shall", "will", "according", "based", "during", "after", "before", "given", "let", "note", "see", "from", "with", "without", "into", "onto", "upon", "about", "around", }) def _all_answer_specifics(answer: str) -> list[str]: """Every specific (proper noun + numeric) the answer asserts. Symmetric with :func:`_specifics_not_in_gold` but unfiltered — used to check 'this answer asserted SOMETHING specific' before we decide whether to call it ABSTAINED.""" out: list[str] = [] seen: set[str] = set() def _add(tok: str): k = tok.lower() if k in seen: return out.append(tok) seen.add(k) for noun in extract_proper_nouns(answer): _add(noun) for m in _SINGLE_CAP_TOKEN_RE.finditer(answer): tok = m.group(0) if tok.lower() in _CAP_TOKEN_STOPWORDS: continue _add(tok) for m in _NUMERIC_SPECIFIC_RE.finditer(answer): _add(m.group(0)) return out # Stopwords for the question-subject-anchor extractor. Mirrors the # inspect-module convention: drop function words / WH-question words / # generic role nouns so the LAST remaining content token is the # question's primary subject ("Iceland" in "who is the president of # Iceland?"; "Australia" in "according to the reference knowledge base, # who is the prime minister of Australia?"). The anchor is also # tolerant of the variant-rewriting prefixes (`According to ...`, # `As of 2010 ...`) because those add to the front, not the back. _SUBJ_STOPWORDS = frozenset({ "who", "what", "when", "where", "why", "how", "which", "whose", "whom", "is", "are", "was", "were", "be", "been", "being", "has", "have", "had", "do", "does", "did", "the", "a", "an", "of", "in", "on", "at", "for", "by", "to", "from", "as", "according", "reference", "knowledge", "base", "current", "currently", "today", "now", "presently", "office", "head", "leader", # Generic role nouns kept OUT of the subject — they describe the # *role*, the polity is the subject. "who is the prime minister of # Australia" → subject = australia, not minister. "president", "prime", "minister", "chancellor", "premier", "governor", "general", "secretary", "king", "queen", "emperor", "leader", "chairman", "chair", "speaker", "monarch", "ruler", }) _TOKEN_RE = re.compile(r"\b\w+\b", re.U) def _question_subject_anchor(question: str) -> str: """Last content token after stopword-strip — the inspect-module fallback used when the question carries no proper-noun-shaped terms. Returns lowercase or empty string.""" toks = [t.lower() for t in _TOKEN_RE.findall(question or "")] content = [t for t in toks if t not in _SUBJ_STOPWORDS and not t.isdigit()] return content[-1] if content else "" def _question_subject_terms(question: str) -> list[str]: """Proper-noun-shaped subject terms from the question. Tries extract_proper_nouns (multi-word PNs) first, then layers single capitalised tokens (modulo stopwords). For "who is the president of Iceland?" → ["Iceland"]. For "who founded the Higgs boson cafe in 1066?" → ["Higgs"]. For "who is the president of the Quorum of the Twelve Apostles?" → ["Quorum", "Twelve", "Apostles"] (whichever the multi-word extractor surfaces + single caps the extractor misses). Used as the strong-anchor signal for ``_subject_in_gold``: a proper-noun-shaped term in the question is the topic's identity, not just a co-occurring stopword ("cafe" matching gold's "any cafe" denial is the bug this avoids). Returns [] when the question has no proper-noun-shaped term — caller falls back to the last-content-token anchor.""" out: list[str] = [] seen: set[str] = set() def _add(t: str): k = t.lower() if k in seen or k in _SUBJ_STOPWORDS or k in _CAP_TOKEN_STOPWORDS: return out.append(t) seen.add(k) for pn in extract_proper_nouns(question or ""): # Multi-word PNs land whole — record both the phrase and its # head tokens for substring matching against gold. _add(pn) for tok in pn.split(): _add(tok) for m in _SINGLE_CAP_TOKEN_RE.finditer(question or ""): _add(m.group(0)) return out def _subject_in_gold(question: str, gold: str) -> tuple[bool, str]: """True iff the question's subject is present in gold. Strong anchor: proper-noun-shaped terms from the question — ALL such terms must appear in gold (case-insensitive substring). Conservative because the WRONG-vs-FABRICATED tie-break demotes FABRICATED → WRONG when this fires, and we'd rather under-fire (keep FABRICATED) than over-fire (the Higgs-boson-cafe gold coincidentally contains "cafe" in a denial — single-token anchor would mis-fire, multi-PN anchor doesn't). Returns ``(found, anchor)`` where ``anchor`` records which term decided it (for the decision trace). Falls back to the last- content-token heuristic when the question has no proper-noun- shaped terms (rare on this corpus).""" gold_lc = gold.lower() terms = _question_subject_terms(question) if terms: # Require every proper-noun-shaped term to appear in gold — # all the topic's identity tokens, not just one. missing = [t for t in terms if t.lower() not in gold_lc] if not missing: return True, terms[0] return False, missing[0] # Fallback for proper-noun-free questions: the last-content-token # anchor (was the original heuristic before reconciliation). anchor = _question_subject_anchor(question) if not anchor: return False, "" return (anchor in gold_lc), anchor def _is_short_answer(answer: str) -> bool: """The verifier's strategy-2 span extractor needs prose; very short answers fall through to UNGROUNDED-no-specifics. Treat as 'short' for the entity-grounding fast path.""" return len(_TOKEN_RE.findall(answer or "")) <= _CODE_JUDGE_SHORT_ANSWER_TOKENS def _specifics_not_in_gold(answer: str, gold: str) -> list[str]: """Proper-noun and numeric specifics that the answer asserts but the gold source never mentions. Three layers: 1. ``extract_proper_nouns`` from the verifier — multi-word capitalised entities (e.g. "Higgs Boson", "World War II"). 2. Local single-word capitalised-token scanner — catches "Napoleon" / "Mars" / "Mozart" that the verifier's multi-word gate filters out. 3. Numeric specifics — dates, years, large counts, money. Used only on the UNGROUNDED branch: a STRICT or HYBRID answer pre-empts this path and does NOT get tagged as FABRICATED.""" out: list[str] = [] seen: set[str] = set() gold_lc = gold.lower() def _add(tok: str): k = tok.lower() if k in seen: return if k not in gold_lc: out.append(tok) seen.add(k) # Layer 1: verifier's multi-word proper-noun extractor. for noun in extract_proper_nouns(answer): _add(noun) # Layer 2: single-word capitalised tokens (modulo sentence-leading # noise — drop tokens immediately following a sentence terminator # OR at position 0, AND in the stopword set). sentence_lead_positions: set[int] = {0} for m in re.finditer(r"[.!?]\s+(?=[A-Z])", answer): sentence_lead_positions.add(m.end()) for m in _SINGLE_CAP_TOKEN_RE.finditer(answer): tok = m.group(0) if tok.lower() in _CAP_TOKEN_STOPWORDS: continue # If sentence-leading AND a 3-letter-or-less stop-shaped token, # ignore. But "Napoleon" sentence-leading is still a specific. # We accept any sentence-leading non-stopword that's >=3 chars. _add(tok) # Layer 3: numerics. for m in _NUMERIC_SPECIFIC_RE.finditer(answer): _add(m.group(0)) return out # ----------------------------------------------------------- judge() def judge(question: str, answer: str, gold_source: str) -> Verdict: """Deterministic verdict on (question, answer, gold_source). Same shape & verdict vocabulary as ``bench.judge.judge``; no network, no LLM, no quota cost. JUDGE_ERROR is reserved for the HYBRID-without-NLI-corroboration class so a downstream LLM judge (Opus batched, Grok) can pick that residue up later. Rule order (reconciled against Opus 2026-05-19): 1. empty / no-gold guards 2. explicit abstention phrases (lexical regex; broad set covering the Hermes "I do not have ... information / access" family that the original narrow patterns missed) 3. strong NLI contradiction (theta_contra=0.85, raised from 0.5 after measuring 114 false-positives in the 0.5-0.75 band) 4. short-answer entity-grounding fast path — for terse answers where the verifier's prose-shape extractor would whiff, accept CG when every asserted specific is in gold AND the question's subject anchor is in gold (catches "Pratibha Patil" → gold "Pratibha Devisingh Patil ..." → CG) 5. lexical verifier (verify_quotes) STRICT → CG HYBRID + NLI entail ≥ 0.55 → CG HYBRID + subject-anchor + all-grounded → CG (entity- grounding rescue) HYBRID otherwise → JUDGE_ERROR (residue) UNGROUNDED + specifics + subject-in-gold → WRONG (source contradicts — Opus's WRONG vs FABRICATED distinction) UNGROUNDED + specifics, no subject → FABRICATED UNGROUNDED + no specifics → ABSTAINED """ raw_answer = (answer or "").strip() trace: dict = {"rules_fired": []} # Preprocess — unwrap claim-lattice JSON envelopes BEFORE any # downstream check. After this point ``a`` is plain prose for both # arms (solo & arborist-claim_lattice), and the verifier / NLI / # specifics paths grade them on equal terms. a = _unwrap_claim_lattice_json(raw_answer) if a != raw_answer: trace["lattice_unwrapped"] = True trace["unwrapped_len"] = len(a) g = (gold_source or "").strip() # Rule 1 — empty / no-gold guards. if not a: trace["rules_fired"].append("empty_answer") return _v("ABSTAINED", "empty answer", trace) if not g: trace["rules_fired"].append("no_gold") return _v("JUDGE_ERROR", "no gold source — cannot ground", trace) # Rule 2 — explicit abstention phrase. is_abs, abs_match = _is_abstention(a) if is_abs: trace["rules_fired"].append("abstention_phrase") trace["abstain_match"] = abs_match return _v("ABSTAINED", f"explicit refusal: {abs_match!r}", trace) # Pre-compute shared signals once. nli = _nli_check(a, g) nli_avail = nli is not None and nli.available trace["nli_available"] = nli_avail if nli_avail: trace["nli_max_contra"] = nli.max_contradiction trace["nli_max_entail"] = nli.max_entailment subj_in_gold, subj_anchor = _subject_in_gold(question, g) trace["subject_anchor"] = subj_anchor trace["subject_in_gold"] = subj_in_gold # Rule 3 — short-answer entity-grounding fast path. RUNS BEFORE # NLI contradiction (since 2026-05-19 Poland-Tusk smoke): when the # answer asserts only specifics that are ALL in gold AND the # question's subject anchor is in gold, that is strong positive # evidence — NLI clause-level noise cannot override it. The # Poland case (qwen+arborist answer "Donald Tusk is listed as the # Prime Minister of Poland.", gold lists multiple PMs across # decades) measured NLI contradiction p=0.89 ABOVE the 0.85 # threshold while NLI entailment was also 0.74 on the correct # clause — mixed signal. The fast path's positive-evidence # combination (specifics-in-gold + subject-in-gold) discriminates # truth from contradiction without depending on NLI's clause- # level aggregation. Rescue when: (a) answer is short, (b) every # specific asserted is present in gold (no unsourced specifics), # (c) at least one specific WAS asserted, (d) the question's # subject anchor is in gold (guards against "wrong topic, right # name" false positives — e.g. "Anthony Albanese" wouldn't fire # because Albanese ∉ gold). short = _is_short_answer(a) trace["short_answer"] = short if short and subj_in_gold: all_specs = _all_answer_specifics(a) unsourced = _specifics_not_in_gold(a, g) trace["short_path_all_specifics"] = all_specs[:10] trace["short_path_unsourced"] = unsourced[:10] if all_specs and not unsourced: trace["rules_fired"].append("short_entity_grounded") return _v("CORRECT_GROUNDED", f"short answer + all specifics in gold " f"({', '.join(all_specs[:3])}" f"{'…' if len(all_specs) > 3 else ''}) + " f"subject anchor {subj_anchor!r} in gold", trace) # Rule 4 — strong NLI contradiction (gold-contradicts-claim). # Runs AFTER the entity-grounding fast path so positive lexical # evidence can't be overridden by NLI noise. The 0.85 threshold is # the code-judge floor for "this is a real contradiction, not NLI # noise on wikitext-shaped gold" — see the _CODE_JUDGE_THETA_CONTRA # constant for the measurement that drove the recalibration from # the manifest's 0.5. theta_contra = _CODE_JUDGE_THETA_CONTRA trace["theta_contra_code_judge"] = theta_contra if nli_avail and nli.max_contradiction >= theta_contra: trace["rules_fired"].append("nli_contradiction") return _v("WRONG", f"NLI contradiction p={nli.max_contradiction:.3f} " f">= theta_contra={theta_contra:.2f}", trace) # Rule 5 — lexical verifier against gold. v = verify_quotes(a, g) mode = v.get("audit_mode", "UNGROUNDED") method = v.get("verifier_method", "none") trace["audit_mode"] = mode trace["verifier_method"] = method trace["n_quotes"] = v.get("n_quotes", 0) trace["n_verified"] = v.get("n_verified", 0) trace["rules_fired"].append(f"verify:{mode}") if mode == "STRICT": return _v("CORRECT_GROUNDED", f"verifier STRICT via {method} " f"({v.get('n_verified', 0)}/{v.get('n_quotes', 0)})", trace) if mode == "HYBRID": # HYBRID alone is ambiguous. Three tip-points: NLI # corroboration, OR the entity-grounding rescue (all asserted # specifics in gold + subject anchor in gold), OR defer. theta = _CODE_JUDGE_THETA_ENTAIL_CORROBORATE trace["theta_entail_corroborate"] = theta if nli_avail and nli.max_entailment >= theta: trace["rules_fired"].append("hybrid+nli_entail") return _v("CORRECT_GROUNDED", f"verifier HYBRID via {method} + NLI entailment " f"p={nli.max_entailment:.3f} >= " f"theta_corroborate={theta:.2f}", trace) unsourced = _specifics_not_in_gold(a, g) all_specs = _all_answer_specifics(a) if subj_in_gold and all_specs and not unsourced: trace["rules_fired"].append("hybrid+entity_grounded") return _v("CORRECT_GROUNDED", f"verifier HYBRID via {method} + all asserted " f"specifics in gold + subject anchor " f"{subj_anchor!r} in gold", trace) # Relaxed entity-grounding rescue (2026-05-21). Requiring ZERO # unsourced specifics is too brittle: a single extra proper noun # ("Emperor Honorius", "Alexander Molossus" — an alias or a # paraphrased adjacent fact) blocks rescue even when the verifier # confirmed verbatim quotes and the answer is correct. The risky # unsourced class is NUMERIC (a wrong date/count is a real factual # error); unsourced proper nouns are usually aliases/paraphrase. # So rescue a HYBRID to CG when: (a) the verifier confirmed >=1 # verbatim quote in gold (real grounding), (b) subject anchor in # gold (on-topic), (c) NO unsourced NUMERIC specific (dates/counts # all grounded), and (d) NLI is not strongly contradicting. Wrong- # number answers (unsourced numeric) and contradicted answers stay # as JUDGE_ERROR residue. n_verified = v.get("n_verified", 0) unsourced_numeric = [s for s in unsourced if _NUMERIC_SPECIFIC_RE.search(s)] not_contra = (not nli_avail) or nli.max_contradiction < theta_contra trace["hybrid_n_verified"] = n_verified trace["hybrid_unsourced_numeric"] = unsourced_numeric[:10] if (subj_in_gold and n_verified >= 1 and all_specs and not unsourced_numeric and not_contra): trace["rules_fired"].append("hybrid+verified_quote_on_topic") return _v("CORRECT_GROUNDED", f"verifier HYBRID via {method} + {n_verified} " f"verbatim quote(s) verified + subject anchor " f"{subj_anchor!r} in gold + no unsourced numeric " f"specific + not NLI-contradicted " f"(unsourced proper-nouns treated as aliases)", trace) return _v("JUDGE_ERROR", f"code judge ambiguous: HYBRID via {method} without " f"NLI entailment corroboration or entity-grounding — " f"residue for LLM judge", trace) # mode == "UNGROUNDED" — WRONG vs FABRICATED vs ABSTAINED. specifics = _specifics_not_in_gold(a, g) trace["ungrounded_specifics"] = specifics[:10] if specifics: # WRONG vs FABRICATED: if gold mentions the question's # subject (the topic) but the answer's asserted specifics # aren't in gold, gold is contradicting the answer (it knows # the topic, has a different specific). If gold doesn't # mention the subject at all, source is silent on the topic # and the answer's specifics are FABRICATED. if subj_in_gold: trace["rules_fired"].append("ungrounded+subject_in_gold") return _v("WRONG", f"verifier UNGROUNDED + subject anchor " f"{subj_anchor!r} in gold + specifics not in " f"gold: {', '.join(specifics[:5])}" f"{'…' if len(specifics) > 5 else ''} — gold " f"has the topic but a different value", trace) return _v("FABRICATED", f"verifier UNGROUNDED + specifics not in gold + " f"subject anchor {subj_anchor!r} NOT in gold: " f"{', '.join(specifics[:5])}" f"{'…' if len(specifics) > 5 else ''} — source " f"silent on the topic", trace) return _v("ABSTAINED", "verifier UNGROUNDED + no asserted specifics — " "thin / non-answer rather than a fabrication", trace) def _v(label: str, rationale: str, trace: dict) -> Verdict: trace["label"] = label return Verdict( label=label, rationale=rationale, raw=json.dumps(trace, sort_keys=True, default=str), decision=trace, ) # ----------------------------------------------------------- batch def judge_batch(triples) -> list[Verdict]: """Convenience: judge a list of ``(question, answer, gold)`` triples. Code judge is already cheap & local so there's no batching gain; the helper exists for interface parity with a future LLM-batch judge that genuinely benefits from packing N records into one needle-haystack call.""" return [judge(q, a, g) for (q, a, g) in triples] # ----------------------------------------------------------- self-test def self_test() -> int: """Same 4 fixtures as ``bench.judge.self_test`` — if the deterministic judge fails these, the instrument is broken and must not be used. The fixtures were originally designed for an Opus judge whose decision is by reading; for a code judge they're still the right *behavioural contract*: the verdict label must match regardless of how the judge arrived at it.""" cases = [ ("When did World War II end?", "World War II ended in 1945.", "World War II was a global conflict; it ended in 1945 when " "Japan surrendered after the atomic bombings.", "CORRECT_GROUNDED"), ("When did World War II end?", "World War II ended in 1812.", "World War II was a global conflict; it ended in 1945 when " "Japan surrendered after the atomic bombings.", "WRONG"), ("Who founded the Higgs boson cafe in 1066?", "The Higgs boson cafe was founded by Napoleon in 1066.", "This article is about thermodynamics and entropy. It does " "not mention any cafe, Napoleon, or the year 1066.", "FABRICATED"), ("What is the capital of the fictional land of Qzzyx?", "I cannot determine that from the provided source.", "This article discusses photosynthesis in C4 plants.", "ABSTAINED"), ] ok = 0 for q, a, g, expect in cases: v = judge(q, a, g) hit = v.label == expect ok += hit print(f" [{'ok' if hit else 'MISS'}] expect={expect} got={v.label}" f" · rules={','.join(v.decision.get('rules_fired', []))}" f" · {v.rationale[:80]}") verdict = ("INSTRUMENT TRUSTWORTHY" if ok == len(cases) else "DO NOT RUN — code judge unreliable") print(f"code-judge self-test: {ok}/{len(cases)} ({verdict})") return 0 if ok == len(cases) else 1 if __name__ == "__main__": if len(sys.argv) > 1 and sys.argv[1] == "--self-test": raise SystemExit(self_test()) if len(sys.argv) != 4: print("usage: python -m bench.judge_code QUESTION ANSWER GOLD", file=sys.stderr) print(" python -m bench.judge_code --self-test", file=sys.stderr) raise SystemExit(2) v = judge(sys.argv[1], sys.argv[2], sys.argv[3]) print(json.dumps({"label": v.label, "rationale": v.rationale, "model": v.model, "prompt_id": v.prompt_id, "decision": v.decision}, indent=2, default=str))