russell@unturf.com
866e67f43d
Reorder rule 3 (short-answer entity grounding) above rule 4 (NLI contradiction) so positive lexical evidence cannot be overridden by NLI clause-level noise. Surfaced by the 2026-05-19 arborist+qwen- nothink smoke: i=3 · who is the prime minister of Poland? ans: 'Donald Tusk is listed as the Prime Minister of Poland.' gold: ...lists Tusk + Marcinkiewicz + Belka + Kaczynski + Kopacz... NLI contradiction p=0.892 (above 0.85 threshold) NLI entailment p=0.744 (also high on the correct clause) Tusk WAS PM in 2010 (served 2007-2014); answer is correct against the corpus-vintage gold. The NLI contradiction signal came from clause-level candidate selection picking a NON-Tusk PM the source also mentions; entailment was high on the Tusk clause. Mixed signal that the WRONG rule then over-confidently resolved. The fix is a rule reorder, not a threshold change — the fast path's positive-evidence combination (specifics-in-gold AND subject-in-gold) is a strictly stronger signal than NLI's clause-level max contradiction, so when it fires it should win. The combination discriminates Poland-Tusk (Tusk ∈ gold, Poland ∈ gold → CG) from Anthony-Albanese (Albanese ∉ gold → fast path declines → falls through to UNGROUNDED-subject-in-gold → WRONG, unchanged). Self-test 4/4 INSTRUMENT TRUSTWORTHY unchanged. pytest 27/27. Poland-Tusk regression smoke: now CG via short_entity_grounded ✓. No regression risk on the existing reconciliation cells: - Iceland CG: short_entity_grounded was already winning (was rule 4, now rule 3 — same outcome, earlier exit) - WWII-1812 WRONG: '1812' ∉ gold → fast path declines, NLI fires ✓ - Higgs-cafe FABRICATED: 'Higgs' ∉ gold → fast path declines ✓ - Anthony Albanese WRONG: 'Albanese' ∉ gold → fast path declines ✓ - Abstention phrases: rule 2 still fires first ✓
775 lines
35 KiB
Python
775 lines
35 KiB
Python
#!/usr/bin/env python3
|
||
"""Code-only judge for the #000057 control experiment — no LLM, no quota.
|
||
|
||
Drop-in alternative to ``bench/judge.py`` (Opus headless via
|
||
``claude -p``). Same :class:`bench.judge.Verdict`-shaped dataclass,
|
||
same closed verdict vocabulary
|
||
(``CORRECT_GROUNDED`` / ``WRONG`` / ``FABRICATED`` / ``ABSTAINED`` /
|
||
``JUDGE_ERROR``), same ``judge(question, answer, gold_source)`` entry
|
||
point — so a caller can swap one for the other without changing its
|
||
scoring/aggregation code.
|
||
|
||
**Why a code judge first** (fox 2026-05-19): the Opus judge burned our
|
||
Anthropic quota on the huge-N sweep. A deterministic code judge gives
|
||
us the data-collection arm we need RIGHT NOW (zero quota, free,
|
||
replayable, no model-family hygiene threat — same-family judging is
|
||
the Opus-judge's live limitation per ``judge.py:26-29``). LLM-based
|
||
judging (Opus batched needle-haystack, or Grok credit-card) is a
|
||
later, separate concern; it operates on the residue this judge can
|
||
not classify deterministically.
|
||
|
||
The pipeline, in fixed order — first hit decides:
|
||
|
||
1. **Empty / no-gold guard.** Whitespace answer → ABSTAINED.
|
||
Empty gold → JUDGE_ERROR ("no gold source").
|
||
2. **Abstention phrases.** Lexical scan for explicit refusals
|
||
("I cannot determine", "the source does not state", …). Hit →
|
||
ABSTAINED. ABSTAINED is NOT failure — it's honest about not
|
||
answering and matches the Opus-judge prompt's definition.
|
||
3. **NLI contradiction.** ``arborist.qa.nli.shadow_check`` against
|
||
the gold source. If ``max_contradiction >= theta_contra`` (pinned
|
||
manifest threshold, default 0.5) → WRONG. Strongest possible
|
||
signal: the source *contradicts* the answer.
|
||
4. **Lexical verifier vs gold.** ``arborist.qa.verify.verify_quotes``
|
||
classifies the answer's grounding against the gold source as
|
||
STRICT / HYBRID / UNGROUNDED (the existing four-strategy ladder:
|
||
quote → span → entity → paraphrase). Maps to:
|
||
* STRICT → CORRECT_GROUNDED
|
||
* HYBRID + NLI entailment ≥ theta_entail → CORRECT_GROUNDED
|
||
* UNGROUNDED + specifics (proper-nouns or
|
||
dates not in gold) → FABRICATED
|
||
* UNGROUNDED + no specifics → ABSTAINED
|
||
(thin, unsupported,
|
||
but not a fabrication)
|
||
* HYBRID without NLI corroboration → JUDGE_ERROR
|
||
(ambiguous; residue
|
||
for LLM judge)
|
||
|
||
The JUDGE_ERROR class is a feature: ``rationale`` carries
|
||
``"code judge ambiguous: <reason>"`` so the residue is greppable for
|
||
later promotion to an LLM judge. Existing callers
|
||
(``control_sweep.py``, ``control_ab.py``) already treat JUDGE_ERROR
|
||
non-fatally and count it in the JE bucket.
|
||
|
||
Threats to validity, stated:
|
||
* Lexical grounding is a *proxy* for "supported by source" — the
|
||
Opus judge can read & reason. Code judge can not. STRICT means
|
||
"verbatim/span present in gold"; an answer can be STRICT but
|
||
still subtly wrong if the gold's claim differs in interpretation.
|
||
NLI guards against that (an answer whose proposition gold
|
||
contradicts will trip rule 3 BEFORE rule 4 sees STRICT).
|
||
* NLI is optional (``[nli]`` extra). When the model is unavailable
|
||
we fall back to rules 1+2+4 only and label the rationale so the
|
||
operator sees the degradation.
|
||
* Specificity heuristic for FABRICATED has false-negatives on
|
||
answers that fabricate WITHOUT proper nouns or numbers
|
||
("the answer to that is yes" with no grounding) — those will land
|
||
in ABSTAINED, the conservative class. False-positives on STRICT
|
||
answers cannot happen (STRICT pre-empts the FABRICATED branch).
|
||
|
||
Self-test (``python -m bench.judge_code --self-test``) covers the
|
||
same 4 fixtures as ``judge.py:self_test()`` so the two instruments
|
||
can be cross-checked when fox does eventually re-fire the Opus
|
||
judge on the residue.
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import json
|
||
import os
|
||
import re
|
||
import sys
|
||
from dataclasses import dataclass, field
|
||
|
||
from arborist.qa.verify import verify_quotes, extract_proper_nouns
|
||
|
||
JUDGE_RULESET_ID = "code-judge-v1"
|
||
JUDGE_MODEL = "code (deterministic, no LLM)"
|
||
|
||
# Opus-judge parity: same closed vocabulary so a Verdict from this
|
||
# module aggregates against the same buckets without translation.
|
||
_VERDICTS = ("CORRECT_GROUNDED", "WRONG", "FABRICATED", "ABSTAINED")
|
||
|
||
# Code-judge NLI corroboration threshold (rule 4 — HYBRID lexical +
|
||
# NLI entailment → CORRECT_GROUNDED). Distinct from the NLI manifest's
|
||
# ``entailment_block_veto=0.9``, which is calibrated for the OPPOSITE
|
||
# use case (high bar to override a STRICT lexical signal — additive
|
||
# negative evidence needs strong confidence). Corroborating a HYBRID
|
||
# lexical signal is additive POSITIVE evidence on an already-positive
|
||
# anchor; a moderate threshold is appropriate. Self-test fixture #1
|
||
# ("World War II ended in 1945." vs gold mentioning "it ended in 1945")
|
||
# measures NLI entailment=0.769 — clearly entailed, well clear of the
|
||
# 0.55 floor used here. Anything below 0.55 defers to JUDGE_ERROR
|
||
# (residue for an LLM judge later) rather than risk a false promote.
|
||
_CODE_JUDGE_THETA_ENTAIL_CORROBORATE = 0.55
|
||
|
||
# Code-judge NLI contradiction threshold (rule 3 — gold contradicts
|
||
# the answer → WRONG). The 2026-05-19 reconciliation against the
|
||
# trusted Opus judge measured **114 records** where the code judge
|
||
# fired WRONG via NLI contradiction p ∈ [0.51, 0.74] on answers Opus
|
||
# correctly graded CG — clear factual claims like "Ólafur Ragnar
|
||
# Grímsson is the president of Iceland" against the corpus infobox
|
||
# that DOES list him as incumbent. NLI is noisy in the 0.5–0.8 band
|
||
# on wikitext-shaped gold (clause-level candidate selection picks up
|
||
# co-mentioned earlier office-holders & reads temporal-frame mismatch
|
||
# as contradiction). For TRUE contradictions the signal is far higher
|
||
# (the WW2 self-test fixture #2 measures p=0.985). Threshold raised
|
||
# from the manifest's 0.5 → 0.85 to keep the strong contradictions
|
||
# (cleanly above the floor) while dropping the noise band.
|
||
_CODE_JUDGE_THETA_CONTRA = 0.85
|
||
|
||
# Short-answer threshold (rule 3.5 — terse correct-noun answers).
|
||
# The verifier's strategy 2 needs prose-shaped spans; very short
|
||
# answers ("Pratibha Patil", "Jalal Talabani", "In 1945.") fall
|
||
# through to UNGROUNDED-no-specifics → ABSTAINED, missing valid CG.
|
||
# The reconciliation against Opus identified 13 such records.
|
||
# Compensate by accepting a short answer as grounded when every
|
||
# specific (proper noun + numeric) it asserts is present in gold AND
|
||
# the question's subject anchor is present in gold (catches "wrong
|
||
# topic, right name" false-positives).
|
||
_CODE_JUDGE_SHORT_ANSWER_TOKENS = 15
|
||
|
||
|
||
@dataclass
|
||
class Verdict:
|
||
label: str # one of _VERDICTS, or "JUDGE_ERROR"
|
||
rationale: str
|
||
raw: str # serialised decision trace (logged for replay)
|
||
prompt_id: str = JUDGE_RULESET_ID
|
||
model: str = JUDGE_MODEL
|
||
# Surface the decision trace separately for programmatic callers
|
||
# who want to bucket by which rule fired. `raw` is the same data
|
||
# JSON-serialised — both are kept so the human-facing ``judge.py``
|
||
# interface is preserved.
|
||
decision: dict = field(default_factory=dict)
|
||
|
||
|
||
# ----------------------------------------------------------- preprocess
|
||
|
||
# Claim-lattice JSON envelope detector. Arborist's `claim_lattice`
|
||
# answer_mode emits {"claims":[{"text":"...","evidence_ids":["E1"]},...]}.
|
||
# Without unwrapping, the verifier sees brace + key syntax noise
|
||
# instead of the actual claim prose — the proper-noun & span
|
||
# extractors give weak signal, every Arborist record falls to
|
||
# UNGROUNDED. Unwrapping to concatenated sentences lets the existing
|
||
# lexical paths handle these uniformly with plain-prose answers,
|
||
# preserving the judge's per-record cost (no special-case grader).
|
||
def _unwrap_claim_lattice_json(answer: str) -> str:
|
||
"""Extract claim text(s) from claim-lattice JSON envelopes. Returns
|
||
the original answer when it isn't claim-lattice shaped, so
|
||
plain-prose answers pass through unchanged.
|
||
|
||
The detection is conservative: must start with ``{`` AND contain
|
||
``"claims"`` AND contain ``"text"`` — three independent signals
|
||
that this is the Arborist envelope rather than coincidental JSON.
|
||
Parse failures fall back to the original answer (let the
|
||
downstream verifier see what it gets — no silent rewriting on
|
||
malformed input)."""
|
||
a = (answer or "").strip()
|
||
if not (a.startswith("{") and '"claims"' in a and '"text"' in a):
|
||
return answer
|
||
try:
|
||
obj = json.loads(a)
|
||
except Exception: # noqa: BLE001 — malformed JSON, let it through
|
||
return answer
|
||
if not isinstance(obj, dict):
|
||
return answer
|
||
claims = obj.get("claims")
|
||
if not isinstance(claims, list):
|
||
return answer
|
||
texts: list[str] = []
|
||
for c in claims:
|
||
if isinstance(c, dict):
|
||
t = c.get("text")
|
||
if isinstance(t, str) and t.strip():
|
||
texts.append(t.strip())
|
||
if not texts:
|
||
return answer
|
||
# Concatenate as discrete sentences so verify.extract_claim_spans
|
||
# can pick each one up as its own span. Append a period only when
|
||
# the claim text doesn't already end in sentence punctuation.
|
||
return " ".join(
|
||
t if t[-1:] in ".!?:;" else (t + ".")
|
||
for t in texts
|
||
)
|
||
|
||
|
||
# ----------------------------------------------------------- rule 2
|
||
|
||
# Lexical abstention phrases. Kept conservative: only fires when the
|
||
# answer EXPLICITLY says it can not / will not answer or that the
|
||
# source does not contain the information. A model that answers
|
||
# "I think it is X" is NOT abstaining — it is making a claim.
|
||
#
|
||
# Pattern set expanded 2026-05-19 from the Opus-reconciliation
|
||
# residue: Hermes's most common abstention phrasings under
|
||
# `--variant source_relative` are "I do not have accurate information
|
||
# about", "I do not have access to real-time / a reference knowledge
|
||
# base to", "I do not have enough information to". The original
|
||
# patterns missed all three (they require a verb like determine/know
|
||
# right after "do not"). Eighteen records mis-classified as WRONG
|
||
# trace to this gap.
|
||
_ABSTAIN_PATTERNS = (
|
||
re.compile(r"\b(?:i\s+(?:cannot|can[' ]?t|do\s+not|don[' ]?t)\s+"
|
||
r"(?:determine|know|tell|say|find|locate|verify))\b", re.I),
|
||
# NEW: "I do not have <qualifier?> information / access / data ..."
|
||
re.compile(r"\bi\s+(?:do\s+not|don[' ]?t|cannot|can[' ]?t)\s+"
|
||
r"have\s+(?:\w+\s+){0,4}"
|
||
r"(?:information|access|data|knowledge|details)\b", re.I),
|
||
# NEW: "I do not have access to ..." stand-alone
|
||
re.compile(r"\bi\s+(?:do\s+not|don[' ]?t)\s+have\s+access\b", re.I),
|
||
# NEW: "I lack (access / information / knowledge)"
|
||
re.compile(r"\bi\s+lack\s+(?:\w+\s+){0,3}"
|
||
r"(?:access|information|knowledge|data|details)\b", re.I),
|
||
re.compile(r"\b(?:not\s+(?:enough|sufficient)\s+information)\b", re.I),
|
||
re.compile(r"\b(?:no\s+information\s+(?:is\s+)?(?:available|provided|given))\b", re.I),
|
||
re.compile(r"\b(?:the\s+(?:source|article|text|context|passage)\s+"
|
||
r"(?:does\s+not|doesn[' ]?t|fails\s+to)\s+"
|
||
r"(?:contain|mention|state|specify|provide|include))\b", re.I),
|
||
re.compile(r"\b(?:unable\s+to\s+(?:determine|answer|verify|locate))\b", re.I),
|
||
re.compile(r"\b(?:no\s+(?:answer|response)\s+(?:can\s+be|is)\s+(?:given|provided|made))\b", re.I),
|
||
re.compile(r"\b(?:cannot\s+be\s+(?:determined|answered|found|verified)\s+from)\b", re.I),
|
||
)
|
||
|
||
|
||
def _is_abstention(answer: str) -> tuple[bool, str]:
|
||
"""Hit on any of the explicit-refusal patterns above.
|
||
|
||
Returns ``(is_abstention, matched_pattern_excerpt)``."""
|
||
for pat in _ABSTAIN_PATTERNS:
|
||
m = pat.search(answer)
|
||
if m:
|
||
return True, m.group(0)
|
||
return False, ""
|
||
|
||
|
||
# ----------------------------------------------------------- rule 3
|
||
|
||
def _nli_check(answer: str, gold: str):
|
||
"""Run NLI shadow on (gold, answer). Returns the
|
||
:class:`arborist.qa.nli.ShadowResult` or ``None`` when the
|
||
``[nli]`` extra is unavailable.
|
||
|
||
Note: ``shadow_check`` itself returns an unavailable-result
|
||
object rather than raising, so the actual ``None`` branch
|
||
here is only the import-failure path."""
|
||
try:
|
||
from arborist.qa.nli import shadow_check
|
||
except Exception: # noqa: BLE001 — fail-open to lexical-only judging
|
||
return None
|
||
try:
|
||
return shadow_check(claim=answer, source=gold)
|
||
except Exception: # noqa: BLE001
|
||
return None
|
||
|
||
|
||
# ----------------------------------------------------------- rule 4
|
||
|
||
# Tokens that are evidence the answer asserts SPECIFICS without
|
||
# grounding — proper nouns (multi-word via verify.extract_proper_nouns
|
||
# AND single-word via the local scanner below) and bare numerics that
|
||
# look like dates / counts / measurements.
|
||
_NUMERIC_SPECIFIC_RE = re.compile(
|
||
r"\b(?:\d{2,4}(?:st|nd|rd|th)?|\d+\.\d+|\d{3,}|"
|
||
# 4-digit year, ordinal year, decimal, large count
|
||
r"\d{1,2}/\d{1,2}(?:/\d{2,4})?|" # date 1/2 or 1/2/34
|
||
r"\$\d[\d,]*(?:\.\d+)?)\b"
|
||
)
|
||
|
||
# Single-word capitalised tokens. The verifier's
|
||
# ``extract_proper_nouns`` is deliberately multi-word-only (to keep
|
||
# the verifier's lexical contract conservative — single capitalised
|
||
# tokens are noisy as evidence of grounding). For the FABRICATION
|
||
# check we go the other direction: we want to catch "Napoleon" alone
|
||
# when it appears in an UNGROUNDED answer with no source backing.
|
||
# Sentence-leading tokens get filtered as a soft-noise reduction.
|
||
_SINGLE_CAP_TOKEN_RE = re.compile(r"\b[A-Z][a-z]{2,}\b")
|
||
# Common English words that frequently appear sentence-leading and
|
||
# would otherwise produce noise (the FABRICATED class is about asserted
|
||
# proper-noun-shaped specifics, not "The"/"This"/"It"). Lower-case
|
||
# comparison so the filter is shape-agnostic.
|
||
_CAP_TOKEN_STOPWORDS = frozenset({
|
||
"the", "this", "that", "these", "those", "it", "its", "they", "their",
|
||
"there", "then", "thus", "such", "some", "all", "any", "many",
|
||
"most", "much", "more", "less", "few", "several", "each", "every",
|
||
"and", "but", "or", "nor", "yet", "so", "for", "however", "moreover",
|
||
"also", "though", "although", "while", "since", "because", "if",
|
||
"when", "where", "what", "which", "who", "whom", "whose", "why", "how",
|
||
"is", "are", "was", "were", "be", "been", "being", "has", "have",
|
||
"had", "do", "does", "did", "can", "could", "should", "would", "may",
|
||
"might", "must", "shall", "will", "according", "based", "during",
|
||
"after", "before", "given", "let", "note", "see", "from", "with",
|
||
"without", "into", "onto", "upon", "about", "around",
|
||
})
|
||
|
||
|
||
def _all_answer_specifics(answer: str) -> list[str]:
|
||
"""Every specific (proper noun + numeric) the answer asserts.
|
||
Symmetric with :func:`_specifics_not_in_gold` but unfiltered —
|
||
used to check 'this answer asserted SOMETHING specific' before
|
||
we decide whether to call it ABSTAINED."""
|
||
out: list[str] = []
|
||
seen: set[str] = set()
|
||
|
||
def _add(tok: str):
|
||
k = tok.lower()
|
||
if k in seen:
|
||
return
|
||
out.append(tok)
|
||
seen.add(k)
|
||
|
||
for noun in extract_proper_nouns(answer):
|
||
_add(noun)
|
||
for m in _SINGLE_CAP_TOKEN_RE.finditer(answer):
|
||
tok = m.group(0)
|
||
if tok.lower() in _CAP_TOKEN_STOPWORDS:
|
||
continue
|
||
_add(tok)
|
||
for m in _NUMERIC_SPECIFIC_RE.finditer(answer):
|
||
_add(m.group(0))
|
||
return out
|
||
|
||
|
||
# Stopwords for the question-subject-anchor extractor. Mirrors the
|
||
# inspect-module convention: drop function words / WH-question words /
|
||
# generic role nouns so the LAST remaining content token is the
|
||
# question's primary subject ("Iceland" in "who is the president of
|
||
# Iceland?"; "Australia" in "according to the reference knowledge base,
|
||
# who is the prime minister of Australia?"). The anchor is also
|
||
# tolerant of the variant-rewriting prefixes (`According to ...`,
|
||
# `As of 2010 ...`) because those add to the front, not the back.
|
||
_SUBJ_STOPWORDS = frozenset({
|
||
"who", "what", "when", "where", "why", "how", "which", "whose", "whom",
|
||
"is", "are", "was", "were", "be", "been", "being", "has", "have", "had",
|
||
"do", "does", "did", "the", "a", "an", "of", "in", "on", "at", "for",
|
||
"by", "to", "from", "as", "according", "reference", "knowledge", "base",
|
||
"current", "currently", "today", "now", "presently", "office",
|
||
"head", "leader",
|
||
# Generic role nouns kept OUT of the subject — they describe the
|
||
# *role*, the polity is the subject. "who is the prime minister of
|
||
# Australia" → subject = australia, not minister.
|
||
"president", "prime", "minister", "chancellor", "premier", "governor",
|
||
"general", "secretary", "king", "queen", "emperor", "leader",
|
||
"chairman", "chair", "speaker", "monarch", "ruler",
|
||
})
|
||
|
||
|
||
_TOKEN_RE = re.compile(r"\b\w+\b", re.U)
|
||
|
||
|
||
def _question_subject_anchor(question: str) -> str:
|
||
"""Last content token after stopword-strip — the inspect-module
|
||
fallback used when the question carries no proper-noun-shaped
|
||
terms. Returns lowercase or empty string."""
|
||
toks = [t.lower() for t in _TOKEN_RE.findall(question or "")]
|
||
content = [t for t in toks if t not in _SUBJ_STOPWORDS and not t.isdigit()]
|
||
return content[-1] if content else ""
|
||
|
||
|
||
def _question_subject_terms(question: str) -> list[str]:
|
||
"""Proper-noun-shaped subject terms from the question. Tries
|
||
extract_proper_nouns (multi-word PNs) first, then layers single
|
||
capitalised tokens (modulo stopwords). For "who is the president
|
||
of Iceland?" → ["Iceland"]. For "who founded the Higgs boson
|
||
cafe in 1066?" → ["Higgs"]. For "who is the president of the
|
||
Quorum of the Twelve Apostles?" → ["Quorum", "Twelve", "Apostles"]
|
||
(whichever the multi-word extractor surfaces + single caps the
|
||
extractor misses).
|
||
|
||
Used as the strong-anchor signal for ``_subject_in_gold``: a
|
||
proper-noun-shaped term in the question is the topic's identity,
|
||
not just a co-occurring stopword ("cafe" matching gold's "any
|
||
cafe" denial is the bug this avoids). Returns [] when the
|
||
question has no proper-noun-shaped term — caller falls back to
|
||
the last-content-token anchor."""
|
||
out: list[str] = []
|
||
seen: set[str] = set()
|
||
|
||
def _add(t: str):
|
||
k = t.lower()
|
||
if k in seen or k in _SUBJ_STOPWORDS or k in _CAP_TOKEN_STOPWORDS:
|
||
return
|
||
out.append(t)
|
||
seen.add(k)
|
||
|
||
for pn in extract_proper_nouns(question or ""):
|
||
# Multi-word PNs land whole — record both the phrase and its
|
||
# head tokens for substring matching against gold.
|
||
_add(pn)
|
||
for tok in pn.split():
|
||
_add(tok)
|
||
for m in _SINGLE_CAP_TOKEN_RE.finditer(question or ""):
|
||
_add(m.group(0))
|
||
return out
|
||
|
||
|
||
def _subject_in_gold(question: str, gold: str) -> tuple[bool, str]:
|
||
"""True iff the question's subject is present in gold. Strong
|
||
anchor: proper-noun-shaped terms from the question — ALL such
|
||
terms must appear in gold (case-insensitive substring).
|
||
Conservative because the WRONG-vs-FABRICATED tie-break demotes
|
||
FABRICATED → WRONG when this fires, and we'd rather under-fire
|
||
(keep FABRICATED) than over-fire (the Higgs-boson-cafe gold
|
||
coincidentally contains "cafe" in a denial — single-token anchor
|
||
would mis-fire, multi-PN anchor doesn't).
|
||
|
||
Returns ``(found, anchor)`` where ``anchor`` records which term
|
||
decided it (for the decision trace). Falls back to the last-
|
||
content-token heuristic when the question has no proper-noun-
|
||
shaped terms (rare on this corpus)."""
|
||
gold_lc = gold.lower()
|
||
terms = _question_subject_terms(question)
|
||
if terms:
|
||
# Require every proper-noun-shaped term to appear in gold —
|
||
# all the topic's identity tokens, not just one.
|
||
missing = [t for t in terms if t.lower() not in gold_lc]
|
||
if not missing:
|
||
return True, terms[0]
|
||
return False, missing[0]
|
||
# Fallback for proper-noun-free questions: the last-content-token
|
||
# anchor (was the original heuristic before reconciliation).
|
||
anchor = _question_subject_anchor(question)
|
||
if not anchor:
|
||
return False, ""
|
||
return (anchor in gold_lc), anchor
|
||
|
||
|
||
def _is_short_answer(answer: str) -> bool:
|
||
"""The verifier's strategy-2 span extractor needs prose; very
|
||
short answers fall through to UNGROUNDED-no-specifics. Treat as
|
||
'short' for the entity-grounding fast path."""
|
||
return len(_TOKEN_RE.findall(answer or "")) <= _CODE_JUDGE_SHORT_ANSWER_TOKENS
|
||
|
||
|
||
def _specifics_not_in_gold(answer: str, gold: str) -> list[str]:
|
||
"""Proper-noun and numeric specifics that the answer asserts but
|
||
the gold source never mentions. Three layers:
|
||
|
||
1. ``extract_proper_nouns`` from the verifier — multi-word
|
||
capitalised entities (e.g. "Higgs Boson", "World War II").
|
||
2. Local single-word capitalised-token scanner — catches
|
||
"Napoleon" / "Mars" / "Mozart" that the verifier's multi-word
|
||
gate filters out.
|
||
3. Numeric specifics — dates, years, large counts, money.
|
||
|
||
Used only on the UNGROUNDED branch: a STRICT or HYBRID answer
|
||
pre-empts this path and does NOT get tagged as FABRICATED."""
|
||
out: list[str] = []
|
||
seen: set[str] = set()
|
||
gold_lc = gold.lower()
|
||
|
||
def _add(tok: str):
|
||
k = tok.lower()
|
||
if k in seen:
|
||
return
|
||
if k not in gold_lc:
|
||
out.append(tok)
|
||
seen.add(k)
|
||
|
||
# Layer 1: verifier's multi-word proper-noun extractor.
|
||
for noun in extract_proper_nouns(answer):
|
||
_add(noun)
|
||
# Layer 2: single-word capitalised tokens (modulo sentence-leading
|
||
# noise — drop tokens immediately following a sentence terminator
|
||
# OR at position 0, AND in the stopword set).
|
||
sentence_lead_positions: set[int] = {0}
|
||
for m in re.finditer(r"[.!?]\s+(?=[A-Z])", answer):
|
||
sentence_lead_positions.add(m.end())
|
||
for m in _SINGLE_CAP_TOKEN_RE.finditer(answer):
|
||
tok = m.group(0)
|
||
if tok.lower() in _CAP_TOKEN_STOPWORDS:
|
||
continue
|
||
# If sentence-leading AND a 3-letter-or-less stop-shaped token,
|
||
# ignore. But "Napoleon" sentence-leading is still a specific.
|
||
# We accept any sentence-leading non-stopword that's >=3 chars.
|
||
_add(tok)
|
||
# Layer 3: numerics.
|
||
for m in _NUMERIC_SPECIFIC_RE.finditer(answer):
|
||
_add(m.group(0))
|
||
return out
|
||
|
||
|
||
# ----------------------------------------------------------- judge()
|
||
|
||
def judge(question: str, answer: str, gold_source: str) -> Verdict:
|
||
"""Deterministic verdict on (question, answer, gold_source).
|
||
|
||
Same shape & verdict vocabulary as ``bench.judge.judge``; no
|
||
network, no LLM, no quota cost. JUDGE_ERROR is reserved for the
|
||
HYBRID-without-NLI-corroboration class so a downstream LLM judge
|
||
(Opus batched, Grok) can pick that residue up later.
|
||
|
||
Rule order (reconciled against Opus 2026-05-19):
|
||
|
||
1. empty / no-gold guards
|
||
2. explicit abstention phrases (lexical regex; broad set
|
||
covering the Hermes "I do not have ... information / access"
|
||
family that the original narrow patterns missed)
|
||
3. strong NLI contradiction (theta_contra=0.85, raised from 0.5
|
||
after measuring 114 false-positives in the 0.5-0.75 band)
|
||
4. short-answer entity-grounding fast path — for terse answers
|
||
where the verifier's prose-shape extractor would whiff,
|
||
accept CG when every asserted specific is in gold AND the
|
||
question's subject anchor is in gold (catches "Pratibha
|
||
Patil" → gold "Pratibha Devisingh Patil ..." → CG)
|
||
5. lexical verifier (verify_quotes)
|
||
STRICT → CG
|
||
HYBRID + NLI entail ≥ 0.55 → CG
|
||
HYBRID + subject-anchor + all-grounded → CG (entity-
|
||
grounding rescue)
|
||
HYBRID otherwise → JUDGE_ERROR
|
||
(residue)
|
||
UNGROUNDED + specifics + subject-in-gold → WRONG (source
|
||
contradicts —
|
||
Opus's WRONG
|
||
vs FABRICATED
|
||
distinction)
|
||
UNGROUNDED + specifics, no subject → FABRICATED
|
||
UNGROUNDED + no specifics → ABSTAINED
|
||
"""
|
||
raw_answer = (answer or "").strip()
|
||
trace: dict = {"rules_fired": []}
|
||
|
||
# Preprocess — unwrap claim-lattice JSON envelopes BEFORE any
|
||
# downstream check. After this point ``a`` is plain prose for both
|
||
# arms (solo & arborist-claim_lattice), and the verifier / NLI /
|
||
# specifics paths grade them on equal terms.
|
||
a = _unwrap_claim_lattice_json(raw_answer)
|
||
if a != raw_answer:
|
||
trace["lattice_unwrapped"] = True
|
||
trace["unwrapped_len"] = len(a)
|
||
g = (gold_source or "").strip()
|
||
|
||
# Rule 1 — empty / no-gold guards.
|
||
if not a:
|
||
trace["rules_fired"].append("empty_answer")
|
||
return _v("ABSTAINED", "empty answer", trace)
|
||
if not g:
|
||
trace["rules_fired"].append("no_gold")
|
||
return _v("JUDGE_ERROR", "no gold source — cannot ground", trace)
|
||
|
||
# Rule 2 — explicit abstention phrase.
|
||
is_abs, abs_match = _is_abstention(a)
|
||
if is_abs:
|
||
trace["rules_fired"].append("abstention_phrase")
|
||
trace["abstain_match"] = abs_match
|
||
return _v("ABSTAINED", f"explicit refusal: {abs_match!r}", trace)
|
||
|
||
# Pre-compute shared signals once.
|
||
nli = _nli_check(a, g)
|
||
nli_avail = nli is not None and nli.available
|
||
trace["nli_available"] = nli_avail
|
||
if nli_avail:
|
||
trace["nli_max_contra"] = nli.max_contradiction
|
||
trace["nli_max_entail"] = nli.max_entailment
|
||
subj_in_gold, subj_anchor = _subject_in_gold(question, g)
|
||
trace["subject_anchor"] = subj_anchor
|
||
trace["subject_in_gold"] = subj_in_gold
|
||
|
||
# Rule 3 — short-answer entity-grounding fast path. RUNS BEFORE
|
||
# NLI contradiction (since 2026-05-19 Poland-Tusk smoke): when the
|
||
# answer asserts only specifics that are ALL in gold AND the
|
||
# question's subject anchor is in gold, that is strong positive
|
||
# evidence — NLI clause-level noise cannot override it. The
|
||
# Poland case (qwen+arborist answer "Donald Tusk is listed as the
|
||
# Prime Minister of Poland.", gold lists multiple PMs across
|
||
# decades) measured NLI contradiction p=0.89 ABOVE the 0.85
|
||
# threshold while NLI entailment was also 0.74 on the correct
|
||
# clause — mixed signal. The fast path's positive-evidence
|
||
# combination (specifics-in-gold + subject-in-gold) discriminates
|
||
# truth from contradiction without depending on NLI's clause-
|
||
# level aggregation. Rescue when: (a) answer is short, (b) every
|
||
# specific asserted is present in gold (no unsourced specifics),
|
||
# (c) at least one specific WAS asserted, (d) the question's
|
||
# subject anchor is in gold (guards against "wrong topic, right
|
||
# name" false positives — e.g. "Anthony Albanese" wouldn't fire
|
||
# because Albanese ∉ gold).
|
||
short = _is_short_answer(a)
|
||
trace["short_answer"] = short
|
||
if short and subj_in_gold:
|
||
all_specs = _all_answer_specifics(a)
|
||
unsourced = _specifics_not_in_gold(a, g)
|
||
trace["short_path_all_specifics"] = all_specs[:10]
|
||
trace["short_path_unsourced"] = unsourced[:10]
|
||
if all_specs and not unsourced:
|
||
trace["rules_fired"].append("short_entity_grounded")
|
||
return _v("CORRECT_GROUNDED",
|
||
f"short answer + all specifics in gold "
|
||
f"({', '.join(all_specs[:3])}"
|
||
f"{'…' if len(all_specs) > 3 else ''}) + "
|
||
f"subject anchor {subj_anchor!r} in gold",
|
||
trace)
|
||
|
||
# Rule 4 — strong NLI contradiction (gold-contradicts-claim).
|
||
# Runs AFTER the entity-grounding fast path so positive lexical
|
||
# evidence can't be overridden by NLI noise. The 0.85 threshold is
|
||
# the code-judge floor for "this is a real contradiction, not NLI
|
||
# noise on wikitext-shaped gold" — see the _CODE_JUDGE_THETA_CONTRA
|
||
# constant for the measurement that drove the recalibration from
|
||
# the manifest's 0.5.
|
||
theta_contra = _CODE_JUDGE_THETA_CONTRA
|
||
trace["theta_contra_code_judge"] = theta_contra
|
||
if nli_avail and nli.max_contradiction >= theta_contra:
|
||
trace["rules_fired"].append("nli_contradiction")
|
||
return _v("WRONG",
|
||
f"NLI contradiction p={nli.max_contradiction:.3f} "
|
||
f">= theta_contra={theta_contra:.2f}",
|
||
trace)
|
||
|
||
# Rule 5 — lexical verifier against gold.
|
||
v = verify_quotes(a, g)
|
||
mode = v.get("audit_mode", "UNGROUNDED")
|
||
method = v.get("verifier_method", "none")
|
||
trace["audit_mode"] = mode
|
||
trace["verifier_method"] = method
|
||
trace["n_quotes"] = v.get("n_quotes", 0)
|
||
trace["n_verified"] = v.get("n_verified", 0)
|
||
trace["rules_fired"].append(f"verify:{mode}")
|
||
|
||
if mode == "STRICT":
|
||
return _v("CORRECT_GROUNDED",
|
||
f"verifier STRICT via {method} "
|
||
f"({v.get('n_verified', 0)}/{v.get('n_quotes', 0)})",
|
||
trace)
|
||
|
||
if mode == "HYBRID":
|
||
# HYBRID alone is ambiguous. Three tip-points: NLI
|
||
# corroboration, OR the entity-grounding rescue (all asserted
|
||
# specifics in gold + subject anchor in gold), OR defer.
|
||
theta = _CODE_JUDGE_THETA_ENTAIL_CORROBORATE
|
||
trace["theta_entail_corroborate"] = theta
|
||
if nli_avail and nli.max_entailment >= theta:
|
||
trace["rules_fired"].append("hybrid+nli_entail")
|
||
return _v("CORRECT_GROUNDED",
|
||
f"verifier HYBRID via {method} + NLI entailment "
|
||
f"p={nli.max_entailment:.3f} >= "
|
||
f"theta_corroborate={theta:.2f}",
|
||
trace)
|
||
unsourced = _specifics_not_in_gold(a, g)
|
||
all_specs = _all_answer_specifics(a)
|
||
if subj_in_gold and all_specs and not unsourced:
|
||
trace["rules_fired"].append("hybrid+entity_grounded")
|
||
return _v("CORRECT_GROUNDED",
|
||
f"verifier HYBRID via {method} + all asserted "
|
||
f"specifics in gold + subject anchor "
|
||
f"{subj_anchor!r} in gold",
|
||
trace)
|
||
return _v("JUDGE_ERROR",
|
||
f"code judge ambiguous: HYBRID via {method} without "
|
||
f"NLI entailment corroboration or entity-grounding — "
|
||
f"residue for LLM judge",
|
||
trace)
|
||
|
||
# mode == "UNGROUNDED" — WRONG vs FABRICATED vs ABSTAINED.
|
||
specifics = _specifics_not_in_gold(a, g)
|
||
trace["ungrounded_specifics"] = specifics[:10]
|
||
if specifics:
|
||
# WRONG vs FABRICATED: if gold mentions the question's
|
||
# subject (the topic) but the answer's asserted specifics
|
||
# aren't in gold, gold is contradicting the answer (it knows
|
||
# the topic, has a different specific). If gold doesn't
|
||
# mention the subject at all, source is silent on the topic
|
||
# and the answer's specifics are FABRICATED.
|
||
if subj_in_gold:
|
||
trace["rules_fired"].append("ungrounded+subject_in_gold")
|
||
return _v("WRONG",
|
||
f"verifier UNGROUNDED + subject anchor "
|
||
f"{subj_anchor!r} in gold + specifics not in "
|
||
f"gold: {', '.join(specifics[:5])}"
|
||
f"{'…' if len(specifics) > 5 else ''} — gold "
|
||
f"has the topic but a different value",
|
||
trace)
|
||
return _v("FABRICATED",
|
||
f"verifier UNGROUNDED + specifics not in gold + "
|
||
f"subject anchor {subj_anchor!r} NOT in gold: "
|
||
f"{', '.join(specifics[:5])}"
|
||
f"{'…' if len(specifics) > 5 else ''} — source "
|
||
f"silent on the topic",
|
||
trace)
|
||
return _v("ABSTAINED",
|
||
"verifier UNGROUNDED + no asserted specifics — "
|
||
"thin / non-answer rather than a fabrication",
|
||
trace)
|
||
|
||
|
||
def _v(label: str, rationale: str, trace: dict) -> Verdict:
|
||
trace["label"] = label
|
||
return Verdict(
|
||
label=label,
|
||
rationale=rationale,
|
||
raw=json.dumps(trace, sort_keys=True, default=str),
|
||
decision=trace,
|
||
)
|
||
|
||
|
||
# ----------------------------------------------------------- batch
|
||
|
||
def judge_batch(triples) -> list[Verdict]:
|
||
"""Convenience: judge a list of ``(question, answer, gold)`` triples.
|
||
|
||
Code judge is already cheap & local so there's no batching gain;
|
||
the helper exists for interface parity with a future LLM-batch
|
||
judge that genuinely benefits from packing N records into one
|
||
needle-haystack call."""
|
||
return [judge(q, a, g) for (q, a, g) in triples]
|
||
|
||
|
||
# ----------------------------------------------------------- self-test
|
||
|
||
def self_test() -> int:
|
||
"""Same 4 fixtures as ``bench.judge.self_test`` — if the
|
||
deterministic judge fails these, the instrument is broken and
|
||
must not be used.
|
||
|
||
The fixtures were originally designed for an Opus judge whose
|
||
decision is by reading; for a code judge they're still the right
|
||
*behavioural contract*: the verdict label must match regardless
|
||
of how the judge arrived at it."""
|
||
cases = [
|
||
("When did World War II end?",
|
||
"World War II ended in 1945.",
|
||
"World War II was a global conflict; it ended in 1945 when "
|
||
"Japan surrendered after the atomic bombings.",
|
||
"CORRECT_GROUNDED"),
|
||
("When did World War II end?",
|
||
"World War II ended in 1812.",
|
||
"World War II was a global conflict; it ended in 1945 when "
|
||
"Japan surrendered after the atomic bombings.",
|
||
"WRONG"),
|
||
("Who founded the Higgs boson cafe in 1066?",
|
||
"The Higgs boson cafe was founded by Napoleon in 1066.",
|
||
"This article is about thermodynamics and entropy. It does "
|
||
"not mention any cafe, Napoleon, or the year 1066.",
|
||
"FABRICATED"),
|
||
("What is the capital of the fictional land of Qzzyx?",
|
||
"I cannot determine that from the provided source.",
|
||
"This article discusses photosynthesis in C4 plants.",
|
||
"ABSTAINED"),
|
||
]
|
||
ok = 0
|
||
for q, a, g, expect in cases:
|
||
v = judge(q, a, g)
|
||
hit = v.label == expect
|
||
ok += hit
|
||
print(f" [{'ok' if hit else 'MISS'}] expect={expect} got={v.label}"
|
||
f" · rules={','.join(v.decision.get('rules_fired', []))}"
|
||
f" · {v.rationale[:80]}")
|
||
verdict = ("INSTRUMENT TRUSTWORTHY" if ok == len(cases)
|
||
else "DO NOT RUN — code judge unreliable")
|
||
print(f"code-judge self-test: {ok}/{len(cases)} ({verdict})")
|
||
return 0 if ok == len(cases) else 1
|
||
|
||
|
||
if __name__ == "__main__":
|
||
if len(sys.argv) > 1 and sys.argv[1] == "--self-test":
|
||
raise SystemExit(self_test())
|
||
if len(sys.argv) != 4:
|
||
print("usage: python -m bench.judge_code QUESTION ANSWER GOLD",
|
||
file=sys.stderr)
|
||
print(" python -m bench.judge_code --self-test",
|
||
file=sys.stderr)
|
||
raise SystemExit(2)
|
||
v = judge(sys.argv[1], sys.argv[2], sys.argv[3])
|
||
print(json.dumps({"label": v.label, "rationale": v.rationale,
|
||
"model": v.model, "prompt_id": v.prompt_id,
|
||
"decision": v.decision},
|
||
indent=2, default=str))
|