fix(#000057): code judge — short-entity-grounded fast path runs before NLI contradiction

Reorder rule 3 (short-answer entity grounding) above rule 4 (NLI
contradiction) so positive lexical evidence cannot be overridden by
NLI clause-level noise. Surfaced by the 2026-05-19 arborist+qwen-
nothink smoke:

  i=3 · who is the prime minister of Poland?
  ans: 'Donald Tusk is listed as the Prime Minister of Poland.'
  gold: ...lists Tusk + Marcinkiewicz + Belka + Kaczynski + Kopacz...
  NLI contradiction p=0.892  (above 0.85 threshold)
  NLI entailment   p=0.744   (also high on the correct clause)

Tusk WAS PM in 2010 (served 2007-2014); answer is correct against
the corpus-vintage gold. The NLI contradiction signal came from
clause-level candidate selection picking a NON-Tusk PM the source
also mentions; entailment was high on the Tusk clause. Mixed signal
that the WRONG rule then over-confidently resolved.

The fix is a rule reorder, not a threshold change — the fast path's
positive-evidence combination (specifics-in-gold AND subject-in-gold)
is a strictly stronger signal than NLI's clause-level max
contradiction, so when it fires it should win. The combination
discriminates Poland-Tusk (Tusk ∈ gold, Poland ∈ gold → CG) from
Anthony-Albanese (Albanese ∉ gold → fast path declines → falls
through to UNGROUNDED-subject-in-gold → WRONG, unchanged).

Self-test 4/4 INSTRUMENT TRUSTWORTHY unchanged. pytest 27/27.
Poland-Tusk regression smoke: now CG via short_entity_grounded ✓.

No regression risk on the existing reconciliation cells:
- Iceland CG: short_entity_grounded was already winning (was rule
  4, now rule 3 — same outcome, earlier exit)
- WWII-1812 WRONG: '1812' ∉ gold → fast path declines, NLI fires ✓
- Higgs-cafe FABRICATED: 'Higgs' ∉ gold → fast path declines ✓
- Anthony Albanese WRONG: 'Albanese' ∉ gold → fast path declines ✓
- Abstention phrases: rule 2 still fires first ✓
This commit is contained in:
russell@unturf.com 2026-05-19 20:03:21 -04:00
parent c6621ee700
commit 866e67f43d
No known key found for this signature in database

View file

@ -565,28 +565,24 @@ def judge(question: str, answer: str, gold_source: str) -> Verdict:
trace["subject_anchor"] = subj_anchor
trace["subject_in_gold"] = subj_in_gold
# Rule 3 — strong NLI contradiction (gold-contradicts-claim).
# The 0.85 threshold is the code-judge floor for "this is a real
# contradiction, not NLI noise on wikitext-shaped gold" — see the
# _CODE_JUDGE_THETA_CONTRA constant for the measurement that drove
# the recalibration from the manifest's 0.5.
theta_contra = _CODE_JUDGE_THETA_CONTRA
trace["theta_contra_code_judge"] = theta_contra
if nli_avail and nli.max_contradiction >= theta_contra:
trace["rules_fired"].append("nli_contradiction")
return _v("WRONG",
f"NLI contradiction p={nli.max_contradiction:.3f} "
f">= theta_contra={theta_contra:.2f}",
trace)
# Rule 4 — short-answer entity-grounding fast path. Verifier's
# strategy-2 needs prose shape; terse-name answers fall through
# to UNGROUNDED with no specifics → ABSTAINED, missing valid CG.
# Rescue when: (a) answer is short, (b) every specific asserted
# is present in gold (no unsourced specifics), (c) at least one
# specific WAS asserted (not just "ok" or other empty content),
# (d) the question's subject anchor is in gold (guards against
# "wrong topic, right name" false positives).
# Rule 3 — short-answer entity-grounding fast path. RUNS BEFORE
# NLI contradiction (since 2026-05-19 Poland-Tusk smoke): when the
# answer asserts only specifics that are ALL in gold AND the
# question's subject anchor is in gold, that is strong positive
# evidence — NLI clause-level noise cannot override it. The
# Poland case (qwen+arborist answer "Donald Tusk is listed as the
# Prime Minister of Poland.", gold lists multiple PMs across
# decades) measured NLI contradiction p=0.89 ABOVE the 0.85
# threshold while NLI entailment was also 0.74 on the correct
# clause — mixed signal. The fast path's positive-evidence
# combination (specifics-in-gold + subject-in-gold) discriminates
# truth from contradiction without depending on NLI's clause-
# level aggregation. Rescue when: (a) answer is short, (b) every
# specific asserted is present in gold (no unsourced specifics),
# (c) at least one specific WAS asserted, (d) the question's
# subject anchor is in gold (guards against "wrong topic, right
# name" false positives — e.g. "Anthony Albanese" wouldn't fire
# because Albanese ∉ gold).
short = _is_short_answer(a)
trace["short_answer"] = short
if short and subj_in_gold:
@ -603,6 +599,22 @@ def judge(question: str, answer: str, gold_source: str) -> Verdict:
f"subject anchor {subj_anchor!r} in gold",
trace)
# Rule 4 — strong NLI contradiction (gold-contradicts-claim).
# Runs AFTER the entity-grounding fast path so positive lexical
# evidence can't be overridden by NLI noise. The 0.85 threshold is
# the code-judge floor for "this is a real contradiction, not NLI
# noise on wikitext-shaped gold" — see the _CODE_JUDGE_THETA_CONTRA
# constant for the measurement that drove the recalibration from
# the manifest's 0.5.
theta_contra = _CODE_JUDGE_THETA_CONTRA
trace["theta_contra_code_judge"] = theta_contra
if nli_avail and nli.max_contradiction >= theta_contra:
trace["rules_fired"].append("nli_contradiction")
return _v("WRONG",
f"NLI contradiction p={nli.max_contradiction:.3f} "
f">= theta_contra={theta_contra:.2f}",
trace)
# Rule 5 — lexical verifier against gold.
v = verify_quotes(a, g)
mode = v.get("audit_mode", "UNGROUNDED")