fix(#000057): code judge — short-entity-grounded fast path runs before NLI contradiction
Reorder rule 3 (short-answer entity grounding) above rule 4 (NLI contradiction) so positive lexical evidence cannot be overridden by NLI clause-level noise. Surfaced by the 2026-05-19 arborist+qwen- nothink smoke: i=3 · who is the prime minister of Poland? ans: 'Donald Tusk is listed as the Prime Minister of Poland.' gold: ...lists Tusk + Marcinkiewicz + Belka + Kaczynski + Kopacz... NLI contradiction p=0.892 (above 0.85 threshold) NLI entailment p=0.744 (also high on the correct clause) Tusk WAS PM in 2010 (served 2007-2014); answer is correct against the corpus-vintage gold. The NLI contradiction signal came from clause-level candidate selection picking a NON-Tusk PM the source also mentions; entailment was high on the Tusk clause. Mixed signal that the WRONG rule then over-confidently resolved. The fix is a rule reorder, not a threshold change — the fast path's positive-evidence combination (specifics-in-gold AND subject-in-gold) is a strictly stronger signal than NLI's clause-level max contradiction, so when it fires it should win. The combination discriminates Poland-Tusk (Tusk ∈ gold, Poland ∈ gold → CG) from Anthony-Albanese (Albanese ∉ gold → fast path declines → falls through to UNGROUNDED-subject-in-gold → WRONG, unchanged). Self-test 4/4 INSTRUMENT TRUSTWORTHY unchanged. pytest 27/27. Poland-Tusk regression smoke: now CG via short_entity_grounded ✓. No regression risk on the existing reconciliation cells: - Iceland CG: short_entity_grounded was already winning (was rule 4, now rule 3 — same outcome, earlier exit) - WWII-1812 WRONG: '1812' ∉ gold → fast path declines, NLI fires ✓ - Higgs-cafe FABRICATED: 'Higgs' ∉ gold → fast path declines ✓ - Anthony Albanese WRONG: 'Albanese' ∉ gold → fast path declines ✓ - Abstention phrases: rule 2 still fires first ✓
This commit is contained in:
parent
c6621ee700
commit
866e67f43d
1 changed files with 34 additions and 22 deletions
|
|
@ -565,28 +565,24 @@ def judge(question: str, answer: str, gold_source: str) -> Verdict:
|
|||
trace["subject_anchor"] = subj_anchor
|
||||
trace["subject_in_gold"] = subj_in_gold
|
||||
|
||||
# Rule 3 — strong NLI contradiction (gold-contradicts-claim).
|
||||
# The 0.85 threshold is the code-judge floor for "this is a real
|
||||
# contradiction, not NLI noise on wikitext-shaped gold" — see the
|
||||
# _CODE_JUDGE_THETA_CONTRA constant for the measurement that drove
|
||||
# the recalibration from the manifest's 0.5.
|
||||
theta_contra = _CODE_JUDGE_THETA_CONTRA
|
||||
trace["theta_contra_code_judge"] = theta_contra
|
||||
if nli_avail and nli.max_contradiction >= theta_contra:
|
||||
trace["rules_fired"].append("nli_contradiction")
|
||||
return _v("WRONG",
|
||||
f"NLI contradiction p={nli.max_contradiction:.3f} "
|
||||
f">= theta_contra={theta_contra:.2f}",
|
||||
trace)
|
||||
|
||||
# Rule 4 — short-answer entity-grounding fast path. Verifier's
|
||||
# strategy-2 needs prose shape; terse-name answers fall through
|
||||
# to UNGROUNDED with no specifics → ABSTAINED, missing valid CG.
|
||||
# Rescue when: (a) answer is short, (b) every specific asserted
|
||||
# is present in gold (no unsourced specifics), (c) at least one
|
||||
# specific WAS asserted (not just "ok" or other empty content),
|
||||
# (d) the question's subject anchor is in gold (guards against
|
||||
# "wrong topic, right name" false positives).
|
||||
# Rule 3 — short-answer entity-grounding fast path. RUNS BEFORE
|
||||
# NLI contradiction (since 2026-05-19 Poland-Tusk smoke): when the
|
||||
# answer asserts only specifics that are ALL in gold AND the
|
||||
# question's subject anchor is in gold, that is strong positive
|
||||
# evidence — NLI clause-level noise cannot override it. The
|
||||
# Poland case (qwen+arborist answer "Donald Tusk is listed as the
|
||||
# Prime Minister of Poland.", gold lists multiple PMs across
|
||||
# decades) measured NLI contradiction p=0.89 ABOVE the 0.85
|
||||
# threshold while NLI entailment was also 0.74 on the correct
|
||||
# clause — mixed signal. The fast path's positive-evidence
|
||||
# combination (specifics-in-gold + subject-in-gold) discriminates
|
||||
# truth from contradiction without depending on NLI's clause-
|
||||
# level aggregation. Rescue when: (a) answer is short, (b) every
|
||||
# specific asserted is present in gold (no unsourced specifics),
|
||||
# (c) at least one specific WAS asserted, (d) the question's
|
||||
# subject anchor is in gold (guards against "wrong topic, right
|
||||
# name" false positives — e.g. "Anthony Albanese" wouldn't fire
|
||||
# because Albanese ∉ gold).
|
||||
short = _is_short_answer(a)
|
||||
trace["short_answer"] = short
|
||||
if short and subj_in_gold:
|
||||
|
|
@ -603,6 +599,22 @@ def judge(question: str, answer: str, gold_source: str) -> Verdict:
|
|||
f"subject anchor {subj_anchor!r} in gold",
|
||||
trace)
|
||||
|
||||
# Rule 4 — strong NLI contradiction (gold-contradicts-claim).
|
||||
# Runs AFTER the entity-grounding fast path so positive lexical
|
||||
# evidence can't be overridden by NLI noise. The 0.85 threshold is
|
||||
# the code-judge floor for "this is a real contradiction, not NLI
|
||||
# noise on wikitext-shaped gold" — see the _CODE_JUDGE_THETA_CONTRA
|
||||
# constant for the measurement that drove the recalibration from
|
||||
# the manifest's 0.5.
|
||||
theta_contra = _CODE_JUDGE_THETA_CONTRA
|
||||
trace["theta_contra_code_judge"] = theta_contra
|
||||
if nli_avail and nli.max_contradiction >= theta_contra:
|
||||
trace["rules_fired"].append("nli_contradiction")
|
||||
return _v("WRONG",
|
||||
f"NLI contradiction p={nli.max_contradiction:.3f} "
|
||||
f">= theta_contra={theta_contra:.2f}",
|
||||
trace)
|
||||
|
||||
# Rule 5 — lexical verifier against gold.
|
||||
v = verify_quotes(a, g)
|
||||
mode = v.get("audit_mode", "UNGROUNDED")
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue