engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(#000057): stronger code judge — resolve HYBRID with verified quote + on-topic

Browse source 
The code judge bailed to JUDGE_ERROR on 40% of in-corpus answers: HYBRID
(partial grounding) with low NLI entail, where the entity-grounding
rescue needs ZERO unsourced specifics. A single extra proper noun
('Emperor Honorius', 'Alexander Molossus' — an alias/paraphrase) blocked
rescue even with verbatim quotes verified and the answer correct.

New HYBRID resolution tier: rescue to CORRECT_GROUNDED when the verifier
confirmed >=1 verbatim quote, the subject anchor is in gold (on-topic),
there is NO unsourced NUMERIC specific (wrong dates/counts stay residue),
and NLI isn't strongly contradicting. Unsourced proper nouns are treated
as aliases/paraphrase; unsourced numerics (the real factual-error class)
keep the answer as JUDGE_ERROR. Validated on the 12 real residue cases:
9 -> CORRECT (all genuinely right), 3 stay residue (unsourced numerics).
JUDGE_ERROR 40% -> ~10%. self-test 4/4; 2 new tier tests; suite 2549.
This commit is contained in:
russell@unturf.com 2026-05-21 13:10:01 -04:00
parent 39c040cacc
commit 2d3186669f
No known key found for this signature in database
2 changed files with 65 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

36
tests/test_judge_code.py
View file

@ -237,3 +237,39 @@ def test_judge_handles_nli_unavailable(monkeypatch):
"This article is about thermodynamics.",
)
assert v.label == "FABRICATED"
# --- relaxed HYBRID rescue (2026-05-21): tolerate unsourced proper nouns
# (aliases/paraphrase) but still block unsourced numerics (date/count
# errors), gated on a verified quote + on-topic + no contradiction. -----
_EPIRUS_GOLD = ("Alexander I of Epirus was a king of Epirus from 350 to "
"331 BC, belonging to the Aeacid dynasty. He was the son "
"of Neoptolemus I and brother of Olympias.")
def test_hybrid_rescued_when_only_unsourced_specific_is_proper_noun(monkeypatch):
# NLI off so the lexical tier is isolated (not the NLI corroboration
# path). One verbatim quote (verified) + one quote NOT in gold
# (-> HYBRID); the only unsourced specifics are proper nouns.
import bench.judge_code as jc
monkeypatch.setattr(jc, "_nli_check", lambda a, g: None)
answer = ('Alexander I of Epirus "was a king of Epirus from 350 to '
'331 BC". Also called Alexander Molossus, he "led famed '
'campaigns in southern Italy against the Romans".')
v = jc.judge("who was Alexander the first of epirus?", answer, _EPIRUS_GOLD)
assert v.label == "CORRECT_GROUNDED", (v.label, v.rationale)
assert v.decision.get("audit_mode") == "HYBRID"
assert "hybrid+verified_quote_on_topic" in v.decision["rules_fired"]
def test_hybrid_stays_residue_when_unsourced_specific_is_numeric(monkeypatch):
# NLI off. One verified quote + an unverified quote with a WRONG date
# -> unsourced numeric -> NOT rescued (date/count errors stay residue).
import bench.judge_code as jc
monkeypatch.setattr(jc, "_nli_check", lambda a, g: None)
answer = ('Alexander I of Epirus "was a king of Epirus". He '
'"reigned from 999 to 888 BC".')
v = jc.judge("who was Alexander the first of epirus?", answer, _EPIRUS_GOLD)
assert v.label == "JUDGE_ERROR", (v.label, v.rationale)
assert "hybrid+verified_quote_on_topic" not in v.decision["rules_fired"]