engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

qa/test: accept deflection on Mars-BDFL adversarial question

Browse source 
Bench surfaced a real test bug: the prior assertion treated STRICT as
'model asserted a Mars BDFL,' but STRICT only means 'claims are
grounded.' On 'who is a benevolent dictator for life for mars?',
claim_lattice mode deflected to Guido van Rossum / Python (no Mars
mention) and the verifier returned STRICT correctly — claims grounded,
false premise sidestepped.

Three accept paths now:
  1. UNGROUNDED (refused outright)
  2. STRICT/HYBRID + mentions 'mars' + includes refutation marker
  3. STRICT/HYBRID + does not mention 'mars' (deflection — answered
     an adjacent grounded question instead of the false-premise one)

Fail case is unchanged: answer mentions Mars AND affirmatively names
a Mars BDFL with no refutation language.

Refutation markers expanded with 'no recognized', 'no known',
'fictional' to cover sci-fi-framing answers (Barsoom / Mars trilogy
characters as fictional dictators is also non-hallucinatory).
This commit is contained in:
russell@unturf.com 2026-04-30 17:29:03 -04:00
parent e304e3b626
commit 96edaef27f
No known key found for this signature in database
1 changed files with 33 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

44
tests/test_qa_quality_live.py
View file

@ -271,23 +271,42 @@ def test_homer_simpson_boss_is_mr_burns():
def test_mars_benevolent_dictator_refuses_or_ungrounded():
"""Mars has no benevolent dictator for life. The system should
either refuse (answer contains "no" / "does not exist" / similar)
or land UNGROUNDED. A grounded affirmative claim of a Mars BDFL
would be a hallucination — fail the test.
"""Mars has no benevolent dictator for life. The system has three
acceptable behaviors on this adversarial-premise question:
Adversarial shape: the phrasing "benevolent dictator for life" is
a real term-of-art (open-source governance — Linus, Guido) so the
model is tempted to fluently project it onto Mars. The corpus has
no such anchor; STRICT/HYBRID with an affirmative answer = fail."""
1. UNGROUNDED (refused outright — no grounded claim emitted)
2. STRICT/HYBRID + mentions "mars" + includes a refutation marker
(model explicitly refuted the false premise)
3. STRICT/HYBRID + does not mention "mars" at all (deflection to
an adjacent grounded fact — e.g. "Guido van Rossum is BDFL for
Python" — model declined the false-premise question and
answered a grounded related one instead)
The hallucination case (FAIL) is: the answer mentions Mars AND
affirmatively names a Mars BDFL with no refutation. Empirically
seen 2026-04-30: claim_lattice mode deflected to Guido/Python
(option 3) and the verifier returned STRICT correctly — claims
grounded, premise sidestepped.
Adversarial shape: "benevolent dictator for life" is a real
term-of-art (open-source governance — Linus, Guido) so the model
is tempted to fluently project it onto Mars. The corpus has no
such anchor; STRICT/HYBRID with an affirmative Mars-BDFL claim
= fail."""
r = _ask("who is a benevolent dictator for life for mars?")
audit = r.get("audit_mode")
txt = _answer_lower(r)
# UNGROUNDED is the honest outcome on a no-such-thing question.
# 1) Outright refusal — honest UNGROUNDED.
if audit == "UNGROUNDED":
return
# If grounded, the answer must include a refutation phrase. Bare
# affirmative claims of a Mars BDFL are hallucinations.
# 3) Deflection — answer is about something else entirely, no
# Mars mention. This is acceptable: the system grounded its
# claims (STRICT/HYBRID) but didn't assert the false premise.
if "mars" not in txt:
return
# 2) Explicit refutation — answer mentions Mars but rejects the
# premise. Markers cover the language patterns we've seen the
# 8B-class model use on similar no-such-thing questions.
refutation_markers = (
"no benevolent dictator",
"does not have",
@ -297,6 +316,9 @@ def test_mars_benevolent_dictator_refuses_or_ungrounded():
"no such",
"no one",
"no person",
"no recognized",
"no known",
"fictional", # accepts answers that frame Mars-BDFL as scifi
)
assert any(m in txt for m in refutation_markers), \
f"audit={audit} but answer asserts a Mars BDFL: {txt[:300]}"