qa(verify): Rule 9 — SUBJECT_TOKENS_ABSENT premise-parroting demote
Closes the first confirmed EVIDENCE-WARRANTED false-positive surfaced by the 200-cycle bench-emergent run on `steer/reply/correcter` (Ticket #000006 amend 2026-05-02b). The model parroted three question-distinctive tokens (correcter, steer, reply) into its claim while citing a glossary article whose 33.5K-char content contains ZERO occurrences of any of them. Generic linguistic vocabulary (language, communication, terms, relationships) carried Rule 5's citation-coverage check on its own; the actual subject tokens rode along unverified. New per-claim check `_parroted_subject_tokens_absent`: for each resolving claim, compute the question∩claim content-token set, then check substring presence in the union of cited evidence spans (lower-cased, mirroring Rule 5). When ≥ threshold parroted tokens are absent, emit `SUBJECT_TOKENS_ABSENT` and demote STRICT → HYBRID. Default threshold = 3 — single-token absence is often stem-variant noise; three+ is the parrot fingerprint. Plumbing: - New default `DEFAULT_SUBJECT_TOKENS_ABSENT_THRESHOLD = 3` - Both `verify_claim_lattice` and `verify_claim_lattice_json` gain `subject_tokens_absent_threshold` kwarg + per-claim check block (mirrors TITLE_MISMATCH plumbing, sits right after it in the rule order) - `claim_lattice_subject_tokens_absent_threshold` policy field added to `DEFAULT_QUERY_POLICY` and `DEFAULT_POLICY`; folds into `governance_policy_hash` and (via _VERIFIER_POLICY_FIELDS) `verifier_policy_hash` - All four runner/query call sites pass the policy-derived value Live verification (cache-split cleanly via policy-hash bump): pre-fix cache_key 08dbd2c1… : STRICT (false positive) post-fix cache_key 6a519636… : UNGROUNDED Three new unit tests in `tests/test_verify_json.py`: - threshold-meeting parrot demotes STRICT → HYBRID - no-op when question is None - below-threshold absence stays STRICT Full suite: 776 passed, 34 skipped.
This commit is contained in:
parent
d3ad520529
commit
39bebe3fdb
6 changed files with 311 additions and 13 deletions
|
|
@ -394,3 +394,132 @@ def test_runner_ask_json_mode_passes_guided_json_extra_body(tmp_path):
|
|||
assert "guided_json" in kwargs["extra_body"]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- Rule 9
|
||||
# Subject-tokens-absent / premise-parroting check (Ticket #000006 amend
|
||||
# 2026-05-02b). Surfaced by the 200-cycle bench-emergent finding on
|
||||
# `steer/reply/correcter`: claim affirmed three question-distinctive
|
||||
# tokens that appeared zero times in the cited 33.5K-char glossary,
|
||||
# while generic linguistic vocabulary carried Rule 5's coverage check.
|
||||
|
||||
|
||||
def test_verify_json_subject_tokens_absent_demotes_strict_to_hybrid():
|
||||
"""Reproduces the steer/reply/correcter false-positive shape:
|
||||
the claim parrots the question's distinctive subject tokens
|
||||
(correcter, steer, reply) but the cited evidence contains only
|
||||
generic linguistic vocabulary (language, communication, terms).
|
||||
Rule 5 passes on the generic overlap; Rule 9 catches that the
|
||||
question-distinctive tokens are absent and demotes."""
|
||||
cited_span = (
|
||||
"Sociolinguistics is the study of language in society and how "
|
||||
"social factors influence communication. The exchange of meaning "
|
||||
"between speakers depends on shared terms and discourse "
|
||||
"relationships. Different aspects of language interact with "
|
||||
"communication norms in any given community."
|
||||
)
|
||||
evidence = [
|
||||
_ev(
|
||||
"Eparrot01",
|
||||
cited_span,
|
||||
pointer_id="E1",
|
||||
title="Glossary of language teaching terms and ideas",
|
||||
),
|
||||
]
|
||||
answer = json.dumps({
|
||||
"claims": [
|
||||
{
|
||||
# Three question-distinctive tokens (correcter, steer,
|
||||
# reply) parroted from question into claim — but ZERO
|
||||
# of these tokens appear in cited_span.
|
||||
"text": (
|
||||
"A correcter can be used to steer a reply by "
|
||||
"identifying errors in language and communication "
|
||||
"between speakers, which involves the exchange of "
|
||||
"meaning across discourse relationships."
|
||||
),
|
||||
"evidence_ids": ["E1"],
|
||||
}
|
||||
]
|
||||
})
|
||||
question = (
|
||||
"How might a correcter be used to steer a reply in a "
|
||||
"conversation, and what aspects of language or communication "
|
||||
"do these terms encompass?"
|
||||
)
|
||||
v = verify_claim_lattice_json(answer, evidence, question=question)
|
||||
assert v["audit_mode"] == "HYBRID", (
|
||||
f"expected HYBRID via SUBJECT_TOKENS_ABSENT demote; got "
|
||||
f"{v['audit_mode']} with violations "
|
||||
f"{[vio['kind'] for vio in v['violations']]}"
|
||||
)
|
||||
assert any(vio["kind"] == "SUBJECT_TOKENS_ABSENT" for vio in v["violations"])
|
||||
# The parroted-but-absent tokens should be reported.
|
||||
sta = next(vio for vio in v["violations"] if vio["kind"] == "SUBJECT_TOKENS_ABSENT")
|
||||
absent = set(sta["absent_tokens"])
|
||||
assert {"correcter", "steer", "reply"}.issubset(absent), (
|
||||
f"expected correcter/steer/reply in absent_tokens; got {absent}"
|
||||
)
|
||||
|
||||
|
||||
def test_verify_json_subject_tokens_absent_no_question_skips_check():
|
||||
"""No question text → Rule 9 is a no-op. STRICT stays STRICT
|
||||
when every other check passes. Pins that the check requires
|
||||
question text to operate."""
|
||||
evidence = [
|
||||
_ev(
|
||||
"Enoq00001",
|
||||
"Brachiosaurus appears in the Jurassic Park film as a herbivore.",
|
||||
pointer_id="E1",
|
||||
title="Jurassic Park (film)",
|
||||
),
|
||||
]
|
||||
answer = json.dumps({
|
||||
"claims": [
|
||||
{"text": "Brachiosaurus appears in the film", "evidence_ids": ["E1"]},
|
||||
]
|
||||
})
|
||||
v = verify_claim_lattice_json(answer, evidence, question=None)
|
||||
assert v["audit_mode"] == "STRICT"
|
||||
assert not any(vio["kind"] == "SUBJECT_TOKENS_ABSENT" for vio in v["violations"])
|
||||
|
||||
|
||||
def test_verify_json_subject_tokens_absent_below_threshold_passes():
|
||||
"""One absent parroted token → below default threshold of 3 →
|
||||
no demote. Pins the threshold semantics: single-token absence
|
||||
is acceptable noise, three+ is the parrot fingerprint."""
|
||||
cited_span = (
|
||||
"Brachiosaurus appears in the Jurassic Park film as a herbivore. "
|
||||
"The dinosaurs in the film were rendered with practical effects "
|
||||
"and CGI by Industrial Light and Magic."
|
||||
)
|
||||
evidence = [
|
||||
_ev(
|
||||
"Eonebelow",
|
||||
cited_span,
|
||||
pointer_id="E1",
|
||||
title="Jurassic Park (film)",
|
||||
),
|
||||
]
|
||||
# Question token "extinction" doesn't appear in cited; "brachiosaurus"
|
||||
# and "film" do. Only 1 parroted-token absent → below threshold 3.
|
||||
answer = json.dumps({
|
||||
"claims": [
|
||||
{
|
||||
"text": (
|
||||
"Brachiosaurus appears in the film alongside other "
|
||||
"dinosaurs after a long extinction"
|
||||
),
|
||||
"evidence_ids": ["E1"],
|
||||
}
|
||||
]
|
||||
})
|
||||
question = (
|
||||
"How does Brachiosaurus appear in the Jurassic Park film "
|
||||
"after extinction?"
|
||||
)
|
||||
v = verify_claim_lattice_json(answer, evidence, question=question)
|
||||
assert v["audit_mode"] == "STRICT", (
|
||||
f"expected STRICT (below threshold); got {v['audit_mode']} "
|
||||
f"with violations {[vio['kind'] for vio in v['violations']]}"
|
||||
)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue