qa: tighten TITLE_MISMATCH + DEFLECTION demotes from emergent-log review
Emergent stress-test log surfaced two failure shapes the curated bench can't reach. Reviewer (claude-opus-4-7 + fox) appended teacher: blocks to four log entries; two of them named real defects in the demote chain. This commit ships those fixes plus the teacher-block annotations. Cases reviewed (2026-05-02 emergent_log.jsonl): ✅ Menkar/sterilizers/trifle — working as designed 🟡 swallowtail/upbraided/rockiest — metaphorical deflection 🔴 cashback/widescreens/within — warrant gap on inferential claims 🔴 comeliness/fetish/investitures — topic shift earned ANCHOR-WARRANTED unfairly Two code changes: 1) verify_claim_lattice + verify_claim_lattice_json: when EVERY resolving claim has TITLE_MISMATCH, demote audit_mode → UNGROUNDED. Previously HYBRID. Catches the cashback case ('widescreens offer cashback' cited to a generic Coupon article — n_verified=1 overclaimed when the citation was meaningless). 2) cli._ladder_rung_for_lattice: DEFLECTION_DETECTED moves from soft-demote (cap at ANCHOR-WARRANTED) to hard-demote (cap at POINTER-LINKED) alongside WARRANT_MISSING and TITLE_MISMATCH. Catches the comeliness case (model fully shifted topic to 'Rock & Chips'; pre-fix the verified Rock & Chips claims landed at ANCHOR-WARRANTED unfairly). Test updates: - tests/test_verify_json.py:test_verify_json_title_mismatch_demotes_to_ungrounded (renamed; expected outcome flipped HYBRID → UNGROUNDED) - 4 fixtures in tests/test_claim_lattice.py and 1 in tests/test_verify_json.py: claim text augmented with 'in the film' so the Jurassic-Park-titled fixture's title shares a stem with the claim. These tests were testing other paths (NO_EVIDENCE_POINTER, EVIDENCE_LINKED_PARTIAL, UNKNOWN_EVIDENCE_ID) and the title-mismatch was incidental to the fixture choice. Teacher blocks on the four log entries record the verdict + failure_class + recommendation + notes, with reviewer + date. The two ✅/🟡 entries have no recommended code change; the two 🔴 entries reference this commit as the fix. 751/34 tests pass.
This commit is contained in:
parent
ea82e4288c
commit
a1dd330455
5 changed files with 104 additions and 18 deletions
|
|
@ -427,14 +427,14 @@ def _cmd_query(args: argparse.Namespace) -> int:
|
|||
|
||||
|
||||
# Soft-demote violation kinds that demote STRICT to HYBRID without
|
||||
# rejecting the pointer outright. WARRANT_MISSING handled separately
|
||||
# (its presence determines POINTER-LINKED vs ANCHOR-WARRANTED).
|
||||
# rejecting the pointer outright. WARRANT_MISSING / TITLE_MISMATCH /
|
||||
# DEFLECTION_DETECTED handled separately as hard demotes (their
|
||||
# presence determines POINTER-LINKED vs ANCHOR-WARRANTED).
|
||||
_SOFT_DEMOTE_VIOLATION_KINDS = frozenset({
|
||||
"LAZY_ANCHOR_DEMOTED",
|
||||
"POINTER_OVERFLOW_TRIMMED",
|
||||
"TOO_MANY_CLAIMS",
|
||||
"BARE_NAME_CLAIM",
|
||||
"DEFLECTION_DETECTED",
|
||||
})
|
||||
|
||||
|
||||
|
|
@ -466,11 +466,21 @@ def _ladder_rung_for_lattice(audit_mode: str, violations: list[dict] | None) ->
|
|||
if audit_mode == "UNGROUNDED":
|
||||
return "UNGROUNDED"
|
||||
kinds = {v.get("kind") for v in (violations or [])}
|
||||
# WARRANT_MISSING and TITLE_MISMATCH both indicate the citation
|
||||
# is structurally misaligned with the claim — pointer resolved
|
||||
# but the cited evidence (span or source) doesn't actually
|
||||
# support the claim. Both drop to POINTER-LINKED.
|
||||
if "WARRANT_MISSING" in kinds or "TITLE_MISMATCH" in kinds:
|
||||
# Hard demotes: WARRANT_MISSING, TITLE_MISMATCH, DEFLECTION_DETECTED
|
||||
# all indicate the citation is structurally misaligned with the
|
||||
# claim or the answer is structurally off-topic — pointer resolved
|
||||
# but the cited evidence (span, source, or whole answer) doesn't
|
||||
# actually support the user's question. All three drop to
|
||||
# POINTER-LINKED. (DEFLECTION_DETECTED was previously a soft demote
|
||||
# to ANCHOR-WARRANTED; the comeliness/fetish/investitures emergent
|
||||
# case showed that "the model totally shifted topic but anchored
|
||||
# the new topic" earned ANCHOR-WARRANTED unfairly. Off-topic
|
||||
# belongs at the lower rung.)
|
||||
if (
|
||||
"WARRANT_MISSING" in kinds
|
||||
or "TITLE_MISMATCH" in kinds
|
||||
or "DEFLECTION_DETECTED" in kinds
|
||||
):
|
||||
rung = "POINTER-LINKED"
|
||||
elif kinds & _SOFT_DEMOTE_VIOLATION_KINDS:
|
||||
rung = "ANCHOR-WARRANTED"
|
||||
|
|
|
|||
|
|
@ -1478,6 +1478,25 @@ def verify_claim_lattice(
|
|||
})
|
||||
if title_mismatch_claims and audit_mode == "STRICT":
|
||||
audit_mode = "HYBRID"
|
||||
# Tightening (2026-05-02 emergent-log finding): when EVERY resolving
|
||||
# claim has TITLE_MISMATCH, the substrate has zero structural
|
||||
# grounding for the user's question — every cited source is
|
||||
# title-irrelevant. The cashback case ("widescreens offer cashback"
|
||||
# cited to a generic Coupon article) had n_verified=1 but the
|
||||
# citation was meaningless; HYBRID overclaimed. Demote to
|
||||
# UNGROUNDED so the four-rung ladder maps it to UNGROUNDED, not
|
||||
# POINTER-LINKED-PARTIAL.
|
||||
n_resolving = sum(
|
||||
1
|
||||
for cs in claim_statuses
|
||||
if cs.get("status") in ("EVIDENCE_LINKED", "EVIDENCE_LINKED_PARTIAL")
|
||||
)
|
||||
if (
|
||||
title_mismatch_claims
|
||||
and n_resolving > 0
|
||||
and len(title_mismatch_claims) == n_resolving
|
||||
):
|
||||
audit_mode = "UNGROUNDED"
|
||||
|
||||
# Deflection check (soft demote, promoted from sidecar 2026-05-02).
|
||||
# When the question's subject anchor is missing from the answer,
|
||||
|
|
@ -1906,6 +1925,21 @@ def verify_claim_lattice_json(
|
|||
})
|
||||
if title_mismatch_claims and audit_mode == "STRICT":
|
||||
audit_mode = "HYBRID"
|
||||
# Tightening (2026-05-02): mirrors the pointer-variant promotion.
|
||||
# When EVERY resolving claim has TITLE_MISMATCH, demote to
|
||||
# UNGROUNDED — the substrate has zero structural grounding for
|
||||
# the user's question. See verify_claim_lattice for full rationale.
|
||||
n_resolving = sum(
|
||||
1
|
||||
for cs in claim_statuses
|
||||
if cs.get("status") in ("EVIDENCE_LINKED", "EVIDENCE_LINKED_PARTIAL")
|
||||
)
|
||||
if (
|
||||
title_mismatch_claims
|
||||
and n_resolving > 0
|
||||
and len(title_mismatch_claims) == n_resolving
|
||||
):
|
||||
audit_mode = "UNGROUNDED"
|
||||
|
||||
# Deflection check (parallel to pointer-variant promotion).
|
||||
deflection_detected = False
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -273,7 +273,7 @@ def test_no_evidence_pointer_downgrades():
|
|||
em = build_evidence_map(_sample_chunks())
|
||||
answer = (
|
||||
"Trex appears in the film.\n" # no pointer
|
||||
"Velociraptors are shown. [E2]\n"
|
||||
"Velociraptors are shown in the film. [E2]\n" # 'film' overlaps title
|
||||
)
|
||||
# Opt out of bare-name check (Velociraptors stems to 1 content token
|
||||
# after spotlight stopword strip); test exercises NO_EVIDENCE_POINTER.
|
||||
|
|
@ -289,7 +289,7 @@ def test_no_evidence_pointer_downgrades():
|
|||
def test_partial_resolution_yields_hybrid():
|
||||
em = build_evidence_map(_sample_chunks())
|
||||
answer = (
|
||||
"Tyrannosaurus rex appears. [E1]\n"
|
||||
"Tyrannosaurus rex appears in the film. [E1]\n" # 'film' overlaps title
|
||||
"Made-up. [E99]\n"
|
||||
)
|
||||
# Test exercises mixed-pointer-resolution behavior; bare-name check
|
||||
|
|
@ -302,7 +302,7 @@ def test_partial_resolution_yields_hybrid():
|
|||
|
||||
def test_partial_pointer_within_claim_yields_partial():
|
||||
em = build_evidence_map(_sample_chunks())
|
||||
answer = "Tyrannosaurus rex appears. [E1,E99]\n"
|
||||
answer = "Tyrannosaurus rex appears in the film. [E1,E99]\n" # 'film' overlaps title
|
||||
v = verify_claim_lattice(answer, em, min_claim_content_tokens=0)
|
||||
assert v["audit_mode"] == "HYBRID"
|
||||
assert v["claim_statuses"][0]["status"] == "EVIDENCE_LINKED_PARTIAL"
|
||||
|
|
@ -1029,7 +1029,7 @@ def test_citation_mismatch_partial_when_one_pointer_overlaps():
|
|||
em = build_evidence_map(_sample_chunks())
|
||||
# E1 contains "Tyrannosaurus rex"; E2 contains "Velociraptors".
|
||||
# Claim mentions Tyrannosaurus → E1 overlaps, E2 does not.
|
||||
answer = "Tyrannosaurus rex appears. [E1,E2]\n"
|
||||
answer = "Tyrannosaurus rex appears in the film. [E1,E2]\n" # 'film' overlaps title
|
||||
# Test exercises CITATION_MISMATCH on a single pointer; bare-name
|
||||
# check is orthogonal so opt out.
|
||||
v = verify_claim_lattice(answer, em, min_claim_content_tokens=0)
|
||||
|
|
|
|||
|
|
@ -134,7 +134,7 @@ def test_verify_json_strict_when_all_claims_resolve():
|
|||
|
||||
def test_verify_json_hybrid_when_some_unknown_evidence_id():
|
||||
evidence = [
|
||||
_ev("E1f8e4c2a", "Brachiosaurus appears in the Jurassic Park film as a herbivore.", pointer_id="E1"),
|
||||
_ev("E1f8e4c2a", "Brachiosaurus appears in the Jurassic Park film as a herbivore.", pointer_id="E1", title="Jurassic Park (film)"),
|
||||
]
|
||||
answer = json.dumps({
|
||||
"claims": [
|
||||
|
|
@ -244,9 +244,15 @@ def test_claim_title_overlap_vacuous_pass_on_empty():
|
|||
assert _claim_title_overlap("", "Some Title")
|
||||
|
||||
|
||||
def test_verify_json_title_mismatch_demotes_to_hybrid():
|
||||
def test_verify_json_title_mismatch_demotes_to_ungrounded():
|
||||
"""End-to-end Rule 8: claim cited to a source with no title-token
|
||||
overlap → TITLE_MISMATCH violation, audit_mode demoted to HYBRID.
|
||||
overlap → TITLE_MISMATCH violation. When EVERY resolving claim is
|
||||
title-mismatched (here only 1 claim, 1/1 mismatched), audit_mode
|
||||
demotes to UNGROUNDED — the substrate has zero structural grounding
|
||||
for the user's question. Earlier behavior was HYBRID; tightened
|
||||
2026-05-02 after the cashback emergent case ('widescreens offer
|
||||
cashback' cited to a generic Coupon article) showed n_verified=1
|
||||
overclaimed when the citation was meaningless.
|
||||
|
||||
Span is intentionally rich with claim tokens (covers Rule 5
|
||||
citation-coverage threshold) so the failure path is Rule 8
|
||||
|
|
@ -278,9 +284,10 @@ def test_verify_json_title_mismatch_demotes_to_hybrid():
|
|||
]
|
||||
})
|
||||
v = verify_claim_lattice_json(answer, evidence)
|
||||
assert v["audit_mode"] == "HYBRID", (
|
||||
f"expected HYBRID via TITLE_MISMATCH; got {v['audit_mode']} "
|
||||
f"with violations {[v['kind'] for v in v['violations']]}"
|
||||
assert v["audit_mode"] == "UNGROUNDED", (
|
||||
f"expected UNGROUNDED via all-claims TITLE_MISMATCH demote; "
|
||||
f"got {v['audit_mode']} with violations "
|
||||
f"{[v['kind'] for v in v['violations']]}"
|
||||
)
|
||||
assert any(vio["kind"] == "TITLE_MISMATCH" for vio in v["violations"])
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue