qa/inspect: deflection sidecar — subject-anchor topic-shift detection

Empirical finding 2026-04-30 on 'who is a benevolent dictator for life
for mars?': JSON mode returned STRICT with answer 'Guido van Rossum is
a Benevolent Dictator For Life (BDFL) for the Python programming
language.' The verifier did its job (claims grounded), but the answer
never mentioned Mars — pure topic-shift. A real failure mode for
adversarial-premise questions where the model deflects rather than
refuses.

Detection rule: subject-anchor heuristic. The LAST content token in the
question (after stopword strip) is treated as the question's primary
subject ('mars' for BDFL, 'beatles' for 'who are the members of the
beatles?'). If the subject anchor is missing from the answer, classify
as 'deflection' regardless of generic-vocabulary overlap. The Mars-BDFL
case has 3/4 generic overlap (benevolent, dictator, life) but missing
subject — exactly the pattern overlap-ratio alone can't catch.

Returns dict with kind ∈ {deflection, partial_overlap, on_topic,
no_question_tokens} plus subject_anchor + subject_in_answer for
operator legibility.

Sidecar-only — wired into inspect_cache_key() but never feeds back
into providence_cache or audit_events. Per the verifier-stays-binary
discipline (CLAUDE.md): soft signals live in inspect verbs, never in
the hard chain.

aborist/qa/inspect.py

@@ -333,6 +333,137 @@ def _repair_for_trailing_artifact(span: str, diag: dict[str, Any]) -> dict[str,
     return {"action": "trim_trailing_artifact"}
 
 
+# ---------------------------------------------------------------- deflection signal
+
+
+# Stopword set for question/answer token overlap. Mirrors the title-search
+# stopwords in aborist.qa.query._TITLE_STOPWORDS but kept local so this
+# sidecar never reaches into the verifier's machinery for soft computations.
+_DEFLECTION_STOPWORDS = frozenset(
+    """
+    the a an is are was were be been being of to in on at for with by from
+    as about into through during and or but not no nor so yet too very also just
+    what who where when why how which this that these those such i you he she
+    it we they me him her us them do does did have has had can could should
+    would will may might
+    tell show describe explain summarize say give list find make please
+    name names called known does did can could would all there
+    """.split()
+)
+
+
+def _stem_for_deflection(t: str) -> str:
+    """Mirror of aborist.qa.query._stem_token_for_match: strip trailing
+    `'s` (possessive) then trailing `s` on tokens >4 chars (skip
+    `ss`-enders). Keeps 4-char words like 'mars' intact while
+    collapsing 'mars's' → 'mars' and 'rivers' → 'river'."""
+    if t.endswith("'s") and len(t) > 3:
+        t = t[:-2]
+    if len(t) > 4 and t.endswith("s") and not t.endswith("ss"):
+        t = t[:-1]
+    return t
+
+
+def _content_tokens_for_deflection(text: str) -> set[str]:
+    """Lower-cased content tokens (≥3 chars, non-stopword) for soft
+    question/answer overlap analysis.
+
+    Lower minimum length than the verifier's ``_content_tokens`` (≥4)
+    so that 3-char proper-noun stems ("amd", "bsd", "fox") still
+    register on the deflection signal — the cost of false-positives
+    is just a soft sidecar advisory, not a verifier promotion."""
+    return set(_content_tokens_in_order(text))
+
+
+def _content_tokens_in_order(text: str) -> list[str]:
+    """Same content-token extraction as ``_content_tokens_for_deflection``
+    but preserves source order. Used by the subject-anchor heuristic
+    where the LAST content token is treated as the question's primary
+    subject ('who is a benevolent dictator for life for mars?' → 'mars')."""
+    import re
+    out: list[str] = []
+    for raw in re.findall(r"[A-Za-z][A-Za-z'\-]+", text):
+        t = _stem_for_deflection(raw.lower())
+        if len(t) < 3 or t in _DEFLECTION_STOPWORDS:
+            continue
+        out.append(t)
+    return out
+
+
+def diagnose_deflection(question_text: str, answer_text: str) -> dict[str, Any]:
+    """Soft signal: did the answer change topic?
+
+    Empirically observed 2026-04-30 on 'who is a benevolent dictator
+    for life for mars?' — JSON mode grounded its claims (STRICT) but
+    the answer mentioned only Guido van Rossum and Python, never Mars.
+    Verifier did its job (claims are grounded); the user's question
+    wasn't answered. A real failure mode for adversarial-premise
+    questions where the model deflects rather than refuses.
+
+    Detection rule:
+
+    1. **Subject anchor**: the LAST content token in the question
+       (after stopword strip) is a strong heuristic for the question's
+       primary subject ('mars' in the BDFL case, 'beatles' in 'who
+       are the members of the beatles?'). If the subject anchor
+       doesn't appear in the answer → ``deflection``.
+    2. Otherwise, fall back to the overlap-ratio heuristic (1.0 →
+       ``on_topic``, partial → ``partial_overlap``, 0 → ``deflection``).
+
+    Subject-anchor is more robust than zero-overlap because the
+    BDFL-Mars case has 3/4 generic-vocabulary overlap (benevolent,
+    dictator, life) while still being a clear topic shift. The signal
+    we want is "the *subject* is missing", not "all tokens are missing."
+
+    Returns:
+        {
+            "kind": "deflection" | "partial_overlap" | "on_topic" | "no_question_tokens",
+            "question_tokens": [...],
+            "answer_tokens": [...],
+            "overlap": [...],
+            "overlap_ratio": float,  # |overlap| / |question_tokens|
+            "subject_anchor": str | None,
+            "subject_in_answer": bool,
+        }
+
+    Sidecar-only — never enters the binary verifier output."""
+    q_in_order = _content_tokens_in_order(question_text or "")
+    qtok = set(q_in_order)
+    atok = _content_tokens_for_deflection(answer_text or "")
+    if not qtok:
+        return {
+            "kind": "no_question_tokens",
+            "question_tokens": [],
+            "answer_tokens": sorted(atok),
+            "overlap": [],
+            "overlap_ratio": 0.0,
+            "subject_anchor": None,
+            "subject_in_answer": False,
+        }
+    subject_anchor = q_in_order[-1] if q_in_order else None
+    subject_in_answer = subject_anchor in atok if subject_anchor else False
+    overlap = qtok & atok
+    if subject_anchor and not subject_in_answer:
+        # Subject token missing dominates — even if 3/4 generic vocab
+        # overlapped, the actual topic isn't in the answer.
+        kind = "deflection"
+    elif not overlap:
+        kind = "deflection"
+    elif len(overlap) < len(qtok):
+        kind = "partial_overlap"
+    else:
+        kind = "on_topic"
+    return {
+        "kind": kind,
+        "question_tokens": sorted(qtok),
+        "answer_tokens": sorted(atok),
+        "overlap": sorted(overlap),
+        "overlap_ratio": len(overlap) / len(qtok),
+        "subject_anchor": subject_anchor,
+        "subject_in_answer": subject_in_answer,
+    }
+
+
 def inspect_cache_key(
     cache_key: str,
     *,
@@ -415,6 +546,13 @@ def inspect_cache_key(
         d = _classify_span(q, norm_base_ctx=n_base, norm_raw_ctx=n_raw)
         diagnoses.append({"span": q, **d})
 
+    # Soft deflection signal — see diagnose_deflection. Computed even
+    # for STRICT records: a STRICT record where the answer never
+    # mentions the question's subject is the Mars-BDFL pattern (model
+    # deflected to a related grounded fact rather than refusing or
+    # answering). Sidecar-only, never feeds back into providence.
+    deflection = diagnose_deflection(row["question_text"], row["answer_text"])
+
     return {
         "status": "ok",
         "record": {
@@ -434,4 +572,5 @@ def inspect_cache_key(
             "wikitext_strip_active": _wikitext_to_base is not None,
         },
         "unverified": diagnoses,
+        "deflection": deflection,
     }

tests/test_inspect.py

@@ -18,7 +18,12 @@ from pathlib import Path
 
 import pytest
 
-from aborist.qa.inspect import _classify_span, _normalize, inspect_cache_key
+from aborist.qa.inspect import (
+    _classify_span,
+    _normalize,
+    diagnose_deflection,
+    inspect_cache_key,
+)
 from aborist.store import append_audit, connect, transaction
 
 
@@ -408,3 +413,74 @@ def test_inspect_does_not_mutate_state(tmp_path):
         conn.close()
     assert after_audit == before_audit
     assert after_prov == before_prov
+
+
+# ---------------------------------------------------------------------------
+# deflection diagnostic
+# ---------------------------------------------------------------------------
+
+
+def test_deflection_mars_bdfl_to_python_guido():
+    """Empirical 2026-04-30: 'who is a benevolent dictator for life
+    for mars?' returned STRICT with answer 'Guido van Rossum is a BDFL
+    for the Python programming language.' Verifier did its job (claims
+    grounded) but answer never mentions mars — pure topic-shift.
+
+    Subject-anchor heuristic catches this: 3/4 of question tokens
+    (benevolent, dictator, life) match the answer, but the SUBJECT
+    anchor (last content token, 'mars') is missing. That's the signal
+    that a generic-vocabulary overlap can't paper over."""
+    d = diagnose_deflection(
+        "who is a benevolent dictator for life for mars?",
+        "Guido van Rossum is a Benevolent Dictator For Life (BDFL) "
+        "for the Python programming language.",
+    )
+    assert d["kind"] == "deflection", d
+    assert "mars" in d["question_tokens"]
+    assert d["subject_anchor"] == "mars"
+    assert d["subject_in_answer"] is False
+    assert "mars" not in d["overlap"]
+    # Generic-vocabulary overlap is high (3/4) but subject is missing
+    # — that's the deflection signal subject-anchor catches.
+    assert d["overlap_ratio"] > 0.5
+
+
+def test_deflection_partial_overlap_when_some_subjects_match():
+    """Some content tokens overlap, others don't — soft signal."""
+    d = diagnose_deflection(
+        "what is the relationship between linux and unix?",
+        "Linux is a Unix-like operating system inspired by Unix design.",
+    )
+    assert d["kind"] == "partial_overlap" or d["kind"] == "on_topic", d
+    assert "linux" in d["overlap"]
+    assert "unix" in d["overlap"]
+    # 'relationship' is a question-shape word that may or may not appear
+    # in answer; key is that core subjects overlap.
+    assert d["overlap_ratio"] > 0.0
+
+
+def test_deflection_on_topic_when_all_subjects_present():
+    """All question content tokens appear in answer — clean on-topic."""
+    d = diagnose_deflection(
+        "who painted the mona lisa?",
+        "Leonardo da Vinci painted the Mona Lisa.",
+    )
+    assert d["kind"] == "on_topic"
+    assert d["overlap_ratio"] == 1.0
+
+
+def test_deflection_handles_empty_question_or_answer():
+    """Vacuous case: question with no content tokens — no signal to give."""
+    d = diagnose_deflection("?", "Some answer text here.")
+    assert d["kind"] == "no_question_tokens"
+    assert d["overlap_ratio"] == 0.0
+
+
+def test_deflection_strips_possessive_s_for_overlap():
+    """Possessive 'mars's' should match 'mars' in the answer."""
+    d = diagnose_deflection(
+        "what is mars's atmosphere?",
+        "Mars has a thin atmosphere of carbon dioxide.",
+    )
+    assert "mars" in d["overlap"]
+    assert "atmosphere" in d["overlap"]