#000068 Phase 1: verifier-blind missed-answer falsification guard

Adds a deterministic read-only sidecar to detect a class of failure the
binary verifier is structurally blind to:

  Evidence contains the answer.
  Model says the evidence does not contain the answer.
  Verifier sees no unsupported positive claim -> marks run clean.
  User receives a false negative under EVIDENCE-WARRANTED.

The motivating case: "songs by veronica ballestrini" against the 2010
Wikipedia corpus. Hermes-3-8B under user_payload_layout=tail returned
"the specific songs by her are not mentioned in the provided evidence
blocks" when evidence E2 literally contained "Amazing", "Out There
Somewhere", "Fascinated", "What's Up With That", "Don't Say". Verifier
correctly returned EVIDENCE-WARRANTED 2/2 because the existing layered
verifier (quote / span / entity / paraphrase + Rule 8 title-relevance +
Rule 9 subject-tokens-absent + claim-count ceiling) guards unsupported
*presence*, has no hook for unsupported *absence*.

Layout fixes attention placement on the specific instance (the 5/27
n=3x75q bench confirms bookend/per_chunk recover Ballestrini); layout
alone cannot close the class -- adversarial phrasing or a bigger prompt
resurfaces the failure under any layout. The right substrate move is to
falsify "not mentioned" as a testable claim.

Detection rule (three-clause conjunction, all must fire):

  A. Denial pattern in answer (sealed v1 phrase list: "not mentioned",
     "not provided", "the evidence does not say", "does not mention",
     "no specific", "no evidence", "cannot determine from the provided
     evidence", "is not stated", "is not specified"). Casefolded +
     whitespace-normalized substring match.

  B. Question is extraction/list-shaped. Either a surface cue ("songs
     by", "works by", "books by", "who wrote", "who composed", "what
     year", "list of", "name all", ...) matches, OR the existing
     arborist.qa.quantifier classifier returns intensity in {ALL,
     COMPREHENSIVE, OPEN_REQUEST, MANY, PLURAL}.

  C. Evidence contains candidate spans matching the answer_type within
     a proximity window (default 600 chars) of cleaned subject tokens.
     Candidate kinds aligned to answer_type:
       title_like -> quoted_string, title_case_span, comma_list_item
       person     -> title_case_span
       date       -> year, date

Hardenings folded in from the 2026-05-27 Dav1d de-novo review:

  1. Subject tokens strip cue/relation/stop words. For "songs by
     veronica ballestrini" the cleaned subject is ["veronica",
     "ballestrini"], NOT all four tokens. Without this the guard
     false-triggers on "Harvard University" or "New York" near
     proper-noun subjects.

  2. Answer-type alignment. Candidate span kind must match query type
     so "songs by John Smith" + evidence about Harvard/NY does not
     strong-trigger.

  3. Confidence class is deterministic (weak | medium | strong), not
     boolean. Strong requires quoted_string near exact subject mention
     + multiple type-matched candidates. Phase 3 demote will gate on
     confidence_class.

  4. Cap output at 10 candidates (the per_chunk-quote-inflation
     lesson). Prevents the guard becoming another claim amplifier.

  5. Offsets are offset_start + offset_end + offset_basis=
     "evidence_object_text", never an ambiguous single offset.

  6. Cache-hit path returns answerability: None. Cached records do not
     carry the evidence_map, only the rendered sources summary, so the
     sidecar cannot recompute candidate spans without re-running
     retrieval. Operators wanting fresh diagnostics use --burn.

  7. Phase 1 stays out of verifier_policy_hash. The
     answerability_sidecar_enabled / answerability_threshold /
     denial_patterns_version / extraction_cues_version fields fold
     into governance_policy_hash only. Phase 3 demote flag
     (answerability_demote_enabled, default False) will move the
     verifier hash WHEN ON because it changes the rendered audit_mode
     (EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL).

Sidecar discipline (matches arborist.qa.inspect.diagnose_* sister
functions deflection / coherence / title_relevance):

  - no model calls (no LLM-as-judge, no NLI, no translation)
  - no audit_events writes
  - no providence_cache writes
  - no answer text mutation
  - no claim promotion -- the trigger conjunction makes promotion
    structurally impossible (only fires on denial answers)
  - byte-deterministic: same (question, answer, evidence, policy) ->
    same output every time

Result-dict integration: result["answerability"] is None when the
guard did not fire, or a structured diagnostic dict when it did
(diagnostic_version, confidence_class, triggered_clauses,
denial_pattern_matched, extraction_cue_matched, extraction_shape,
answer_type, subject_tokens, candidate_count, threshold,
missed_answer_candidate_spans). Read by bench_qa (Phase 2 will add
warning-count aggregate to bench rows) and CLI render.

Three return points carry the key:
  - miss-path (full retrieval + verify): computed from evidence_map
  - cache-hit: None (Dav1d cache-hit recompute discipline -- evidence
    not stored, recompute requires re-retrieval)
  - reject-broad early-return: None (no evidence examined)

Tests: 36 new pinning the three-clause logic, positive (Ballestrini)
regression, negative control (John Smith + Harvard/NY), each-clause-
alone silence, schema integrity, byte-determinism, sidecar-disabled
short-circuit, dict-shaped evidence support. All pass; existing
inspect tests (60) all pass.

End-to-end verified live via the CLI on the real corpus (2010 ~/.arborist
/shards): 3 fresh Hermes-3-8B runs on "songs by veronica ballestrini",
run 1 hit the failure (sidecar fired with confidence: strong, 351
candidates, denial: "not mentioned"), runs 2-3 model extracted
correctly and sidecar correctly silent.

Phase 2 (bench + threshold tuning) and Phase 3 (opt-in demote flag)
are open as follow-ups. Per Dav1d: NO-GO on default demote-on until
benchmark + human spot-check confirms low false-positive rate.

Full spec in docs/tickets/ticket-000068-verifier-blind-missed-answer-
guard.md (post-review hardenings section at top names the seven
load-bearing changes from the Dav1d 2026-05-27 review).

tests/test_missed_answer_guard.py

@@ -0,0 +1,500 @@
+"""Tests for the Ticket #000068 Phase 1 missed-answer falsification guard.
+
+The guard is a deterministic read-only sidecar that fires on the three-clause
+conjunction: denial pattern + extraction-shaped question + candidate spans
+near cleaned subject tokens. These tests pin each clause in isolation, the
+positive Ballestrini regression case, the negative John-Smith control, and
+the discipline invariants (no model call, no audit write, no claim promotion).
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import pytest
+
+from arborist.qa.inspect import (
+    _DENIAL_PATTERNS_V1,
+    _extract_subject_tokens,
+    _match_denial_pattern,
+    _classify_extraction_shape,
+    _extract_candidate_spans,
+    _score_answerability_candidates,
+    diagnose_missed_answer,
+    ANSWERABILITY_DIAGNOSTIC_VERSION,
+    DENIAL_PATTERNS_VERSION,
+)
+
+
+# ---------------------------------------------------------------------------
+# Fixtures
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class FakeEv:
+    pointer_id: str
+    evidence_id: str
+    title: str
+    document_uri: str
+    span: str
+
+
+def _ballestrini_evidence() -> list[FakeEv]:
+    return [
+        FakeEv(
+            pointer_id="E1",
+            evidence_id="E652b125b",
+            title="Veronica Ballestrini",
+            document_uri="https://en.wikipedia.org/wiki/Veronica_Ballestrini",
+            span=(
+                "Veronica Jean Ballestrini (born October 29, 1991) is an "
+                "Italian-American country music singer and songwriter from "
+                "Waterford, Connecticut."
+            ),
+        ),
+        FakeEv(
+            pointer_id="E2",
+            evidence_id="E19912984",
+            title="Veronica Ballestrini",
+            document_uri="https://en.wikipedia.org/wiki/Veronica_Ballestrini",
+            span=(
+                "Ballestrini went on to record her debut album "
+                "\"What I'm All About\" which was released August 21, 2009. "
+                "You can buy her CD and hear amazing 11 songs and 6 were "
+                "written by her. Her first single \"Amazing\" charted on "
+                "the Music Row Country chart and the music video debuted at "
+                "#3 on CMT Pure. In January 2010 Timbob records partnered "
+                "with Lofton Creek Records president Mike Borchetta for the "
+                "promotions of Veronica's single \"Out There Somewhere\"."
+            ),
+        ),
+    ]
+
+
+# ---------------------------------------------------------------------------
+# Clause A — denial pattern
+# ---------------------------------------------------------------------------
+
+
+def test_denial_pattern_positive():
+    assert _match_denial_pattern("the specific songs are not mentioned in evidence") == "not mentioned"
+
+
+def test_denial_pattern_negative():
+    assert _match_denial_pattern("Veronica has released several songs including Amazing.") is None
+
+
+def test_denial_pattern_casefold():
+    assert _match_denial_pattern("THIS INFORMATION IS NOT PROVIDED IN THE TEXT") == "not provided"
+
+
+def test_denial_pattern_whitespace_normalized():
+    assert _match_denial_pattern("specific  songs  are\n not  mentioned\nhere") == "not mentioned"
+
+
+def test_denial_pattern_sealed_list_intact():
+    """Adding a new phrase requires bumping DENIAL_PATTERNS_VERSION.
+    This test pins the v1 set so silent additions can't slip through."""
+    expected = {
+        "not mentioned",
+        "not provided",
+        "the evidence does not say",
+        "does not mention",
+        "no specific",
+        "no evidence",
+        "cannot determine from the provided evidence",
+        "is not stated",
+        "is not specified",
+    }
+    assert set(_DENIAL_PATTERNS_V1) == expected
+
+
+# ---------------------------------------------------------------------------
+# Clause B — extraction shape
+# ---------------------------------------------------------------------------
+
+
+def test_extraction_cue_songs_by():
+    info = _classify_extraction_shape("songs by veronica ballestrini", None)
+    assert info is not None
+    assert info["cue"] == "songs by"
+    assert info["answer_type"] == "title_like"
+    assert info["shape"] == "list"
+
+
+def test_extraction_cue_who_wrote():
+    info = _classify_extraction_shape("who wrote ulysses?", None)
+    assert info is not None
+    assert info["cue"] == "who wrote"
+    assert info["answer_type"] == "person"
+
+
+def test_extraction_cue_what_year():
+    info = _classify_extraction_shape("what year did the war end?", None)
+    assert info is not None
+    assert info["cue"] == "what year"
+    assert info["answer_type"] == "date"
+
+
+def test_extraction_shape_quantifier_fallback():
+    """No surface cue but broad quantifier intensity → still extraction shape."""
+    info = _classify_extraction_shape("tell me about beatles members", "OPEN_REQUEST")
+    assert info is not None
+    assert info["cue"].startswith("quantifier:")
+    assert info["answer_type"] == "title_like"
+
+
+def test_extraction_shape_narrow_question():
+    """Narrow factoid: no cue, no broad quantifier → not extraction-shaped."""
+    info = _classify_extraction_shape("who is veronica ballestrini?", "SINGULAR")
+    assert info is None
+
+
+# ---------------------------------------------------------------------------
+# Subject-token extraction (Dav1d §7 — non-negotiable)
+# ---------------------------------------------------------------------------
+
+
+def test_subject_tokens_strip_cue_words():
+    """`songs by veronica ballestrini` → just the proper-noun run.
+    Without this hardening the guard false-triggers on generic spans."""
+    tokens = _extract_subject_tokens("songs by veronica ballestrini")
+    low = [t.lower() for t in tokens]
+    assert "songs" not in low
+    assert "by" not in low
+    assert "veronica" in low
+    assert "ballestrini" in low
+
+
+def test_subject_tokens_strip_who_wrote():
+    tokens = _extract_subject_tokens("who wrote ulysses?")
+    low = [t.lower() for t in tokens]
+    assert "who" not in low
+    assert "wrote" not in low
+    assert "ulysses" in low
+
+
+def test_subject_tokens_preserve_proper_noun_runs():
+    """Multi-word proper nouns survive as separate tokens (matched by either)."""
+    tokens = _extract_subject_tokens("books by f scott fitzgerald")
+    low = [t.lower() for t in tokens]
+    assert "books" not in low
+    assert "by" not in low
+    assert "fitzgerald" in low
+    # f and scott are short — `f` drops (single letter), `scott` survives
+    assert "scott" in low
+
+
+def test_subject_tokens_preserve_hyphenated():
+    tokens = _extract_subject_tokens("songs by jean-luc picard")
+    low = [t.lower() for t in tokens]
+    assert "jean-luc" in low or "jean" in low  # implementation may split or preserve
+    assert "picard" in low
+
+
+def test_subject_tokens_dedupe():
+    tokens = _extract_subject_tokens("ballestrini ballestrini songs")
+    low = [t.lower() for t in tokens]
+    assert low.count("ballestrini") == 1
+
+
+# ---------------------------------------------------------------------------
+# Candidate span extraction
+# ---------------------------------------------------------------------------
+
+
+def test_candidate_quoted_string():
+    text = "She wrote her debut single \"Amazing\" in 2009."
+    spans = _extract_candidate_spans(text, "title_like")
+    quoted = [s for s in spans if s["kind"] == "quoted_string"]
+    assert any(s["text"] == "Amazing" for s in quoted)
+
+
+def test_candidate_title_case_span():
+    text = "Veronica Jean Ballestrini was born in Connecticut."
+    spans = _extract_candidate_spans(text, "title_like")
+    title_case = [s for s in spans if s["kind"] == "title_case_span"]
+    assert any("Veronica Jean Ballestrini" in s["text"] for s in title_case)
+
+
+def test_candidate_year_for_date_query():
+    text = "She was born in 1991 in Connecticut."
+    spans = _extract_candidate_spans(text, "date")
+    years = [s for s in spans if s["kind"] == "year"]
+    assert any(s["text"] == "1991" for s in years)
+
+
+def test_candidate_no_year_for_title_query():
+    """Year extraction only happens for date queries."""
+    text = "She was born in 1991."
+    spans = _extract_candidate_spans(text, "title_like")
+    assert not any(s["kind"] == "year" for s in spans)
+
+
+# ---------------------------------------------------------------------------
+# Full guard — positive case
+# ---------------------------------------------------------------------------
+
+
+def test_ballestrini_tail_failure_triggers():
+    """The motivating regression: Hermes-3-8B under tail layout said
+    'specific songs are not mentioned' when E2 contained 'Amazing',
+    'Out There Somewhere', etc. Guard must catch this."""
+    result = diagnose_missed_answer(
+        question="songs by veronica ballestrini",
+        answer=(
+            "Veronica Ballestrini is a country music singer and songwriter "
+            "who has released several songs. However, the specific songs by "
+            "her are not mentioned in the provided evidence blocks."
+        ),
+        evidence=_ballestrini_evidence(),
+    )
+    assert result is not None
+    assert result["answerability_warning"] is True
+    assert result["confidence_class"] in ("medium", "strong")
+    assert result["denial_pattern_matched"] == "not mentioned"
+    assert result["extraction_cue_matched"] == "songs by"
+    assert "ballestrini" in [t.lower() for t in result["subject_tokens"]]
+    assert "songs" not in [t.lower() for t in result["subject_tokens"]]
+    # At least one quoted-string candidate (Amazing / What I'm All About / Out There Somewhere)
+    kinds = {c["candidate_kind"] for c in result["missed_answer_candidate_spans"]}
+    assert "quoted_string" in kinds
+
+
+def test_ballestrini_diagnostic_version_pinned():
+    result = diagnose_missed_answer(
+        question="songs by veronica ballestrini",
+        answer="The evidence does not say what songs she released.",
+        evidence=_ballestrini_evidence(),
+    )
+    assert result["diagnostic_version"] == ANSWERABILITY_DIAGNOSTIC_VERSION
+    assert result["denial_patterns_version"] == DENIAL_PATTERNS_VERSION
+
+
+def test_offsets_are_start_end_basis():
+    """Dav1d §10: offsets must be precise — start + end + basis, not ambiguous."""
+    result = diagnose_missed_answer(
+        question="songs by veronica ballestrini",
+        answer="Songs are not mentioned in the evidence.",
+        evidence=_ballestrini_evidence(),
+    )
+    for c in result["missed_answer_candidate_spans"]:
+        assert "offset_start" in c
+        assert "offset_end" in c
+        assert c["offset_basis"] == "evidence_object_text"
+        assert c["offset_end"] > c["offset_start"]
+
+
+def test_candidate_cap_at_10():
+    """Phase 1 must cap output at 10 candidates (the per_chunk lesson)."""
+    long_evidence = FakeEv(
+        pointer_id="E1",
+        evidence_id="EBIG",
+        title="Veronica Ballestrini",
+        document_uri="https://x",
+        # 12 quoted titles
+        span=" ".join(f'Her single "Song{i}" charted.' for i in range(12))
+              + " Ballestrini wrote them all.",
+    )
+    result = diagnose_missed_answer(
+        question="songs by veronica ballestrini",
+        answer="The evidence does not mention specific songs.",
+        evidence=[long_evidence],
+    )
+    assert result is not None
+    assert len(result["missed_answer_candidate_spans"]) <= 10
+
+
+# ---------------------------------------------------------------------------
+# Negative controls (Dav1d §14)
+# ---------------------------------------------------------------------------
+
+
+def test_genuine_absence_no_strong_trigger():
+    """Dav1d §14 negative control: 'songs by John Smith' + evidence
+    about Harvard/NY → must NOT strong-trigger."""
+    ev = FakeEv(
+        pointer_id="E1",
+        evidence_id="EJS",
+        title="John Smith",
+        document_uri="https://x",
+        span="John Smith studied at Harvard University and lived in New York.",
+    )
+    result = diagnose_missed_answer(
+        question="songs by john smith",
+        answer="The evidence does not mention songs by John Smith.",
+        evidence=[ev],
+    )
+    # Per Dav1d: 'no strong warning; ideally no warning; if warning
+    # exists, confidence_class = weak.' We accept weak or None.
+    if result is not None:
+        assert result["confidence_class"] != "strong", (
+            "false positive at strong confidence — Harvard/NY are not song titles"
+        )
+
+
+def test_no_denial_does_not_trigger():
+    """Clause A failure: extraction question + candidates, but no denial."""
+    result = diagnose_missed_answer(
+        question="songs by veronica ballestrini",
+        answer="Veronica released Amazing and Out There Somewhere.",
+        evidence=_ballestrini_evidence(),
+    )
+    assert result is None
+
+
+def test_not_extraction_shape_does_not_trigger():
+    """Clause B failure: SINGULAR question + denial + candidates."""
+    result = diagnose_missed_answer(
+        question="who is veronica ballestrini?",
+        answer="The evidence does not say who she is.",
+        evidence=_ballestrini_evidence(),
+    )
+    assert result is None
+
+
+def test_no_evidence_does_not_trigger():
+    """Clause C failure: empty evidence list."""
+    result = diagnose_missed_answer(
+        question="songs by veronica ballestrini",
+        answer="Not mentioned.",
+        evidence=[],
+    )
+    assert result is None
+
+
+def test_no_candidate_spans_does_not_trigger():
+    """Clause C failure: evidence exists but contains no candidate spans
+    matching the answer_type (plain prose with no titles)."""
+    ev = FakeEv(
+        pointer_id="E1",
+        evidence_id="X",
+        title="V B",
+        document_uri="https://x",
+        span="she is a singer-songwriter from connecticut.",  # lowercase — no title-case
+    )
+    result = diagnose_missed_answer(
+        question="songs by veronica ballestrini",
+        answer="Not mentioned.",
+        evidence=[ev],
+    )
+    assert result is None
+
+
+# ---------------------------------------------------------------------------
+# Discipline invariants — no model, no audit, no promotion
+# ---------------------------------------------------------------------------
+
+
+def test_sidecar_disabled_returns_none():
+    """answerability_sidecar_enabled=False short-circuits to None."""
+    result = diagnose_missed_answer(
+        question="songs by veronica ballestrini",
+        answer="not mentioned",
+        evidence=_ballestrini_evidence(),
+        policy={"answerability_sidecar_enabled": False},
+    )
+    assert result is None
+
+
+def test_evidence_dict_form_supported():
+    """Phase 1 accepts both EvidenceObject and dict-shaped evidence."""
+    dict_ev = [
+        {
+            "pointer_id": "E2",
+            "evidence_id": "EX",
+            "title": "Veronica Ballestrini",
+            "document_uri": "https://x",
+            "span": 'Her first single "Amazing" by Veronica Ballestrini.',
+        },
+    ]
+    result = diagnose_missed_answer(
+        question="songs by veronica ballestrini",
+        answer="The evidence does not mention specific songs.",
+        evidence=dict_ev,
+    )
+    assert result is not None
+    assert result["answerability_warning"] is True
+
+
+def test_deterministic_byte_for_byte():
+    """Same inputs → same output. Byte-deterministic per Dav1d §13."""
+    inputs = dict(
+        question="songs by veronica ballestrini",
+        answer="not mentioned",
+        evidence=_ballestrini_evidence(),
+    )
+    r1 = diagnose_missed_answer(**inputs)
+    r2 = diagnose_missed_answer(**inputs)
+    assert r1 == r2
+
+
+def test_empty_question_returns_none():
+    result = diagnose_missed_answer(
+        question="",
+        answer="not mentioned",
+        evidence=_ballestrini_evidence(),
+    )
+    assert result is None
+
+
+def test_empty_answer_returns_none():
+    result = diagnose_missed_answer(
+        question="songs by veronica ballestrini",
+        answer="",
+        evidence=_ballestrini_evidence(),
+    )
+    assert result is None
+
+
+# ---------------------------------------------------------------------------
+# Output schema integrity
+# ---------------------------------------------------------------------------
+
+
+def test_output_schema_keys():
+    result = diagnose_missed_answer(
+        question="songs by veronica ballestrini",
+        answer="The specific songs are not mentioned in evidence.",
+        evidence=_ballestrini_evidence(),
+    )
+    required_keys = {
+        "diagnostic_version",
+        "denial_patterns_version",
+        "extraction_cues_version",
+        "answerability_warning",
+        "confidence_class",
+        "triggered_clauses",
+        "denial_pattern_matched",
+        "extraction_cue_matched",
+        "extraction_shape",
+        "answer_type",
+        "subject_tokens",
+        "candidate_count",
+        "threshold",
+        "threshold_report",
+        "missed_answer_candidate_spans",
+    }
+    assert required_keys.issubset(result.keys()), (
+        f"missing keys: {required_keys - set(result.keys())}"
+    )
+
+
+def test_triggered_clauses_all_true_when_firing():
+    result = diagnose_missed_answer(
+        question="songs by veronica ballestrini",
+        answer="not mentioned",
+        evidence=_ballestrini_evidence(),
+    )
+    assert result["triggered_clauses"]["denial"] is True
+    assert result["triggered_clauses"]["extraction_shape"] is True
+    assert result["triggered_clauses"]["candidate_proximity"] is True
+
+
+def test_confidence_class_in_valid_set():
+    result = diagnose_missed_answer(
+        question="songs by veronica ballestrini",
+        answer="not mentioned",
+        evidence=_ballestrini_evidence(),
+    )
+    assert result["confidence_class"] in ("weak", "medium", "strong")