# Ticket #000068 — Verifier-blind missed-answer falsification guard

**Status:** in progress · Phase 1 + Phase 2 + Phase 3 landed 2026-05-27; Phase 4 (default demote-on) NO-GO per Dav1d, waiting on human spot-check across larger sample
**Opened:** 2026-05-27
**Scope:** Deterministic read-only sidecar that detects evidence-neglect / false-negative answers — runs where the LLM said "not mentioned" but the evidence contains candidate answer spans near the question subject. Emits an `answerability_warning` + candidate-span list; optional audit-label demote `EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL` behind a policy flag. No LLM-as-judge. No verifier-policy change.

## Phase 1+2+3 landing notes (2026-05-27)

**Phase 1 (sidecar)** — `arborist/qa/inspect.py:diagnose_missed_answer` + helpers + sealed constants + result-dict wiring. 36 tests. End-to-end verified live on the Ballestrini case (sidecar fires on the failure, silent on success).

**Phase 2 (bench + threshold tuning)** — `bench/qa_results/phase2-sidecar-on/2026-05-27T14-16-22Z.{md,jsonl}`. 76 questions × n=3 × claim_lattice × Hermes-3-8B × tail layout = 228 runs. Headline:

| metric | value |
|---|---|
| sidecar fires | 2/228 (0.88%) |
| confidence distribution | 2 strong / 0 medium / 0 weak |
| precision | 100% (2/2 fires were the Ballestrini regression fixture) |
| recall on Ballestrini (n=3) | 2/3 (third run model extracted correctly — silent is correct) |
| false positives | 0/226 non-Ballestrini runs |
| verifier verdict on the failures | both STRICT (the verifier-blind class, exactly as predicted) |

The detection rule's three-clause conjunction (denial pattern + extraction shape + candidate proximity) is operating at the precision floor: when it fires, it fires correctly. Recall on the Ballestrini class is 100% of the runs where Hermes actually produced the failure (the third run had no denial pattern, so silence is the right behavior).

**Phase 3 (opt-in demote flag)** — `answerability_demote_enabled` added to `_VERIFIER_POLICY_FIELDS` (flipping the flag invalidates prior cache via verifier_policy_hash, the correct discipline since rendered audit_mode changes). `_render_audit_label` extended to demote `EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL` on lattice modes (strong/medium confidence only — weak doesn't demote per Phase 2 data showing zero weak fires on real failures). Lower rungs (POINTER-LINKED, ANCHOR-WARRANTED) get a `· missed-answer` tail tag instead of a rung transition (rung already signals degradation). Non-lattice modes (quote/span/entity/paraphrase) get the same tail-tag treatment on their existing audit_mode token. CLI flag `--demote-on-missed-answer` on both `query` and `ask` subcommands (default OFF). 11 Phase 3 tests + end-to-end verified live (4/4 Hermes runs on Ballestrini with `--demote-on-missed-answer` rendered `EVIDENCE-MISSED-PARTIAL`).

**Phase 4 (default demote-on)** — NO-GO. Per Dav1d 2026-05-27 §3.4: "a false sidecar warning is tolerable; a false audit-label demotion can damage trust in correct abstentions." Phase 2 precision is 100% but n=2 fires is too few samples to claim precision floor empirically; default flip blocks on a wider bench (more diverse failure shapes) + human spot-check of the warnings. Operators wanting the demote behavior set `--demote-on-missed-answer` per-call.

## Post-review hardenings (Dav1d 2026-05-27)

The 2026-05-27 de-novo review (`~/Downloads/RESPONSE_final_ticket-000068-verifier-blind-missed-answer-guard.txt`) was a **GO for Phase 1** with seven specific hardenings folded into the spec below. Summary so a re-read can hit the load-bearing changes at a glance:

1. **Subject-token extraction must strip cue/relation/stopwords.** `"songs by veronica ballestrini"` → `["veronica", "ballestrini"]`, NOT all four tokens. Cue/relation words to strip: `songs, works, books, papers, list, name, who, what, which, by, wrote, composed, directed, acted, examples`. Preserve: quoted phrases, hyphenated terms, proper-noun runs, exact title-like fragments. Without this, generic title-case spans like "Harvard University" trigger on "songs by John Smith". Non-negotiable.
2. **Answer-type alignment.** Candidate span kind must match query type:
   - `songs / works / books / papers` → quoted strings, title-like spans, comma-list items
   - `who wrote / composed / directed / acted in` → person-like title-case spans
   - `when / what year / date` → year/date spans
   - `what are / list / name all` → quoted strings, title-like spans, comma-list items
3. **Confidence class is deterministic, not boolean.** `weak | medium | strong`. Trigger fires only when denial + extraction-shape match AND one of: ≥1 quoted candidate near exact subject, ≥2 comma/list candidates near subject, ≥1 query-type-matched candidate near exact subject mention.
4. **Cap output**: `max_candidates_returned = 10`. Prevents the guard from becoming a quote-amplifier (the per_chunk lesson).
5. **Precise offsets**: `offset_start` + `offset_end` + `offset_basis="evidence_object_text"`. Never an ambiguous single `offset`.
6. **Cache-hit recompute-on-read.** The sidecar output is NOT stored as a proof object. It is recomputable from `(question, answer_text, evidence_objects, answerability_policy)`. If a cached record's evidence is recoverable, recompute on the next lookup so cached false negatives don't stay invisible.
7. **Phase 1 stays out of `verifier_policy_hash`.** Only the demote flag (Phase 3) moves the verifier hash — and only when ON. Phase 1's `answerability_*` policy fields fold into `governance_policy_hash` only.

The body of the ticket below absorbs these into the design. Phase boundaries unchanged: Phase 1 (sidecar read-only, this work) → Phase 2 (bench + threshold tuning) → Phase 3 (opt-in demote) → Phase 4 (default decision, bench-gated). NO-GO on default demote-on until Phase 2 + human spot-check evidence is in.
**Audience:** fox + anyone maintaining `arborist/qa/` + reviewers reading the next Dav1d review pass.
**Hard constraint:** Sidecar discipline. Never writes `providence_cache` / `audit_events`. Never folds into `verifier_policy_hash`. Never promotes claims — can only flag possible missed answerability. Follows the same pattern as `arborist.qa.inspect.diagnose_*` (deflection, coherence, title-relevance).

---

## 1. The failure class — verifier-blind false negatives

Surfaced 2026-05-26 by the Ballestrini case (`docs/user-payload-layout.md`).
Same query, same retrieval, same evidence, Hermes-3-8B under
`user_payload_layout=tail`:

```
Q: songs by veronica ballestrini

A: Veronica Ballestrini is a country music singer and songwriter who
   has released several songs. However, the specific songs by her
   are not mentioned in the provided evidence blocks.
```

Evidence block `E2` literally contained "Amazing", "Out There
Somewhere", "Fascinated", "What's Up With That", "Don't Say".
Verifier verdict: `EVIDENCE-WARRANTED` 2/2 — clean. From the
verifier's view nothing was wrong:

```
Evidence contains the answer.
Model says the evidence does not contain the answer.
Verifier sees no unsupported positive claim → marks run clean.
User receives a false negative under EVIDENCE-WARRANTED.
```

The existing layered verifier (quote / span / entity / paraphrase
+ Rule 8 title-relevance + Rule 9 subject-tokens-absent + claim-
count ceiling) guards against **unsupported positive claims** — it
has no hook for **unsupported absences**. The 2026-05-27 layout
bench (n=3 × 76q on Hermes-3-8B) confirmed that prompt-layout
fixes (`bookend`, `per_chunk`) address the specific Ballestrini
attention-placement instance but cannot close the class — a
sufficiently large prompt or an adversarial phrasing can resurface
the failure under any layout.

This is exactly the class of failure that selects for **adding a
new falsifier into the substrate** rather than for tuning model
inputs. Layout is a model-input policy knob (`#user_payload_layout`,
folds into `governance_policy_hash`); this is a verifier-side
diagnostic adjunct.

Dav1d 2026-05-27 review framing:

```
#0000XY-A  (layout, shipped 2026-05-26)  — user_payload_layout
#0000XY-B  (this ticket, design open)     — missed-answer guard
```

The split is per the Dav1d-audience rule (`feedback_ticket_proliferation`):
the missed-answer guard touches a different layer (verifier /
falsification adjunct) than the layout knob (model-input policy),
and a Dav1d review needs to read each independently. Same
substrate, two tickets.

## 2. Detection rule — deterministic, no LLM-as-judge

The guard fires when **all three** of the following hold:

```
(A) Denial pattern in the answer.
    Regex / token-class match against a closed list of phrases:
      "not mentioned"
      "not provided"
      "the evidence does not say"
      "does not mention"
      "no specific"
      "no evidence"
      "cannot determine from the provided evidence"
      "is not stated"
      "is not specified"
    Sealed list, versioned via a denial_patterns_version policy field
    so additions cache-partition cleanly.

(B) Question is an extraction/list shape.
    EITHER the existing arborist.qa.quantifier classifier returns
    intensity ∈ {ALL, COMPREHENSIVE, OPEN_REQUEST, MANY, PLURAL},
    OR the question matches one of the surface cues:
      "songs by"     "works by"     "books by"     "papers by"
      "who wrote"    "who composed" "who acted in" "who directed"
      "what are"     "which"        "list"
      "name all"     "name the"
    The classifier already exists and folds into
    governance_policy_hash; reuse it.

(C) Evidence contains candidate answer spans near subject tokens.
    Reuse arborist.qa.verify's entity-proximity primitive
    (entity_proximity_n / entity_proximity_window). For each
    question subject token (content tokens, stemmed), find
    candidate spans within `window` chars:
      - quoted strings  (".+?" pattern)
      - title-case spans (Run of 2+ Capitalized Words)
      - comma-separated title lists ('X, Y, and Z')
    A "candidate span" is a span where the entity-proximity score
    against the subject tokens passes a configurable threshold
    (default: at least N candidate spans within W chars of any
    subject token).
```

All three must fire. Each in isolation is too noisy:

- **(A) alone**: model legitimately says "not mentioned" when
  evidence really doesn't have the answer — that's correct
  behavior, not a failure.
- **(B) alone**: list shape doesn't imply evidence has the answer.
- **(C) alone**: candidate spans near subject tokens are common in
  Wikipedia prose without being the answer to *this* question.

The three-clause conjunction is the fingerprint of the
Ballestrini-class failure: question asks for a list, evidence has
list-like spans clustered near the subject, model still denies.

## 3. Output

Sidecar emits a structured warning:

```python
{
    "answerability_warning": bool,
    "missed_answer_candidate_spans": [
        {
            "evidence_id": "E2",        # pointer-mode or content-addressed
            "offset": 142,              # char offset within the span
            "text": "Amazing",          # the candidate token/phrase
            "subject_proximity_chars": 67,
        },
        ...
    ],
    "denial_pattern_matched": "not mentioned in the provided evidence",
    "extraction_cue_matched": "songs by",
    "quantifier_intensity": "ALL",
}
```

Optional audit-label demote (gated by `answerability_demote_enabled`,
default False to preserve cache):

```
EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL
```

The display layer (`_render_audit_label`) gets one new four-rung
suffix when demote fires. Programmatic callers see the
underlying `audit_mode` unchanged unless the demote flag is on —
same projection discipline as the existing four-rung ladder.

## 4. Hash discipline (the load-bearing constraint)

This is where the Dav1d review is most precise:

```
user_payload_layout      → governance_policy_hash   (model input)
denial_patterns_version  → governance_policy_hash   (sidecar trigger)
extraction_cues_version  → governance_policy_hash   (sidecar trigger)
answerability_threshold  → governance_policy_hash   (sidecar trigger)
answerability_demote_enabled → governance_policy_hash + verifier_policy_hash IF on
                         (it changes the rendered audit_mode, which
                         IS a verifier output, so the verifier hash
                         must move when demote is enabled. Hence
                         default-off — opt-in flips the verifier
                         hash deliberately, sidecar-only stays out)
```

When the demote flag is off (default), the guard is a pure read-
only diagnostic — same status as `diagnose_deflection`,
`diagnose_coherence`, `diagnose_title_relevance`. When on, it
becomes a verifier-policy change and partitions the cache
accordingly. The default is off so existing operators don't see
audit_mode mutate without an explicit opt-in.

Alternative design considered: a separate
`answerability_policy_hash` (a 10th cache_key dimension). Rejected
— it adds a hash to scan with no correctness coverage beyond what
`governance_policy_hash` already gives (the trigger fields are a
subset of the policy dict; flipping them already bumps
`governance_policy_hash`). The "audit legibility" argument made
for the optional 9th-dim `verifier_policy_hash` (#000058) doesn't
recur here — sidecar warnings are read off the result dict, not
the cache_key.

## 5. Where it lives

```
arborist/qa/inspect.py
    + def diagnose_missed_answer(
        question: str,
        answer: str,
        evidence: list[EvidenceObject],
        *,
        policy: dict,
      ) -> dict | None:
        ...

arborist/qa/query.py + runner.py
    after verify_*() returns, before persist, call the sidecar:
        warning = diagnose_missed_answer(
            question, answer_text, evidence, policy=policy
        )
        result["answerability"] = warning  # may be None
        if policy.get("answerability_demote_enabled") and warning:
            if audit_mode == "EVIDENCE-WARRANTED":
                audit_mode = "EVIDENCE-MISSED-PARTIAL"
                # bump verifier_policy_hash partition (handled by
                # the policy field already being in the verifier set)
```

Sidecar code follows the existing inspect.py pattern verbatim — no
new module, no new audit-event type, no schema change.

## 6. Acceptance criteria

```
1. Ballestrini-class fixture (added 2026-05-27 to bench/qa_questions.txt
   as "songs by veronica ballestrini") produces a warning when the
   model under tail layout returns the negation. Sidecar test pins this.

2. Genuine-absence cases do not false-trigger. Negative-control
   fixture: question asks for a fact the corpus does not contain,
   evidence genuinely lacks the answer, model correctly says
   "not mentioned" → warning MUST NOT fire (the (C) clause must
   be the filter that keeps these out — candidate spans near
   subject must be absent).

3. No model call. Pure-Python, deterministic. Same input → same
   output, byte-for-byte (canonical-projection discipline).

4. Candidate spans are cited by evidence_id and char offset within
   the span. Forensically traceable to the source bytes.

5. Sidecar output is read off `result["answerability"]` — never
   written to `providence_cache` (it's policy-versioned, not
   audit-chained).

6. Guard never promotes unsupported claims. The trigger conjunction
   makes promotion structurally impossible — it only fires on
   denial answers.

7. Demote-mode flag (`answerability_demote_enabled`) is default-False.
   When True, folds into both governance_policy_hash AND
   verifier_policy_hash (since it changes rendered audit_mode).
   When False, governance_policy_hash only (sidecar diagnostic).

8. Tests:
   - denial pattern match (positive)
   - extraction cue match (positive)
   - candidate proximity match (positive)
   - all-three conjunction (positive)
   - each clause in isolation (must NOT trigger alone)
   - genuine absence (negative control)
   - hash partitioning: flipping demote flag changes verifier hash
   - hash partitioning: flipping denial_patterns_version changes
     governance hash, not verifier hash
```

## 7. Bench

After landing, run the same n=3 × 76q claim_lattice sweep against
Hermes-3-8B that the user_payload_layout work used (the curated
set already has the Ballestrini regression fixture). Compare:

- baseline (no guard): use the 2026-05-27 layout-tail results
  on disk at `bench/qa_results/layout-tail/2026-05-27T10-07-29Z.{md,jsonl}`
- guard-on (sidecar only, demote off): expect 0 STRICT-rate
  change (sidecar is read-only); expect warning to fire on the
  Ballestrini fixture
- guard-on (demote enabled): expect STRICT count to drop by the
  number of fixtures where the model produced a denial — that
  drop is the visible measurement of the failure class

Promotion rule (when/if making the demote default-on later):
honest report on what fraction of the curated set produced
denials, how many of those were true vs false negatives by
human review of a sample.

## 8. Out of scope (explicitly)

- LLM-as-judge of any kind. The cost / reproducibility / chain-of-
  custody constraints in `arborist/qa/verify.py` apply — no model
  in the proof path (per the #000049 cage), and no model in the
  sidecar either by choice.
- Active retrieval re-runs ("the answer wasn't in the top-k, let
  me re-fetch"). That's a different ticket. This guard only looks
  at evidence already in the prompt.
- Fixing the model's behavior under tail layout. That's what
  `user_payload_layout` does (and the 2026-05-27 bench confirms
  it doesn't generalize as a default). The guard is the
  *substrate-side* recovery — catch the failure even when the
  model and the layout both let it through.

## 9. Relationship to other tickets / docs

- `docs/user-payload-layout.md` — sibling work. Layout is the
  model-input lever; this is the verifier-side adjunct. Read
  together for the full picture of the Ballestrini failure.
- `docs/seven-point-program.md` — when this lands it extends **D2
  (pointer-grounding)** by covering *unsupported absence* in
  addition to *unsupported presence*. Same falsifier discipline,
  new failure shape.
- `arborist/qa/inspect.py` — existing sidecar pattern. Follow it
  verbatim.
- `arborist/qa/quantifier.py` — reuse the broad-quantifier
  classifier for clause (B). Already folds into
  `governance_policy_hash`.
- `arborist/qa/verify.py` — reuse `entity_proximity_n` /
  `entity_proximity_window` primitives for clause (C). No new
  proximity code.
- 5F-Falsification framework (Dav1dPrometheus, see
  `~/.claude/projects/-home-fox-git-arborist/memory/dav1dprometheus_framework.md`)
  — the Ballestrini case is exactly the kind of fixture that
  feeds the falsification loop: controller proposes (layout fix),
  battery exposes the residual class (verifier-blind absence),
  fixture lands here.

## 10. Open questions

- **Threshold tuning for clause (C).** Default proximity window?
  Default minimum candidate-span count? Should be benched on the
  curated set + a few synthetic negative controls before locking
  defaults.
- **Phrase patterns vs token patterns** in clause (A). The denial
  list above is phrase-based ("not mentioned"). Tokenized
  matching ("absent", "lack", "no information") might over-catch.
  Conservative phrase-based start; widen only on measured FN.
- **Interaction with the existing `DEFLECTION_DETECTED` signal.**
  Deflection detector (`diagnose_deflection`) already flags
  topic-shift via subject-anchor heuristic. There's overlap: an
  answer that deflects to "I can't help" also denies. Sidecar
  outputs should be independent — both can fire on the same
  row — but cross-checking on the curated set is worth doing to
  make sure neither is a wholly-contained subset of the other.

## 11. Implementation order (proposed)

```
Phase 1 — sidecar (read-only, no demote)
  arborist/qa/inspect.py:diagnose_missed_answer
  policy fields: denial_patterns_version, extraction_cues_version,
                 answerability_threshold (all in governance hash)
  unit tests (acceptance criteria 1, 2, 3, 4, 5, 6, 8.a-f)
  wire into query.py + runner.py result dict

Phase 2 — bench + threshold tuning
  Re-run the layout-tail bench with the sidecar enabled, read-only;
  count fires on the curated set; spot-check the warnings.

Phase 3 — demote flag (opt-in)
  Add answerability_demote_enabled (governance + verifier hashes)
  Wire EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL in the
  _render_audit_label projection.
  Tests for hash partitioning (acceptance criteria 7, 8.g-h).

Phase 4 — default decision
  After Phase 2 evidence in hand: keep demote default-off (read-
  only diagnostic) or flip to on. Bench-gated, not opinion-gated.
```
