# Model-based vetoes in arborist's proof-adjacent path — dav1d synthesis (2026-05-13)

Written for an independent reviewer who has not walked the commit log
or read the per-ticket §7 / §3 narratives. Synthesizes three parallel
investigations into a single decision-ready brief.

**Scope.** Three model/sidecar additions to arborist that all live in
the proof-adjacent path (consume verifier output; potentially
influence `audit_mode` if promoted; per #000049 §7 #2 discipline,
SHADOW until fox+dav1d sign-off + folding into
`governance_policy_hash`):

| ticket | what | failure mode it catches | status |
|---|---|---|---|
| **#000049** | NLI veto (cross-encoder entailment) | recombination — answer's tokens are all in the source but re-paired with the wrong subject ("Mercury is the largest" from a source saying Jupiter is largest and Mercury smallest) | partial closure (~48% real-haystack recall at 0 FP) |
| **#000052 §3.1** | `diagnose_coherence` (lexical, no model) | structural — answer is word-salad / circular ("X is X") / vacuously true / phrase-component-reuse | landed, advisory-only, 1.1% real-STRICT FP |
| **#000052 §3.2** | relevance reranker (cross-encoder aboutness) | topic-collision / mis-cite — claim about X cited to source about Y, X≠Y but shared tokens (the "Zionist entity" field case) | **viable runtime soft-veto: 100% mis-cite, 55% deflection catch at 0 real-STRICT FP** |

Each tool covers a structurally distinct failure shape. They are
**orthogonal** (§3.2.2 step 3C measurement: combining lexical
sidecars with the relevance reranker gives no lift — each owns its
own slice). The verifier (`verify_quotes`) remains binary and first-
line; all three additions are layered on top and demote-only.

## The recommended operating points

```
#000052 §3.1 diagnose_coherence:
    LANDED (advisory only, off the proof path)
    1.1% FP rate on 808-cell pooled bench-qa STRICT
    (was 5.4% pre-patch; 80% relative reduction in this session)

#000052 §3.2 relevance reranker (PRIMARY RECOMMENDATION):
    model:    BAAI/bge-reranker-large (~560MB)
    preprocessing: clean_for_relevance (strip claim-lattice metadata)
    threshold: θ ≤ -2.42 (the smallest threshold that yields fp=0 on
              the 808-cell pooled bench-qa STRICT)
    measured: 100% mis-cite catch (20/20 fixtures)
              55% deflection catch (11/20 fixtures)
              0% real-STRICT false positive (0/808)
    mis-cite max score -3.81 vs STRICT min -2.42 → 1.4-pt margin

#000052 §3.2 cost-pick alternate:
    model:    cross-encoder/ms-marco-MiniLM-L-6-v2 (~80MB, 5× smaller)
    threshold: θ = +3 (cleaned)
    measured: 100% mis-cite, 65% deflection, 0.4% STRICT FP
              (3 false-demotes per 808 STRICT)

#000049 NLI veto:
    model:    facebook/bart-large-mnli (~407MB)
    config:   k=12 (top-k candidate clauses by token overlap),
              agg=max, θc=0.999
    measured: 48% recall on 25 hand-crafted real-haystack
              recombination fixtures, 0/808 STRICT FP
    architectural bottleneck:  the candidate-clause selector
              (top-k by token overlap) misses the contradicting
              clause when its tokens aren't lexically close to the
              answer's subject — a semantic candidate selector
              (vec-driven, sibling of #000050/#000051) would lift
              this ceiling. See #000049 §7 #27.
```

## How we got there (the meta-lesson)

The §3.2 arc, in five steps, with the verdict moving each time:

```
1. Candidate-bench (n=26 contrived POS/NEG)
   → "all rerankers clean-separate"  (overoptimistic — contrived data)

2. Real STRICT precision sweep (n=808 pooled bench-qa STRICT)
   → "candidate-bench θ doesn't survive — runtime veto NOT VIABLE"
   (wrong recall denominator — used candidate-bench NEG which
    overlapped real STRICT distribution)

3A. Real-context deflection fixtures (n=20 hand-built against real
    bench-qa contexts)
   → "bge-large catches 55% of deflections at strict fp=0"
   (positive reversal — real-data NEG is well-separated from real STRICT)

3B. Real-context mis-cite fixtures (n=20)
   → "bge-large catches 100% of mis-cite at strict fp=0"
   (full closure of the motivating Zionist-shape failure)

3C. Multi-signal combination (relevance × diagnose_deflection ×
    diagnose_coherence)
   → "lexical sidecars silent on these shapes; each tool owns its
    own slice cleanly"
```

The #000049 arc was a parallel six-step zigzag through n=1 → n=3 →
n=5 → pooled-808 STRICT samples, with the same pattern (candidate-
bench → over-optimistic; smaller sample → wrong threshold; full
sample → walk-back twice). Settled at bart-large-mnli / k=12 /
margin / θ=0.999 with 48% real-haystack recall.

**Eight meta-lesson instances across both arcs** (#000049 §7 #18→#27
+ #000052 §3.2.1→§3.2.2 steps 2/3A/3B/3C), with the sharpest one
yet: **clean candidate-bench can mis-predict in both directions** —
over-optimistic on threshold (step 2 caught this on §3.2) AND
over-pessimistic on viability (step 3 reverses this). Real-data
fixtures on BOTH precision and recall axes are the only
load-bearing measurement. Codified in `CLAUDE.md` bench-maxing
section.

## The architectural bottleneck — shared between #000049 and #000052 §3.2

Both arcs converged on the same observation: **the lexical-candidate
selector (top-k by token overlap) is the structural ceiling**.

- For NLI (#000049): the selector misses the contradicting clause
  when it's not lexically close to the answer's subject (a
  Kilimanjaro answer with a buried Mount Kenya contradicting clause
  — the Mount Kenya clause only shares "Kenya", gets ranked low,
  NLI never sees it, can't fire). Ceiling: 48% real-haystack recall.

- For relevance (#000052 §3.2): the same issue limits the *deflection*
  recall (55% at fp=0) — though the *mis-cite* case is structurally
  easier (the mis-cited source is a wholly different topic, so the
  relevance score is robustly low even without sophisticated candidate
  selection).

**The Phase-3 lever for both is a semantic candidate selector** —
embedding-similarity-driven (sibling of #000050/#000051's vec-hybrid
retrieval work), not lexical. Untested but architecturally clear.

## Discipline cage (inherited from #000049 §7 #2; applies to both
model-based additions)

```
1. SHADOW-FIRST.   demote_below_score / runtime promotion null until
                   fox+dav1d sign-off + per-step bench-gating.
2. DEMOTION-ONLY.  models can lower confidence (STRICT→HYBRID or
                   demote), never raise it. No model manufactures
                   proof. (No-LLM-as-judge invariant satisfied:
                   answering LLM never grades itself; these models
                   are fixed-checkpoint, content-addressed, and only
                   in the demote-direction.)
3. GOVERNANCE-HASH. relevance_policy_hash / nli_policy_hash
                   (model + preprocessing + threshold + aggregation)
                   folds into governance_policy_hash the moment
                   audit_mode is affected. Two deployments with
                   different model identities can't share cache rows.
4. OPTIONAL EXTRA. [nli] pip extra; CPU + torch + transformers.
                   Fresh checkout stays python3.12 + venv + sqlite3.
                   Heavy passes can run off-device (4090 GPU, sibling
                   of the #000051 vecpack producer pattern).
```

## Decisions for fox + dav1d

```
[ ] §3.1 diagnose_coherence: keep advisory-only (current state)?
    Or wire a demote-policy hook for the 1.1% FP rate? Probably
    leave advisory — the 1.1% FP rate would corrupt audit_mode
    even for the trivial gain it'd buy. Status quo recommended.

[ ] §3.2 relevance reranker: promote to runtime demotion-only veto?
    The bge-large / cleaned / θ ≤ -2.42 operating point is clean
    (100% mis-cite, 0% STRICT FP measured on 808). Sign-off
    folds relevance_policy_hash into governance_policy_hash.
    Caveats: n=20 mis-cite + n=20 deflection fixtures; recall
    numbers will move with bigger sets. The fp=0 floor is the
    safest pick; aggressive operators can take θ ≈ 0 for higher
    deflection recall at ~2% FP.

[ ] #000049 NLI veto: promote at 48% real-haystack recall, or
    wait for a semantic candidate selector (Phase 3)?
    The 48% catch is strictly better than the current 0%, but
    well short of full closure. Phase 3 work is gated on
    #000050/#000051 vec-hybrid retrieval landing (semantic
    selector substrate). Recommend: wait for Phase 3 unless
    fox wants the partial coverage now.

[ ] Bench-maxing methodology codification: the CLAUDE.md
    discipline ("clean eval ≠ bench-qa precision OR recall;
    re-confirm config choice when denominator grows") is the
    transferable artifact. Worth keeping as the working
    discipline for any future model-based addition.
```

## Receipts

```
#000049 (NLI):
  docs/tickets/ticket-000049-attribution-aware-grounding-check.md
    §7 #18 → #27 (the full bench-maxing arc with grid JSONs at each step)
  arborist/qa/nli/ + bench/scripts/nli_shadow_grid.py
  bench/results/nli-shadow-grid-* (5 JSON sweep results)
  bench/fixtures/5f/recombination-realcontext-v1.jsonl

#000052 §3.1 (coherence sidecar):
  docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md §3.1
  arborist/qa/inspect.py:diagnose_coherence + the 5 round-2 rule tightenings
  tests/test_inspect.py (19 coherence tests + 5 ex-xfail-now-passing regressions
    + 1 remaining xfail + the 808-cell pooled-STRICT FP regression test)

#000052 §3.2 (relevance reranker):
  docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md §3.2
  arborist/qa/relevance/ (manifest + ShadowRelevance class
    + clean_for_relevance preprocessor + 20 tests)
  bench/scripts/relevance_shadow_grid.py + relevance_shadow_sweep.py
  bench/fixtures/5f/relevance-aboutness-v1.jsonl (candidate-bench)
  bench/fixtures/5f/relevance-deflection-realcontext-v1.jsonl (step 3A)
  bench/fixtures/5f/relevance-miscite-realcontext-v1.jsonl (step 3B)
  bench/results/relevance-shadow-* (4 JSON sweep results)

Bench-maxing discipline:
  CLAUDE.md "Bench-maxing — measure deltas, not opinions" section
    (the codified discipline distilled from this work)
```

Production verifier is unchanged. Nothing in `audit_mode`. All work
is SHADOW until the runtime-promotion decisions above. Standing by
for fox+dav1d sign-off (or for further measurement if either of you
wants to push on a specific axis).
