diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 852467e..94a13a8 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -91,6 +91,16 @@ proposing change. Examples that stay un-numbered: between "ingest + search runs on a phone" and an NPU hand-warmer. Generalizes the "soft hash vs hard hash" discipline; the numbers come from #000039 §14.6. Written 2026-05-12.) +- `relevance-and-veto-synthesis-for-dav1d.md` (2026-05-13 dav1d + handoff: synthesis of #000049 [NLI recombination veto] + #000052 + §3.1 [coherence sidecar] + §3.2 [relevance reranker] into one + decision-ready brief. Three orthogonal sidecars / vetoes + covering non-overlapping failure shapes; recommended runtime + operating points; per-step bench-maxing receipts; the + eight-instance meta-lesson "clean candidate-bench mis-predicts in + both directions, real-data on both axes is the only load-bearing + measurement". Awaiting fox+dav1d sign-off on the three runtime- + promotion decisions.) If a doc proposes change AND awaits a decision AND has scoped implementation cost, it's a ticket. Otherwise it's reference. diff --git a/docs/relevance-and-veto-synthesis-for-dav1d.md b/docs/relevance-and-veto-synthesis-for-dav1d.md new file mode 100644 index 0000000..fed06ba --- /dev/null +++ b/docs/relevance-and-veto-synthesis-for-dav1d.md @@ -0,0 +1,216 @@ +# Model-based vetoes in arborist's proof-adjacent path — dav1d synthesis (2026-05-13) + +Written for an independent reviewer who has not walked the commit log +or read the per-ticket §7 / §3 narratives. Synthesizes three parallel +investigations into a single decision-ready brief. + +**Scope.** Three model/sidecar additions to arborist that all live in +the proof-adjacent path (consume verifier output; potentially +influence `audit_mode` if promoted; per #000049 §7 #2 discipline, +SHADOW until fox+dav1d sign-off + folding into +`governance_policy_hash`): + +| ticket | what | failure mode it catches | status | +|---|---|---|---| +| **#000049** | NLI veto (cross-encoder entailment) | recombination — answer's tokens are all in the source but re-paired with the wrong subject ("Mercury is the largest" from a source saying Jupiter is largest and Mercury smallest) | partial closure (~48% real-haystack recall at 0 FP) | +| **#000052 §3.1** | `diagnose_coherence` (lexical, no model) | structural — answer is word-salad / circular ("X is X") / vacuously true / phrase-component-reuse | landed, advisory-only, 1.1% real-STRICT FP | +| **#000052 §3.2** | relevance reranker (cross-encoder aboutness) | topic-collision / mis-cite — claim about X cited to source about Y, X≠Y but shared tokens (the "Zionist entity" field case) | **viable runtime soft-veto: 100% mis-cite, 55% deflection catch at 0 real-STRICT FP** | + +Each tool covers a structurally distinct failure shape. They are +**orthogonal** (§3.2.2 step 3C measurement: combining lexical +sidecars with the relevance reranker gives no lift — each owns its +own slice). The verifier (`verify_quotes`) remains binary and first- +line; all three additions are layered on top and demote-only. + +## The recommended operating points + +``` +#000052 §3.1 diagnose_coherence: + LANDED (advisory only, off the proof path) + 1.1% FP rate on 808-cell pooled bench-qa STRICT + (was 5.4% pre-patch; 80% relative reduction in this session) + +#000052 §3.2 relevance reranker (PRIMARY RECOMMENDATION): + model: BAAI/bge-reranker-large (~560MB) + preprocessing: clean_for_relevance (strip claim-lattice metadata) + threshold: θ ≤ -2.42 (the smallest threshold that yields fp=0 on + the 808-cell pooled bench-qa STRICT) + measured: 100% mis-cite catch (20/20 fixtures) + 55% deflection catch (11/20 fixtures) + 0% real-STRICT false positive (0/808) + mis-cite max score -3.81 vs STRICT min -2.42 → 1.4-pt margin + +#000052 §3.2 cost-pick alternate: + model: cross-encoder/ms-marco-MiniLM-L-6-v2 (~80MB, 5× smaller) + threshold: θ = +3 (cleaned) + measured: 100% mis-cite, 65% deflection, 0.4% STRICT FP + (3 false-demotes per 808 STRICT) + +#000049 NLI veto: + model: facebook/bart-large-mnli (~407MB) + config: k=12 (top-k candidate clauses by token overlap), + agg=max, θc=0.999 + measured: 48% recall on 25 hand-crafted real-haystack + recombination fixtures, 0/808 STRICT FP + architectural bottleneck: the candidate-clause selector + (top-k by token overlap) misses the contradicting + clause when its tokens aren't lexically close to the + answer's subject — a semantic candidate selector + (vec-driven, sibling of #000050/#000051) would lift + this ceiling. See #000049 §7 #27. +``` + +## How we got there (the meta-lesson) + +The §3.2 arc, in five steps, with the verdict moving each time: + +``` +1. Candidate-bench (n=26 contrived POS/NEG) + → "all rerankers clean-separate" (overoptimistic — contrived data) + +2. Real STRICT precision sweep (n=808 pooled bench-qa STRICT) + → "candidate-bench θ doesn't survive — runtime veto NOT VIABLE" + (wrong recall denominator — used candidate-bench NEG which + overlapped real STRICT distribution) + +3A. Real-context deflection fixtures (n=20 hand-built against real + bench-qa contexts) + → "bge-large catches 55% of deflections at strict fp=0" + (positive reversal — real-data NEG is well-separated from real STRICT) + +3B. Real-context mis-cite fixtures (n=20) + → "bge-large catches 100% of mis-cite at strict fp=0" + (full closure of the motivating Zionist-shape failure) + +3C. Multi-signal combination (relevance × diagnose_deflection × + diagnose_coherence) + → "lexical sidecars silent on these shapes; each tool owns its + own slice cleanly" +``` + +The #000049 arc was a parallel six-step zigzag through n=1 → n=3 → +n=5 → pooled-808 STRICT samples, with the same pattern (candidate- +bench → over-optimistic; smaller sample → wrong threshold; full +sample → walk-back twice). Settled at bart-large-mnli / k=12 / +margin / θ=0.999 with 48% real-haystack recall. + +**Eight meta-lesson instances across both arcs** (#000049 §7 #18→#27 ++ #000052 §3.2.1→§3.2.2 steps 2/3A/3B/3C), with the sharpest one +yet: **clean candidate-bench can mis-predict in both directions** — +over-optimistic on threshold (step 2 caught this on §3.2) AND +over-pessimistic on viability (step 3 reverses this). Real-data +fixtures on BOTH precision and recall axes are the only +load-bearing measurement. Codified in `CLAUDE.md` bench-maxing +section. + +## The architectural bottleneck — shared between #000049 and #000052 §3.2 + +Both arcs converged on the same observation: **the lexical-candidate +selector (top-k by token overlap) is the structural ceiling**. + +- For NLI (#000049): the selector misses the contradicting clause + when it's not lexically close to the answer's subject (a + Kilimanjaro answer with a buried Mount Kenya contradicting clause + — the Mount Kenya clause only shares "Kenya", gets ranked low, + NLI never sees it, can't fire). Ceiling: 48% real-haystack recall. + +- For relevance (#000052 §3.2): the same issue limits the *deflection* + recall (55% at fp=0) — though the *mis-cite* case is structurally + easier (the mis-cited source is a wholly different topic, so the + relevance score is robustly low even without sophisticated candidate + selection). + +**The Phase-3 lever for both is a semantic candidate selector** — +embedding-similarity-driven (sibling of #000050/#000051's vec-hybrid +retrieval work), not lexical. Untested but architecturally clear. + +## Discipline cage (inherited from #000049 §7 #2; applies to both +model-based additions) + +``` +1. SHADOW-FIRST. demote_below_score / runtime promotion null until + fox+dav1d sign-off + per-step bench-gating. +2. DEMOTION-ONLY. models can lower confidence (STRICT→HYBRID or + demote), never raise it. No model manufactures + proof. (No-LLM-as-judge invariant satisfied: + answering LLM never grades itself; these models + are fixed-checkpoint, content-addressed, and only + in the demote-direction.) +3. GOVERNANCE-HASH. relevance_policy_hash / nli_policy_hash + (model + preprocessing + threshold + aggregation) + folds into governance_policy_hash the moment + audit_mode is affected. Two deployments with + different model identities can't share cache rows. +4. OPTIONAL EXTRA. [nli] pip extra; CPU + torch + transformers. + Fresh checkout stays python3.12 + venv + sqlite3. + Heavy passes can run off-device (4090 GPU, sibling + of the #000051 vecpack producer pattern). +``` + +## Decisions for fox + dav1d + +``` +[ ] §3.1 diagnose_coherence: keep advisory-only (current state)? + Or wire a demote-policy hook for the 1.1% FP rate? Probably + leave advisory — the 1.1% FP rate would corrupt audit_mode + even for the trivial gain it'd buy. Status quo recommended. + +[ ] §3.2 relevance reranker: promote to runtime demotion-only veto? + The bge-large / cleaned / θ ≤ -2.42 operating point is clean + (100% mis-cite, 0% STRICT FP measured on 808). Sign-off + folds relevance_policy_hash into governance_policy_hash. + Caveats: n=20 mis-cite + n=20 deflection fixtures; recall + numbers will move with bigger sets. The fp=0 floor is the + safest pick; aggressive operators can take θ ≈ 0 for higher + deflection recall at ~2% FP. + +[ ] #000049 NLI veto: promote at 48% real-haystack recall, or + wait for a semantic candidate selector (Phase 3)? + The 48% catch is strictly better than the current 0%, but + well short of full closure. Phase 3 work is gated on + #000050/#000051 vec-hybrid retrieval landing (semantic + selector substrate). Recommend: wait for Phase 3 unless + fox wants the partial coverage now. + +[ ] Bench-maxing methodology codification: the CLAUDE.md + discipline ("clean eval ≠ bench-qa precision OR recall; + re-confirm config choice when denominator grows") is the + transferable artifact. Worth keeping as the working + discipline for any future model-based addition. +``` + +## Receipts + +``` +#000049 (NLI): + docs/tickets/ticket-000049-attribution-aware-grounding-check.md + §7 #18 → #27 (the full bench-maxing arc with grid JSONs at each step) + arborist/qa/nli/ + bench/scripts/nli_shadow_grid.py + bench/results/nli-shadow-grid-* (5 JSON sweep results) + bench/fixtures/5f/recombination-realcontext-v1.jsonl + +#000052 §3.1 (coherence sidecar): + docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md §3.1 + arborist/qa/inspect.py:diagnose_coherence + the 5 round-2 rule tightenings + tests/test_inspect.py (19 coherence tests + 5 ex-xfail-now-passing regressions + + 1 remaining xfail + the 808-cell pooled-STRICT FP regression test) + +#000052 §3.2 (relevance reranker): + docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md §3.2 + arborist/qa/relevance/ (manifest + ShadowRelevance class + + clean_for_relevance preprocessor + 20 tests) + bench/scripts/relevance_shadow_grid.py + relevance_shadow_sweep.py + bench/fixtures/5f/relevance-aboutness-v1.jsonl (candidate-bench) + bench/fixtures/5f/relevance-deflection-realcontext-v1.jsonl (step 3A) + bench/fixtures/5f/relevance-miscite-realcontext-v1.jsonl (step 3B) + bench/results/relevance-shadow-* (4 JSON sweep results) + +Bench-maxing discipline: + CLAUDE.md "Bench-maxing — measure deltas, not opinions" section + (the codified discipline distilled from this work) +``` + +Production verifier is unchanged. Nothing in `audit_mode`. All work +is SHADOW until the runtime-promotion decisions above. Standing by +for fox+dav1d sign-off (or for further measurement if either of you +wants to push on a specific axis).