docs: relevance-and-veto-synthesis-for-dav1d.md — single decision brief synthesizing #000049 + #000052 §3.1 + §3.2 for forward review
Synthesizes the bench-maxing work across: #000049 NLI recombination veto (bart-large-mnli/k=12/margin/θ=0.999 → 48% real-haystack recall at 0/808 STRICT FP — partial closure, Phase-3 semantic candidate selector for full closure) #000052 §3.1 diagnose_coherence (lexical sidecar, advisory-only, 1.1% real-STRICT FP after round-2 patch) #000052 §3.2 relevance reranker (bge-reranker-large + cleaned + θ ≤ -2.42 → 100% mis-cite / 55% deflection / 0% STRICT FP — motivating Zionist-shape failure fully covered) The three are architecturally orthogonal (§3.2.2 step 3C verified: combining lexical sidecars with the relevance reranker gives no lift; each owns its own failure-shape slice). Three structurally distinct demote-only signals layered on the binary verifier. Three runtime-promotion decisions for fox+dav1d: - §3.1: keep advisory or wire policy hook? (probably advisory) - §3.2: promote at the fp=0 operating point? (sign-off folds relevance_policy_hash into governance_policy_hash) - #000049: promote at 48% partial closure, or wait for Phase 3 semantic candidate selector? Bench-maxing methodology codified in CLAUDE.md is the transferable artifact: 'clean candidate-bench can mis-predict in BOTH directions — real-data fixtures on both precision AND recall axes are the only load-bearing measurement'. Eight instances across the two arcs; the discipline applies to any future model-based addition. Indexed in docs/TICKETS.md 'Distinction from other docs' section as a non-ticket reference doc. Production verifier unchanged; nothing in audit_mode; all work SHADOW pending sign-off.
This commit is contained in:
parent
7910440e50
commit
90d5c54c0a
2 changed files with 226 additions and 0 deletions
|
|
@ -91,6 +91,16 @@ proposing change. Examples that stay un-numbered:
|
|||
between "ingest + search runs on a phone" and an NPU hand-warmer.
|
||||
Generalizes the "soft hash vs hard hash" discipline; the numbers
|
||||
come from #000039 §14.6. Written 2026-05-12.)
|
||||
- `relevance-and-veto-synthesis-for-dav1d.md` (2026-05-13 dav1d
|
||||
handoff: synthesis of #000049 [NLI recombination veto] + #000052
|
||||
§3.1 [coherence sidecar] + §3.2 [relevance reranker] into one
|
||||
decision-ready brief. Three orthogonal sidecars / vetoes
|
||||
covering non-overlapping failure shapes; recommended runtime
|
||||
operating points; per-step bench-maxing receipts; the
|
||||
eight-instance meta-lesson "clean candidate-bench mis-predicts in
|
||||
both directions, real-data on both axes is the only load-bearing
|
||||
measurement". Awaiting fox+dav1d sign-off on the three runtime-
|
||||
promotion decisions.)
|
||||
|
||||
If a doc proposes change AND awaits a decision AND has scoped
|
||||
implementation cost, it's a ticket. Otherwise it's reference.
|
||||
|
|
|
|||
216
docs/relevance-and-veto-synthesis-for-dav1d.md
Normal file
216
docs/relevance-and-veto-synthesis-for-dav1d.md
Normal file
|
|
@ -0,0 +1,216 @@
|
|||
# Model-based vetoes in arborist's proof-adjacent path — dav1d synthesis (2026-05-13)
|
||||
|
||||
Written for an independent reviewer who has not walked the commit log
|
||||
or read the per-ticket §7 / §3 narratives. Synthesizes three parallel
|
||||
investigations into a single decision-ready brief.
|
||||
|
||||
**Scope.** Three model/sidecar additions to arborist that all live in
|
||||
the proof-adjacent path (consume verifier output; potentially
|
||||
influence `audit_mode` if promoted; per #000049 §7 #2 discipline,
|
||||
SHADOW until fox+dav1d sign-off + folding into
|
||||
`governance_policy_hash`):
|
||||
|
||||
| ticket | what | failure mode it catches | status |
|
||||
|---|---|---|---|
|
||||
| **#000049** | NLI veto (cross-encoder entailment) | recombination — answer's tokens are all in the source but re-paired with the wrong subject ("Mercury is the largest" from a source saying Jupiter is largest and Mercury smallest) | partial closure (~48% real-haystack recall at 0 FP) |
|
||||
| **#000052 §3.1** | `diagnose_coherence` (lexical, no model) | structural — answer is word-salad / circular ("X is X") / vacuously true / phrase-component-reuse | landed, advisory-only, 1.1% real-STRICT FP |
|
||||
| **#000052 §3.2** | relevance reranker (cross-encoder aboutness) | topic-collision / mis-cite — claim about X cited to source about Y, X≠Y but shared tokens (the "Zionist entity" field case) | **viable runtime soft-veto: 100% mis-cite, 55% deflection catch at 0 real-STRICT FP** |
|
||||
|
||||
Each tool covers a structurally distinct failure shape. They are
|
||||
**orthogonal** (§3.2.2 step 3C measurement: combining lexical
|
||||
sidecars with the relevance reranker gives no lift — each owns its
|
||||
own slice). The verifier (`verify_quotes`) remains binary and first-
|
||||
line; all three additions are layered on top and demote-only.
|
||||
|
||||
## The recommended operating points
|
||||
|
||||
```
|
||||
#000052 §3.1 diagnose_coherence:
|
||||
LANDED (advisory only, off the proof path)
|
||||
1.1% FP rate on 808-cell pooled bench-qa STRICT
|
||||
(was 5.4% pre-patch; 80% relative reduction in this session)
|
||||
|
||||
#000052 §3.2 relevance reranker (PRIMARY RECOMMENDATION):
|
||||
model: BAAI/bge-reranker-large (~560MB)
|
||||
preprocessing: clean_for_relevance (strip claim-lattice metadata)
|
||||
threshold: θ ≤ -2.42 (the smallest threshold that yields fp=0 on
|
||||
the 808-cell pooled bench-qa STRICT)
|
||||
measured: 100% mis-cite catch (20/20 fixtures)
|
||||
55% deflection catch (11/20 fixtures)
|
||||
0% real-STRICT false positive (0/808)
|
||||
mis-cite max score -3.81 vs STRICT min -2.42 → 1.4-pt margin
|
||||
|
||||
#000052 §3.2 cost-pick alternate:
|
||||
model: cross-encoder/ms-marco-MiniLM-L-6-v2 (~80MB, 5× smaller)
|
||||
threshold: θ = +3 (cleaned)
|
||||
measured: 100% mis-cite, 65% deflection, 0.4% STRICT FP
|
||||
(3 false-demotes per 808 STRICT)
|
||||
|
||||
#000049 NLI veto:
|
||||
model: facebook/bart-large-mnli (~407MB)
|
||||
config: k=12 (top-k candidate clauses by token overlap),
|
||||
agg=max, θc=0.999
|
||||
measured: 48% recall on 25 hand-crafted real-haystack
|
||||
recombination fixtures, 0/808 STRICT FP
|
||||
architectural bottleneck: the candidate-clause selector
|
||||
(top-k by token overlap) misses the contradicting
|
||||
clause when its tokens aren't lexically close to the
|
||||
answer's subject — a semantic candidate selector
|
||||
(vec-driven, sibling of #000050/#000051) would lift
|
||||
this ceiling. See #000049 §7 #27.
|
||||
```
|
||||
|
||||
## How we got there (the meta-lesson)
|
||||
|
||||
The §3.2 arc, in five steps, with the verdict moving each time:
|
||||
|
||||
```
|
||||
1. Candidate-bench (n=26 contrived POS/NEG)
|
||||
→ "all rerankers clean-separate" (overoptimistic — contrived data)
|
||||
|
||||
2. Real STRICT precision sweep (n=808 pooled bench-qa STRICT)
|
||||
→ "candidate-bench θ doesn't survive — runtime veto NOT VIABLE"
|
||||
(wrong recall denominator — used candidate-bench NEG which
|
||||
overlapped real STRICT distribution)
|
||||
|
||||
3A. Real-context deflection fixtures (n=20 hand-built against real
|
||||
bench-qa contexts)
|
||||
→ "bge-large catches 55% of deflections at strict fp=0"
|
||||
(positive reversal — real-data NEG is well-separated from real STRICT)
|
||||
|
||||
3B. Real-context mis-cite fixtures (n=20)
|
||||
→ "bge-large catches 100% of mis-cite at strict fp=0"
|
||||
(full closure of the motivating Zionist-shape failure)
|
||||
|
||||
3C. Multi-signal combination (relevance × diagnose_deflection ×
|
||||
diagnose_coherence)
|
||||
→ "lexical sidecars silent on these shapes; each tool owns its
|
||||
own slice cleanly"
|
||||
```
|
||||
|
||||
The #000049 arc was a parallel six-step zigzag through n=1 → n=3 →
|
||||
n=5 → pooled-808 STRICT samples, with the same pattern (candidate-
|
||||
bench → over-optimistic; smaller sample → wrong threshold; full
|
||||
sample → walk-back twice). Settled at bart-large-mnli / k=12 /
|
||||
margin / θ=0.999 with 48% real-haystack recall.
|
||||
|
||||
**Eight meta-lesson instances across both arcs** (#000049 §7 #18→#27
|
||||
+ #000052 §3.2.1→§3.2.2 steps 2/3A/3B/3C), with the sharpest one
|
||||
yet: **clean candidate-bench can mis-predict in both directions** —
|
||||
over-optimistic on threshold (step 2 caught this on §3.2) AND
|
||||
over-pessimistic on viability (step 3 reverses this). Real-data
|
||||
fixtures on BOTH precision and recall axes are the only
|
||||
load-bearing measurement. Codified in `CLAUDE.md` bench-maxing
|
||||
section.
|
||||
|
||||
## The architectural bottleneck — shared between #000049 and #000052 §3.2
|
||||
|
||||
Both arcs converged on the same observation: **the lexical-candidate
|
||||
selector (top-k by token overlap) is the structural ceiling**.
|
||||
|
||||
- For NLI (#000049): the selector misses the contradicting clause
|
||||
when it's not lexically close to the answer's subject (a
|
||||
Kilimanjaro answer with a buried Mount Kenya contradicting clause
|
||||
— the Mount Kenya clause only shares "Kenya", gets ranked low,
|
||||
NLI never sees it, can't fire). Ceiling: 48% real-haystack recall.
|
||||
|
||||
- For relevance (#000052 §3.2): the same issue limits the *deflection*
|
||||
recall (55% at fp=0) — though the *mis-cite* case is structurally
|
||||
easier (the mis-cited source is a wholly different topic, so the
|
||||
relevance score is robustly low even without sophisticated candidate
|
||||
selection).
|
||||
|
||||
**The Phase-3 lever for both is a semantic candidate selector** —
|
||||
embedding-similarity-driven (sibling of #000050/#000051's vec-hybrid
|
||||
retrieval work), not lexical. Untested but architecturally clear.
|
||||
|
||||
## Discipline cage (inherited from #000049 §7 #2; applies to both
|
||||
model-based additions)
|
||||
|
||||
```
|
||||
1. SHADOW-FIRST. demote_below_score / runtime promotion null until
|
||||
fox+dav1d sign-off + per-step bench-gating.
|
||||
2. DEMOTION-ONLY. models can lower confidence (STRICT→HYBRID or
|
||||
demote), never raise it. No model manufactures
|
||||
proof. (No-LLM-as-judge invariant satisfied:
|
||||
answering LLM never grades itself; these models
|
||||
are fixed-checkpoint, content-addressed, and only
|
||||
in the demote-direction.)
|
||||
3. GOVERNANCE-HASH. relevance_policy_hash / nli_policy_hash
|
||||
(model + preprocessing + threshold + aggregation)
|
||||
folds into governance_policy_hash the moment
|
||||
audit_mode is affected. Two deployments with
|
||||
different model identities can't share cache rows.
|
||||
4. OPTIONAL EXTRA. [nli] pip extra; CPU + torch + transformers.
|
||||
Fresh checkout stays python3.12 + venv + sqlite3.
|
||||
Heavy passes can run off-device (4090 GPU, sibling
|
||||
of the #000051 vecpack producer pattern).
|
||||
```
|
||||
|
||||
## Decisions for fox + dav1d
|
||||
|
||||
```
|
||||
[ ] §3.1 diagnose_coherence: keep advisory-only (current state)?
|
||||
Or wire a demote-policy hook for the 1.1% FP rate? Probably
|
||||
leave advisory — the 1.1% FP rate would corrupt audit_mode
|
||||
even for the trivial gain it'd buy. Status quo recommended.
|
||||
|
||||
[ ] §3.2 relevance reranker: promote to runtime demotion-only veto?
|
||||
The bge-large / cleaned / θ ≤ -2.42 operating point is clean
|
||||
(100% mis-cite, 0% STRICT FP measured on 808). Sign-off
|
||||
folds relevance_policy_hash into governance_policy_hash.
|
||||
Caveats: n=20 mis-cite + n=20 deflection fixtures; recall
|
||||
numbers will move with bigger sets. The fp=0 floor is the
|
||||
safest pick; aggressive operators can take θ ≈ 0 for higher
|
||||
deflection recall at ~2% FP.
|
||||
|
||||
[ ] #000049 NLI veto: promote at 48% real-haystack recall, or
|
||||
wait for a semantic candidate selector (Phase 3)?
|
||||
The 48% catch is strictly better than the current 0%, but
|
||||
well short of full closure. Phase 3 work is gated on
|
||||
#000050/#000051 vec-hybrid retrieval landing (semantic
|
||||
selector substrate). Recommend: wait for Phase 3 unless
|
||||
fox wants the partial coverage now.
|
||||
|
||||
[ ] Bench-maxing methodology codification: the CLAUDE.md
|
||||
discipline ("clean eval ≠ bench-qa precision OR recall;
|
||||
re-confirm config choice when denominator grows") is the
|
||||
transferable artifact. Worth keeping as the working
|
||||
discipline for any future model-based addition.
|
||||
```
|
||||
|
||||
## Receipts
|
||||
|
||||
```
|
||||
#000049 (NLI):
|
||||
docs/tickets/ticket-000049-attribution-aware-grounding-check.md
|
||||
§7 #18 → #27 (the full bench-maxing arc with grid JSONs at each step)
|
||||
arborist/qa/nli/ + bench/scripts/nli_shadow_grid.py
|
||||
bench/results/nli-shadow-grid-* (5 JSON sweep results)
|
||||
bench/fixtures/5f/recombination-realcontext-v1.jsonl
|
||||
|
||||
#000052 §3.1 (coherence sidecar):
|
||||
docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md §3.1
|
||||
arborist/qa/inspect.py:diagnose_coherence + the 5 round-2 rule tightenings
|
||||
tests/test_inspect.py (19 coherence tests + 5 ex-xfail-now-passing regressions
|
||||
+ 1 remaining xfail + the 808-cell pooled-STRICT FP regression test)
|
||||
|
||||
#000052 §3.2 (relevance reranker):
|
||||
docs/tickets/ticket-000052-relevance-coherence-meta-cognition.md §3.2
|
||||
arborist/qa/relevance/ (manifest + ShadowRelevance class
|
||||
+ clean_for_relevance preprocessor + 20 tests)
|
||||
bench/scripts/relevance_shadow_grid.py + relevance_shadow_sweep.py
|
||||
bench/fixtures/5f/relevance-aboutness-v1.jsonl (candidate-bench)
|
||||
bench/fixtures/5f/relevance-deflection-realcontext-v1.jsonl (step 3A)
|
||||
bench/fixtures/5f/relevance-miscite-realcontext-v1.jsonl (step 3B)
|
||||
bench/results/relevance-shadow-* (4 JSON sweep results)
|
||||
|
||||
Bench-maxing discipline:
|
||||
CLAUDE.md "Bench-maxing — measure deltas, not opinions" section
|
||||
(the codified discipline distilled from this work)
|
||||
```
|
||||
|
||||
Production verifier is unchanged. Nothing in `audit_mode`. All work
|
||||
is SHADOW until the runtime-promotion decisions above. Standing by
|
||||
for fox+dav1d sign-off (or for further measurement if either of you
|
||||
wants to push on a specific axis).
|
||||
Loading…
Add table
Add a link
Reference in a new issue