#000049 §7 #18: harder-set 5-candidate NLI bench — 28+26 cases; the two-threshold rule is load-bearing (entailment guard filters spurious competing-superlative contradictions, 0% guarded FP vs ~4% single-threshold); MiniLM-82M cost-pick

docs/TICKETS.md

@@ -103,7 +103,7 @@ Newest first. Update on every open/close.
 |----------|------------------------------------------------|-----------------------|------------|-----------|
 | #000051  | Federated vecpack distribution (gossip the embedding backfill) | open · awaiting go/no-go · doc-only scaffold. Makes `chunk_vecs` a distributable artifact: backfill once on any CPU box (cloud / Prometheus-Σ sweep — #000037 §3.1), publish a **vecpack** `(shard_root, vec_backend_version, [(leaf_hash, embedding_blob)…])` over the mesh wire layer, every peer pulls + bulk-loads (sub-ms/chunk on the receiver — the laptop never runs the transformer). Keyed on `leaf_hash` (portable) not `chunk_id` (shard-local). Vecpacks are **soft data** — embeddings are `UNGROUNDED`, never proof path — so a cheap structural sanity gate (chunk exists locally w/ matching leaf_hash, right blob length for (dim,quant), finite norm, backend_version matches) suffices, no Merkle-proof-grade verification needed. Supplies #000050's prereq #1 ("a vecpack exists & is imported on the bench box", not "fox embedded the corpus locally"). GPU producer (the fast path): bge-small-en-v1.5 batched on a CUDA box (4090) ≈ 10³–10⁴ chunks/s → full 6.24M-chunk corpus in *minutes*, not days — drop a CUDA `Embedder` into `default_embedder()`; CUDA stack lives only on the producer box, never in arborist's `python+sqlite3` core. The mechanism behind whitepaper §1's "the embedding pass runs off the device". #000039 / #000050 sibling | 2026-05-12 | —         |
 | #000050  | Vec RRF hybrid fusion (#000039 Phase 2) | open · awaiting go/no-go · doc-only scaffold; design in #000039 §4.2 (RRF) + §8 (the gate). Wire `VecBackend` as a 5th retrieval route in `query.py`, RRF-merged (route provenance carried) with the 4 FTS5 routes; UNGROUNDED hits, additive not replacement. Phase-2 sub-items now explicit: **accept-path-5** in `_filter_by_title_relevance` (low-title-overlap vec hits survive only via a stronger span-level warrant, never similarity-score alone — else the title gate drops exactly the semantic candidates vec exists for & the bench shows no lift); **six** vec config fields fold into `governance_policy_hash` (recipe-named quant `int8sym`) **+ a cache-write guard** blocking `providence_cache` persistence for vec/hybrid runs until that's wired; **run-DAG records the vec stage** (backend version, six fields, top_k, query-embedding hash, candidate chunk_ids+distances). **Gated** on (a) a corpus backfill **distributed via #000051** AND (b) a **four-condition** recall bench (A FTS5-only / B vec-only / C RRF hybrid / D candidate-union-no-RRF) clearing the 5pp floor incl. C-beats-D, on semantic-allusion + curated + **adversarial-semantic-neighbor** fixtures (else park, vec stays opt-in `--backend vec`; if C≈D ship the union, drop RRF). #000039 follow-up | 2026-05-12 | —         |
-| #000049  | Attribution-aware grounding check (the recombination boundary) | open · boundary accepted · production no-go · shadow-path approved (de novo review 2026-05-13 — ticket §7) · doc-only; the home for #000048's deferred §2.3 — closing the 2 recombination over-grounds in `falsification-hard` (hard-003 Mercury / hard-005 Einstein) needs an attribution / dependency-parse or mini-NLI check, which is *not lexical* (#000048 §5). Discipline question answered: a small fixed purpose-built NLI/entailment *model* may influence `audit_mode` only as an opt-in, hash-pinned, governance-hashed, **demotion-only contradiction veto** after shadow-mode evidence (never promotes — `MODEL_ASSISTED_DEMOTION`, never `MODEL_ASSISTED_PROMOTION`). Production verifier unchanged; `falsification-hard` stays 10/12 as an honest boundary marker. Roadmap: Phase 0 (this amendment) → Phase 1 (shadow design: NLI manifest, fetch/verify, `nli_pair@v1` canonicalization, recombination-risk trigger) → Phase 2 (bench-only shadow impl, `[nli]` extra, `make fetch-nli`) → Phase 3 (demotion-only runtime, gated) → Phase 4 (mesh blob sync); §7 #12 six-condition bench gate required before Phases 2–4; if NLI ever affects `audit_mode`, `nli_policy_hash` folds into `governance_policy_hash`. **Phase-2 candidate bench started 2026-05-12** (`~/git/arborist-nli-bench/`, commit `829f9a4`; ticket §7 #18): `deberta-v3-base-mnli-fever-anli` (184M, 251ms p50 CPU) and `nli-MiniLM2-L6-H768` (82M, 28ms) both catch 18/18 synthesized recombination cases (incl. the 2 fixtures) with 0/18 false positives on true cross-sentence summaries, contra-score gap ~0.93–0.97, both deterministic — picture: recombination is an *easy* task for any modern NLI checkpoint, so the small MiniLM is the cost-pick; open risk = real-traffic FP rate, which only a shadow run on actual `bench-qa` measures (gate item 4, TODO). #000048 follow-up | 2026-05-12 | —         |
+| #000049  | Attribution-aware grounding check (the recombination boundary) | open · boundary accepted · production no-go · shadow-path approved (de novo review 2026-05-13 — ticket §7) · doc-only; the home for #000048's deferred §2.3 — closing the 2 recombination over-grounds in `falsification-hard` (hard-003 Mercury / hard-005 Einstein) needs an attribution / dependency-parse or mini-NLI check, which is *not lexical* (#000048 §5). Discipline question answered: a small fixed purpose-built NLI/entailment *model* may influence `audit_mode` only as an opt-in, hash-pinned, governance-hashed, **demotion-only contradiction veto** after shadow-mode evidence (never promotes — `MODEL_ASSISTED_DEMOTION`, never `MODEL_ASSISTED_PROMOTION`). Production verifier unchanged; `falsification-hard` stays 10/12 as an honest boundary marker. Roadmap: Phase 0 (this amendment) → Phase 1 (shadow design: NLI manifest, fetch/verify, `nli_pair@v1` canonicalization, recombination-risk trigger) → Phase 2 (bench-only shadow impl, `[nli]` extra, `make fetch-nli`) → Phase 3 (demotion-only runtime, gated) → Phase 4 (mesh blob sync); §7 #12 six-condition bench gate required before Phases 2–4; if NLI ever affects `audit_mode`, `nli_policy_hash` folds into `governance_policy_hash`. **Phase-2 candidate bench done 2026-05-12** (`~/git/arborist-nli-bench/`, commits `829f9a4` + `a1cb28d`; ticket §7 #18): checkpoint-agnostic harness runs the §7 #5 clause-level algorithm over 28 synth recombination cases (incl. the 2 fixtures + harder shapes) + 26 legit cases (true summaries + near-miss decoys). 4 working candidates; `nli-MiniLM2-L6-H768` (82M, 45ms p50 CPU), `deberta-v3-base-mnli-fever-anli` (184M, 223ms), `bart-large-mnli` (407M, 259ms) all 28/28 catch · 0/26 FP with the standard θe=0.9 entailment guard; `cross-encoder/nli-deberta-v3-base` 27/28; deberta-large repo-id TODO. **Key finding: the §7 #5 two-threshold rule is load-bearing** — 3 of 4 candidates argmax-contradict 1/26 legit cases on the *wrong* source clause (competing-superlative confusion, e.g. "largest hot desert" vs "largest desert overall"); the entailment guard filters every one because another clause restates the claim → 0% guarded FP vs ~4% single-threshold. Picture: recombination is *easy* for any modern NLI checkpoint — differentiator is cost/robustness, MiniLM is the cost-pick, bart-large the threshold-robust pick. Open risk = real-traffic FP rate, measurable only by a shadow run on actual `bench-qa` (gate item 4, TODO). #000048 follow-up | 2026-05-12 | —         |
 | #000048  | Verifier upgrade — recombination-aware grounding + clause segmentation | **closed · 2026-05-12** — steps 2.1 + 2.4 landed 2026-05-11 (12 of 16 residual items: 4 HYBRID_ENTITY over-grounds + 8 Formulate mis-segments → `formulate-hard` 12/12, `falsification-hard` 10/12; each bench-gated, no STRICT-rate regression — 2.1's gate fired on 0 QA answers, 2.4's segmenter touched 7 of 450 lattice cells both verdict changes correct). Step 2.2 (single-clause-containment paraphrase check) attempted + reverted — catches the 2 recombination fixtures but also rejects legit cross-sentence summaries with no threshold separating the two; recombination-vs-summary isn't lexical (§5 "What we learned"). The attribution-aware path moved to **#000049** (fox 2026-05-12). 2 live-pack `expected_reason` updated HYBRID_ENTITY→UNGROUNDED; 12+ tests; `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`. #000046 follow-up; #000047 closed | 2026-05-11 | —         |
 | #000047  | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | —         |
 | #000046  | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom now down to 2 falsification-hard over-grounds (#000048 step 2.1 closed the 4 entity over-grounds; step 2.4 closed the 8 Formulate mis-segments → that pack 12/12; step 2.2 attempted + reverted — the last 2 recombination fixtures need an attribution-aware verifier, now tracked as **#000049**, and stand as documented residue). `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed; #000048 closed | 2026-05-11 | —         |

docs/tickets/ticket-000049-attribution-aware-grounding-check.md

@@ -2,9 +2,11 @@
 
 **Status:** open · boundary accepted · production no-go · shadow-path
 approved (de novo review 2026-05-13 — see §7) · Phase-2 candidate bench
-done (§7 #18 — `~/git/arborist-nli-bench/`; 2/4 candidates, both
-18/18 catch · 0/18 FP on a synthetic 36-case set; real-QA shadow FP
-rate still TODO)
+done (§7 #18 — `~/git/arborist-nli-bench/`; 4 working candidates, 3 hit
+28/28 catch · 0/26 FP on a 54-case synth set incl. harder shapes;
+confirmed the §7 #5 entailment guard is load-bearing — filters spurious
+competing-superlative contradictions; MiniLM-82M is the cost-pick;
+real-QA shadow FP rate still TODO = gate item 4)
 **Opened:** 2026-05-12
 **Scope:** Decide whether — and if so how — to add a verifier check
 that catches a *recombination*: a claim whose content tokens are all
@@ -561,45 +563,81 @@ is bench-gated and auditable) while respecting the danger (a model in
 the verifier path is still a model in the verifier path — a
 controlled exception, not a casual dependency).
 
-**18. Phase-2 candidate bench — first run (2026-05-12).** Harness:
+**18. Phase-2 candidate bench (2026-05-12).** Harness:
 `~/git/arborist-nli-bench/` (sibling repo — keeps `transformers`/
-`torch` out of arborist's `python+sqlite3` core; commit `829f9a4`).
-Runs the §7 #5 clause-level algorithm checkpoint-agnostically (reads
-`id2label` from each model's config) over 18 synthesized recombination
-cases (incl. the two `5f-fal-hard-003/-005` fixtures — the veto
-*should* fire) + 18 true cross-sentence summaries (the veto must
-*not* fire). First two candidates:
+`torch` out of arborist's `python+sqlite3` core; commits `829f9a4`
+[harness] + `a1cb28d` [harder eval + 5-candidate scorecard]). Runs
+the §7 #5 clause-level algorithm checkpoint-agnostically (reads
+`id2label` from each model's config — handles the cross-encoder vs
+MNLI vs BART label-ordering differences with no per-model wiring)
+over **28 synthesized recombination cases** (incl. the two
+`5f-fal-hard-003/-005` fixtures + harder shapes: longer sources with
+the swap buried mid-paragraph, paraphrased predicates, numeric swaps,
+qualifier swaps — the veto *should* fire) + **26 legit cases** (true
+cross-sentence summaries + near-miss decoys: same-subject-across-
+clauses, qualifier-dropping-but-still-true, competing-superlative,
+negation-paraphrase — the veto must *not* fire). CPU only, float32,
+no sampling.
 
-| candidate | params | p50 / p95 latency (CPU) | catch / FP | min(contra\|recomb) − max(contra\|legit) |
-|---|---|---|---|---|
-| `MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli` (Apache-2.0) | 184M | 251 / 465 ms | **18/18 · 0/18** | 0.977 − 0.005 = **0.97** |
-| `cross-encoder/nli-MiniLM2-L6-H768` (Apache-2.0) | 82M | **28 / 42 ms** | **18/18 · 0/18** | 0.981 − 0.046 = **0.93** |
+| candidate | params | p50 / p95 (CPU) | catch / FP (guarded) | raw legit-contra | threshold-free gap min(c\|rec)−max(c\|leg) |
+|---|---|---|---|---|---|
+| `MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli` (Apache) | 184M | 223 / 279 ms | **28/28 · 0/26** @ θc 0.5, θe 0.9 | 1/26 | 0.977 − 0.986 = −0.01 |
+| `cross-encoder/nli-deberta-v3-base` (Apache) | 184M | 235 / 275 ms | 27/28 · 0/26 @ θc 0.5, θe 0.3 | 1/26 | 0.991 − 0.999 = −0.01 |
+| `cross-encoder/nli-MiniLM2-L6-H768` (Apache) | 82M | **45 / 60 ms** | **28/28 · 0/26** @ θc 0.5, θe 0.9 | 1/26 | 0.759 − 0.946 = −0.19 |
+| `facebook/bart-large-mnli` (MIT) | 407M | 259 / 350 ms | **28/28 · 0/26** @ θc 0.5, θe 1.01 | **0/26** | 0.648 − 0.412 = **+0.24** |
+| `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli` (MIT) | — | — | repo id failed to resolve under transformers 5.x (TODO — base variant of the same family loaded fine, so it's an id/availability hiccup, not fundamental) | — | — |
 
-Both deterministic across re-runs. *Every* θ_c in roughly [0.1, 0.95]
-separates the two classes perfectly (best operating point reported as
-θ_c = 0.5, θ_e = 0.9 — the θ_e entailment-guard barely matters on this
-set, only `recomb-004` Mauna Kea has high `max_entail`). Null baseline
-(current lexical verifier, no NLI) is 0/2 catch · 0 FP — so on *this*
-eval set both candidates beat it net (full catch, FP still 0). The
-differentiator is **cost, not capability**: contradiction recombination
-is an *easy* task for any modern sentence-pair NLI checkpoint, so the
-82M MiniLM (9× faster, half the size) is the natural pick for a check
-that runs only on a small unresolved subset.
+All four working candidates deterministic across re-runs.
 
-**Caveats (why this is not yet the §7 #12 gate, only gate items 1–3 +
-6).** (a) The eval set is *synthetic and clean* — short single-sentence
-sources, textbook-contradiction shapes; real Wikipedia QA has long
-messy contexts, partial overlaps, hedged claims. Gate item 4
-(shadow-mode FP rate on a real `bench-qa` sample) is the actual test
-and is *not* done here. (b) n = 36 total — better than n = 2, still
-small; needs the harder, more-realistic recombination/summary cases
-before it's load-bearing. (c) Only 2 of 4 manifest candidates run
-(the deberta-large + bart-large reference points pending) — but the
+**Key finding — the §7 #5 two-threshold rule is load-bearing, not
+decorative.** On the *threshold-free* argmax view, three of four
+candidates argmax-label one legit case as `contradiction` (1/26 ≈
+4%): the model picks the *wrong source clause* — e.g. `legit-025`
+("The Sahara is the largest hot desert on Earth. Antarctica is the
+largest desert overall." / claim "The Sahara is the largest hot
+desert on Earth.") triggers a spurious contradiction on the
+*Antarctica* clause because the model reads "largest hot desert" vs
+"largest desert overall" as competing superlatives. **The
+entailment guard catches every one of these**: another source clause
+(here the Sahara one) directly *entails* the claim, so `max_entail`
+is ≈0.99 ≥ θe and the `Demote()` rule does not fire. That is exactly
+the failure mode §7 #5 step 6's `max_entail < θ_e` conjunct was
+written for ("contradiction alone should not demote if another clause
+entails the answer"), now empirically confirmed. A single-threshold
+contradiction veto would carry ~4% FP; the two-threshold rule
+carries 0% on this set.
+
+**Picture.** Contradiction recombination is an *easy* task for any
+modern sentence-pair NLI checkpoint — the differentiator is **cost
+and threshold-robustness, not capability**. The 82M
+`nli-MiniLM2-L6-H768` (45 ms p50, 6× faster) hits full catch · 0 FP
+with the standard θe=0.9 guard — the natural pick for a check that
+runs only on a small unresolved subset. `bart-large-mnli` (407M, 259
+ms) is the only one with a *clean threshold-free* separation (no
+entail guard needed, +0.24 gap) — the pick if latency budget allows
+and threshold-robustness is prized over speed. `cross-encoder/
+nli-deberta-v3-base` is the weakest of the working four (needs θe=0.3,
+loses one catch). Null baseline (current lexical verifier, no NLI):
+0/2 catch on the fixtures · 0 FP — so all four working candidates
+beat it *net* on this eval set.
+
+**Caveats (why this clears §7 #12 gate items 1–3 + 6 only — not
+item 4).** (a) The eval set is *synthesized and still relatively
+clean* — single-/few-sentence sources, encyclopedia-shaped facts;
+real Wikipedia QA has long messy contexts, partial overlaps, hedged
+claims, and the paraphrase/entity path it'd attach to rarely fires
+on real traffic anyway. **Gate item 4 — shadow-mode FP rate on a
+real `bench-qa` sample — is the actual test and is not done here.**
+(b) n = 54 total — better than n = 2, still small; the harder shapes
+help but a real-traffic-derived set would be more load-bearing. (c)
+4 of 5 manifest candidates ran (deberta-large repo-id TODO) — but the
 picture is already clear enough that "which checkpoint" is a
-cost question, not an accuracy one. **None of Phases 2 (full shadow
-implementation in arborist) / 3 / 4 are unblocked by this run** — it
-unblocks the *decision to build a shadow mode at all* with real
-numbers instead of a contrived 2/12, and the answer it points to is
-"yes, a shadow mode is worth building; use a small cross-encoder NLI;
-the open risk is real-traffic false positives, which only a shadow
-run on actual QA can measure."
+cost/robustness trade, not an accuracy one. **None of Phases 2 (full
+shadow implementation *in arborist*) / 3 / 4 are unblocked by this
+run** — it unblocks the *decision to build a shadow mode at all*,
+with real numbers replacing a contrived 2/12, and the answer it
+points to is: yes, a shadow mode is worth building; the §7 #5
+two-threshold algorithm is the right shape (the entailment guard
+earns its place); a small cross-encoder NLI suffices; the one open
+risk is real-traffic false positives, measurable only by a shadow
+run on actual QA (Phase 2 → gate item 4).