#000049 §7 #26: mining for real recombinations + the pooled-808-STRICT check — verdict settles on margin@0.995

(a) Mined the pooled n=1+3+5 bench-qa runs (808 distinct STRICT answers)
for natural recombinations at lowered θc≥0.7 → 37 would-fires, ALL
token-collision FPs on inspection (Mount Kenya pulled into a Kilimanjaro
answer, Dalí into da Vinci, Donovan into Superman). ZERO genuine
recombination errors — the boundary is theoretical-in-practice; the
failure mode is the candidate selector (top-k by token overlap) pulling
different-entity same-subject-area clauses.

(b) Re-ran the grid against the 808-cell pooled STRICT set: the §7 #25
'max@0.96' was itself a small-sample artifact — the n=5 444-cell set
lacked the high-confidence spurious hits the pooled set has. On 808
cells θc goes back to ~0.995, and at θc=0.995 only agg=margin still
catches 28/28 (max gets 27/28). microsoft/deberta-large-mnli / k=3 /
agg=margin / θc=0.995 → 28/28 synthetic recombinations · 0/808 pooled
real STRICT FP · 0/26 synthetic legit — the ONLY config in the
7-model×full-grid sweep that hits 1.0/0.0 on 808 cells, held at n=3
too. recommended_operating_point reverted to margin@0.995.

Realistic next check: ~20-30 hand-built synthetic-recombination-vs-
real-bench-qa-context fixtures (real haystack, deliberate re-attribution).
Still SHADOW; runtime promotion fox+dav1d-decides. Production verifier
unchanged; falsification-hard stays 10/12.

Meta-lesson instance five: a bigger sample can vindicate a config a
smaller one made look unnecessary — re-confirm the config choice (not
just the threshold) each time the denominator grows.

arborist/qa/nli/manifest.json

@@ -20,19 +20,19 @@
   },
   "thresholds_provenance": "the bench-validated operating point — arborist-nli-bench commit a1cb28d, ticket #000049 §7 #18; on the 28+26-case synth set this point gives 28/28 catch, 0/26 false positives, and the entailment guard is what filters the spurious competing-superlative contradictions",
   "recommended_operating_point": {
-    "_comment": "ticket #000049 §7 #25 — n=5-confirmed (444 real STRICT cells; also held at n=3=275). The {model × candidate-cap k × aggregation × guard × θc × θe} mega-grid found THIS robustly clears the §7 #12 gate: microsoft/deberta-large-mnli + k=2 + agg='max' (over the top-2 candidate clauses) + guard='max_entail' (max p_entail over those clauses) → catches all 28 synthetic recombinations (incl. both 5f-fal-hard fixtures), 0 false positives on 444 real STRICT answers, 0 on the 26 synthetic legit summaries — with ~4 pts of θc headroom (it stays 0 FP up to ~θc 0.99 on this sample; the config below uses 0.96 with margin). roberta-large-mnli is an equally-good alternate (k=2/max/θc=0.95). NOTE the journey: the §7 #23 candidate (deberta-base k=2 θc=0.99) was a small-FP-sample artifact — it's 27/28 here but the §7 #24 n=3 sample drove it to 11/28; it sits at the cliff. The §7 #18 'MiniLM-82M is the cost-pick' is overturned — at the fp=0 bar MiniLM caps at ~0.82, the large model gets 1.0; cost was the wrong axis. Still NOT the active default — pending (a) a bench-qa-derived recombination set (real false answers, not synthetic), (b) optionally n=9 STRICT (~825 cells), (c) fox+dav1d sign-off. When promoted: switch nli_model_version/hf_repo, wire 'aggregation'+'guard' into ShadowNLI.check, set max_candidate_clauses + thresholds, and fold nli_policy_hash (model+k+agg+guard+θc) into governance_policy_hash before any audit_mode effect (§7 #2).",
+    "_comment": "ticket #000049 §7 #26 — confirmed across n=3=275 AND the pooled n=1+n=3+n=5 = 808 real STRICT cells (the §7 #25 'agg=max @ θc=0.96' was a small-sample artifact — the n=5 444-cell set lacked the high-confidence token-collision spurious hits the pooled set has; on 808 cells θc must go back to ~0.995, and at θc=0.995 'margin' is the ONLY aggregation that still catches 28/28 — 'max' gets 27/28, 'mean' ≤0.75). The {model × k × agg × guard × θc × θe} mega-grid: microsoft/deberta-large-mnli + k=3 + agg='margin' (= max over the top-3 candidate clauses of (p_contra - p_entail) of THAT clause — a single threshold that discriminates a real recombination [source genuinely contradicts → that clause's entail≈0 → margin≈0.996-0.998] from a spurious token-collision FP [the source ALSO contains an entailing clause → its margin drags to ~0.95-0.99]) + θc=0.995 → catches all 28 synthetic recombinations (incl. both 5f-fal-hard fixtures), 0 false positives on 808 real STRICT answers, 0 on the 26 synthetic legit summaries. This is the ONLY config (across the 7-model × full-grid sweep) that hits 1.0/0.0 on the 808-cell denominator. Mining the 808 real STRICT answers found ZERO natural recombination errors — the boundary is theoretical-in-practice; the failure mode the mining surfaced is the candidate selector (top-k by token overlap) pulling different-entity same-subject-area clauses (Mount Kenya into a Kilimanjaro answer, Dalí into da Vinci, Donovan into Superman), which 'margin' survives but a Phase-3 verifier-matched-clause hook would eliminate. Still NOT the active default — pending (a) a synthetic-recombination-vs-real-bench-qa-context set (~20-30 hand-built fixtures: real haystack, deliberate re-attribution — the recall-side realism check), (b) fox+dav1d sign-off. When promoted: switch nli_model_version/hf_repo, wire 'aggregation'='margin'+'guard' into ShadowNLI.check, set max_candidate_clauses=3 + thresholds, and fold nli_policy_hash (model+k+agg+guard+θc) into governance_policy_hash before any audit_mode effect (§7 #2).",
     "nli_model_version": "nli-shadow-v1-deberta-large-mnli",
     "hf_repo": "microsoft/deberta-large-mnli",
-    "max_candidate_clauses": 2,
-    "aggregation": "max",
-    "guard": "max_entail",
-    "thresholds": {"contradiction_veto": 0.96, "entailment_block_veto": 0.9},
-    "thresholds_headroom_note": "0 FP held up to θc≈0.99 on the 444-cell n=5 STRICT sample; 0.96 is the conservative-recall choice (still 28/28). At n=3 the same model was 1.0/0.0 with agg='margin' @ θc=0.995 — both score-shapes work; 'max'+'max_entail' is the simpler/more-robust pick at n=5.",
-    "alternates_at_fp0_proper_n": [
-      {"hf_repo": "roberta-large-mnli", "config": "k=2/agg=max/guard=max_entail/θc=0.95", "catch_n5": "28/28", "note": "355M; equally good as the primary at n=5; not separately validated at n=3"},
-      {"hf_repo": "facebook/bart-large-mnli", "config": "k=3/agg=mean/θc=0.94", "catch_n5": "20/28", "note": "407M; only ~0.71 recall at fp=0 — bigger ≠ better, it's the specific checkpoint"},
-      {"hf_repo": "cross-encoder/nli-MiniLM2-L6-H768", "config": "k=2/agg=max/θc=0.98", "catch_n5": "23/28", "note": "82M; ~0.82 recall at fp=0 at proper n — NOT the cost-pick once you demand 0 FP"},
-      {"hf_repo": "MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli", "config": "k=2/agg=max/θc=0.99", "catch_n5": "27/28", "note": "184M; sits at the cliff — 27/28 here, 11/28 on the n=3 sample; sample-sensitive, do not use"}
+    "max_candidate_clauses": 3,
+    "aggregation": "margin",
+    "guard": "max_entail (informational only — 'margin' folds the guard into the score, so θe is N/A)",
+    "thresholds": {"contradiction_minus_entailment_margin_veto": 0.995},
+    "simpler_alternate": {"config": "k=2/agg=max/guard=max_entail/θc=0.995/θe=0.9", "catch_pooled808": "27/28", "note": "the two-threshold variant — one recombination short of margin, but simpler; also 0/808 FP"},
+    "model_alternates_proper_n": [
+      {"hf_repo": "roberta-large-mnli", "best_fp0_pooled808": "21/28 = 0.75", "note": "355M; second-best on the pooled set — does NOT reach 28/28 there (the n=5-only 28/28 was a small-sample read)"},
+      {"hf_repo": "facebook/bart-large-mnli", "best_fp0_pooled808": "~20/28", "note": "407M; bigger ≠ better — it's the specific checkpoint (and the right aggregation)"},
+      {"hf_repo": "MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli", "best_fp0_pooled808": "11/28", "note": "184M; collapses at proper-pooled-n — do not use"},
+      {"hf_repo": "cross-encoder/nli-MiniLM2-L6-H768", "best_fp0_pooled808": "14/28", "note": "82M; the §7 #18 'cost-pick' is overturned — caps low once you demand 0 FP at a real denominator"}
     ]
   },
   "alternates": [

docs/tickets/ticket-000049-attribution-aware-grounding-check.md

@@ -20,44 +20,51 @@ trigger; do NOT enable runtime NLI demotion on the current scaffold.
 Candidate-clause restriction (§7 #21) + speedup (§7 #22 — batched
 forwards, device auto-detect, ONNX-int8 export, ~4× on CPU / seconds
 on a 4090) + a `{model × k × agg × guard × θc × θe}` grid sweep
-(§7 #23–#25, `bench/scripts/nli_shadow_grid.py`, run against bench-qa
-STRICT samples of growing size n=1=89 → n=3=275 → n=5=444) landed;
-the verdict moved **three times then settled**: §7 #22's "fails the
-gate" was a *default-config* artifact (k=6/θc=0.5/θe=0.9); §7 #23's
-"deberta-base k=2/θc=0.99 passes" was a *small-FP-sample* artifact
-(27/28 at n=5 but the n=3 sample drove it to 11/28 — sits at the
-cliff); §7 #24 (n=3) found a large model passes; **§7 #25
-(n=5-confirmed, 444 real STRICT cells)**: the lexical-candidate NLI
-veto **robustly clears the §7 #12 gate** — **`microsoft/deberta-large-mnli`
-/ k=2 / agg=max / guard=max_entail / θc≈0.96 / θe=0.9 → catches all
-28 synthetic recombinations (incl. both `5f-fal-hard` fixtures) ·
-0/444 real STRICT FP · 0/26 synthetic legit FP, ~4 pts of θc
-headroom**; `roberta-large-mnli` is an equally-good alternate
-(k=2/max/θc=0.95). Resolved findings: `agg=max + max_entail guard` is
-the robust score-shape across both proper-n samples (§7 #24's `margin`
-win was a sample tie); the *specific large checkpoint* (~350–400M:
-deberta-large-mnli / roberta-large-mnli) is what hits 1.0/0.0 —
-`bart-large` (similar size) only ~0.71, `deberta-base-184M` at the
-cliff, the small models (MiniLM-82M, deberta-v3-small) cap at ~0.82,
-so the §7 #18 "MiniLM is the cost-pick" is **overturned by the
-proper-n evidence**; k=2 is the consistent winner; int8-ONNX costs
-≈1 catch vs fp32. Remaining caveats (narrow now): the 28/28 recall is
-a *28-case* denominator — a bench-qa-derived recombination set (real
-false answers that took the paraphrase/entity path) is the one
-load-bearing check not yet done; the FP side is 444 cells at 0 FP
-(solid); n=9 (~825 cells) would add confidence but, given the ~4–5 pt
-θc margin, is "if dav1d wants more"; still SHADOW — flipping to a
-runtime demotion-only veto is fox+dav1d-decides (then `nli_policy_hash`
-= model+k+agg+guard+θc folds into `governance_policy_hash` per §7 #2);
+(§7 #23–#26, `bench/scripts/nli_shadow_grid.py`, run against bench-qa
+STRICT samples of growing size: n=1=89 → n=3=275 → n=5=444 → pooled
+n=1+3+5 = 808) + a mining pass over the 808 real STRICT answers for
+natural recombinations landed; the verdict moved **five times then
+settled**: §7 #22 "fails" = default-config artifact; §7 #23
+"deberta-base passes" = small-FP-sample artifact; §7 #24 found a large
+model passes at n=3 with `margin`; §7 #25 said "no, at n=5 plain
+`max`@0.96 works" — but §7 #26 (pooled 808, the biggest real FP
+denominator) showed *that* was the small-sample artifact: on 808 cells
+θc must go back to ~0.995 and at θc=0.995 only `margin` still catches
+28/28. **Settled: `microsoft/deberta-large-mnli` / k=3 / agg=`margin`
+(= max over top-3 candidate clauses of `p_contra − p_entail`) / θc=0.995
+→ catches all 28 synthetic recombinations (incl. both `5f-fal-hard`
+fixtures) · 0/808 pooled real STRICT FP · 0/26 synthetic legit FP** —
+the *only* config in the 7-model × full-grid sweep that hits 1.0/0.0
+on 808 cells, and it held at n=3 too (the simpler two-threshold variant
+k=2/`max`/θc=0.995 gets 27/28 · 0/808). Findings: `margin` is the right
+score-shape (discriminates a real recombination — source genuinely
+contradicts, that clause's entail≈0, margin≈0.997 — from a spurious
+token-collision FP — source also has an entailing clause, margin dragged
+to ~0.95–0.99); the specific *large* checkpoint matters (deberta-large-mnli
+wins; bart-large/deberta-base/MiniLM all collapse at proper-pooled-n,
+so the §7 #18 "MiniLM cost-pick" is overturned); int8-ONNX costs ≈1
+catch. **Mining the 808 real STRICT answers found ZERO natural
+recombination errors** — the boundary is theoretical-in-practice; the
+37 would-fires at lowered θc are all token-collision FPs (Mount Kenya
+pulled into a Kilimanjaro answer, Dalí into da Vinci, Donovan into
+Superman) — i.e. the candidate selector (top-k by token overlap) pulls
+different-entity same-subject-area clauses; `margin@0.995` survives it,
+a Phase-3 verifier-matched-clause hook would eliminate it. Remaining
+caveats: the 28/28 recall is a 28-case denominator and (since there are
+no natural recombinations to mine) the realistic next check is ~20–30
+hand-built synthetic-recombination-claim-vs-real-bench-qa-context
+fixtures (real haystack, deliberate re-attribution); still SHADOW —
+runtime promotion is fox+dav1d-decides (then `nli_policy_hash` =
+model+k+agg+guard+θc folds into `governance_policy_hash` per §7 #2);
 deberta-large is ~400M (GPU-fast). Manifest active defaults stay
-k=6/θc=0.5; the `recommended_operating_point` (deberta-large-mnli /
-k=2 / max / θc=0.96 / θe=0.9) is documented in the manifest. Standing
-lesson, sharpened thrice (codified in CLAUDE.md bench-maxing): the
-clean synthetic eval doesn't predict bench-qa precision (§7 #18→#20),
-the default config doesn't predict the best config (§7 #22→#23),
-**and a small FP-side sample doesn't predict the large-sample FP rate
-(§7 #23→#24→#25)** — every gate number is provisional until the
-denominators are big enough, and
+k=6/θc=0.5; `recommended_operating_point` = the `margin`@0.995 config.
+Standing lesson, sharpened five times (codified in CLAUDE.md
+bench-maxing): the clean synthetic eval doesn't predict bench-qa
+precision (§7 #18→#20), the default config doesn't predict the best
+config (§7 #22→#23), **and a small FP-side sample doesn't predict the
+large-sample FP rate — and a *bigger* sample can vindicate a config a
+smaller one made look unnecessary (§7 #23→#24→#25→#26)** — every gate
+number is provisional until the denominators are big enough, and
 the sweep has to be wide enough to include the config that survives
 them. Production verifier unchanged; `falsification-hard` stays 10/12.
 **Opened:** 2026-05-12
@@ -1155,3 +1162,96 @@ ones and the small ones cap at 0.82 — three samples, three sharper
 answers, ~2 hours of GPU+LLM total. A guessed config (MiniLM, per the
 n=1 read) would have shipped a 0.82-recall veto when a 1.0-recall one
 was available.
+
+**26. Mining for *real* recombinations + the pooled-808-STRICT check
+— the §7 #25 "max@0.96" was *itself* a small-sample artifact; `margin`
+is reinstated as the right aggregation, and the boundary is
+theoretical-in-practice (2026-05-12).** §7 #25 left one open item: the
+28/28 recall is on a 28-case *synthetic* set; do real LLM answers
+actually recombine? Two passes:
+
+*(a) Mine the pooled n=1+n=3+n=5 bench-qa runs (808 distinct STRICT
+answers) for natural recombinations.* Ran the candidate veto over all
+808 at a *lowered* θc≥0.7: 37 STRICT-fires (`bench/results/nli-strict-fires-mining-pooled.jsonl`).
+On inspection, **all 37 are token-collision false positives, not
+genuine recombination errors** — the candidate-clause selector
+(top-k by token overlap) pulls in *different-entity, same-subject-area*
+clauses and NLI false-fires: e.g. "where is Kilimanjaro located?" /
+"Kilimanjaro is in Tanzania" gets the clause "Mount Kenya is located
+in central Kenya" pulled in (shares `Mount`, `located`, a country
+name) → NLI reads it as a contradiction (contra 0.99); "who painted
+the Mona Lisa?" / "da Vinci painted the Mona Lisa" gets "Salvador
+Dalí … painted …" (contra 0.95); "Superman's girlfriend" gets a
+Donovan *Sunshine Superman* album-tracklist clause (contra 0.99).
+**Zero of the 808 real STRICT answers look like an LLM-recombined
+falsehood the verifier missed** — the STRICT label holds; the
+recombination boundary is, in practice on well-covered factual
+questions, a *contrived/theoretical* concern (the 2 fixtures are
+hand-built). What the mining *does* prove is that the failure mode is
+**systematic in the candidate selector** — the token-overlap proxy
+pulls subject-area-sharing-but-different-entity clauses, and NLI then
+reads "statement about a different entity" as "contradicts the answer".
+
+*(b) Re-run the grid against the pooled 808-STRICT set (the biggest
+real FP denominator we have)* (`bench/results/nli-shadow-grid-pooled808-synth-vs-realstrict.json`):
+
+| config | recombination catch | real-STRICT FP (pooled, 808 cells) |
+|---|---|---|
+| **`deberta-large-mnli` / k=3 / agg=`margin` / θc=0.995** | **28/28 = 1.000** | **0/808** |
+| `deberta-large-mnli` / k=2 / agg=`max` / θc=0.995 | 27/28 = 0.964 | 0/808 |
+| `deberta-large-mnli` / k=2 / agg=`max` / θc=**0.96** *(the §7 #25 config)* | — | **FP > 0** (the Superman/Kilimanjaro/Dalí hits at contra 0.95–0.99) |
+| `roberta-large-mnli` (best fp=0) | 21/28 = 0.750 | 0/808 |
+| `bart-large-mnli` / `deberta-base` / MiniLM (best fp=0) | ≤ 20/28 | 0/808 |
+
+**The §7 #25 "`max` @ θc=0.96" was the small-sample artifact this
+time** — the n=5 444-cell STRICT set happened not to contain the
+high-confidence spurious hits the pooled n=1+n=3 runs do, so it let
+θc drop to 0.96; on 808 cells θc has to go back up to ~0.995. And at
+θc=0.995, **`agg=margin` is the only thing that still catches 28/28**
+(`max` gets 27/28; `mean` and the others ≤0.75). So **§7 #24's
+`margin` finding is reinstated**: `margin = max_clause(p_contra −
+p_entail)` is the right score-shape because it discriminates a *real*
+recombination (the source genuinely contradicts → that clause's
+entail ≈ 0 → margin ≈ 0.996–0.998) from a *spurious token-collision*
+FP (the source *also* contains the entailing clause → its margin is
+dragged to ~0.95–0.99) — a narrow gap, but it held on 808 cells where
+plain contradiction-thresholding did not.
+
+**Settled verdict.** The lexical-candidate NLI veto clears the §7 #12
+gate with **`microsoft/deberta-large-mnli` / k=3 / agg=`margin` /
+θc=0.995** → **28/28 synthetic recombinations (incl. both `5f-fal-hard`
+fixtures) · 0/808 pooled real STRICT FP · 0/26 synthetic legit FP** —
+this is the *only* config (across the 7-model × full-grid sweep) that
+hits 1.0/0.0 on the 808-cell denominator, and it has held across n=3
+(275 cells) and the pooled 808; the §7 #25 "max@0.96" only held on the
+n=5 444-cell sample. **Open items now minimal:** (a) the 28/28 recall
+is still a 28-case denominator — but since there are *zero* natural
+recombinations in 808 real STRICT answers, a "bench-qa-derived
+recombination set" can't be mined (there's nothing to mine); the
+realistic next step is *synthetic recombination claims constructed
+against real bench-qa contexts* (real haystack, deliberate
+re-attribution) — that tests "the candidate-selector + veto vs a real
+30 KB haystack" on the recall side, and is doable as ~20–30
+hand-built fixtures. (b) The systematic candidate-selector FP failure
+(different-entity clauses pulled in by token overlap) is *survived*
+by `margin@0.995` on real traffic but would be *eliminated* by a
+Phase-3 verifier-matched-clause hook (pass NLI only the clause the
+verifier actually grounded the answer against) — `margin@0.995` is
+robust enough to ship as-is, the Phase-3 hook is a "clean it up
+later". (c) Still SHADOW; runtime promotion is fox+dav1d-decides
+(then `nli_policy_hash` = model+k+agg+guard+θc folds into
+`governance_policy_hash`); deberta-large is ~400M (GPU-fast).
+Manifest active defaults stay k=6/θc=0.5; `recommended_operating_point`
+reverted to the `margin`@0.995 config. Production verifier unchanged;
+`falsification-hard` stays 10/12.
+
+*Meta-lesson, instance five (and the one that cuts the other way):*
+§7 #18→#20→#22→#23→#24→#25→#26 — clean eval ≠ bench-qa precision; the
+default config ≠ the best config; a small FP sample ≠ the large-sample
+FP rate **— and a *bigger* sample can vindicate a config a smaller
+one made look unnecessary** (n=5=444 said "drop θc to 0.96, use plain
+`max`"; pooled=808 said "no — `margin`@0.995 was right all along").
+The honest discipline: don't trust *any* denominator until it's the
+biggest you can afford, and re-confirm the *config choice* (not just
+the threshold) every time the denominator grows. Pooled 808 STRICT ×
+the full grid was ~4 min on the 4090.