#000049 §7 #25: n=5 confirmation (444 STRICT cells) — verdict settles; the large models are the robust ones, MiniLM-cost-pick overturned
ARBORIST_NLI_SHADOW=1 make bench-qa BENCH_QA_N=5 → 1125 cells, 444 real STRICT (5x n=1, 1.6x n=3). Re-ran the 7-model mega-grid: the lexical- candidate NLI veto robustly clears the §7 #12 gate with microsoft/deberta-large-mnli / k=2 / agg=max / guard=max_entail / θc≈0.96 / θe=0.9 → 28/28 synthetic recombinations (incl. both fixtures) · 0/444 real STRICT FP · 0/26 synthetic legit FP, ~4 pts θc headroom; roberta-large-mnli equally good (k=2/max/θc=0.95). Resolved: agg=max + max_entail guard is the robust score-shape (§7 #24's margin win was a sample tie); the large checkpoints (~350-400M deberta-large-mnli / roberta-large-mnli) hit 1.0/0.0, bart-large (similar size) only ~0.71, deberta-base at the cliff (27/28 here, 11/28 at n=3), small models (MiniLM-82M, deberta-v3-small) cap at ~0.82 — so §7 #18's 'MiniLM is the cost-pick' is OVERTURNED by the proper-n evidence; k=2 consistent winner; int8-ONNX costs ~1 catch. recommended_operating_point updated. Remaining: a bench-qa-derived recombination set (the one check not done); n=9 if dav1d wants more; still SHADOW; runtime promotion is fox+dav1d-decides. Production verifier unchanged; falsification-hard stays 10/12. This run is the worked example behind CLAUDE.md's new bench-maxing line.
This commit is contained in:
parent
221b784a80
commit
d9760bc5bb
3 changed files with 10098 additions and 40 deletions
|
|
@ -20,38 +20,44 @@ trigger; do NOT enable runtime NLI demotion on the current scaffold.
|
|||
Candidate-clause restriction (§7 #21) + speedup (§7 #22 — batched
|
||||
forwards, device auto-detect, ONNX-int8 export, ~4× on CPU / seconds
|
||||
on a 4090) + a `{model × k × agg × guard × θc × θe}` grid sweep
|
||||
(§7 #23–#24, `bench/scripts/nli_shadow_grid.py`) landed; the verdict
|
||||
moved **twice**: §7 #22's "fails the gate" was a *default-config*
|
||||
artifact (k=6/θc=0.5/θe=0.9); §7 #23's "deberta-base k=2/θc=0.99 →
|
||||
27/28 · 0/89 passes" was a *small-FP-sample* artifact (it collapses
|
||||
to 11/28 on the n=3 275-STRICT sample, which forces θc up to ~0.995);
|
||||
the n=3-confirmed truth (§7 #24, the "no stone unturned" mega-grid —
|
||||
7 models incl. 4 extra xsmall→large, the new `margin` aggregation =
|
||||
max over clauses of `p_contra − p_entail`): **`microsoft/deberta-large-mnli`
|
||||
/ k=3 / agg=`margin` / θc=0.995 → catches all 28 synthetic
|
||||
recombinations (incl. both `5f-fal-hard` fixtures) + 8/12 of the
|
||||
`falsification-hard` pack, with 0 false positives on 275 real STRICT
|
||||
answers and 0 on the 26 synthetic legit summaries** — a passing config
|
||||
at proper n. Findings: (1) `margin` is the right score-shape (single
|
||||
threshold, folds the entailment guard in); (2) the *specific
|
||||
checkpoint* matters more than parameter count — `microsoft/deberta-large-mnli`
|
||||
wins clean, `deberta-base` collapses, `roberta-large`/`bart-large`
|
||||
(similar size) only reach ~0.71–0.75, there's no "bigger is better"
|
||||
law. Caveats: still SHADOW; the 28/28 recall is a 28-case denominator
|
||||
(100%-of-28 ≠ 100%-in-general — a bench-qa-derived recombination set
|
||||
is the next check), and θc=0.995/0 FP is "on the largest STRICT sample
|
||||
we have (275 cells)" not "0 FP, period" — a bigger STRICT sample
|
||||
should be the other next check before runtime promotion; flipping to
|
||||
a runtime demotion-only veto is fox+dav1d-decides (then `nli_policy_hash`
|
||||
= model+k+agg+θc folds into `governance_policy_hash` per §7 #2);
|
||||
deberta-large is ~400M (GPU-fast, slow on CPU). Manifest active
|
||||
defaults stay k=6/θc=0.5; the `recommended_operating_point`
|
||||
(deberta-large-mnli / k=3 / margin / θc=0.995) is documented in the
|
||||
manifest. Standing lesson, sharpened twice: the clean synthetic eval
|
||||
doesn't predict bench-qa precision (§7 #18→#20), the default config
|
||||
doesn't predict the best config (§7 #22→#23), **and a small FP-side
|
||||
sample doesn't predict the large-sample FP rate (§7 #23→#24)** — every
|
||||
gate number is provisional until the denominators are big enough, and
|
||||
(§7 #23–#25, `bench/scripts/nli_shadow_grid.py`, run against bench-qa
|
||||
STRICT samples of growing size n=1=89 → n=3=275 → n=5=444) landed;
|
||||
the verdict moved **three times then settled**: §7 #22's "fails the
|
||||
gate" was a *default-config* artifact (k=6/θc=0.5/θe=0.9); §7 #23's
|
||||
"deberta-base k=2/θc=0.99 passes" was a *small-FP-sample* artifact
|
||||
(27/28 at n=5 but the n=3 sample drove it to 11/28 — sits at the
|
||||
cliff); §7 #24 (n=3) found a large model passes; **§7 #25
|
||||
(n=5-confirmed, 444 real STRICT cells)**: the lexical-candidate NLI
|
||||
veto **robustly clears the §7 #12 gate** — **`microsoft/deberta-large-mnli`
|
||||
/ k=2 / agg=max / guard=max_entail / θc≈0.96 / θe=0.9 → catches all
|
||||
28 synthetic recombinations (incl. both `5f-fal-hard` fixtures) ·
|
||||
0/444 real STRICT FP · 0/26 synthetic legit FP, ~4 pts of θc
|
||||
headroom**; `roberta-large-mnli` is an equally-good alternate
|
||||
(k=2/max/θc=0.95). Resolved findings: `agg=max + max_entail guard` is
|
||||
the robust score-shape across both proper-n samples (§7 #24's `margin`
|
||||
win was a sample tie); the *specific large checkpoint* (~350–400M:
|
||||
deberta-large-mnli / roberta-large-mnli) is what hits 1.0/0.0 —
|
||||
`bart-large` (similar size) only ~0.71, `deberta-base-184M` at the
|
||||
cliff, the small models (MiniLM-82M, deberta-v3-small) cap at ~0.82,
|
||||
so the §7 #18 "MiniLM is the cost-pick" is **overturned by the
|
||||
proper-n evidence**; k=2 is the consistent winner; int8-ONNX costs
|
||||
≈1 catch vs fp32. Remaining caveats (narrow now): the 28/28 recall is
|
||||
a *28-case* denominator — a bench-qa-derived recombination set (real
|
||||
false answers that took the paraphrase/entity path) is the one
|
||||
load-bearing check not yet done; the FP side is 444 cells at 0 FP
|
||||
(solid); n=9 (~825 cells) would add confidence but, given the ~4–5 pt
|
||||
θc margin, is "if dav1d wants more"; still SHADOW — flipping to a
|
||||
runtime demotion-only veto is fox+dav1d-decides (then `nli_policy_hash`
|
||||
= model+k+agg+guard+θc folds into `governance_policy_hash` per §7 #2);
|
||||
deberta-large is ~400M (GPU-fast). Manifest active defaults stay
|
||||
k=6/θc=0.5; the `recommended_operating_point` (deberta-large-mnli /
|
||||
k=2 / max / θc=0.96 / θe=0.9) is documented in the manifest. Standing
|
||||
lesson, sharpened thrice (codified in CLAUDE.md bench-maxing): the
|
||||
clean synthetic eval doesn't predict bench-qa precision (§7 #18→#20),
|
||||
the default config doesn't predict the best config (§7 #22→#23),
|
||||
**and a small FP-side sample doesn't predict the large-sample FP rate
|
||||
(§7 #23→#24→#25)** — every gate number is provisional until the
|
||||
denominators are big enough, and
|
||||
the sweep has to be wide enough to include the config that survives
|
||||
them. Production verifier unchanged; `falsification-hard` stays 10/12.
|
||||
**Opened:** 2026-05-12
|
||||
|
|
@ -1082,3 +1088,70 @@ sweep has to be wide enough to include the config that survives them.
|
|||
The 4090 is what makes that affordable: this whole §7 #24 run — 7
|
||||
models × the full {k × agg × guard × θc × θe} grid × 303 records — was
|
||||
~3.5 min.
|
||||
|
||||
**25. The n=5 confirmation — it firms up, and the right config + model
|
||||
crystallize (2026-05-12).** `ARBORIST_NLI_SHADOW=1 make bench-qa
|
||||
BENCH_QA_N=5` → 1125 cells, **444 STRICT** (5× the n=1 sample, 1.6×
|
||||
the n=3). Re-ran the full 7-model mega-grid against the synthetic-28
|
||||
recombination set vs the 444 real STRICT cells
|
||||
(`bench/results/nli-shadow-grid-n5-synth-vs-realstrict.json`):
|
||||
|
||||
| NLI model (size) | best `fp=0` config | recombination catch | real-STRICT FP (n=5, 444 cells) | θc margin |
|
||||
|---|---|---|---|---|
|
||||
| `roberta-large-mnli` (~355M) | k=2, agg=max, θc=0.95 | **28/28 = 1.000** | **0/444** | ~5 pts |
|
||||
| `microsoft/deberta-large-mnli` (~400M) | k=2, agg=max, θc=0.96 | **28/28 = 1.000** | **0/444** | ~4 pts |
|
||||
| `deberta-base-184M` | k=2, agg=max, θc=0.99 | 27/28 = 0.964 | 0/444 | **edge** (28/28 needs θc=0.975 → fp 0.004) |
|
||||
| MiniLM-82M | k=2, agg=max, θc=0.98 | 23/28 = 0.821 | 0/444 | — |
|
||||
| `nli-deberta-v3-small` (~140M) | k=2, agg=max, θc=0.998 | 23/28 = 0.821 | 0/444 | — |
|
||||
| `bart-large-mnli` (~407M) | k=3, agg=mean, θc=0.94 | 20/28 = 0.714 | 0/444 | — |
|
||||
|
||||
**What resolved:**
|
||||
- *Model.* The **large models (~350–400M) are the robust ones**:
|
||||
`roberta-large-mnli` and `microsoft/deberta-large-mnli` both hit
|
||||
**28/28 · 0/444 at modest θc (0.95–0.96)** — i.e. ~4–5 points of θc
|
||||
margin before they'd start false-demoting. `deberta-large-mnli` is
|
||||
the more-validated pick (1.0/0.0 at *both* n=3 [θc=0.995, margin
|
||||
agg] and n=5 [θc=0.96, plain max]). **`deberta-base-184M` sits right
|
||||
at the cliff** — 27/28 at θc=0.99 on this sample, but the §7 #24 n=3
|
||||
sample drove it to 11/28; its config is sample-sensitive. The small
|
||||
models cap at ~0.82. So the §7 #18 "MiniLM-82M is the cost-pick" is
|
||||
**overturned by the proper-n evidence** — at the fp=0 bar MiniLM
|
||||
caps at 0.82, the large model gets 1.0; cost was the wrong axis to
|
||||
optimize once you demand 0 false-demote.
|
||||
- *Aggregation.* `agg=max` + the `max_entail` guard + θc≈0.95 is the
|
||||
robust config across *all* models and both proper-n samples; §7 #24's
|
||||
`margin` win was a sample-specific tie — keep `max`.
|
||||
- *k.* `k=2` is the consistent winner everywhere.
|
||||
|
||||
**Verdict (n=5-confirmed).** The lexical-candidate NLI veto **passes
|
||||
the §7 #12 gate, robustly**, with **`microsoft/deberta-large-mnli` /
|
||||
k=2 / agg=max / guard=max_entail / θc≈0.96 / θe=0.9** → 28/28 synthetic
|
||||
recombinations (incl. both `5f-fal-hard` fixtures) · 0/444 real STRICT
|
||||
FP, ~4 pts of θc headroom; `roberta-large-mnli` is an equally-good
|
||||
alternate. The remaining caveats are now narrow: (a) the 28/28 recall
|
||||
is still a *28-case* denominator — a bench-qa-derived recombination set
|
||||
(real false answers that took the paraphrase/entity path) is the one
|
||||
load-bearing check not yet done (those are rare in real traffic — the
|
||||
whole reason this is a "boundary"); (b) the FP side is 444 cells at 0
|
||||
FP, a solid denominator — n=9 (~825 cells) would add confidence but,
|
||||
given the ~4–5 pt θc margin, is unlikely to move the verdict (n=9 is
|
||||
"if dav1d wants more"); (c) still SHADOW — flipping to a runtime
|
||||
demotion-only veto is fox+dav1d-decides, then `nli_policy_hash` =
|
||||
model+k+agg+guard+θc folds into `governance_policy_hash` (§7 #2);
|
||||
deberta-large is ~400M (GPU-fast; ~200–300 ms/pair on CPU on the now-
|
||||
small k=2 candidate set; ONNX-int8 would cost a catch or two). The
|
||||
manifest's `recommended_operating_point` is updated to this config;
|
||||
active defaults stay k=6/θc=0.5 until fox promotes. Production verifier
|
||||
unchanged; `falsification-hard` stays 10/12.
|
||||
|
||||
**This `make bench-qa BENCH_QA_N=5` run (~50 min Hermes) + the 7-model
|
||||
mega-grid on it (~3 min on the 4090) is the worked example behind
|
||||
CLAUDE.md's new bench-maxing line** ("when researching a config/model
|
||||
decision, sweep wide+deep on real-traffic-sized samples; the GPU makes
|
||||
it cheap; burn the electricity"). The n=1→n=3→n=5 sequence is exactly
|
||||
why: n=1 said MiniLM/deberta-base pass at high recall; n=3 said
|
||||
deberta-base collapses; n=5 said the *large* models are the robust
|
||||
ones and the small ones cap at 0.82 — three samples, three sharper
|
||||
answers, ~2 hours of GPU+LLM total. A guessed config (MiniLM, per the
|
||||
n=1 read) would have shipped a 0.82-recall veto when a 1.0-recall one
|
||||
was available.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue