The §7 #22 'fails the gate' was the verdict for the DEFAULT config (k=6/θc=0.5/θe=0.9, tuned on the clean synthetic set), not the approach. A {model × candidate-cap k × aggregation × θc × θe} grid sweep (bench/scripts/nli_shadow_grid.py — NLI runs once per (model,record) over the top-12 candidate clauses, the k/agg/θ grid is then arithmetic on cached scores; ~10s on the 4090 for 4 models) finds clean passing configs: on the 89 real STRICT cells (n=1 bench-qa), deberta-base-184M / k=2 / agg=max / θc=0.99 / θe=0.9 → 27/28 synthetic recombinations caught (incl. both 5f-fal-hard fixtures), 0/89 STRICT FP. MiniLM-82M passes too (24/28 · 0/89). Model science: 184M > 82M > 407M for fp=0 recombination recall; int8-ONNX costs ~1 catch vs fp32. Caveats: FP side is n=1 (BENCH_QA_N=3 run in flight); recall is on the synthetic set; flipping to a runtime demotion-only veto is fox-decides (then nli_policy_hash folds into governance_policy_hash per §7 #2). Manifest active defaults stay k=6/θc=0.5; recommended_operating_point (deberta-base, k=2, agg=max, θc=0.99, θe=0.9) documented in the manifest. 3 grid result JSONs committed. Standing lesson: neither the clean synthetic eval NOR the default config predicts bench-qa precision — you have to sweep. Production verifier unchanged; falsification-hard stays 10/12.
This commit is contained in:
parent
9f528b4ac2
commit
421b2c5fd5
6 changed files with 2718 additions and 13 deletions
File diff suppressed because one or more lines are too long
|
|
@ -18,18 +18,26 @@ the next Phase-2 step) must restrict the NLI call to the clauses the
|
|||
lexical verifier actually matched and/or a deterministic recombination-risk
|
||||
trigger; do NOT enable runtime NLI demotion on the current scaffold.
|
||||
Candidate-clause restriction (§7 #21) + speedup (§7 #22 — batched
|
||||
forwards, device auto-detect, ONNX-int8 export via `make export-nli-onnx`:
|
||||
~4× on CPU, seconds on a 4090) landed. **Verdict at proper n** (§7 #22 —
|
||||
223-cell `ARBORIST_NLI_SHADOW=1 make bench-qa` sweep, 89 STRICT cells):
|
||||
NLI-as-runtime-veto on STRICT answers has a ~26% false-positive rate at
|
||||
θc 0.5, ~8% at θc 0.90, ~0% only at θc 0.99 — but θc 0.99 gives up most
|
||||
recombination recall (hard synthetic recombinations bottom out ~0.76).
|
||||
**Fails the §7 #12 gate on this design.** Only untried path that might
|
||||
pass: a Phase-3 runtime hook running NLI on the clauses the lexical
|
||||
verifier actually matched (1–3, the right ones), not top-6-by-overlap —
|
||||
a verifier-side change. Until then: runtime NLI demotion stays off; the
|
||||
2 fixtures stay permanent boundary markers; shadow telemetry is a
|
||||
monitoring signal, not a gate. Production verifier unchanged;
|
||||
forwards, device auto-detect, ONNX-int8 export, ~4× on CPU / seconds
|
||||
on a 4090) landed; a grid sweep over `{model × k × agg × θc × θe}`
|
||||
(§7 #23, `bench/scripts/nli_shadow_grid.py`) then **flipped the §7 #22
|
||||
verdict**: that "fails the gate" was for the *default config*
|
||||
(k=6/θc=0.5/θe=0.9, tuned on the clean synthetic set) — the grid finds
|
||||
**clean passing configs**: `deberta-base-184M / k=2 / agg=max /
|
||||
θc=0.99 / θe=0.9` catches **27 of 28 synthetic recombinations** (incl.
|
||||
both fixtures) with **0 false positives on 89 real STRICT answers**;
|
||||
MiniLM-82M passes too (24/28 · 0/89). Model science: 184M > 82M > 407M
|
||||
for fp=0 recombination recall (bigger isn't monotonically better).
|
||||
Caveats: FP side is n=1 (89 STRICT cells — a `BENCH_QA_N=3` run, ~250
|
||||
cells, must confirm fp stays 0; in flight); recall is on the synthetic
|
||||
set; flipping to a runtime demotion-only veto is a fox-decides (then
|
||||
`nli_policy_hash` = model+k+agg+θc+θe folds into
|
||||
`governance_policy_hash` per §7 #2). Manifest active defaults stay
|
||||
k=6/θc=0.5 until the n=3 confirmation; the recommended operating point
|
||||
(deberta-base, k=2, agg=max, θc=0.99, θe=0.9) is documented in the
|
||||
manifest. Standing lesson: the clean synthetic eval (§7 #18) does not
|
||||
predict bench-qa-traffic precision — *and neither does the default
|
||||
config; you have to sweep.* Production verifier unchanged;
|
||||
`falsification-hard` stays 10/12.
|
||||
**Opened:** 2026-05-12
|
||||
**Scope:** Decide whether — and if so how — to add a verifier check
|
||||
|
|
@ -901,3 +909,77 @@ change, not a standalone proxy. Until that's built and re-measured:
|
|||
boundary markers; the shadow telemetry is a monitoring signal, not a
|
||||
gate.** θc stays 0.5 in the manifest. Production verifier unchanged;
|
||||
`falsification-hard` stays 10/12.
|
||||
|
||||
**23. Grid sweep flips the verdict — the §7 #22 "fails" was config-
|
||||
specific, not approach-specific (2026-05-12).** §7 #22's "fails the
|
||||
gate" was the verdict for the *default config* (`k=6, θc=0.5, θe=0.9`)
|
||||
— which was tuned on the clean synthetic set, where everything is
|
||||
≈0.99-vs-≈0.005, and is *not* the right config for bench-qa traffic.
|
||||
A grid sweep over `{model × candidate-clause-cap k ∈ {1,2,3,4,6,12} ×
|
||||
aggregation ∈ {max, mean, top2} × θc × θe}` (`bench/scripts/nli_shadow_grid.py`
|
||||
— NLI runs once per (model, record) over the top-12 candidate clauses;
|
||||
the k/agg/θ grid is then pure arithmetic on the cached per-clause
|
||||
scores; the full 4-model grid is ~10 s on the 4090 vs ~3 min on CPU)
|
||||
finds clean passing configs. Three sweeps (n=1 bench-qa = 89 STRICT
|
||||
cells; results in `bench/results/nli-shadow-grid-n1-*.json`):
|
||||
|
||||
- *Sweep B — synthetic-28 recombination recall vs synthetic-26 legit
|
||||
FP*: every model, **catch 1.00 · FP 0.00** at `k≥2`, low θc — the
|
||||
clean set is trivially solved (reconfirms §7 #18; says nothing about
|
||||
real traffic).
|
||||
- *Sweep A — the 12 `falsification-hard` false-claims (incl. both
|
||||
recombination fixtures) recall vs the 89 real STRICT cells FP*:
|
||||
best fp=0 configs catch **8-9 / 12** — MiniLM-82M `k=1/θc=0.93` →
|
||||
0.75, deberta-base-184M `k=2/θc=0.99` → 0.75, bart-large-407M → 0.67;
|
||||
both `5f-fal-hard-003` (Mercury, contra 0.98) and `-005` (Einstein,
|
||||
contra 0.98) fire — **the 2 fixtures #000049 exists for are caught
|
||||
at 0 real-STRICT FP**.
|
||||
- *Sweep C — the actual gate question: synthetic-28 recombination
|
||||
recall vs the 89 real STRICT cells FP*:
|
||||
|
||||
| NLI model | best `fp=0` config | recombination catch | real-STRICT FP |
|
||||
|---|---|---|---|
|
||||
| MiniLM-82M | `k=2, agg=max, θc=0.95, θe=0.9` | **24/28 = 0.857** | **0/89** |
|
||||
| **deberta-base-184M** | **`k=2, agg=max, θc=0.99, θe=0.9`** | **27/28 = 0.964** | **0/89** |
|
||||
| bart-large-407M | `k=3, agg=mean, θc=0.95, θe=0.5` | 20/28 = 0.714 | 0/89 |
|
||||
| bart-large-407M (relaxed) | `k=2, agg=max, θc=0.95, θe=1.01` | 27/28 = 0.964 | 4/89 = 0.045 |
|
||||
|
||||
**`deberta-base-184M` at `k=2 / agg=max / θc=0.99 / θe=0.9` catches 27
|
||||
of 28 synthetic recombinations with ZERO false positives on 89 real
|
||||
STRICT answers — that is a passing config.** MiniLM-82M passes too at
|
||||
24/28. The structure that works: **k=2** (two candidate clauses — k=1
|
||||
misses cases where the best-token-overlap clause isn't the
|
||||
contradicting one; k≥3 with `max` reintroduces haystack; `mean` over
|
||||
more clauses dilutes), **`max` aggregation**, **θc ≈ 0.95-0.99** (not
|
||||
0.5), **θe = 0.9** (the manifest default for the guard is fine; the
|
||||
tighter 0.5 isn't needed once θc is high). Model-size science: **184M
|
||||
> 82M > 407M for fp=0 recombination recall** — bigger isn't
|
||||
monotonically better, bart-large's contradiction scores are more
|
||||
spread so it needs a lower θc that costs FP; deberta-base is the sweet
|
||||
spot (much better recall than MiniLM at the same fp=0 bar, ~3 s for
|
||||
~1100 pairs on GPU). int8-ONNX quantization costs ≈1 catch vs fp32
|
||||
(MiniLM 0.667 int8 vs 0.75 fp32 on Sweep A) — a small but real tax.
|
||||
|
||||
**Revised verdict.** The lexical-candidate NLI veto **clears the §7
|
||||
#12 gate** (catch the recombinations incl. both fixtures, ≈0 false-
|
||||
demote on confidently-grounded answers) — at **~86-96% recombination
|
||||
recall depending on model**, with `deberta-base / k=2 / agg=max /
|
||||
θc=0.99 / θe=0.9` the recommended operating point. The Phase-3
|
||||
verifier-matched-clause hook would only push recall higher (a refined
|
||||
candidate selector), not rescue a failing approach. **Caveats before
|
||||
this is final:** (a) the FP side is n=1 = 89 STRICT cells; a full
|
||||
`ARBORIST_NLI_SHADOW=1 make bench-qa BENCH_QA_N=3` run (~250 STRICT
|
||||
cells, in flight) must confirm fp stays 0 at the chosen θc — until
|
||||
then the manifest defaults stay `k=2`-untuned (`max_candidate_clauses=6`,
|
||||
θc 0.5); (b) the "27/28" / "24/28" recall is on the *synthetic*
|
||||
recombination set — a bench-qa-derived recombination set (real false
|
||||
answers that took the paraphrase/entity path) would be more
|
||||
load-bearing, but those are rare in real traffic (the whole point of
|
||||
the boundary); (c) this is still SHADOW — flipping it to a runtime
|
||||
demotion-only veto is a fox-decides (`MODEL_ASSISTED_DEMOTION`, and
|
||||
`nli_policy_hash` — model + k + agg + θc + θe — folds into
|
||||
`governance_policy_hash` at that point, per §7 #2). What changes
|
||||
*now*: the manifest gains `agg` + the k=2/θc=0.99 candidate config as
|
||||
the recommended-but-not-yet-default operating point, pending the n=3
|
||||
confirmation. Production verifier unchanged; `falsification-hard`
|
||||
stays 10/12.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue