engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000049 §7 #24: n=3 confirmation — §7 #23 was a small-FP-sample artifact; the no-stone-unturned mega-grid found the config that survives

Browse source 
ARBORIST_NLI_SHADOW=1 make bench-qa BENCH_QA_N=3 → 275 real STRICT cells
(3x the n=1 sample). Re-ran the expanded grid (7 aggregations incl.
margin = max_clause(p_contra - p_entail), paired-entail guard variant,
θc to 0.999, --extra-models) over all manifest models + 4 extra
xsmall→large (microsoft/deberta-large-mnli, roberta-large-mnli,
nli-deberta-v3-small, deberta-v3-xsmall), synth-28 recombination vs the
275 STRICT cells. Result: the §7 #23 deberta-base/k=2/θc=0.99 config
does NOT survive — it catches only 11/28 at θc=0.995 (which the larger
STRICT sample forces). BUT the broader sweep found the config that does:
microsoft/deberta-large-mnli / k=3 / agg=margin / θc=0.995 → 28/28
synthetic recombinations (incl. both fixtures) + 8/12 falsification-hard,
0/275 real STRICT FP, 0/26 synthetic legit FP — a passing config at
proper n. Findings: margin is the right score-shape (single threshold,
folds the guard in); the specific checkpoint matters more than param
count (deberta-large-mnli wins clean, deberta-base collapses,
roberta/bart ~0.71-0.75 — no 'bigger is better' law). recommended_operating_point
updated. Still SHADOW; runtime promotion needs a bigger STRICT sample +
a bigger recombination set + fox/dav1d sign-off. Production verifier
unchanged; falsification-hard stays 10/12.

Meta-lesson sharpened twice: clean eval ≠ bench-qa precision (§7 #18→#20),
default config ≠ best config (§7 #22→#23), small FP sample ≠ large FP
rate (§7 #23→#24).
This commit is contained in:
russell@unturf.com 2026-05-12 18:38:43 -04:00
parent c452f0d58a
commit 824b1bc21c
No known key found for this signature in database
4 changed files with 10149 additions and 29 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/TICKETS.md
View file

File diff suppressed because one or more lines are too long