diff --git a/CLAUDE.md b/CLAUDE.md index e11ad6f..1ab9144 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -451,6 +451,29 @@ Full discipline + worked examples in `docs/bench-maxing.md`. Headlines: the runtime wins. Pointer IDs, runtime-interpolated spans, evidence maps, policy hashes, hard verifier checks all move authority OUT of the model's prior and INTO runtime artifacts. +- **When researching a config/model decision: sweep wide, sweep + deep, on real-traffic-sized samples — don't guess, and don't trust + small denominators.** Three times running (ticket #000049 §7 + #18→#24) the answer flipped: a clean synthetic eval said one thing, + bench-qa pipeline output said another; the *default* config said one + thing, a `{model × hyperparams}` grid sweep said another; an n=1 FP + sample said "this config passes", an n=3 FP sample said "that one + collapses — a *different* config passes". So: (1) gate numbers come + from bench-qa pipeline output, never contrived fixtures; (2) the + sweep must be wide enough — multiple checkpoints, the full + hyperparam grid; there is no "bigger model is better" law, the + *specific checkpoint* and the *score-shape* (e.g. a single-threshold + margin vs a two-threshold rule) dominate — wide enough to include + the config that survives; (3) every gate number is provisional until + the denominators are big enough — bump `BENCH_QA_N` and re-confirm + before promoting anything. The GPU box (`ai`, the 4090 — `make + bootstrap-nli-only`, then `ARBORIST_NLI_DEVICE` auto-detects cuda) + makes this cheap: a 7-model × full-grid × ~300-record sweep is + ~3.5 min. **That electricity is well spent — burn it; a guessed + config that ships is far more expensive than a sweep that doesn't.** + Heavy off-device passes (NLI sweeps, embedding backfills) run on the + GPU producer, never in arborist's `python+sqlite3` core (cf. the + #000051 vecpack pattern). ## Docs index