CLAUDE.md bench-maxing: when researching a config/model decision, sweep wide+deep on real-traffic-sized samples — the #000049 §7 #18→#24 lesson (clean eval ≠ bench-qa precision; default ≠ best config; small FP sample ≠ large FP rate; no 'bigger model' law — specific checkpoint + score-shape dominate; gate numbers provisional until denominators are big — bump BENCH_QA_N; the 4090 makes wide sweeps cheap, burn the electricity; heavy passes on the GPU producer not the core)

CLAUDE.md

@@ -451,6 +451,29 @@ Full discipline + worked examples in `docs/bench-maxing.md`. Headlines:
   the runtime wins. Pointer IDs, runtime-interpolated spans, evidence
   maps, policy hashes, hard verifier checks all move authority OUT
   of the model's prior and INTO runtime artifacts.
+- **When researching a config/model decision: sweep wide, sweep
+  deep, on real-traffic-sized samples — don't guess, and don't trust
+  small denominators.** Three times running (ticket #000049 §7
+  #18→#24) the answer flipped: a clean synthetic eval said one thing,
+  bench-qa pipeline output said another; the *default* config said one
+  thing, a `{model × hyperparams}` grid sweep said another; an n=1 FP
+  sample said "this config passes", an n=3 FP sample said "that one
+  collapses — a *different* config passes". So: (1) gate numbers come
+  from bench-qa pipeline output, never contrived fixtures; (2) the
+  sweep must be wide enough — multiple checkpoints, the full
+  hyperparam grid; there is no "bigger model is better" law, the
+  *specific checkpoint* and the *score-shape* (e.g. a single-threshold
+  margin vs a two-threshold rule) dominate — wide enough to include
+  the config that survives; (3) every gate number is provisional until
+  the denominators are big enough — bump `BENCH_QA_N` and re-confirm
+  before promoting anything. The GPU box (`ai`, the 4090 — `make
+  bootstrap-nli-only`, then `ARBORIST_NLI_DEVICE` auto-detects cuda)
+  makes this cheap: a 7-model × full-grid × ~300-record sweep is
+  ~3.5 min. **That electricity is well spent — burn it; a guessed
+  config that ships is far more expensive than a sweep that doesn't.**
+  Heavy off-device passes (NLI sweeps, embedding backfills) run on the
+  GPU producer, never in arborist's `python+sqlite3` core (cf. the
+  #000051 vecpack pattern).
 
 ## Docs index