From a34ba8be712cc4d349b9d7fef8ad4b379eb4caa3 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Tue, 12 May 2026 18:45:25 -0400 Subject: [PATCH] =?UTF-8?q?CLAUDE.md=20bench-maxing:=20when=20researching?= =?UTF-8?q?=20a=20config/model=20decision,=20sweep=20wide+deep=20on=20real?= =?UTF-8?q?-traffic-sized=20samples=20=E2=80=94=20the=20#000049=20=C2=A77?= =?UTF-8?q?=20#18=E2=86=92#24=20lesson=20(clean=20eval=20=E2=89=A0=20bench?= =?UTF-8?q?-qa=20precision;=20default=20=E2=89=A0=20best=20config;=20small?= =?UTF-8?q?=20FP=20sample=20=E2=89=A0=20large=20FP=20rate;=20no=20'bigger?= =?UTF-8?q?=20model'=20law=20=E2=80=94=20specific=20checkpoint=20+=20score?= =?UTF-8?q?-shape=20dominate;=20gate=20numbers=20provisional=20until=20den?= =?UTF-8?q?ominators=20are=20big=20=E2=80=94=20bump=20BENCH=5FQA=5FN;=20th?= =?UTF-8?q?e=204090=20makes=20wide=20sweeps=20cheap,=20burn=20the=20electr?= =?UTF-8?q?icity;=20heavy=20passes=20on=20the=20GPU=20producer=20not=20the?= =?UTF-8?q?=20core)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CLAUDE.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index e11ad6f..1ab9144 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -451,6 +451,29 @@ Full discipline + worked examples in `docs/bench-maxing.md`. Headlines: the runtime wins. Pointer IDs, runtime-interpolated spans, evidence maps, policy hashes, hard verifier checks all move authority OUT of the model's prior and INTO runtime artifacts. +- **When researching a config/model decision: sweep wide, sweep + deep, on real-traffic-sized samples — don't guess, and don't trust + small denominators.** Three times running (ticket #000049 §7 + #18→#24) the answer flipped: a clean synthetic eval said one thing, + bench-qa pipeline output said another; the *default* config said one + thing, a `{model × hyperparams}` grid sweep said another; an n=1 FP + sample said "this config passes", an n=3 FP sample said "that one + collapses — a *different* config passes". So: (1) gate numbers come + from bench-qa pipeline output, never contrived fixtures; (2) the + sweep must be wide enough — multiple checkpoints, the full + hyperparam grid; there is no "bigger model is better" law, the + *specific checkpoint* and the *score-shape* (e.g. a single-threshold + margin vs a two-threshold rule) dominate — wide enough to include + the config that survives; (3) every gate number is provisional until + the denominators are big enough — bump `BENCH_QA_N` and re-confirm + before promoting anything. The GPU box (`ai`, the 4090 — `make + bootstrap-nli-only`, then `ARBORIST_NLI_DEVICE` auto-detects cuda) + makes this cheap: a 7-model × full-grid × ~300-record sweep is + ~3.5 min. **That electricity is well spent — burn it; a guessed + config that ships is far more expensive than a sweep that doesn't.** + Heavy off-device passes (NLI sweeps, embedding backfills) run on the + GPU producer, never in arborist's `python+sqlite3` core (cf. the + #000051 vecpack pattern). ## Docs index