docs(CLAUDE.md): bench-maxing — mine ground-truth recall@k when a class is sub-noise-floor
Compressed headline: when a lever's failure class is ≤3-5/75 q the curated n=3 audit_mode bench can't resolve it (four 2026-05-18 hypotheses died there). Fix the instrument: mine ground-truth- carrying questions (bench/mine_questions.py) graded by deterministic retrieval recall@k via `query --dry-run` (bench/recall_at_k.py) — no LLM/verifier/n=3-noise/5pp-floor, ±1-q resolution. Caveats codified: measures retrieval surfacing (necessary-not-sufficient for STRICT); complements never replaces the curated adversarial trap gate.
This commit is contained in:
parent
a3ac6539c1
commit
98a4d61de9
1 changed files with 14 additions and 0 deletions
14
CLAUDE.md
14
CLAUDE.md
|
|
@ -469,6 +469,20 @@ steps 3-5.
|
|||
Full discipline + worked examples in `docs/bench-maxing.md`. Headlines:
|
||||
|
||||
- Bench before AND after every change (n=3, signal under 5pp is noise).
|
||||
- **When a lever's failure class is sub-noise-floor, fix the
|
||||
instrument, not just the lever.** The curated n=3 audit_mode bench
|
||||
can't resolve a class that is ≤3-5 of 75 q (four 2026-05-18
|
||||
hypotheses died there — round-trip, entity-mask, disambiguation).
|
||||
Mine **ground-truth-carrying** questions from corpus titles
|
||||
(`bench/mine_questions.py`) and grade by **deterministic retrieval
|
||||
recall@k** via `query --dry-run` (`bench/recall_at_k.py`): no LLM,
|
||||
no verifier, no n=3 noise, no 5pp floor, scalable to the corpus.
|
||||
Resolves a single lever to ±1 question (worked example: numeral-
|
||||
fold recall 55→75%, `a3ac653`). Caveats: measures *retrieval
|
||||
surfacing* — necessary-not-sufficient for STRICT; mined fixtures
|
||||
are answerable-by-construction so they **complement, never
|
||||
replace**, the curated adversarial set (the verifier-honesty/trap
|
||||
gate). Conflating the two is itself a bench-maxing error.
|
||||
- Avoid negation in prompts (Hermes-3-8B inverts under attention).
|
||||
- Bench is the scoreboard; live fixtures are the gates.
|
||||
- Self-heal beats retry (preserve partial output, never fabricate).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue