diff --git a/CLAUDE.md b/CLAUDE.md index f8fd993..bc38128 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -469,6 +469,20 @@ steps 3-5. Full discipline + worked examples in `docs/bench-maxing.md`. Headlines: - Bench before AND after every change (n=3, signal under 5pp is noise). +- **When a lever's failure class is sub-noise-floor, fix the + instrument, not just the lever.** The curated n=3 audit_mode bench + can't resolve a class that is ≤3-5 of 75 q (four 2026-05-18 + hypotheses died there — round-trip, entity-mask, disambiguation). + Mine **ground-truth-carrying** questions from corpus titles + (`bench/mine_questions.py`) and grade by **deterministic retrieval + recall@k** via `query --dry-run` (`bench/recall_at_k.py`): no LLM, + no verifier, no n=3 noise, no 5pp floor, scalable to the corpus. + Resolves a single lever to ±1 question (worked example: numeral- + fold recall 55→75%, `a3ac653`). Caveats: measures *retrieval + surfacing* — necessary-not-sufficient for STRICT; mined fixtures + are answerable-by-construction so they **complement, never + replace**, the curated adversarial set (the verifier-honesty/trap + gate). Conflating the two is itself a bench-maxing error. - Avoid negation in prompts (Hermes-3-8B inverts under attention). - Bench is the scoreboard; live fixtures are the gates. - Self-heal beats retry (preserve partial output, never fabricate).