From 98a4d61de9750cef0023fe715882525cd43c245e Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Mon, 18 May 2026 15:30:01 -0400 Subject: [PATCH] =?UTF-8?q?docs(CLAUDE.md):=20bench-maxing=20=E2=80=94=20m?= =?UTF-8?q?ine=20ground-truth=20recall@k=20when=20a=20class=20is=20sub-noi?= =?UTF-8?q?se-floor?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Compressed headline: when a lever's failure class is ≤3-5/75 q the curated n=3 audit_mode bench can't resolve it (four 2026-05-18 hypotheses died there). Fix the instrument: mine ground-truth- carrying questions (bench/mine_questions.py) graded by deterministic retrieval recall@k via `query --dry-run` (bench/recall_at_k.py) — no LLM/verifier/n=3-noise/5pp-floor, ±1-q resolution. Caveats codified: measures retrieval surfacing (necessary-not-sufficient for STRICT); complements never replaces the curated adversarial trap gate. --- CLAUDE.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index f8fd993..bc38128 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -469,6 +469,20 @@ steps 3-5. Full discipline + worked examples in `docs/bench-maxing.md`. Headlines: - Bench before AND after every change (n=3, signal under 5pp is noise). +- **When a lever's failure class is sub-noise-floor, fix the + instrument, not just the lever.** The curated n=3 audit_mode bench + can't resolve a class that is ≤3-5 of 75 q (four 2026-05-18 + hypotheses died there — round-trip, entity-mask, disambiguation). + Mine **ground-truth-carrying** questions from corpus titles + (`bench/mine_questions.py`) and grade by **deterministic retrieval + recall@k** via `query --dry-run` (`bench/recall_at_k.py`): no LLM, + no verifier, no n=3 noise, no 5pp floor, scalable to the corpus. + Resolves a single lever to ±1 question (worked example: numeral- + fold recall 55→75%, `a3ac653`). Caveats: measures *retrieval + surfacing* — necessary-not-sufficient for STRICT; mined fixtures + are answerable-by-construction so they **complement, never + replace**, the curated adversarial set (the verifier-honesty/trap + gate). Conflating the two is itself a bench-maxing error. - Avoid negation in prompts (Hermes-3-8B inverts under attention). - Bench is the scoreboard; live fixtures are the gates. - Self-heal beats retry (preserve partial output, never fabricate).