docs(CLAUDE.md): bench-maxing — mine ground-truth recall@k when a class is sub-noise-floor

Compressed headline: when a lever's failure class is ≤3-5/75 q the
curated n=3 audit_mode bench can't resolve it (four 2026-05-18
hypotheses died there). Fix the instrument: mine ground-truth-
carrying questions (bench/mine_questions.py) graded by deterministic
retrieval recall@k via `query --dry-run` (bench/recall_at_k.py) —
no LLM/verifier/n=3-noise/5pp-floor, ±1-q resolution. Caveats codified:
measures retrieval surfacing (necessary-not-sufficient for STRICT);
complements never replaces the curated adversarial trap gate.
This commit is contained in:
russell@unturf.com 2026-05-18 15:30:01 -04:00
parent a3ac6539c1
commit 98a4d61de9
No known key found for this signature in database

View file

@ -469,6 +469,20 @@ steps 3-5.
Full discipline + worked examples in `docs/bench-maxing.md`. Headlines: Full discipline + worked examples in `docs/bench-maxing.md`. Headlines:
- Bench before AND after every change (n=3, signal under 5pp is noise). - Bench before AND after every change (n=3, signal under 5pp is noise).
- **When a lever's failure class is sub-noise-floor, fix the
instrument, not just the lever.** The curated n=3 audit_mode bench
can't resolve a class that is ≤3-5 of 75 q (four 2026-05-18
hypotheses died there — round-trip, entity-mask, disambiguation).
Mine **ground-truth-carrying** questions from corpus titles
(`bench/mine_questions.py`) and grade by **deterministic retrieval
recall@k** via `query --dry-run` (`bench/recall_at_k.py`): no LLM,
no verifier, no n=3 noise, no 5pp floor, scalable to the corpus.
Resolves a single lever to ±1 question (worked example: numeral-
fold recall 55→75%, `a3ac653`). Caveats: measures *retrieval
surfacing* — necessary-not-sufficient for STRICT; mined fixtures
are answerable-by-construction so they **complement, never
replace**, the curated adversarial set (the verifier-honesty/trap
gate). Conflating the two is itself a bench-maxing error.
- Avoid negation in prompts (Hermes-3-8B inverts under attention). - Avoid negation in prompts (Hermes-3-8B inverts under attention).
- Bench is the scoreboard; live fixtures are the gates. - Bench is the scoreboard; live fixtures are the gates.
- Self-heal beats retry (preserve partial output, never fabricate). - Self-heal beats retry (preserve partial output, never fabricate).