russell@unturf.com
b573c592d8
Second MEASURED fold-search win, and the instrument correcting my own
premature call. accent-fold ON vs OFF on the mined accent fixture:
recall@1 55% -> 85% (+30pp), rank-1 22/40 -> 34/40. recall@8 was
flat (95->98) — a too-lenient k nearly got a real lever wrongly
reverted; @1/@3 is the resolution that drives primary-source
selection. _accent_fold_variants: ASCII-fold then re-tokenise so a
diacritic title ("Béla Bartók", which _TITLE_TOKEN_RE otherwise
fragments to junk) matches the ASCII form a user types. Additive+
symmetric, no-op on pure-ASCII (zero effect on non-accent
queries/titles), mirrors _hyphen_fold_variants (#000007).
Also fixes a defect I shipped in a3ac653: an orphaned duplicate
body left as dead code after `return base` in _title_query_tokens
(unreachable — numeral-fold behaviour/measurement were valid — but
cruft; removed).
Fold-search factory, fanned out across the full survey backlog
(deterministic recall, no LLM, parallel — serial-by-caution was
halting in disguise):
- recall_at_k.py: returns rank -> recall@1/@3/@k from one retrieval
(verified offline). A coarse k hides rank-only lifts.
- mine_questions.py: numeral/accent/hyphen/honorific/amp/brit
ground-truth classes; fixtures committed.
- Measured @1 headroom verdicts: accent SHIP (this commit);
honorific 45% / brit 50% = real headroom (build next); hyphen
90% = existing #000007 already delivers, NOTHING to build (the
measure-the-unmeasured-thing check pays off); amp 82% = no fold
needed (prevalence-overranked, instrument kills it cheaply).
CLAUDE.md bench-maxing: two measured lessons codified — report
recall@1/@3/@k (a lenient k hides rank lifts; prevalence != miss-
rate), and fan out independent measurements (serial-by-caution is
halting). Full suite 2488 passed, 0 regressions (accent-fold is
hot-path in _title_query_tokens); real-path test (FakeSource->
ingest->query()->real _Hit).
43 lines
1.7 KiB
Text
43 lines
1.7 KiB
Text
# AUTO-MINED (brit class) from corpus titles via bench/mine_questions.py — ground-truth-carrying.
|
|
# Graded by deterministic retrieval recall@k (bench/recall_at_k.py), NOT audit_mode. Not adversarial; complements (never replaces) qa_questions.txt.
|
|
|
|
what is Finnish defense Forces?
|
|
what is Hopewell center, Hong Kong?
|
|
what is labor economics?
|
|
what is World organization for Animal Health?
|
|
what is Papua New Guinea defense Force?
|
|
what is programr?
|
|
what is Tonga defense Services?
|
|
what is Trinidad and Tobago defense Force?
|
|
what is Zambian defense Force?
|
|
what is defense Signals Directorate?
|
|
what is English Renaissance theater?
|
|
what is labor Party (Netherlands)?
|
|
what is The Luzhin defense?
|
|
what is Asia-Pacific Network Information center?
|
|
what is Soyuz program?
|
|
what is center Georges Pompidou?
|
|
what is Amphitheater?
|
|
what is Australian defense Force?
|
|
what is center County, Pennsylvania?
|
|
what is organization internationale de la Francophonie?
|
|
what is Luna program?
|
|
what is Man and the Biosphere program?
|
|
what is centerville, Maryland?
|
|
what is Boggs Township, center County, Pennsylvania?
|
|
what is Ferguson Township, center County, Pennsylvania?
|
|
what is Harris Township, center County, Pennsylvania?
|
|
what is Huston Township, center County, Pennsylvania?
|
|
what is Patton Township, center County, Pennsylvania?
|
|
what is Union Township, center County, Pennsylvania?
|
|
what is Worth Township, center County, Pennsylvania?
|
|
what is center Township, Perry County, Pennsylvania?
|
|
what is center Township, Pennsylvania?
|
|
what is Tricorn center?
|
|
what is Living color?
|
|
what is Philipsburg, center County, Pennsylvania?
|
|
what is German labor Front?
|
|
what is Rogers center?
|
|
what is Ulster defense Association?
|
|
what is Guiana Space center?
|
|
what is Community theater?
|