russell@unturf.com
b573c592d8
Second MEASURED fold-search win, and the instrument correcting my own
premature call. accent-fold ON vs OFF on the mined accent fixture:
recall@1 55% -> 85% (+30pp), rank-1 22/40 -> 34/40. recall@8 was
flat (95->98) — a too-lenient k nearly got a real lever wrongly
reverted; @1/@3 is the resolution that drives primary-source
selection. _accent_fold_variants: ASCII-fold then re-tokenise so a
diacritic title ("Béla Bartók", which _TITLE_TOKEN_RE otherwise
fragments to junk) matches the ASCII form a user types. Additive+
symmetric, no-op on pure-ASCII (zero effect on non-accent
queries/titles), mirrors _hyphen_fold_variants (#000007).
Also fixes a defect I shipped in a3ac653: an orphaned duplicate
body left as dead code after `return base` in _title_query_tokens
(unreachable — numeral-fold behaviour/measurement were valid — but
cruft; removed).
Fold-search factory, fanned out across the full survey backlog
(deterministic recall, no LLM, parallel — serial-by-caution was
halting in disguise):
- recall_at_k.py: returns rank -> recall@1/@3/@k from one retrieval
(verified offline). A coarse k hides rank-only lifts.
- mine_questions.py: numeral/accent/hyphen/honorific/amp/brit
ground-truth classes; fixtures committed.
- Measured @1 headroom verdicts: accent SHIP (this commit);
honorific 45% / brit 50% = real headroom (build next); hyphen
90% = existing #000007 already delivers, NOTHING to build (the
measure-the-unmeasured-thing check pays off); amp 82% = no fold
needed (prevalence-overranked, instrument kills it cheaply).
CLAUDE.md bench-maxing: two measured lessons codified — report
recall@1/@3/@k (a lenient k hides rank lifts; prevalence != miss-
rate), and fan out independent measurements (serial-by-caution is
halting). Full suite 2488 passed, 0 regressions (accent-fold is
hot-path in _title_query_tokens); real-path test (FakeSource->
ingest->query()->real _Hit).
43 lines
1.3 KiB
Text
43 lines
1.3 KiB
Text
# AUTO-MINED (honorific class) from corpus titles via bench/mine_questions.py — ground-truth-carrying.
|
|
# Graded by deterministic retrieval recall@k (bench/recall_at_k.py), NOT audit_mode. Not adversarial; complements (never replaces) qa_questions.txt.
|
|
|
|
what is Dr Who?
|
|
what is Dr Syn?
|
|
what is Dr V64?
|
|
what is Ft Wayne, Indiana?
|
|
what is Ft Collins, Colorado?
|
|
what is Pres of France?
|
|
what is St Lawrence Seaway?
|
|
what is St Kitts and Nevis?
|
|
what is St Lucia?
|
|
what is St Adrian?
|
|
what is St Boniface?
|
|
what is St Ninian?
|
|
what is St Andrew?
|
|
what is St Timothy?
|
|
what is St Anselm?
|
|
what is St Paul, Minnesota?
|
|
what is Mt Everest?
|
|
what is Ft Montgomery (Hudson River)?
|
|
what is St Nicholas?
|
|
what is St Patrick's Battalion?
|
|
what is Mt Sinai?
|
|
what is St Charles?
|
|
what is Mt Ephraim?
|
|
what is Gen Electric Company plc?
|
|
what is Mt Joy, Pennsylvania?
|
|
what is Ft Pitt?
|
|
what is Mt Clunie National Park?
|
|
what is Mt Nothofagus National Park?
|
|
what is Mt Field National Park?
|
|
what is Mt Richmond National Park?
|
|
what is Mt Aberdeen National Park?
|
|
what is Mt Colosseum National Park?
|
|
what is Mt Etna Caves National Park?
|
|
what is Mt O'Connell National Park?
|
|
what is Mt Webb National Park?
|
|
what is Ft Rucker?
|
|
what is Mt Olive, Alabama?
|
|
what is Ft Yukon, Alaska?
|
|
what is Mt Ida, Arkansas?
|
|
what is Ft Smith, Arkansas?
|