russell@unturf.com
b573c592d8
Second MEASURED fold-search win, and the instrument correcting my own
premature call. accent-fold ON vs OFF on the mined accent fixture:
recall@1 55% -> 85% (+30pp), rank-1 22/40 -> 34/40. recall@8 was
flat (95->98) — a too-lenient k nearly got a real lever wrongly
reverted; @1/@3 is the resolution that drives primary-source
selection. _accent_fold_variants: ASCII-fold then re-tokenise so a
diacritic title ("Béla Bartók", which _TITLE_TOKEN_RE otherwise
fragments to junk) matches the ASCII form a user types. Additive+
symmetric, no-op on pure-ASCII (zero effect on non-accent
queries/titles), mirrors _hyphen_fold_variants (#000007).
Also fixes a defect I shipped in a3ac653: an orphaned duplicate
body left as dead code after `return base` in _title_query_tokens
(unreachable — numeral-fold behaviour/measurement were valid — but
cruft; removed).
Fold-search factory, fanned out across the full survey backlog
(deterministic recall, no LLM, parallel — serial-by-caution was
halting in disguise):
- recall_at_k.py: returns rank -> recall@1/@3/@k from one retrieval
(verified offline). A coarse k hides rank-only lifts.
- mine_questions.py: numeral/accent/hyphen/honorific/amp/brit
ground-truth classes; fixtures committed.
- Measured @1 headroom verdicts: accent SHIP (this commit);
honorific 45% / brit 50% = real headroom (build next); hyphen
90% = existing #000007 already delivers, NOTHING to build (the
measure-the-unmeasured-thing check pays off); amp 82% = no fold
needed (prevalence-overranked, instrument kills it cheaply).
CLAUDE.md bench-maxing: two measured lessons codified — report
recall@1/@3/@k (a lenient k hides rank lifts; prevalence != miss-
rate), and fan out independent measurements (serial-by-caution is
halting). Full suite 2488 passed, 0 regressions (accent-fold is
hot-path in _title_query_tokens); real-path test (FakeSource->
ingest->query()->real _Hit).
43 lines
1.2 KiB
Text
43 lines
1.2 KiB
Text
# AUTO-MINED (accent class) from corpus titles via bench/mine_questions.py — ground-truth-carrying.
|
||
# Graded by deterministic retrieval recall@k (bench/recall_at_k.py), NOT audit_mode. Not adversarial; complements (never replaces) qa_questions.txt.
|
||
|
||
what is Casa Batllo?
|
||
what is Andre-Marie Ampere?
|
||
what is Antoni Gaudi?
|
||
what is Alcobaca (Portugal)?
|
||
what is Bifrost?
|
||
what is Bootes?
|
||
what is Bela Bartok?
|
||
what is Blue Oyster Cult?
|
||
what is Bjorn Borg?
|
||
what is Barıs Manco?
|
||
what is Transport in Cote d'Ivoire?
|
||
what is Crannog?
|
||
what is Emily Bronte?
|
||
what is Evariste Galois?
|
||
what is Elblag?
|
||
what is Erwin Schrodinger?
|
||
what is Elisabeth-Louise Vigee-Le Brun?
|
||
what is Emperor Shomu?
|
||
what is Emperor Yomei?
|
||
what is Emperor Kotoku?
|
||
what is Empress Koken?
|
||
what is Emperor Go-En'yu?
|
||
what is Emperor Koan?
|
||
what is Emperor Chuai?
|
||
what is Francisco Alvares?
|
||
what is Felix Guattari?
|
||
what is Gota Canal?
|
||
what is Goran Bregovic?
|
||
what is Godel's completeness theorem?
|
||
what is Gilbert Arthur a Beckett?
|
||
what is GEANT?
|
||
what is Herge?
|
||
what is Ismet Inonu?
|
||
what is Gyula Andrassy?
|
||
what is Jean-Francois Millet?
|
||
what is House of Karađorđevic?
|
||
what is Kunstlerroman?
|
||
what is Musee du Louvre?
|
||
what is Lubeck?
|
||
what is La Tene culture?
|