russell@unturf.com
b573c592d8
Second MEASURED fold-search win, and the instrument correcting my own
premature call. accent-fold ON vs OFF on the mined accent fixture:
recall@1 55% -> 85% (+30pp), rank-1 22/40 -> 34/40. recall@8 was
flat (95->98) — a too-lenient k nearly got a real lever wrongly
reverted; @1/@3 is the resolution that drives primary-source
selection. _accent_fold_variants: ASCII-fold then re-tokenise so a
diacritic title ("Béla Bartók", which _TITLE_TOKEN_RE otherwise
fragments to junk) matches the ASCII form a user types. Additive+
symmetric, no-op on pure-ASCII (zero effect on non-accent
queries/titles), mirrors _hyphen_fold_variants (#000007).
Also fixes a defect I shipped in a3ac653: an orphaned duplicate
body left as dead code after `return base` in _title_query_tokens
(unreachable — numeral-fold behaviour/measurement were valid — but
cruft; removed).
Fold-search factory, fanned out across the full survey backlog
(deterministic recall, no LLM, parallel — serial-by-caution was
halting in disguise):
- recall_at_k.py: returns rank -> recall@1/@3/@k from one retrieval
(verified offline). A coarse k hides rank-only lifts.
- mine_questions.py: numeral/accent/hyphen/honorific/amp/brit
ground-truth classes; fixtures committed.
- Measured @1 headroom verdicts: accent SHIP (this commit);
honorific 45% / brit 50% = real headroom (build next); hyphen
90% = existing #000007 already delivers, NOTHING to build (the
measure-the-unmeasured-thing check pays off); amp 82% = no fold
needed (prevalence-overranked, instrument kills it cheaply).
CLAUDE.md bench-maxing: two measured lessons codified — report
recall@1/@3/@k (a lenient k hides rank lifts; prevalence != miss-
rate), and fan out independent measurements (serial-by-caution is
halting). Full suite 2488 passed, 0 regressions (accent-fold is
hot-path in _title_query_tokens); real-path test (FakeSource->
ingest->query()->real _Hit).
242 lines
8 KiB
JSON
242 lines
8 KiB
JSON
[
|
|
{
|
|
"question": "what is Dr Who?",
|
|
"target_title": "Doctor Who",
|
|
"target_root": "3bda6eaddefbf7c72119270622302f55ed440de84f8dcfbc8c3fa59bf39f2286",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Dr Syn?",
|
|
"target_title": "Doctor Syn",
|
|
"target_root": "f36fe260f689bc797a9afe95fc30a3dd7bd9376b3569fd8df572a2002a35287d",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Dr V64?",
|
|
"target_title": "Doctor V64",
|
|
"target_root": "18972d20548cec458b36984c89aa833ee230e9f696bfed6419ed4172a1413289",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Ft Wayne, Indiana?",
|
|
"target_title": "Fort Wayne, Indiana",
|
|
"target_root": "cf688a2a18a47316030b7c6f6d3f51e2be398607fc4e20542da534daaab323ff",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Ft Collins, Colorado?",
|
|
"target_title": "Fort Collins, Colorado",
|
|
"target_root": "c0c464c0315c4e2dd9ef08cf768f93f074ed21f98df39a65ec3f78cd6030aa69",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Pres of France?",
|
|
"target_title": "President of France",
|
|
"target_root": "382bba1c25dc090e24e4e54a5233509a0eb4b65d5862458fb660e4bf733b3395",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is St Lawrence Seaway?",
|
|
"target_title": "Saint Lawrence Seaway",
|
|
"target_root": "a891a80fef6b9248ecae5cc009a12372738c41b5278716a492e455c2b3509f6b",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is St Kitts and Nevis?",
|
|
"target_title": "Saint Kitts and Nevis",
|
|
"target_root": "c90a8ef06705b60431de77db01a4e2a5964214f51273f54c8e70bdfc5a16e53e",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is St Lucia?",
|
|
"target_title": "Saint Lucia",
|
|
"target_root": "bcc26d405565ba560c68aa2dd21906f3700a3b1d7fef2071af7f853b93ceb5ed",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is St Adrian?",
|
|
"target_title": "Saint Adrian",
|
|
"target_root": "5f8a343d61f4167c96e4e7908bd74e2585ddff4753f6ee29da8832360a395c48",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is St Boniface?",
|
|
"target_title": "Saint Boniface",
|
|
"target_root": "9d36b35a5b47ab475a08626c1eee22c1a6bfe5303b6f8eb036cf9daa8fb2cadd",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is St Ninian?",
|
|
"target_title": "Saint Ninian",
|
|
"target_root": "0ca827e20ca4efabd8509bac52be4f85df7338defb97b1894e5ac869e82986b9",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is St Andrew?",
|
|
"target_title": "Saint Andrew",
|
|
"target_root": "d2294e4c76a3d636ded6e30365ebb1e51a91c671b76603079c024d028ffe9ba1",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is St Timothy?",
|
|
"target_title": "Saint Timothy",
|
|
"target_root": "d0a2f78451581da3ff7146d2e214cd9d0d2b99a6720a2d93d3a86fd3409dd581",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is St Anselm?",
|
|
"target_title": "Saint Anselm",
|
|
"target_root": "7f768c8ccc45617e57f02c9c65bd82c61fb9f43aed4de9abaf67063e746999ac",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is St Paul, Minnesota?",
|
|
"target_title": "Saint Paul, Minnesota",
|
|
"target_root": "22cf7aec47b900ad13865870db6e5e4312da9c9dc4687f33fca9b77e21e1e5e3",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Everest?",
|
|
"target_title": "Mount Everest",
|
|
"target_root": "aaf3b423eb2ccb4e8cabe42f0bbea24547186b079c939562e48383cbec3928ee",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Ft Montgomery (Hudson River)?",
|
|
"target_title": "Fort Montgomery (Hudson River)",
|
|
"target_root": "4b89b15f53a746a077d3bd1f5f8d6f8c6602e9781c4201fed1b15f7871b05ced",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is St Nicholas?",
|
|
"target_title": "Saint Nicholas",
|
|
"target_root": "7e6d8087d3ef4eaaf4b869ec97dfc4a46d8fea78f959211f533e9fff1f3da336",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is St Patrick's Battalion?",
|
|
"target_title": "Saint Patrick's Battalion",
|
|
"target_root": "1e32490a958c1c792e8cca33abe46ca0ce6dc9619d694028b4cc67785bf59717",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Sinai?",
|
|
"target_title": "Mount Sinai",
|
|
"target_root": "b62bac7647911166f1916785423dc42af025a4a0522944ff35053dfa902839e5",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is St Charles?",
|
|
"target_title": "Saint Charles",
|
|
"target_root": "996b4d3dd28814ae12b26553d4746abfd478016a1fe36266fa304f401ffe7abe",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Ephraim?",
|
|
"target_title": "Mount Ephraim",
|
|
"target_root": "148f2298bf9353a1817e63c27333955adc1adfc5640fdd071e65f77d98327de5",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Gen Electric Company plc?",
|
|
"target_title": "General Electric Company plc",
|
|
"target_root": "b59db856e9ee268008282cb3a5b49771452b9d7a1a6ec426c829bfb8c204ca27",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Joy, Pennsylvania?",
|
|
"target_title": "Mount Joy, Pennsylvania",
|
|
"target_root": "4962e4fbe5b0801ae95f81d10c190c51ccaf1df8895e3aec385c95ecbe571328",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Ft Pitt?",
|
|
"target_title": "Fort Pitt",
|
|
"target_root": "61d1a97836cac809be53e674772d23f0bf2b1dd2ae1b48ad38ecc0eaa6b0b9d9",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Clunie National Park?",
|
|
"target_title": "Mount Clunie National Park",
|
|
"target_root": "9ac60e06140c6408071d8cfb8cd0df754631833bbf3e53310fef18c08caf95d1",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Nothofagus National Park?",
|
|
"target_title": "Mount Nothofagus National Park",
|
|
"target_root": "387bdcbe2332bbe299faed4a29cb44253a48bb5a503ce58ceec97c14658aab79",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Field National Park?",
|
|
"target_title": "Mount Field National Park",
|
|
"target_root": "363059003b548b228c9160860650b753b6a0ecdbeaafd67f00f771c6d7935a58",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Richmond National Park?",
|
|
"target_title": "Mount Richmond National Park",
|
|
"target_root": "725b6ce0749788d851c843799608160d32165d2e67dedee6fc68097a6e5f889e",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Aberdeen National Park?",
|
|
"target_title": "Mount Aberdeen National Park",
|
|
"target_root": "2a0423b99faa14704a81c8eecb52162efd1e3a813d0e9ffa865880f3773bda3b",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Colosseum National Park?",
|
|
"target_title": "Mount Colosseum National Park",
|
|
"target_root": "617498381698f11c8dd5188e651992a278fa772b45e275370e4ecfc0fd6dd7ae",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Etna Caves National Park?",
|
|
"target_title": "Mount Etna Caves National Park",
|
|
"target_root": "737b9514367c77ee232e88fd1e3ecb69d4d56bd78543d953e0ffcc966ae3c7c7",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt O'Connell National Park?",
|
|
"target_title": "Mount O'Connell National Park",
|
|
"target_root": "1d7bacc37bce42982e531efd1b3bfa138acdfe3d84cbfebbcb564c5400ed3041",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Webb National Park?",
|
|
"target_title": "Mount Webb National Park",
|
|
"target_root": "b435f7f449dd02cba620200dd860d963ebb993feeb8f2a65b20bb5e4fe9961ca",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Ft Rucker?",
|
|
"target_title": "Fort Rucker",
|
|
"target_root": "630a70bb0e6019709a11891d28960881d25d90cf20de58b533e031d3ce935ba8",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Olive, Alabama?",
|
|
"target_title": "Mount Olive, Alabama",
|
|
"target_root": "3ce537cd880daa793b27d6864bfaeb013b40132fa3ea4770c11a148d3fd6c2db",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Ft Yukon, Alaska?",
|
|
"target_title": "Fort Yukon, Alaska",
|
|
"target_root": "a6dd1002a9fe6827936c461ac4f657fa569a9c5cbc16d56c0b6e50cba91b9c0b",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Mt Ida, Arkansas?",
|
|
"target_title": "Mount Ida, Arkansas",
|
|
"target_root": "e9e5baae06c14cbc1aa2f23fe45f47372ad59d20386d6f023ad7e0a3f978f315",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "what is Ft Smith, Arkansas?",
|
|
"target_title": "Fort Smith, Arkansas",
|
|
"target_root": "e4ebbc4a7b0f1c63001e5247a8aa23b5b9046ae6f16cc4ff63b67537f9ceefd1",
|
|
"shard": "000.db"
|
|
}
|
|
]
|