russell@unturf.com
a3ac6539c1
The first MEASURED, above-noise retrieval win this thread. The 75-q
n=3 audit_mode bench couldn't resolve any single lever (every failure
class <=3-5 q, sub the 5pp floor — four hypotheses died there). Fix
the instrument, not just the lever:
- bench/mine_questions.py + bench/recall_at_k.py: mine questions from
corpus titles (ground-truth target known by construction), grade by
deterministic retrieval recall@k via `query --dry-run` — no LLM, no
verifier, no n=3 noise, scalable to the 22K-deep numeral pool. The
curated qa_questions.txt stays the separate verifier-honesty/trap
gate; mined fixtures measure the answerable long tail per class.
- _numeral_fold_variants in query.py: ordinal-word ("Alexander the
second") <-> multi-char Roman ("Alexander II"), additive+symmetric,
unioned into _title_query_tokens exactly like _hyphen_fold_variants
(#000007). Strict 2..40 Roman set → no English-word collision;
single-char Romans (I/V/X) intentionally out of scope (universal
len>1 token filter — stated before building, ~4 of 10 residual
misses).
Measured on the mined numeral fixture: recall@8 22/40 (55%) -> 30/40
(75%), +20pp; 20 hits now rank-1. Discipline applied end to end:
measured-first, mirrored precedent, full-suite regression run (2482
passed, 0 regressions — numeral-fold is hot-path in
_title_query_tokens), real-path test (FakeSource->ingest->query()->
real _Hit, not a hand-built object), measured-after on a noise-free
instrument. The ~6 multi-char residual misses are a different
downstream cause the instrument now exposes for future iteration.
242 lines
8.2 KiB
JSON
242 lines
8.2 KiB
JSON
[
|
|
{
|
|
"question": "who was Albert the third?",
|
|
"target_title": "Albert III",
|
|
"target_root": "43ff05c3c851e358df34c36d7398fefc53998e5a9cd8654b5c2f8dd9524b9efd",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Ahmed the third?",
|
|
"target_title": "Ahmed III",
|
|
"target_root": "2fc91045e60b915d146f1f4a0f0c72e2df2f644efbc299ea27ca40170e6092aa",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Alaric the first?",
|
|
"target_title": "Alaric I",
|
|
"target_root": "57e7aa58a4fd36b42a920956929e7c852108f2e3d36c87487d411ec4aa7f6686",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Alexander the first of epirus?",
|
|
"target_title": "Alexander I of Epirus",
|
|
"target_root": "db479a5835381d3d94705976dd6adede8fef8e51ea88cc6c3fe4c789e1e1ae35",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Alexander the second of scotland?",
|
|
"target_title": "Alexander II of Scotland",
|
|
"target_root": "4d5885f25ab6d7ebd0054dfdb4783f00db96f210ea33cdff2c0dbb0b92b45a13",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Alexander the second?",
|
|
"target_title": "Alexander II",
|
|
"target_root": "190bf209efb2f096f9b93c7dd2d0b0be9a64d6aa469d27cf2272664e4f624de2",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Alexander the fourth?",
|
|
"target_title": "Alexander IV",
|
|
"target_root": "cc3134f31a0d69f03aa37a5821fc179082f34316e782d464b0685f3ed0e90f0a",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Alyattes the second?",
|
|
"target_title": "Alyattes II",
|
|
"target_root": "464c0878c6fcabaf67f37389aa343b180b2762325a9aa916aa0d95dd7529e853",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Afonso the fourth of portugal?",
|
|
"target_title": "Afonso IV of Portugal",
|
|
"target_root": "88c2a881546a4cfa59c8eba1e037d2092ab73c9f09feb9524801d64763dfbd3e",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Alfonso the second of asturias?",
|
|
"target_title": "Alfonso II of Asturias",
|
|
"target_root": "1b2073183930900d65f1433efb70e8080628b0aa440817a3eb235d70ddf18f61",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Alfonso the fourth of aragon?",
|
|
"target_title": "Alfonso IV of Aragon",
|
|
"target_root": "fb96fb9d074fefd80da864c412a9f07cab26b1a217b54a378442143c147316d7",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Alfonso the third?",
|
|
"target_title": "Alfonso III",
|
|
"target_root": "555a58a9f7c947f32e18dcd6dfb03fb13ba7cc29f144c5b98f8af087892d9ac0",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Alfonso the fifth?",
|
|
"target_title": "Alfonso V",
|
|
"target_root": "c7aaf0276378cefbbb1a44dc1f90b4cb5928411880c7869ea68628f09f1ebde2",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Anastasius the second?",
|
|
"target_title": "Anastasius II",
|
|
"target_root": "2cb452df76d8d15494550c8070b51bca9f369647c95574e6a4f9cb441f865c0c",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Abbas the second of egypt?",
|
|
"target_title": "Abbas II of Egypt",
|
|
"target_root": "8c3378f3375831933b0c0865074747c264599ceb2650f866c5d61d3635126035",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Charles the fifth?",
|
|
"target_title": "Charles V",
|
|
"target_root": "22d8e5eb9bebef543cecda7b33a1f0d300cd24e3a13427c6a297524c9813718f",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Constantius the second?",
|
|
"target_title": "Constantius II",
|
|
"target_root": "5f8bd920302cc83ee5aef59abfda0aafd31b6e1cb48cfc4f5c7d4c3e69bedd4a",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Constantine the second of scotland?",
|
|
"target_title": "Constantine II of Scotland",
|
|
"target_root": "24e398f6b012f01c0e1b1cb7fad7b4ee2fda8898a15975e5d1c75a4ee77bc3df",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Charles the first of england?",
|
|
"target_title": "Charles I of England",
|
|
"target_root": "ad6bc7ccd05e08e3aa9fd3db04dc85e52fc64e54c22532fe1af00ec956bef907",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Frederick the fifth?",
|
|
"target_title": "Frederick V",
|
|
"target_root": "ac3d08404d5e30a1642647512b5e5170253fb46577a244390c909dd8679922c0",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Henry the seventh?",
|
|
"target_title": "Henry VII",
|
|
"target_root": "91e7879a395c0a43c218964ccc28bcff93ec3e3e141483af3b50e2d596b185df",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Mehmed the first?",
|
|
"target_title": "Mehmed I",
|
|
"target_root": "9a526b048fe00f930da0b4128eb30dde55ebfee10490cea2322768c8d1f18023",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Mustafa the first?",
|
|
"target_title": "Mustafa I",
|
|
"target_root": "6d4d3e3d401f3b1651fc3d779d71b8084a5ff69ba2f95516dcefec63f8e6fd20",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Mieszko the first of poland?",
|
|
"target_title": "Mieszko I of Poland",
|
|
"target_root": "fee8e212dc5d6439d1b791eea1bc997d825a4b4abf3bfeeef2cfae57e948d077",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Malcolm the first of scotland?",
|
|
"target_title": "Malcolm I of Scotland",
|
|
"target_root": "1f1983a45ed9b9cfa49425103fed17a1ca0e36284189525a31a9bc73a14d2181",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Osman the second?",
|
|
"target_title": "Osman II",
|
|
"target_root": "71da21f6a76233de3ac45683f8d8b419d9962581cba0f2fb0cf81093b7e41b0a",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Quake the second?",
|
|
"target_title": "Quake II",
|
|
"target_root": "86b1189f25d0fe612423ef88be44885da48f115c27b64024078c2fd257acb66a",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Stephen the third?",
|
|
"target_title": "Stephen III",
|
|
"target_root": "07d31c9abc8c1e8626997b248c82592b09dff67e96069c31c90b95230407e4c2",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Oscar the first of sweden?",
|
|
"target_title": "Oscar I of Sweden",
|
|
"target_root": "09ab96881d295f59dd8131a2d9d15bc2d633bd9e9606c1c27d7e1c7c889cfec0",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Charles the fifteenth of sweden?",
|
|
"target_title": "Charles XV of Sweden",
|
|
"target_root": "381a3cc80ba62a7454ea87bfd56ddefb27229491e3926a88d84968fd791485bb",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Sviatoslav the first of kiev?",
|
|
"target_title": "Sviatoslav I of Kiev",
|
|
"target_root": "99066a1e171e3a3924c115f6010a306743520730d9a0e0ed4f359c0e7239a7dd",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Catherine the second of russia?",
|
|
"target_title": "Catherine II of Russia",
|
|
"target_root": "d7a6a6d1f63b0ddd4977b2c9203fdd6659b0fdcadbb43096910c028ea492a42b",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Childeric the first?",
|
|
"target_title": "Childeric I",
|
|
"target_root": "a21db476b6e4ece9aa1cc7da6b68b199474f82c297eaa3304fd3713d0b7ec682",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Rudolph the first of germany?",
|
|
"target_title": "Rudolph I of Germany",
|
|
"target_root": "7f882ac84cf32819f3af2dda446d18ad7cd818e94b201c56a3a89a0d78bf9161",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Xerxes the second of persia?",
|
|
"target_title": "Xerxes II of Persia",
|
|
"target_root": "edafe8586c5666f8fb38a499ac73f0d954fba6c2d3b869610de16ba33bcc8a3f",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Richard the second of england?",
|
|
"target_title": "Richard II of England",
|
|
"target_root": "d08b7f5eb56fb3352a77facfbc204b53b1a6cef4215d15ac7a2a97f3a30dda8d",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Gustav the first of sweden?",
|
|
"target_title": "Gustav I of Sweden",
|
|
"target_root": "4ff24ede9c8fc1bd401972bef9a3a8e4a4614b8ad289027e9e51b00f16631a04",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Photios the first of constantinople?",
|
|
"target_title": "Photios I of Constantinople",
|
|
"target_root": "1309569c1cb8f37ff5f7d2cfc00080efb5f6846e54b1a45c07357e1b0a2445f2",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was James the fifth of scotland?",
|
|
"target_title": "James V of Scotland",
|
|
"target_root": "6b0e3af072a252daaad838123f5b2c2db3b0ac91bd7833faf17ff4f0a488fbe0",
|
|
"shard": "000.db"
|
|
},
|
|
{
|
|
"question": "who was Basarab the first of wallachia?",
|
|
"target_title": "Basarab I of Wallachia",
|
|
"target_root": "6f543c2b310d31ef08d43e4454a69eb2294ffc412df4cb26bafd92ea9841ca2e",
|
|
"shard": "000.db"
|
|
}
|
|
]
|