diff --git a/docs/tickets/ticket-000072-collapse-legacy-query-into-run-query.md b/docs/tickets/ticket-000072-collapse-legacy-query-into-run-query.md index 7759b68..9744791 100644 --- a/docs/tickets/ticket-000072-collapse-legacy-query-into-run-query.md +++ b/docs/tickets/ticket-000072-collapse-legacy-query-into-run-query.md @@ -98,7 +98,88 @@ on 2/5 smoke questions. Promoting it to default would degrade the user surface. Fox decision 2026-05-31: defer Phase 2; do NOT promote multi_route to default until A lands. -## Path A — port the missing rerank stages (multi-day) +## Path A v1 (attempted 2026-05-31, FAILED — reranks made things WORSE) + +Lifted all 5 reranks from query.py into `arborist/qa/retrieval_routes.py` +(commit d099995), unfroze `Hit` so reranks can mutate `.score` in +place, wired them into `run_query` in legacy order: + + filter_by_title_relevance + → filter_by_body_density + → rerank_by_body_coverage + → rerank_by_source_role + → rerank_by_title_purity + → rerank_by_ordered_token_match + → apply_title_boost (existing tail) + +Smoke probe (5 questions, retrieval-only no LLM, MultiShardSqliteCorpus +on the 4 wikipedia shards): + +| Q | no-policy (body-only) | multi_route | multi_route + 5 reranks | +|---|---|---|---| +| Soviet Union dissolve | Soviet Union ✓ | Soviet Union ✓ | **Soviet Union national bandy team ✗** | +| Mount Kilimanjaro | Mount Kilimanjaro ✓ | Mount Kilimanjaro ✓ | Mount Kilimanjaro ✓ | +| Mona Lisa painter | Mona Lisa ✓ | Mona Lisa ✓ | **Painting Mona Lisa ✗** | +| Mercury Seven astronauts | Mercury Seven ✓ | Sam T. Beddingfield ✗ | 305th Air Mobility Wing ✗ | +| Dinosaurs extinct | Dinosaur ✓ | Paul Austin Kelly ✗ | Paul Austin Kelly ✗ | + +Score: no-policy 5/5, multi_route 3/5, multi_route+reranks 1/5. +The reranks made it strictly worse than multi_route alone. + +**Root cause:** legacy's reranks were tuned against legacy's +candidate-set shape, NOT against my multi_route fan-out's shape: + + - Legacy `_search_corpus` runs per-shard parallel routes with + `over_fetch=32` (vs my `per_route_limit=top_k*4 = 16`), so its + candidate set is much larger and the noisy short-title hits + have more competition from genuine canonical articles before + rerank ever fires. + - Legacy applies body-density filtering EARLIER in the pipeline + (in `_search_one_shard` before merge), so noisy phrase-route + hits with no body coherence are dropped before they can be + boosted by title-purity or title-boost. + - Legacy's rivalry-exclusion + synonym-expansion order differs; + my path applies them inside `filter_by_title_relevance` only. + +Applying the same multipliers to a structurally different +candidate set lands the cascade in a different basin — short noisy +titles with high stem-overlap (e.g. "Soviet Union national bandy +team", "Painting Mona Lisa", "305th Air Mobility Wing") get +amplified into rank-1 territory because they win the +title-purity × ordered-token × source-role compound multiplier even +though their body BM25 starts much higher (less negative) than the +canonical primary's. + +**Reverted the wire-up; helpers stay in tree.** The 5 reranks live +in `arborist/qa/retrieval_routes.py` as importable building blocks. +Hit stays unfrozen (mutable reranks need it). But `run_query` only +applies `filter_by_title_relevance` when `multi_route=True`, not +the 5-stage cascade. multi_route is still worse than body-only +(3/5 vs 5/5), but no longer worse than multi_route+reranks. + +## Path A v2 — possible directions + + 1. **Match legacy's oversample factor.** `per_route_limit=top_k*4` + is too tight. Try `top_k*8` or `over_fetch=32` matching legacy. + More candidates per route = more competition in the merge = + cleaner top-K after reranks. + 2. **Apply body-density filter BEFORE rerank cascade**, not + after `filter_by_title_relevance`. Legacy does this earlier + in `_search_one_shard`; the noisy phrase-route hits without + body coherence get dropped at the SOURCE, not after merge. + 3. **Rerun reranks on per-shard route output**, then merge + final top-K. Preserves per-shard discrimination that the MIN + bm25 merge collapses today. + 4. **Synonym expansion at retrieval time**, not just at filter + time. Legacy passes a synonym pool into FTS5 MATCH; my path + uses raw query tokens. May change which docs surface in the + first place. + +Any of these is a separate focused investigation. The 5-rerank +infrastructure is in tree; v2 is a wiring question, not a +porting question. + +## Path A v1 — port the missing rerank stages (multi-day) [original plan, superseded by the v1 attempt above] Estimated scope: 1. `retrieval_routes.body_density_passes(corpus, hit, qtokens, *,