arborist/docs/tickets/ticket-000072-collapse-legacy-query-into-run-query.md

Ticket #000072 — Collapse legacy query() into the unified run_query

Status: open · Phase 1 foundation shipped 2026-05-31 · Phase 2 BLOCKED on the 5 missing rerank stages. A multi_route smoke probe regressed on 2/5 questions — Phase 1 infrastructure is structurally complete but not production-quality. Promoting policy.multi_route=True to default today would degrade the user surface.

Opened: 2026-05-31 Owner: fox/blackops Directive: cleanup

Why this exists

arborist/qa/query.py is 4280 LOC; the query() function alone runs lines 1975-4196 (2200 LOC, single function). It is the local-CLI entry point for arborist query — every retrieval, rerank, cache lookup, audit-chain append, run-DAG build, and verifier kwarg threads through it. It also picks the WRONG primary source on the dinosaur- extinction smoke fixture: Edwina, the Dinosaur Who Didn't Know She Was Extinct (a children's book) instead of Dinosaur (the encyclopedia article). The cloud-via-slim-FTS5 path landed in d9fb6a9 already picks Dinosaur correctly through the unified run_query orchestrator. Collapsing legacy query() to delegate to the same orchestrator would fix that retrieval bug as a byproduct and delete ~1500 LOC of duplicated retrieval/rerank logic.

What shipped — Phase 1 foundation (2026-05-31)

Ten incremental commits, each behavior-additive, each with the byte-identity gate (step 5) staying green:

commit	step	what landed
9ba6317	1	arborist/qa/source_roles.py — classify_source_role, weights, title-marker tuples extracted from query.py
a83e47b	2	arborist/qa/_text_norm.py gains canonical stem_for_match; query.py + corpus.py both import
056d785	3	arborist/qa/retrieval_routes.py with filter_by_title_relevance (5-accept-path filter); query.py re-exports
e322bbd	4	Corpus protocol gains core_keyword_match + doc_body; SqliteShard implements; SidecarBucket raises NotSupportedError
1822741	5	tests/test_run_query_byte_identity.py + tests/fixtures/byte_identity/claim_lattice.json — pins hash inputs as Phase 2 safety gate
72d1117	6a	run_query gains policy= kwarg threading 12 verifier kwargs
03f248c	6b	role-classified + role-weighted budget when policy provided
5fdd573	6c	multi-route retrieval (body+title+phrase+core_keyword merge) when policy.multi_route=True
6c2ec1b	6c-fil	wire filter_by_title_relevance into multi_route path
b8bd9d6	6d	wikitext-strip when policy.base_version is set
20faae0	P2/1	arborist/qa/providence_query.py — cache-aware wrapper around run_query (skeleton; cache persist not yet done)

Every commit ships with policy=None preserving pre-step byte-identity behavior — existing arborist corpus-query and arborist cloud query callers are unaffected. 264 tests in the query/corpus/sidecar/wallet/bucket/claim_lattice/byte_identity gate pass at HEAD.

What's broken — the multi_route smoke regression (2026-05-31)

Retrieval-only smoke against the 4-shard wikipedia corpus (no LLM, StubClient), comparing policy=None (body-only) vs policy={"multi_route": True} (body+title+phrase+core_keyword merged through filter_by_title_relevance):

Q	no-policy	multi_route	verdict
Soviet Union dissolve	Soviet Union	Soviet Union	preserved
Mount Kilimanjaro	Mount Kilimanjaro	Mount Kilimanjaro	preserved
Mona Lisa painter	Mona Lisa	Mona Lisa	preserved
Mercury Seven astronauts	Mercury Seven	Sam T. Beddingfield	REGRESSED
Dinosaurs extinct	Dinosaur	Paul Austin Kelly	REGRESSED

Diagnosis: the phrase route (fts_phrase on 4-grams from the question) surfaces oddly-titled stubs whose body contains the literal question phrase. The 5-accept-path filter's path-4 (phrase_match_roots) lets them through with no body-density gate, and apply_title_boost doesn't rerank them out because their titles share zero tokens with the question.

Legacy query() suppresses these noisy phrase hits via downstream stages that Phase 1 has NOT ported:

1. _body_density_passes — co-occurrence breadth check on full doc body (3+ token queries require N-1 of N tokens present at ≥3 mentions each)
2. _rerank_by_body_coverage — sqrt-rerank that counters BM25 short-doc bias
3. _rerank_by_source_role — role-weighted score multiplier (primary 2.0×, noisy 0.3×)
4. _rerank_by_title_purity — (1+overlap)*(1+purity) boost
5. _rerank_by_ordered_token_match — longest-ordered-subsequence match between question tokens and title tokens

Phase 2 — BLOCKED until the 5 stages above land

Phase 2 was meant to land:

• providence_query.py cache wrapper (skeleton done, persist deferred)
• Legacy query() collapsed to a thin adapter (50-line shell that delegates to providence_query)
• Bench parity proof on smoke + bench/qa_questions.txt themed subsets

But Phase 2 cannot ship without the 5 missing rerank stages — the new orchestrator currently produces WORSE primary picks than legacy on 2/5 smoke questions. Promoting it to default would degrade the user surface. Fox decision 2026-05-31: defer Phase 2; do NOT promote multi_route to default until A lands.

Path A — port the missing rerank stages (multi-day)

Estimated scope:

1. retrieval_routes.body_density_passes(corpus, hit, qtokens, *, min_mentions=3) — Corpus exposes doc_body(document_root) since step 4 (e322bbd); fn already shaped for hits that carry document_root. ~30 LOC.
2. retrieval_routes.rerank_by_body_coverage(hits, question, corpus) — uses corpus.doc_body per hit; sqrt-rerank multiplier. ~60 LOC.
3. retrieval_routes.rerank_by_source_role(hits, question) — uses classify_source_role from source_roles.py + RANK_WEIGHTS. ~30 LOC.
4. retrieval_routes.rerank_by_title_purity(hits, question, *, shards_dir=None) — lifts the (1+overlap)*(1+purity) shape with stem-aware matching; synonym_expand_strict gated on shards_dir. ~80 LOC.
5. retrieval_routes.rerank_by_ordered_token_match(hits, question) — longest-common-subsequence over query tokens vs title tokens. ~70 LOC.
6. run_query policy-gated chain that invokes them in order: filter → body_density → body_coverage → source_role → title_purity → ordered_token → apply_title_boost (existing tail). ~50 LOC.
7. Re-bench: smoke (must hit 5/5), bench/qa_questions.txt themed subsets (no >5pp drop vs legacy).
8. Only after bench-parity: continue Phase 2 cache persist + legacy collapse.

Risk: byte-identity gate (step 5) catches cache_key drift, but SEMANTIC drift (different primary source picks) requires the bench gate. Legacy bench data in bench/three_way_results/*.jsonl is the reference baseline.

Path B (chosen 2026-05-31)

Stop here. Document the regression. Leave policy.multi_route=False as the default behavior (which it is — policy=None is default and preserves body-only retrieval). Phase 1 infrastructure stays in tree as the FOUNDATION for a future A-path effort; it is not the production user surface today.

Practical implications:

• arborist query (legacy) keeps producing the same answers it always has, INCLUDING the wrong "Edwina" pick on dinosaur Q5.
• arborist corpus-query and arborist cloud query keep producing the better answers via the unified run_query path (no-policy mode, which IS already shipped).
• providence_query exists as a skeleton callers can opt into but is incomplete (cache persist missing).
• No commits revert; the Phase 1 work is genuinely useful even without Phase 2.

When to reopen

When fox wants to either:

• Take a focused multi-day session to land Path A
• Investigate a specific bench regression that suggests the Phase 1 foundation is itself wrong (vs. just incomplete)

Until then this ticket sits open as the gap-tracker. Pattern from CLAUDE.md: "An open ticket whose code already shipped is a stale map" — but here the OPEN-ness is the point. Phase 1 shipped, Phase 2 hasn't. The ticket records the bridge that wasn't built yet.