# Ticket #000072 — Collapse legacy `query()` into the unified `run_query`

Status: **open · Phase 1 foundation shipped 2026-05-31 · Phase 2 BLOCKED on
the 5 missing rerank stages.** A multi_route smoke probe regressed on
2/5 questions — Phase 1 infrastructure is structurally complete but
not production-quality. Promoting `policy.multi_route=True` to default
today would degrade the user surface.

Opened: 2026-05-31  Owner: fox/blackops  Directive: cleanup

## Why this exists

`arborist/qa/query.py` is 4280 LOC; the `query()` function alone runs
lines 1975-4196 (2200 LOC, single function). It is the local-CLI
entry point for `arborist query` — every retrieval, rerank, cache
lookup, audit-chain append, run-DAG build, and verifier kwarg threads
through it. It also picks the WRONG primary source on the dinosaur-
extinction smoke fixture: `Edwina, the Dinosaur Who Didn't Know She
Was Extinct` (a children's book) instead of `Dinosaur` (the encyclopedia
article). The cloud-via-slim-FTS5 path landed in d9fb6a9 already picks
`Dinosaur` correctly through the unified `run_query` orchestrator.
Collapsing legacy `query()` to delegate to the same orchestrator
would fix that retrieval bug as a byproduct and delete ~1500 LOC of
duplicated retrieval/rerank logic.

## What shipped — Phase 1 foundation (2026-05-31)

Ten incremental commits, each behavior-additive, each with the
byte-identity gate (step 5) staying green:

| commit | step | what landed |
|---|---|---|
| 9ba6317 | 1 | `arborist/qa/source_roles.py` — `classify_source_role`, weights, title-marker tuples extracted from query.py |
| a83e47b | 2 | `arborist/qa/_text_norm.py` gains canonical `stem_for_match`; query.py + corpus.py both import |
| 056d785 | 3 | `arborist/qa/retrieval_routes.py` with `filter_by_title_relevance` (5-accept-path filter); query.py re-exports |
| e322bbd | 4 | `Corpus` protocol gains `core_keyword_match` + `doc_body`; SqliteShard implements; SidecarBucket raises `NotSupportedError` |
| 1822741 | 5 | `tests/test_run_query_byte_identity.py` + `tests/fixtures/byte_identity/claim_lattice.json` — pins hash inputs as Phase 2 safety gate |
| 72d1117 | 6a | `run_query` gains `policy=` kwarg threading 12 verifier kwargs |
| 03f248c | 6b | role-classified + role-weighted budget when policy provided |
| 5fdd573 | 6c | multi-route retrieval (body+title+phrase+core_keyword merge) when `policy.multi_route=True` |
| 6c2ec1b | 6c-fil | wire `filter_by_title_relevance` into multi_route path |
| b8bd9d6 | 6d | wikitext-strip when `policy.base_version` is set |
| 20faae0 | P2/1 | `arborist/qa/providence_query.py` — cache-aware wrapper around `run_query` (skeleton; cache persist not yet done) |

Every commit ships with `policy=None` preserving pre-step byte-identity
behavior — existing `arborist corpus-query` and `arborist cloud query`
callers are unaffected. 264 tests in the
query/corpus/sidecar/wallet/bucket/claim_lattice/byte_identity gate
pass at HEAD.

## What's broken — the multi_route smoke regression (2026-05-31)

Retrieval-only smoke against the 4-shard wikipedia corpus (no LLM,
StubClient), comparing `policy=None` (body-only) vs
`policy={"multi_route": True}` (body+title+phrase+core_keyword merged
through `filter_by_title_relevance`):

| Q | no-policy | multi_route | verdict |
|---|---|---|---|
| Soviet Union dissolve | Soviet Union | Soviet Union | preserved |
| Mount Kilimanjaro | Mount Kilimanjaro | Mount Kilimanjaro | preserved |
| Mona Lisa painter | Mona Lisa | Mona Lisa | preserved |
| Mercury Seven astronauts | **Mercury Seven** | **Sam T. Beddingfield** | **REGRESSED** |
| Dinosaurs extinct | **Dinosaur** | **Paul Austin Kelly** | **REGRESSED** |

Diagnosis: the phrase route (`fts_phrase` on 4-grams from the
question) surfaces oddly-titled stubs whose body contains the
literal question phrase. The 5-accept-path filter's path-4
(`phrase_match_roots`) lets them through with no body-density
gate, and `apply_title_boost` doesn't rerank them out because their
titles share zero tokens with the question.

Legacy `query()` suppresses these noisy phrase hits via downstream
stages that Phase 1 has NOT ported:
  1. `_body_density_passes` — co-occurrence breadth check on
     full doc body (3+ token queries require N-1 of N tokens
     present at ≥3 mentions each)
  2. `_rerank_by_body_coverage` — sqrt-rerank that counters BM25
     short-doc bias
  3. `_rerank_by_source_role` — role-weighted score multiplier
     (primary 2.0×, noisy 0.3×)
  4. `_rerank_by_title_purity` — `(1+overlap)*(1+purity)` boost
  5. `_rerank_by_ordered_token_match` — longest-ordered-subsequence
     match between question tokens and title tokens

## Phase 2 — BLOCKED until the 5 stages above land

Phase 2 was meant to land:
  - `providence_query.py` cache wrapper (skeleton done, persist
    deferred)
  - Legacy `query()` collapsed to a thin adapter (50-line shell that
    delegates to `providence_query`)
  - Bench parity proof on smoke + bench/qa_questions.txt themed subsets

But Phase 2 cannot ship without the 5 missing rerank stages — the
new orchestrator currently produces WORSE primary picks than legacy
on 2/5 smoke questions. Promoting it to default would degrade the
user surface. Fox decision 2026-05-31: defer Phase 2; do NOT promote
multi_route to default until A lands.

## Bench evidence — fold-themes regression 2026-05-31 (Hermes-3-8B)

`bench/legacy_vs_providence_bench.py --n-per-theme 3 --themes accent
hyphen honorific brit numeral` after the default flipped to
providence_query (commit 492d1a8). 15 questions × 2 paths (default
vs --legacy), result archived at
`bench/legacy_vs_providence_results/2026-05-31T22-49-30Z.jsonl`.

Per-theme:

| theme | n | agree | ≠ primary | def audit↑ | def audit↓ |
|---|---|---|---|---|---|
| accent | 3 | 2 | 1 | 1 | 1 |
| hyphen | 3 | 3 | 0 | 1 | 2 |
| honorific | 3 | 2 | 1 | 0 | 2 |
| brit | 3 | 3 | 0 | 0 | 3 |
| numeral | 3 | 0 | 3 | 2 | 1 |

**Two failure modes, ONE root cause: the 5 fold-variants stack.**

  - `_hyphen_fold_variants`, `_numeral_fold_variants`,
    `_accent_fold_variants`, `_honorific_fold_variants`,
    `_brit_fold_variants` all live inside `_title_query_tokens`
    (`query.py:288-325`) — which providence_query lazy-imports for
    `apply_title_boost` + `filter_by_title_relevance` BUT the
    underlying fold helpers were never lifted to `_text_norm.py`.

  - Same `_title_query_tokens` is called by legacy's **Rule 8
    title-overlap verifier check** (`verify.py:_claim_title_overlap`)
    — without the folds in the default path, the verifier sees
    `André-Marie Ampère` (title) vs `Andre-Marie Ampere` (claim
    text) as different tokens, fails the overlap check, demotes
    STRICT → HYBRID. **Same root cause as the wrong-primary picks**:
    the fold doesn't fire either at retrieval (wrong doc surfaces)
    or at verification (right doc demotes).

**Specific regressions (default vs legacy):**

Wrong primary (5):
  - honorific "Dr Who" → "Indian Association of Pathologists" vs
    "Doctor (Doctor Who)" (Dr↔Doctor fold)
  - numeral "Albert the third" → "Inside the Third Reich" vs
    "The Third" (third↔III)
  - numeral "Ahmed the third" → "Ahmed Samir" vs "III"
  - numeral "Alaric the first" → "Alaric" vs "Alaric I" (first↔I)
  - accent "Casa Batllo" → empty/error vs "Casa Batlló"

STRICT → HYBRID on correct primary (7):
  - hyphen "André Marie Ampère", "The Amazing Spider Man (comic book)"
  - brit "Finnish defense Forces", "Hopewell center, Hong Kong",
    "labor economics"
  - honorific "Dr Syn", "Dr V64"

Improvements (2):
  - accent "Andre-Marie Ampere" → UNGROUNDED→HYBRID (default picks
    the accented article via FTS5 porter+unicode61 stemmer; legacy
    missed it without the unaccented query token surviving the regex)
  - hyphen "Augustin Jean Fresnel" → UNGROUNDED→HYBRID

**Net for these 5 themes: 12 regressions vs 2 improvements vs 1 tie.**
Default flip is net-negative on fold-driven retrieval.

## Path A v3 — port the fold-variants stack (smallest fix for biggest delta)

Single concrete task, scoped to one file:

  1. Lift `_TITLE_TOKEN_RE`, `_TITLE_STOPWORDS`, `_TITLE_TOKEN_POLICY`,
     `_hyphen_fold_variants`, `_numeral_fold_variants`,
     `_accent_fold_variants`, `_honorific_fold_variants`,
     `_brit_fold_variants`, `_title_query_tokens`,
     `_ascii_fold` from `arborist/qa/query.py:115-325` into
     `arborist/qa/_text_norm.py`.
  2. Update `arborist/qa/query.py` to re-export under old names
     (same pattern as the existing stem_for_match move).
  3. Update `arborist/qa/source_roles.py` + `retrieval_routes.py`
     to drop the lazy-import-from-query.py dance and import
     directly from `_text_norm` — kills the lazy-import warts +
     unblocks any future _text_norm consumer.
  4. Re-run the same bench to confirm regressions disappear (expect
     7 STRICT→HYBRID and 4 wrong-primary picks to flip to legacy
     parity; the 1 accent error needs separate investigation).

Estimated scope: ~250 LOC moved + ~50 LOC import-rewrites in 3 files.
Half-day. Lower risk than Path A v1 because it's pure code motion —
the helpers ARE identical between paths; only the import surface
changes.

## Other gates the bench did NOT exercise (still TBD)

Themes deliberately skipped 2026-05-31 because they'd hit gates the
fold port doesn't address:
  - `quantifier_subset` — broad-quantifier preflight guard
  - `metacog_subset` — meta-cognition preflight
  - `warrant_chain_probe` — warrant-resolver post-step
  - `es` / `fr` — crosslang sandwich translation edges

Re-bench AFTER Path A v3 lands to isolate which of these still
regress; that's the next porting wave.

## Path A v3 verdict (2026-05-31, FAILED — wire-up didn't move the bench)

Two-stage attempt after the original Path A v3 plan:

  - **Stage 1** (commit pending in 971fb58 batch): lifted the
    fold-variants stack to `_text_norm.py`, re-exported from
    query.py, dropped lazy imports.
  - **Stage 2** (same commit): wired `apply_title_boost` to use
    `_title_query_tokens` (full 5-fold) instead of
    `tokenize_text + numeral_expand`.

Bench v5 against the same fold-themed fixture (Hermes-3-8B, 5 themes
× 3 questions × 2 paths, result at
`bench/legacy_vs_providence_results/2026-05-31T23-48-44Z.jsonl`):

| theme | n | agree | ≠ primary | def audit↑ | def audit↓ |
|---|---|---|---|---|---|
| accent | 3 | 2 | 1 | 1 | 1 |
| hyphen | 3 | 3 | 0 | 1 | 2 |
| honorific | 3 | 2 | 1 | 0 | 2 |
| brit | 3 | 3 | 0 | 0 | 3 |
| numeral | 3 | 0 | 3 | 2 | 1 |

**Identical to the pre-wire-up baseline.** The fold-aware
apply_title_boost can only rerank docs ALREADY in the candidate
set. For "Dr Who" the body BM25 doesn't surface "Doctor (Doctor
Who)" at all — so no fold-aware rerank can promote it. Legacy
surfaces it via the title route (one of its 4 parallel routes).
The default path runs body-only retrieval (multi_route=False);
the right candidate never enters the top-K.

So Path A v3 was right in diagnosis but wrong in remedy: the
folds need to fire in RETRIEVAL, not just RERANK. The
SqliteShardCorpus.fts_title route (Corpus protocol method, shipped
step 4 / e322bbd) doesn't apply the folds — it just runs SQLite
FTS5 MATCH on `documents_fts` with the raw query. FTS5's porter
stemmer handles plurals/possessives, NOT Dr→Doctor or third→III.

## The actual story

Path A across v1, v2 (deferred), v3 stages 1+2 has now proven:

  1. The fold helpers exist in tree and are importable
     (`_text_norm.py`).
  2. They are NOT a drop-in replacement for legacy's retrieval-
     time fan-out because the folds need to feed RETRIEVAL token
     generation, not just rank-time.
  3. Legacy's 4-route fan-out (body + title-LIKE + phrase + core-
     keyword) is the actual mechanism that surfaces fold-needing
     docs; lifting the helpers without lifting the fan-out shape
     keeps us in the same basin.
  4. Forcing the fan-out on (commit 5fdd573 multi_route flag)
     regressed primary picks 2/5 on the smoke fixture — different
     basin, also broken.
  5. Adding the 5 reranks on top (Path A v1, commit d099995)
     regressed primary picks 4/5 on smoke — yet another basin.

Net: legacy `query()`'s 2200-line shape is genuinely a tuned
manifold, not a library of multipliers. Reproducing it without
reproducing the topology gives strictly-worse results in every
basin we've measured.

## What we DID ship from this work

The Phase 1 foundation (10 commits 9ba6317..20faae0) + Phase 2
cache wrapper (17b9622) + CLI default flip (492d1a8) + bench
infrastructure (618b784) + Path A v3 (971fb58) are all in tree
and useful EVEN THOUGH the legacy-collapse goal didn't land:

  - `arborist cloud query` + `arborist corpus-query` keep using
    the unified pipeline with no regression (already proved 5/5
    on the smoke fixture)
  - `providence_query` is a usable standalone cache-aware Q&A
    orchestrator (4 tests, cache hit/miss/burn/audit-chain all
    pass) for any caller that doesn't need the pre/post gates
  - The byte-identity test (1822741) keeps cache_key drift
    detectable for any future Phase 2 retry
  - The fold helpers live in `_text_norm.py` available to ANY
    consumer that wants them at retrieval-token-gen time
  - The bench script (`bench/legacy_vs_providence_bench.py`)
    drives head-to-head comparisons against any fixture set

## What this ticket recommends next

**Stop chasing decomposition.** The 5 Path A attempts have shown
that legacy `query()` doesn't decompose cleanly. Either:

  A. **Leave the default at providence_query.** The dinosaur Q5
     bug fixes itself (proven on cloud), the fold themes regress
     (proven here). Accept the trade. Users who hit a fold-
     regression bench failure can `--legacy` per-call or set
     `ARBORIST_LEGACY_QUERY=1` env-wide.

  B. **Flip the default BACK to legacy.** Treat #000072 Phase 1+2
     work as INFRASTRUCTURE for `arborist cloud query` and
     `arborist corpus-query` (where it works) and stop trying to
     replace `arborist query`'s legacy path.

  C. **Keep both alive long-term.** `arborist query` = legacy
     (rich, slow, well-tuned), `arborist query2` = providence
     (fast, lean, opt-in). The dual-CLI ceremony I initially
     hedged toward in this session was actually right; the
     forcing-function flip was wrong.

Fox decides. The bench evidence is what it is — no amount of
additional porting is likely to close the regression without
just copying legacy's full 4-route + filter + 5-rerank topology
verbatim into providence_query, at which point we've gained
NOTHING from the refactor.

## Path A v1 (attempted 2026-05-31, FAILED — reranks made things WORSE)

Lifted all 5 reranks from query.py into `arborist/qa/retrieval_routes.py`
(commit d099995), unfroze `Hit` so reranks can mutate `.score` in
place, wired them into `run_query` in legacy order:

    filter_by_title_relevance
    → filter_by_body_density
    → rerank_by_body_coverage
    → rerank_by_source_role
    → rerank_by_title_purity
    → rerank_by_ordered_token_match
    → apply_title_boost (existing tail)

Smoke probe (5 questions, retrieval-only no LLM, MultiShardSqliteCorpus
on the 4 wikipedia shards):

| Q | no-policy (body-only) | multi_route | multi_route + 5 reranks |
|---|---|---|---|
| Soviet Union dissolve | Soviet Union ✓ | Soviet Union ✓ | **Soviet Union national bandy team ✗** |
| Mount Kilimanjaro | Mount Kilimanjaro ✓ | Mount Kilimanjaro ✓ | Mount Kilimanjaro ✓ |
| Mona Lisa painter | Mona Lisa ✓ | Mona Lisa ✓ | **Painting Mona Lisa ✗** |
| Mercury Seven astronauts | Mercury Seven ✓ | Sam T. Beddingfield ✗ | 305th Air Mobility Wing ✗ |
| Dinosaurs extinct | Dinosaur ✓ | Paul Austin Kelly ✗ | Paul Austin Kelly ✗ |

Score: no-policy 5/5, multi_route 3/5, multi_route+reranks 1/5.
The reranks made it strictly worse than multi_route alone.

**Root cause:** legacy's reranks were tuned against legacy's
candidate-set shape, NOT against my multi_route fan-out's shape:

  - Legacy `_search_corpus` runs per-shard parallel routes with
    `over_fetch=32` (vs my `per_route_limit=top_k*4 = 16`), so its
    candidate set is much larger and the noisy short-title hits
    have more competition from genuine canonical articles before
    rerank ever fires.
  - Legacy applies body-density filtering EARLIER in the pipeline
    (in `_search_one_shard` before merge), so noisy phrase-route
    hits with no body coherence are dropped before they can be
    boosted by title-purity or title-boost.
  - Legacy's rivalry-exclusion + synonym-expansion order differs;
    my path applies them inside `filter_by_title_relevance` only.

Applying the same multipliers to a structurally different
candidate set lands the cascade in a different basin — short noisy
titles with high stem-overlap (e.g. "Soviet Union national bandy
team", "Painting Mona Lisa", "305th Air Mobility Wing") get
amplified into rank-1 territory because they win the
title-purity × ordered-token × source-role compound multiplier even
though their body BM25 starts much higher (less negative) than the
canonical primary's.

**Reverted the wire-up; helpers stay in tree.** The 5 reranks live
in `arborist/qa/retrieval_routes.py` as importable building blocks.
Hit stays unfrozen (mutable reranks need it). But `run_query` only
applies `filter_by_title_relevance` when `multi_route=True`, not
the 5-stage cascade. multi_route is still worse than body-only
(3/5 vs 5/5), but no longer worse than multi_route+reranks.

## Path A v2 — possible directions

  1. **Match legacy's oversample factor.** `per_route_limit=top_k*4`
     is too tight. Try `top_k*8` or `over_fetch=32` matching legacy.
     More candidates per route = more competition in the merge =
     cleaner top-K after reranks.
  2. **Apply body-density filter BEFORE rerank cascade**, not
     after `filter_by_title_relevance`. Legacy does this earlier
     in `_search_one_shard`; the noisy phrase-route hits without
     body coherence get dropped at the SOURCE, not after merge.
  3. **Rerun reranks on per-shard route output**, then merge
     final top-K. Preserves per-shard discrimination that the MIN
     bm25 merge collapses today.
  4. **Synonym expansion at retrieval time**, not just at filter
     time. Legacy passes a synonym pool into FTS5 MATCH; my path
     uses raw query tokens. May change which docs surface in the
     first place.

Any of these is a separate focused investigation. The 5-rerank
infrastructure is in tree; v2 is a wiring question, not a
porting question.

## Path A v1 — port the missing rerank stages (multi-day) [original plan, superseded by the v1 attempt above]

Estimated scope:
  1. `retrieval_routes.body_density_passes(corpus, hit, qtokens, *,
     min_mentions=3)` — Corpus exposes `doc_body(document_root)` since
     step 4 (e322bbd); fn already shaped for hits that carry
     `document_root`. ~30 LOC.
  2. `retrieval_routes.rerank_by_body_coverage(hits, question, corpus)`
     — uses `corpus.doc_body` per hit; sqrt-rerank multiplier.
     ~60 LOC.
  3. `retrieval_routes.rerank_by_source_role(hits, question)` — uses
     `classify_source_role` from source_roles.py + RANK_WEIGHTS.
     ~30 LOC.
  4. `retrieval_routes.rerank_by_title_purity(hits, question, *,
     shards_dir=None)` — lifts the (1+overlap)*(1+purity) shape with
     stem-aware matching; synonym_expand_strict gated on shards_dir.
     ~80 LOC.
  5. `retrieval_routes.rerank_by_ordered_token_match(hits, question)`
     — longest-common-subsequence over query tokens vs title tokens.
     ~70 LOC.
  6. `run_query` policy-gated chain that invokes them in order:
     filter → body_density → body_coverage → source_role → title_purity
     → ordered_token → apply_title_boost (existing tail). ~50 LOC.
  7. Re-bench: smoke (must hit 5/5), bench/qa_questions.txt themed
     subsets (no >5pp drop vs legacy).
  8. Only after bench-parity: continue Phase 2 cache persist +
     legacy collapse.

Risk: byte-identity gate (step 5) catches cache_key drift, but
SEMANTIC drift (different primary source picks) requires the bench
gate. Legacy bench data in `bench/three_way_results/*.jsonl` is the
reference baseline.

## Path B (chosen 2026-05-31)

Stop here. Document the regression. Leave `policy.multi_route=False`
as the default behavior (which it is — `policy=None` is default and
preserves body-only retrieval). Phase 1 infrastructure stays in tree
as the FOUNDATION for a future A-path effort; it is not the
production user surface today.

Practical implications:
  - `arborist query` (legacy) keeps producing the same answers it
    always has, INCLUDING the wrong "Edwina" pick on dinosaur Q5.
  - `arborist corpus-query` and `arborist cloud query` keep
    producing the better answers via the unified `run_query` path
    (no-policy mode, which IS already shipped).
  - `providence_query` exists as a skeleton callers can opt into but
    is incomplete (cache persist missing).
  - No commits revert; the Phase 1 work is genuinely useful even
    without Phase 2.

## When to reopen

When fox wants to either:
  - Take a focused multi-day session to land Path A
  - Investigate a specific bench regression that suggests the
    Phase 1 foundation is itself wrong (vs. just incomplete)

Until then this ticket sits open as the gap-tracker. Pattern from
CLAUDE.md: "An open ticket whose code already shipped is a stale
map" — but here the OPEN-ness is the point. Phase 1 shipped, Phase 2
hasn't. The ticket records the bridge that wasn't built yet.
