qa/retrieval_routes: lift 5 reranks + unfreeze Hit (Path A first attempt — NOT WIRED)

#000072 Path A first attempt: ported the 5 downstream reranks from
legacy query() into arborist.qa.retrieval_routes:
  - body_density_passes / filter_by_body_density (Corpus.doc_body)
  - rerank_by_source_role (SOURCE_ROLE_RANK_WEIGHTS)
  - rerank_by_title_purity ((1+overlap)*(1+purity), shards_dir-gated
    synonym_expand_strict)
  - rerank_by_ordered_token_match (LCS over title tokens)
  - rerank_by_body_coverage (sqrt body coverage, Corpus.doc_body)

Also unfreezes ``arborist.qa.corpus.Hit`` so the reranks can mutate
.score in place (matches legacy _Hit convention). ChunkRow stays
frozen (it's content-addressable evidence). test_hit_is_frozen test
renamed and inverted.

NOT WIRED INTO run_query: smoke probe with all 5 wired in legacy
order (filter → body_density → body_coverage → source_role →
title_purity → ordered_token → apply_title_boost) made the
multi_route regression WORSE:

  pre-reranks:   3/5 correct (Soviet Union ✓, Mt Kilimanjaro ✓,
                              Mona Lisa ✓, Mercury Seven ✗,
                              dinosaurs ✗)
  post-reranks:  1/5 correct (Soviet Union ✗ → "national bandy team",
                              Mt Kilimanjaro ✓,
                              Mona Lisa ✗ → "Painting Mona Lisa",
                              Mercury Seven ✗ → "305th Air Mobility
                              Wing", dinosaurs ✗)

Root cause: legacy's reranks were tuned against legacy's
candidate-set shape (multi-shard parallel _search_corpus with
body-density baked in EARLIER, over_fetch larger than the per_route
limit I'm using, and a different rivalry-exclusion order). Applying
the same multipliers to my multi_route fan-out's candidate set
lands the cascade in a different basin — short noisy titles with
high stem-overlap get amplified into rank-1 territory.

The helpers stay in tree as importable building blocks for a future
Path A v2 attempt. Possible v2 directions: (a) match legacy's
oversample factor (32+ vs my 4×top_k); (b) apply body-density
filter BEFORE rerank cascade (legacy does this earlier in
_search_corpus); (c) rerun against per-shard route output instead
of post-merge candidates so per-shard discrimination survives.

policy=None / experimental multi_route=True paths unchanged in
behavior — multi_route is still strictly worse than body-only
(documented in #000072) but no longer worse than itself with
reranks; reranks aren't auto-applied.

264 tests pass.

arborist/qa/corpus.py

@@ -134,10 +134,20 @@ def apply_title_boost(
     return rescored
 
 
-@dataclass(frozen=True)
+@dataclass
 class Hit:
     """One retrieval result, backend-agnostic.
 
+    Mutable on purpose: the rerank pipeline in
+    ``arborist.qa.retrieval_routes`` mutates ``.score`` (and tags
+    ``.extras["source_role"]``) in place as a sequence of multiplier
+    stages. Frozen would block that and force every rerank to rebuild
+    the list via ``dataclasses.replace``, doubling allocation and
+    losing the LCS-on-sort optimization legacy ``query()`` relies on.
+    Hit objects are short-lived per-request artifacts — they never
+    enter the persistence layer (that's ChunkRow + the verifier's
+    evidence map).
+
     ``score`` semantics differ per backend (FTS5 BM25 ranges negative
     [-20, 0]; sidecar BM25 ranges positive [+25, +60]). Callers that
     merge across backends must use rank or a normalization layer —

arborist/qa/retrieval_routes.py

@@ -1,15 +1,16 @@
 """Retrieval orchestration helpers shared between legacy ``query()``
-and the unified ``run_query()`` (Phase 1 step 3 of #53).
+and the unified ``run_query()`` (#000072 Phase 1).
 
-Functions here take pre-retrieved Hits + a question and return filtered
+Functions here take pre-retrieved Hits + a question and return
-or reranked Hits. They never open a sqlite connection directly — the
+filtered or reranked Hits. They never open a sqlite connection
-caller does the data fetch via the Corpus protocol and hands the
+directly — the caller does the data fetch via the Corpus protocol
-results in. Backwards-compatible with both the legacy ``_Hit``
+and hands the results in. Backwards-compatible with both the legacy
-dataclass and the protocol ``arborist.qa.corpus.Hit`` since both
+``_Hit`` dataclass and the protocol ``arborist.qa.corpus.Hit`` since
-expose ``.title`` and ``.document_root``.
+both expose ``.title``, ``.document_root``, and ``.score``.
 """
 from __future__ import annotations
 
+import math as _math
 import re as _re
 from typing import Callable, Iterable
 
@@ -170,3 +171,267 @@ def filter_by_title_relevance(
     if not kept:
         return hits[: max(1, fallback_top_n)] if hits else []
     return kept
+
+
+# ===========================================================================
+# Body-density gate + 5 reranks (#000072 Phase 2 prep: parity with legacy
+# query()'s downstream suppression of noisy phrase-route hits).
+# ===========================================================================
+
+
+def _body_count_with_stem(body: str, t: str) -> int:
+    """Count mentions of ``t`` in ``body``, falling back to the
+    lite-stemmed form if the literal didn't match. Returns the LARGER
+    of the two counts so a query token that appears under both forms
+    (rare) still scores. Lifted verbatim from legacy
+    arborist.qa.query._body_count_with_stem."""
+    n_literal = body.count(t)
+    if n_literal:
+        return n_literal
+    stem = stem_for_match(t)
+    if stem != t:
+        return body.count(stem)
+    return 0
+
+
+def body_density_passes(
+    corpus, document_root: str, qtokens: set[str],
+    *, min_mentions: int = 3,
+) -> bool:
+    """Body-token CO-OCCURRENCE relevance gate.
+
+    Breadth scales with query length:
+        ≤ 2 tokens   require ALL present in body
+        3+ tokens    require N - 1 (allow one weak signal token to miss)
+    Depth: ``total_mentions >= min_mentions`` also enforced.
+
+    Catches the "Girlfriends (TV show) passing a supermans girlfriend
+    query because body had 'girlfriend' but no 'superman' whatsoever"
+    failure mode. Stem-tolerant via ``_body_count_with_stem``.
+
+    Uses ``corpus.doc_body(document_root)`` (shipped in Corpus protocol
+    step 4 / commit e322bbd) so works against any backend.
+    """
+    if not qtokens:
+        return False
+    body = corpus.doc_body(document_root)
+    if not body:
+        return False
+    body = body.lower()
+    counts = {t: _body_count_with_stem(body, t.lower()) for t in qtokens}
+    distinct_present = sum(1 for n in counts.values() if n > 0)
+    total_mentions = sum(counts.values())
+    if len(qtokens) <= 2:
+        breadth_threshold = len(qtokens)
+    else:
+        breadth_threshold = len(qtokens) - 1
+    return distinct_present >= breadth_threshold and total_mentions >= min_mentions
+
+
+def filter_by_body_density(
+    hits: list, question: str, corpus, *, min_mentions: int = 3,
+) -> list:
+    """Run body_density_passes on each hit; keep only those that
+    pass. Fall-open if all hits fail (don't strand the LLM with no
+    context — legacy convention).
+    """
+    from arborist.qa.query import _title_query_tokens
+    qtokens = _title_query_tokens(question)
+    if not qtokens:
+        return hits
+    kept = [
+        h for h in hits
+        if body_density_passes(
+            corpus, h.document_root, qtokens, min_mentions=min_mentions,
+        )
+    ]
+    return kept if kept else hits
+
+
+def rerank_by_source_role(hits: list, question: str) -> list:
+    """Classify each hit's source_role + rescale .score by
+    SOURCE_ROLE_RANK_WEIGHTS.
+
+    Mutates ``h.extras["source_role"]`` (and ``h.source_role`` on
+    legacy _Hit objects) so downstream context-build can reuse the
+    value. Sort by score (lower = better for FTS5 bm25 negative;
+    DESC if scores are positive). Default-DESC matches legacy
+    behavior; FTS5-negative inputs end up sorted as more-negative-
+    first which is still "best first" if signs stay homogeneous.
+    """
+    from arborist.qa.query import _title_query_tokens
+    from arborist.qa.source_roles import (
+        SOURCE_ROLE_RANK_WEIGHTS,
+        classify_source_role,
+    )
+    qtokens_stem = {
+        stem_for_match(t.lower())
+        for t in _title_query_tokens(question)
+    }
+    for h in hits:
+        role = classify_source_role(
+            h.title, qtokens_stem,
+            document_uri=getattr(h, "document_uri", None),
+        )
+        # Mutate both .source_role (legacy _Hit) and .extras (Hit
+        # dataclass with frozen-ish fields) without breaking either.
+        try:
+            h.source_role = role
+        except (AttributeError, TypeError):
+            pass
+        if hasattr(h, "extras") and isinstance(h.extras, dict):
+            h.extras["source_role"] = role
+        weight = SOURCE_ROLE_RANK_WEIGHTS.get(role, 1.0)
+        # In-place score multiplication. Hit dataclass uses _replace
+        # for immutable fields but our Hit is plain @dataclass and
+        # score is mutable.
+        h.score = h.score * weight
+    hits.sort(key=lambda h: -h.score)
+    return hits
+
+
+def rerank_by_title_purity(
+    hits: list, question: str, *, shards_dir=None,
+) -> list:
+    """Boost titles by purity AND multi-token-match breadth.
+
+    Multiplier ``(1 + overlap_count) * (1 + purity)``:
+      - purity = |title∩query| / |title_tokens| (rewards clean titles)
+      - overlap_count = |title∩query| (rewards multi-token coverage)
+
+    Examples:
+      ``Jurassic Park (film)`` (overlap 2, purity 1.0) → 6.0×
+      ``Jurassic Park (NES game)`` (overlap 2, purity 0.5) → 4.5×
+
+    Optionally expand qtokens via the STRICT synonym view (manual +
+    acronym_parens evidence) when ``shards_dir`` provided. Strict
+    avoids the broad-view link_reciprocity noise tail.
+
+    Stem-aware via stem_for_match (possessive/plural collapse).
+    """
+    from arborist.qa.query import _title_query_tokens
+    qtokens = _title_query_tokens(question)
+    if not qtokens:
+        return hits
+    if shards_dir is not None:
+        try:
+            from arborist.concepts import synonym_expand_strict
+            qtokens = synonym_expand_strict(
+                qtokens, shards_dir=shards_dir,
+            ) or qtokens
+        except (ImportError, Exception):
+            pass
+    qstems = {stem_for_match(t) for t in qtokens}
+    for h in hits:
+        title = getattr(h, "title", "") or ""
+        if not title:
+            continue
+        ttokens = _title_query_tokens(title.replace("_", " "))
+        if not ttokens:
+            continue
+        tstems = {stem_for_match(t) for t in ttokens}
+        overlap = tstems & qstems
+        if not overlap:
+            continue
+        purity = len(overlap) / len(tstems)
+        overlap_count = len(overlap)
+        h.score = h.score * (1.0 + overlap_count) * (1.0 + purity)
+    hits.sort(key=lambda h: -h.score)
+    return hits
+
+
+def _ordered_match_length(
+    query_tokens: list[str], title_tokens: list[str],
+) -> int:
+    """Longest common subsequence length over two token lists. O(N*M).
+    Both lists are typically <10 in practice so cost is negligible.
+    """
+    if not query_tokens or not title_tokens:
+        return 0
+    n = len(query_tokens)
+    m = len(title_tokens)
+    prev = [0] * (m + 1)
+    for i in range(1, n + 1):
+        cur = [0] * (m + 1)
+        for j in range(1, m + 1):
+            if query_tokens[i - 1] == title_tokens[j - 1]:
+                cur[j] = prev[j - 1] + 1
+            else:
+                cur[j] = max(cur[j - 1], prev[j])
+        prev = cur
+    return prev[m]
+
+
+def rerank_by_ordered_token_match(hits: list, question: str) -> list:
+    """Boost titles whose tokens appear in the same order as the
+    query. Multiplier ``1 + 0.5 * (match_length - 1)`` for
+    match_length ≥ 2. Stem-aware. No-op on single-token queries
+    (no order to match).
+
+    Caught the 2026-05-01 "plot of red fish blue fish?" defect:
+    the Dr. Seuss book's title (ordered-match 4) sits cleanly
+    above Red Dwarf / Toronto Blue Jays (ordered-match 1).
+    """
+    from arborist.qa.query import (
+        _title_query_tokens, _TITLE_TOKEN_RE, _TITLE_STOPWORDS,
+    )
+    qtokens = _title_query_tokens(question)
+    if len(qtokens) < 2:
+        return hits
+    qtokens_ordered: list[str] = []
+    seen: set[str] = set()
+    for tok in _TITLE_TOKEN_RE.findall(question):
+        t = tok.lower()
+        if t in _TITLE_STOPWORDS or len(t) <= 1 or t in seen:
+            continue
+        seen.add(t)
+        qtokens_ordered.append(stem_for_match(t))
+    if len(qtokens_ordered) < 2:
+        return hits
+    for h in hits:
+        title = getattr(h, "title", "") or ""
+        if not title:
+            continue
+        ttokens_ordered: list[str] = []
+        title_seen: set[str] = set()
+        for tok in _TITLE_TOKEN_RE.findall(title.replace("_", " ")):
+            t = tok.lower()
+            if t in _TITLE_STOPWORDS or len(t) <= 1 or t in title_seen:
+                continue
+            title_seen.add(t)
+            ttokens_ordered.append(stem_for_match(t))
+        if not ttokens_ordered:
+            continue
+        match_len = _ordered_match_length(qtokens_ordered, ttokens_ordered)
+        if match_len >= 2:
+            h.score = h.score * (1.0 + 0.5 * (match_len - 1))
+    hits.sort(key=lambda h: -h.score)
+    return hits
+
+
+def rerank_by_body_coverage(
+    hits: list, question: str, corpus, *, weight: float = 0.6,
+) -> list:
+    """Boost score by per-token body coverage. sqrt-scaling lets long
+    topical articles meaningfully out-score short tangential ones
+    without runaway domination by enumerative list pages.
+
+    Counters BM25's short-doc bias. Cost: one body fetch per
+    surviving candidate via ``corpus.doc_body`` (cheap on local
+    SQLite; the cloud path pays an HTTP fetch per doc).
+    """
+    from arborist.qa.query import _title_query_tokens
+    qtokens_lower = {t.lower() for t in _title_query_tokens(question)}
+    if not qtokens_lower:
+        return hits
+    for h in hits:
+        body = corpus.doc_body(h.document_root)
+        if not body:
+            continue
+        body_lower = body.lower()
+        coverage = sum(
+            _math.sqrt(body_lower.count(t)) for t in qtokens_lower
+        )
+        h.score += coverage * weight
+    hits.sort(key=lambda h: -h.score)
+    return hits

tests/test_qa_corpus.py

@@ -79,12 +79,17 @@ def shard_conn(tmp_path):
 # ---------------------------------------------------------------------------
 
 
-def test_hit_is_frozen_and_equals_by_value():
+def test_hit_is_mutable_and_equals_by_value():
+    """Hit is mutable on purpose so the rerank pipeline in
+    arborist.qa.retrieval_routes can mutate .score in place (matches
+    legacy _Hit convention). Equality stays value-based."""
     h1 = Hit("root1", "uri1", "title1", 1.5)
     h2 = Hit("root1", "uri1", "title1", 1.5)
-    with pytest.raises(Exception):
-        h1.score = 2.0          # type: ignore[misc]
     assert h1 == h2             # value equality
+    # Mutation works — score is the lever rerank stages use.
+    h1.score = 2.0
+    assert h1.score == 2.0
+    assert h1 != h2             # mutation breaks equality (as expected)
     # extras carries a dict so Hit isn't hashable by design — callers
     # dedup by (document_root, ...) tuples, not by put-in-a-set.