engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

qa/retrieval_routes: lift 5 reranks + unfreeze Hit (Path A first attempt — NOT WIRED)

Browse source 
#000072 Path A first attempt: ported the 5 downstream reranks from
legacy query() into arborist.qa.retrieval_routes:
  - body_density_passes / filter_by_body_density (Corpus.doc_body)
  - rerank_by_source_role (SOURCE_ROLE_RANK_WEIGHTS)
  - rerank_by_title_purity ((1+overlap)*(1+purity), shards_dir-gated
    synonym_expand_strict)
  - rerank_by_ordered_token_match (LCS over title tokens)
  - rerank_by_body_coverage (sqrt body coverage, Corpus.doc_body)

Also unfreezes ``arborist.qa.corpus.Hit`` so the reranks can mutate
.score in place (matches legacy _Hit convention). ChunkRow stays
frozen (it's content-addressable evidence). test_hit_is_frozen test
renamed and inverted.

NOT WIRED INTO run_query: smoke probe with all 5 wired in legacy
order (filter → body_density → body_coverage → source_role →
title_purity → ordered_token → apply_title_boost) made the
multi_route regression WORSE:

  pre-reranks:   3/5 correct (Soviet Union ✓, Mt Kilimanjaro ✓,
                              Mona Lisa ✓, Mercury Seven ✗,
                              dinosaurs ✗)
  post-reranks:  1/5 correct (Soviet Union ✗ → "national bandy team",
                              Mt Kilimanjaro ✓,
                              Mona Lisa ✗ → "Painting Mona Lisa",
                              Mercury Seven ✗ → "305th Air Mobility
                              Wing", dinosaurs ✗)

Root cause: legacy's reranks were tuned against legacy's
candidate-set shape (multi-shard parallel _search_corpus with
body-density baked in EARLIER, over_fetch larger than the per_route
limit I'm using, and a different rivalry-exclusion order). Applying
the same multipliers to my multi_route fan-out's candidate set
lands the cascade in a different basin — short noisy titles with
high stem-overlap get amplified into rank-1 territory.

The helpers stay in tree as importable building blocks for a future
Path A v2 attempt. Possible v2 directions: (a) match legacy's
oversample factor (32+ vs my 4×top_k); (b) apply body-density
filter BEFORE rerank cascade (legacy does this earlier in
_search_corpus); (c) rerun against per-shard route output instead
of post-merge candidates so per-shard discrimination survives.

policy=None / experimental multi_route=True paths unchanged in
behavior — multi_route is still strictly worse than body-only
(documented in #000072) but no longer worse than itself with
reranks; reranks aren't auto-applied.

264 tests pass.
This commit is contained in:
russell@unturf.com 2026-05-31 13:18:33 -04:00
parent 4098e41563
commit d0999957e6
No known key found for this signature in database
3 changed files with 291 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

11
tests/test_qa_corpus.py
View file

@ -79,12 +79,17 @@ def shard_conn(tmp_path):
# ---------------------------------------------------------------------------
def test_hit_is_frozen_and_equals_by_value():
def test_hit_is_mutable_and_equals_by_value():
"""Hit is mutable on purpose so the rerank pipeline in
arborist.qa.retrieval_routes can mutate .score in place (matches
legacy _Hit convention). Equality stays value-based."""
h1 = Hit("root1", "uri1", "title1", 1.5)
h2 = Hit("root1", "uri1", "title1", 1.5)
with pytest.raises(Exception):
h1.score = 2.0 # type: ignore[misc]
assert h1 == h2 # value equality
# Mutation works — score is the lever rerank stages use.
h1.score = 2.0
assert h1.score == 2.0
assert h1 != h2 # mutation breaks equality (as expected)
# extras carries a dict so Hit isn't hashable by design — callers
# dedup by (document_root, ...) tuples, not by put-in-a-set.