engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: catch CLAUDE.md up to the multi-stage retrieval pipeline

Browse source 
The retrieval contract was undocumented — `_filter_by_title_relevance`
has three accept paths, each with its own scaling rules, and the recent
breadth + stem refinements (2026-04-29 catches) lived only in commit
messages. New "Retrieval pipeline" section between Live endpoints and
Hot path / gotchas covers the eight stages in order:

1. Parallel FTS5 (body-BM25 + title-LIKE), unioned per shard
2. Body-coverage sqrt rerank — counter BM25 short-doc bias
3. Title-token boost
4. Three accept paths in _filter_by_title_relevance:
   title overlap (breadth-scaled: ≤2 require ALL, 3+ require N-1),
   TF-IDF core keyword match, body density (same breadth scaling
   plus min_mentions=3 depth)
5. Rivalry exclusion (Intel/AMD groups)
6. Stem-aware token matching: trailing-`s` strip on >4-char tokens
   (skip `ss`-enders) — 'supermans girlfriend' now collapses onto
   'superman'/'girlfriend' instead of admitting `Girlfriends` TV show
7. Per-source context cap = max_context_chars / top_k — prevents
   80 KB+ bibliographies from monopolizing the budget
8. Wikitext base-prose normalization before LLM (cross-ref to
   the Wikitext base prose convention)

Each item includes the commit ref or the catch date so an operator
can trace why the rule exists.
This commit is contained in:
russell@unturf.com 2026-04-29 17:02:38 -04:00
parent e668a8355f
commit 631cf50690
No known key found for this signature in database
1 changed files with 13 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

13
CLAUDE.md
View file

@ -105,6 +105,19 @@ make chain-check-shards # audit-chain break count per shard (0 = intact)
- LLM: `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B-FP8-Dynamic on vLLM, 82K ctx, no auth). `uncloseai.com` is marketing only — has no `/v1`. Override via `--endpoint` or `ABORIST_LLM_ENDPOINT`.
- Wikipedia dumps: `https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`. `robots.txt` returned 404 → no rules.
## Retrieval pipeline (`aborist/qa/query.py`)
Multi-stage. Each stage exists because something earlier wasn't enough; revert at your peril.
1. **Two parallel FTS5 searches per shard, merged.** `_search_corpus` runs body-BM25 AND title-LIKE in parallel and unions hits. The title path closes a recall gap where list-pages with many URLs (e.g. `List_of_HTTP_status_codes`) outrank the actual `HTTP` article on body BM25. See `d608c39 fix retrieval recall`.
2. **Body-coverage `sqrt` rerank.** BM25 systematically favors short docs with rare body tokens (`Tell_(poker)` outranking `Back_to_the_Future` on a film query). Counteract by reranking on `sqrt(body_token_count)` to recover long-doc hits. `1983b79`.
3. **Title-token boost.** `_rerank_by_title` adds `boost × (overlap)` to hits whose title contains query tokens — strong topical signal that BM25 alone misses.
4. **`_filter_by_title_relevance` — three accept paths.** A hit passes if any of: (a) **title-token overlap** meets `title_breadth` (≤2 tokens require ALL; 3+ require N-1); (b) it's in `core_match_roots` (TF-IDF core keywords contain a query token — closes the gap for neologisms like "permacomputer" that never appear in titles); (c) **body density** — `_body_density_passes` requires `distinct_present >= breadth_threshold AND total_mentions >= 3`. Same breadth scaling as titles. Synonym fallback fires only for 1-token queries (otherwise `intel`-titled doc bleeds into AMD queries via the rivalry group).
5. **Rivalry exclusion** (`rivalry_excluded` in `qa/concepts.py`). Intel-titled docs drop from AMD queries; reverse holds. Applies on every accept path.
6. **Stem-aware token matching.** `_stem_token_for_match`: trailing-`s` strip on tokens >4 chars (skip `ss`-enders). Possessive (`superman's` → `supermans` → `superman`) and plural (`girlfriends` → `girlfriend`) collapse onto bare-stem source mentions. Caught the 2026-04-29 *"who is supermans girlfriend?"* defect — pre-stem, the query admitted 7 unrelated `Girlfriends`-titled articles.
7. **Per-source context cap.** Each top-K hit gets at most `max_context_chars / top_k` chars before the global `char_budget` is consumed. Fox's 2026-04-29 catch: `List_of_Batman_comics` (80 KB+ bibliography) was monopolizing the entire 60 KB budget at hit #1, dropping every other source.
8. **Wikitext base prose** runs on the assembled context BEFORE the LLM call (`policy["base_version"]` gates it, see Wikitext base prose convention above).
## Hot path / gotchas
- **Parser is hand-rolled** in `aborist/sources/wikipedia.py` (char-position state machine, escape-aware). After the v9.8 commit it's 4× faster via `str.find` + slicing — easy to break by reverting to char-by-char loops. cProfile any change.