From 631cf506903d13382e13536f6052d7cd1b096002 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Wed, 29 Apr 2026 17:02:38 -0400 Subject: [PATCH] docs: catch CLAUDE.md up to the multi-stage retrieval pipeline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The retrieval contract was undocumented — `_filter_by_title_relevance` has three accept paths, each with its own scaling rules, and the recent breadth + stem refinements (2026-04-29 catches) lived only in commit messages. New "Retrieval pipeline" section between Live endpoints and Hot path / gotchas covers the eight stages in order: 1. Parallel FTS5 (body-BM25 + title-LIKE), unioned per shard 2. Body-coverage sqrt rerank — counter BM25 short-doc bias 3. Title-token boost 4. Three accept paths in _filter_by_title_relevance: title overlap (breadth-scaled: ≤2 require ALL, 3+ require N-1), TF-IDF core keyword match, body density (same breadth scaling plus min_mentions=3 depth) 5. Rivalry exclusion (Intel/AMD groups) 6. Stem-aware token matching: trailing-`s` strip on >4-char tokens (skip `ss`-enders) — 'supermans girlfriend' now collapses onto 'superman'/'girlfriend' instead of admitting `Girlfriends` TV show 7. Per-source context cap = max_context_chars / top_k — prevents 80 KB+ bibliographies from monopolizing the budget 8. Wikitext base-prose normalization before LLM (cross-ref to the Wikitext base prose convention) Each item includes the commit ref or the catch date so an operator can trace why the rule exists. --- CLAUDE.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 7f0a494..a77091c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -105,6 +105,19 @@ make chain-check-shards # audit-chain break count per shard (0 = intact) - LLM: `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B-FP8-Dynamic on vLLM, 82K ctx, no auth). `uncloseai.com` is marketing only — has no `/v1`. Override via `--endpoint` or `ABORIST_LLM_ENDPOINT`. - Wikipedia dumps: `https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`. `robots.txt` returned 404 → no rules. +## Retrieval pipeline (`aborist/qa/query.py`) + +Multi-stage. Each stage exists because something earlier wasn't enough; revert at your peril. + +1. **Two parallel FTS5 searches per shard, merged.** `_search_corpus` runs body-BM25 AND title-LIKE in parallel and unions hits. The title path closes a recall gap where list-pages with many URLs (e.g. `List_of_HTTP_status_codes`) outrank the actual `HTTP` article on body BM25. See `d608c39 fix retrieval recall`. +2. **Body-coverage `sqrt` rerank.** BM25 systematically favors short docs with rare body tokens (`Tell_(poker)` outranking `Back_to_the_Future` on a film query). Counteract by reranking on `sqrt(body_token_count)` to recover long-doc hits. `1983b79`. +3. **Title-token boost.** `_rerank_by_title` adds `boost × (overlap)` to hits whose title contains query tokens — strong topical signal that BM25 alone misses. +4. **`_filter_by_title_relevance` — three accept paths.** A hit passes if any of: (a) **title-token overlap** meets `title_breadth` (≤2 tokens require ALL; 3+ require N-1); (b) it's in `core_match_roots` (TF-IDF core keywords contain a query token — closes the gap for neologisms like "permacomputer" that never appear in titles); (c) **body density** — `_body_density_passes` requires `distinct_present >= breadth_threshold AND total_mentions >= 3`. Same breadth scaling as titles. Synonym fallback fires only for 1-token queries (otherwise `intel`-titled doc bleeds into AMD queries via the rivalry group). +5. **Rivalry exclusion** (`rivalry_excluded` in `qa/concepts.py`). Intel-titled docs drop from AMD queries; reverse holds. Applies on every accept path. +6. **Stem-aware token matching.** `_stem_token_for_match`: trailing-`s` strip on tokens >4 chars (skip `ss`-enders). Possessive (`superman's` → `supermans` → `superman`) and plural (`girlfriends` → `girlfriend`) collapse onto bare-stem source mentions. Caught the 2026-04-29 *"who is supermans girlfriend?"* defect — pre-stem, the query admitted 7 unrelated `Girlfriends`-titled articles. +7. **Per-source context cap.** Each top-K hit gets at most `max_context_chars / top_k` chars before the global `char_budget` is consumed. Fox's 2026-04-29 catch: `List_of_Batman_comics` (80 KB+ bibliography) was monopolizing the entire 60 KB budget at hit #1, dropping every other source. +8. **Wikitext base prose** runs on the assembled context BEFORE the LLM call (`policy["base_version"]` gates it, see Wikitext base prose convention above). + ## Hot path / gotchas - **Parser is hand-rolled** in `aborist/sources/wikipedia.py` (char-position state machine, escape-aware). After the v9.8 commit it's 4× faster via `str.find` + slicing — easy to break by reverting to char-by-char loops. cProfile any change.