diff --git a/CLAUDE.md b/CLAUDE.md index 7f0a494..a77091c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -105,6 +105,19 @@ make chain-check-shards # audit-chain break count per shard (0 = intact) - LLM: `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B-FP8-Dynamic on vLLM, 82K ctx, no auth). `uncloseai.com` is marketing only — has no `/v1`. Override via `--endpoint` or `ABORIST_LLM_ENDPOINT`. - Wikipedia dumps: `https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`. `robots.txt` returned 404 → no rules. +## Retrieval pipeline (`aborist/qa/query.py`) + +Multi-stage. Each stage exists because something earlier wasn't enough; revert at your peril. + +1. **Two parallel FTS5 searches per shard, merged.** `_search_corpus` runs body-BM25 AND title-LIKE in parallel and unions hits. The title path closes a recall gap where list-pages with many URLs (e.g. `List_of_HTTP_status_codes`) outrank the actual `HTTP` article on body BM25. See `d608c39 fix retrieval recall`. +2. **Body-coverage `sqrt` rerank.** BM25 systematically favors short docs with rare body tokens (`Tell_(poker)` outranking `Back_to_the_Future` on a film query). Counteract by reranking on `sqrt(body_token_count)` to recover long-doc hits. `1983b79`. +3. **Title-token boost.** `_rerank_by_title` adds `boost × (overlap)` to hits whose title contains query tokens — strong topical signal that BM25 alone misses. +4. **`_filter_by_title_relevance` — three accept paths.** A hit passes if any of: (a) **title-token overlap** meets `title_breadth` (≤2 tokens require ALL; 3+ require N-1); (b) it's in `core_match_roots` (TF-IDF core keywords contain a query token — closes the gap for neologisms like "permacomputer" that never appear in titles); (c) **body density** — `_body_density_passes` requires `distinct_present >= breadth_threshold AND total_mentions >= 3`. Same breadth scaling as titles. Synonym fallback fires only for 1-token queries (otherwise `intel`-titled doc bleeds into AMD queries via the rivalry group). +5. **Rivalry exclusion** (`rivalry_excluded` in `qa/concepts.py`). Intel-titled docs drop from AMD queries; reverse holds. Applies on every accept path. +6. **Stem-aware token matching.** `_stem_token_for_match`: trailing-`s` strip on tokens >4 chars (skip `ss`-enders). Possessive (`superman's` → `supermans` → `superman`) and plural (`girlfriends` → `girlfriend`) collapse onto bare-stem source mentions. Caught the 2026-04-29 *"who is supermans girlfriend?"* defect — pre-stem, the query admitted 7 unrelated `Girlfriends`-titled articles. +7. **Per-source context cap.** Each top-K hit gets at most `max_context_chars / top_k` chars before the global `char_budget` is consumed. Fox's 2026-04-29 catch: `List_of_Batman_comics` (80 KB+ bibliography) was monopolizing the entire 60 KB budget at hit #1, dropping every other source. +8. **Wikitext base prose** runs on the assembled context BEFORE the LLM call (`policy["base_version"]` gates it, see Wikitext base prose convention above). + ## Hot path / gotchas - **Parser is hand-rolled** in `aborist/sources/wikipedia.py` (char-position state machine, escape-aware). After the v9.8 commit it's 4× faster via `str.find` + slicing — easy to break by reverting to char-by-char loops. cProfile any change.