After the title-search baseline drop, FTS5 BM25 still surfaced short
stub articles ahead of long topical ones for queries with generic
tokens. Intel_4040 (1974 4-bit microcontroller, 1-paragraph stub
mentioning "intel"/"fastest"/"cpu" once each) outscored Intel_Core_2
(real 2010-era CPU article, hundreds of mentions of every query
token) because BM25 normalizes by document length and short docs
that hit every term get an outsized boost.
Add `_rerank_by_body_coverage` between the relevance filter and the
title-overlap rerank. For each surviving candidate (typically 24-32
post-dedup), fetch the full body and sum sqrt(count(t)) over query
tokens, then add `coverage * 0.6` to the score.
Sqrt scaling is intentional: linear sums let enumerative list pages
(e.g. List_of_Intel_microprocessors with 1000+ "intel" mentions)
runaway-dominate, while a hard cap saturates and erases the signal
between short-stub and long-topical articles. Sqrt grows fast enough
to differentiate (5 mentions -> 2.2; 200 mentions -> 14.1) but
sublinearly enough to keep enumerations from overwhelming the merge.
Verified on the 2010-11 enwiki corpus (no distill cores yet):
before: Intel_4040 #1 (43.5), Intel_Core_i7 #6 (39.0)
after: Intel_Core_2 #1 (51.9), Intel_Atom #2 (50.5),
Intel_4040 #3 (49.6)
Body fetch cost is bounded by `over_fetch` (32 candidates per shard,
24-50 unique post-dedup). Each fetch concatenates already-decompressed
chunks via `_load_doc_text` which we use elsewhere in the query path,
so this adds at most one extra DB round-trip per surviving hit.
125 tests still passing.
russell@unturf.com
1983b7928c