The fan-out commit (2b9d1f0) exposed a 13.5s shard-002 fts5_body call
on the Gundremmingen query and hypothesised the synonym OR-pool was
blowing the FTS5 candidate set. Profiling falsified that hypothesis:
synonym_expand returned no synonyms, the OR pool was just the three
query tokens, and the bottleneck was a single high-DF QUERY token
("located": 286,160 matches on a 1.5M-chunk wiki shard) carried into
OR-mode after AND-mode found zero co-occurrences. BM25 ranked all
~290k matches just to pick the top-32.
Two layered fixes:
A. Progressive-AND fallback. When AND returns zero, drop the shortest
token (input order breaks ties) and retry AND. Repeat until hits or
one token left. Only after every chain returns zero do we fall to
OR-mode. On the Gundremmingen case, dropping "located" leaves
"Gundremmingen AND Bavaria" which intersects to 3 docs in 11ms
instead of the 290k-match OR-mode wall.
B. Document-frequency filter at OR-fallback time. ``COUNT(MATCH "tok")``
per OR-pool token; drop any whose corpus DF exceeds
``_OR_FALLBACK_MAX_TOKEN_DF`` (default 50,000). ~15ms warm per
probe. Only fires on the rare path where every progressive-AND
chain still returned zero. Backstops A for queries where the
answer genuinely requires OR (synonym-anchored retrieval, queries
for content that uses different vocabulary than the question) but
one of the OR clauses is a high-DF stopword-adjacent verb.
Both are deletion-first per the five-step algorithm: A deletes the
"jump straight to OR" path, B deletes high-DF tokens that contribute
~zero IDF anyway. No magic constants for A; B has one tunable knob
(threshold).
Bench (cold-cache, n=3, serial workers=1, query "where is
Gundremmingen located? where is Bavaria?"):
metric BEFORE AFTER (A+B) delta
total search wall 57.20s ± 0.22 1.67s ± 0.13 -97% / 34x
shard 002 fts5_body cold 28.07s 0.05s ~560x
shard 002 fts5_body hits 32 3 -29 (the
dropped
were
"located"-
only noise)
Top-K=8 chosen sources unchanged before/after — the dropped fts5_body
candidates were filtered by the title-relevance step downstream
anyway.
Tests (tests/test_search_fts5.py, 11 cases):
- Helper: 5 cases on _progressive_and_token_chains (single-token,
empty, shortest-first, strict length sort, always-keeps-one).
- Search behaviour: 4 cases (progressive-AND drops high-DF token;
full-AND succeeds without progression; OR fallback when no chain
hits; empty result when corpus has neither token nor synonym).
- DF filter: 2 cases (drops high-DF token, keeps input when all
candidates would otherwise be dropped).
Verification:
- make test → 1605 passed, 28 skipped (was 1597 pre-change)
- make chain-check-shards → 0 breaks across all 7 shards
- arborist query "where is Gundremmingen located? where is Bavaria?"
returns the same top-8 sources before/after
russell@unturf.com
416f956734