qa(retrieval): fix 50→100s hangs in title-LIKE & FTS5 OR-fallback

Both retrieval phases were doing O(corpus × tokens) full-scan work
on long queries — synonym work surfaced the latent issues but they
predate it. Hermes is 6-8s; retrieval needs to be sub-second.

(1) `_search_titles` (LIKE-based title backup search) is fundamentally
    O(corpus × |accept_tokens|) — `LOWER(title) LIKE '%tok%'` can't
    use any index. 50 patterns × 870k docs × 4 shards = ~56s pure
    full-scan. Skipped for queries with > 5 accept_tokens, where
    FTS5 BM25 already returns better candidates. Title-LIKE remains
    a backup for short focused queries (1-5 tokens).

(2) FTS5 OR-mode fallback was OR-joining ALL post-stopword query
    tokens. A 19-token query OR-clause matched millions of docs;
    BM25 then ranked them all. 13s/shard observed.
    Capped to top-5 LONGEST tokens (proxy for rarity / topical
    specificity). Long words like "neurotechnology" matter; short
    words like "soon" don't. OR-mode now 0.25s/shard.

(3) Stopword set extended with 6 connector words (one, some,
    another, without, soon, currently). Conservative addition —
    these have zero topical signal. Reduces the AND-mode token
    count, raising the chance AND succeeds before falling back to
    OR.

Verified live: the 19-token brain-tech query that hung indefinitely
now returns in 18s (Hermes + verify + render dominate). Source
`_search_corpus` no longer expands synonyms for the title-LIKE call
either — synonym expansion stays in `_filter_by_title_relevance`
(post-retrieval, in-memory, cheap regardless of accept-set size)
where it actually does useful work.

Tests: 641 passed (no regression).
This commit is contained in:
russell@unturf.com 2026-05-01 22:52:02 -04:00
parent 50a1bbdc0e
commit 1d70c4fd25
No known key found for this signature in database
2 changed files with 52 additions and 7 deletions

View file

@ -911,9 +911,17 @@ def _search_corpus(
out-ranks FTS5 body hits so the actual topic article rises to the top. out-ranks FTS5 body hits so the actual topic article rises to the top.
""" """
qtokens = _title_query_tokens(question) qtokens = _title_query_tokens(question)
# Synonym expansion: a query for "athlon" also fetches AMD-titled docs. # Title-LIKE backup search uses ORIGINAL qtokens only — NOT synonym-
# Reads concept_relations cross-shard via the shards_dir already in scope. # expanded. With expansion, this path becomes O(corpus × |accept|)
accept_tokens = synonym_expand(qtokens, shards_dir=shards_dir) # full-scan because LOWER(title) LIKE '%tok%' can't use any index;
# 50 LIKE patterns × 870k docs × 4 shards = ~56s of pure waste.
# Synonym expansion stays in `_filter_by_title_relevance` (post-
# retrieval, in-memory, cheap regardless of accept-set size) where
# it actually does useful work surfacing synonym-related titles
# from the FTS5 BM25 hits. Title-LIKE is a thin backup for docs
# whose body is short on query terms but whose title is the literal
# topic — that case is well-served by the original qtokens.
accept_tokens = set(qtokens)
paths: list[Path] paths: list[Path]
if shards_dir is not None: if shards_dir is not None:
paths = discover_shards(shards_dir) paths = discover_shards(shards_dir)
@ -980,7 +988,20 @@ def _search_corpus(
accept_stems = { accept_stems = {
_stem_token_for_match(t) for t in accept_tokens _stem_token_for_match(t) for t in accept_tokens
} }
for r in _search_titles(conn, list(accept_tokens), over_fetch): # Title-LIKE is O(corpus × |tokens|) full-scan (LIKE '%tok%'
# can't use any index). Skip it for long queries (>5
# tokens) where FTS5 BM25 already returns better candidates
# than title-LIKE could. Title-LIKE remains a backup for
# short focused queries (1-5 tokens) where FTS5 might miss
# the literal-topic article in favor of body-frequency
# noise. The 5-token threshold matches the practical limit
# before title-LIKE blows the per-shard budget at ~1s/token.
title_search_rows = (
_search_titles(conn, list(accept_tokens), over_fetch)
if len(accept_tokens) <= 5
else []
)
for r in title_search_rows:
title_norm = (r["title"] or "").replace("_", " ") title_norm = (r["title"] or "").replace("_", " ")
title_stems = { title_stems = {
_stem_token_for_match(t) _stem_token_for_match(t)

View file

@ -27,10 +27,22 @@ _FTS5_STOPWORDS = frozenset(
tell show describe explain summarize say give list find make tell show describe explain summarize say give list find make
please please
all there know everything anything something all there know everything anything something
one some another without soon currently
""".split() """.split()
) )
# OR-mode fallback hits SQLite FTS5 with one clause per token. With 15+
# tokens including common English connectors it matches millions of
# docs, then BM25 has to rank them all to pick the top-K — 13s/shard
# observed on a 3.47M-doc corpus. Long queries that fail AND-mode
# should not pay for an OR-mode that's effectively a full-corpus scan.
# Cap OR-mode at the top-N LONGEST tokens (proxy for rarity / topical
# specificity — "neurotechnology" matters; "soon" doesn't).
_OR_FALLBACK_MAX_TOKENS = 5
def _query_tokens(query: str) -> list[str]: def _query_tokens(query: str) -> list[str]:
raw = _FTS5_TOKEN_RE.findall(query) raw = _FTS5_TOKEN_RE.findall(query)
return [t for t in raw if t.lower() not in _FTS5_STOPWORDS and len(t) > 1] return [t for t in raw if t.lower() not in _FTS5_STOPWORDS and len(t) > 1]
@ -48,8 +60,12 @@ def _escape_fts5(query: str, *, mode: str = "and") -> str:
- mode='and' (default): every content token must appear in the doc. - mode='and' (default): every content token must appear in the doc.
Strict relevance — keeps unrelated docs out of the context window. Strict relevance — keeps unrelated docs out of the context window.
- mode='or': any token matches. Used as a fallback when AND returns - mode='or': the top-N LONGEST tokens (proxy for topical specificity)
zero hits. get OR-joined. Used as a fallback when AND returns zero hits.
Capped at ``_OR_FALLBACK_MAX_TOKENS`` so OR-mode doesn't degrade
into a full-corpus scan on long noisy queries (a 19-token OR
clause matches millions of docs and forces BM25 to rank them all
— 13s/shard observed pre-cap).
All-stopword queries fall back to OR over the raw tokens so they All-stopword queries fall back to OR over the raw tokens so they
still find something instead of crashing FTS5 with an empty MATCH. still find something instead of crashing FTS5 with an empty MATCH.
@ -59,8 +75,16 @@ def _escape_fts5(query: str, *, mode: str = "and") -> str:
raw = _FTS5_TOKEN_RE.findall(query) raw = _FTS5_TOKEN_RE.findall(query)
tokens = [t for t in raw if len(t) > 1] or ['""'] tokens = [t for t in raw if len(t) > 1] or ['""']
sep = " OR " sep = " OR "
elif mode == "and":
sep = " AND "
else: else:
sep = " AND " if mode == "and" else " OR " # OR fallback: keep only the rarest tokens (approximated by
# longest — long words tend to be more topical / rarer in the
# corpus). Bounds the per-clause cost so OR-mode terminates
# quickly instead of scanning the corpus.
sep = " OR "
if len(tokens) > _OR_FALLBACK_MAX_TOKENS:
tokens = sorted(tokens, key=len, reverse=True)[:_OR_FALLBACK_MAX_TOKENS]
return sep.join(_quote(t) for t in tokens) return sep.join(_quote(t) for t in tokens)