qa(retrieval): synonym-aware FTS5 OR-fallback restores brain-tech surfacing
The previous fix capped OR-mode at top-5 longest tokens which made
the long brain-tech query fast (118s → 13s) but lost the synonym
retrieval that surfaced Telepathy / Neurotechnology — those titles
weren't reachable via the original query tokens alone.
Fix: pass synonym-expanded tokens to the OR-mode fallback as
``extra_or_tokens``. The merged pool keeps the top-5-longest cap so
retrieval cost is unchanged, but long synonym tokens like
"neurotechnology" (15 chars) and "neuroimaging" (12 chars) now beat
shorter query tokens like "thoughts" (8) by length & surface the
right titles.
OR pool example for the brain-tech query:
Pre-synonym: reconstruct, technology, understand, available, thoughts
Post-synonym: consciousness, neuroimaging, clairvoyance, compensation, reconstruct
→ 3/5 brain-tech terms surface naturally in OR-mode.
AND mode stays unchanged (synonyms in AND would relax the strict-
relevance constraint & pull in noise — wrong tradeoff). The
synonym signal flows ONLY into OR-mode-fallback.
Live verified on the 19-token brain-tech query:
- 13s total (Hermes 6-8s + retrieval ~1s + verify ~2s)
- 2/2 verified pairs
- Cited Functional neuroimaging E5 + FreeSurfer E9 (both real
brain-tech corpus sources, not Universal-pragmatics nonsense)
`_search_corpus` recomputes synonym_expand(qtokens) once into
`or_synonym_pool` (caches in concepts.query._CACHE after first
shard); each shard's FTS5Backend.search() receives the same set
through `extra_or_tokens=`. Cost stays sub-second per shard.
Tests: 641 passed (no regression).
This commit is contained in:
parent
1d70c4fd25
commit
2f7132b9ec
2 changed files with 60 additions and 16 deletions
|
|
@ -911,17 +911,15 @@ def _search_corpus(
|
|||
out-ranks FTS5 body hits so the actual topic article rises to the top.
|
||||
"""
|
||||
qtokens = _title_query_tokens(question)
|
||||
# Title-LIKE backup search uses ORIGINAL qtokens only — NOT synonym-
|
||||
# expanded. With expansion, this path becomes O(corpus × |accept|)
|
||||
# full-scan because LOWER(title) LIKE '%tok%' can't use any index;
|
||||
# 50 LIKE patterns × 870k docs × 4 shards = ~56s of pure waste.
|
||||
# Synonym expansion stays in `_filter_by_title_relevance` (post-
|
||||
# retrieval, in-memory, cheap regardless of accept-set size) where
|
||||
# it actually does useful work surfacing synonym-related titles
|
||||
# from the FTS5 BM25 hits. Title-LIKE is a thin backup for docs
|
||||
# whose body is short on query terms but whose title is the literal
|
||||
# topic — that case is well-served by the original qtokens.
|
||||
# Title-LIKE backup uses ORIGINAL qtokens only (LIKE %tok% can't
|
||||
# use any index — adding synonyms makes it O(corpus × |accept|)).
|
||||
# Synonym expansion stays useful in two places: (1) the FTS5
|
||||
# OR-mode fallback (top-5 longest pool merged with synonyms — long
|
||||
# topical synonyms like "neurotechnology" surface relevant titles
|
||||
# without paying for full-scan), and (2) `_filter_by_title_relevance`
|
||||
# post-retrieval filtering (in-memory, cheap).
|
||||
accept_tokens = set(qtokens)
|
||||
or_synonym_pool = synonym_expand(qtokens, shards_dir=shards_dir)
|
||||
paths: list[Path]
|
||||
if shards_dir is not None:
|
||||
paths = discover_shards(shards_dir)
|
||||
|
|
@ -948,7 +946,9 @@ def _search_corpus(
|
|||
conn = connect(p)
|
||||
try:
|
||||
backend = FTS5Backend(conn)
|
||||
for h in backend.search(question, limit=over_fetch):
|
||||
for h in backend.search(
|
||||
question, limit=over_fetch, extra_or_tokens=or_synonym_pool
|
||||
):
|
||||
raw.append(
|
||||
(
|
||||
h.score,
|
||||
|
|
|
|||
|
|
@ -52,7 +52,12 @@ def _quote(t: str) -> str:
|
|||
return '"' + t.replace('"', '""') + '"'
|
||||
|
||||
|
||||
def _escape_fts5(query: str, *, mode: str = "and") -> str:
|
||||
def _escape_fts5(
|
||||
query: str,
|
||||
*,
|
||||
mode: str = "and",
|
||||
extra_or_tokens: set[str] | None = None,
|
||||
) -> str:
|
||||
"""Build a MATCH expression from a free-text query.
|
||||
|
||||
Tokenizes by alpha runs (so `?` and other punctuation can't break the
|
||||
|
|
@ -67,6 +72,14 @@ def _escape_fts5(query: str, *, mode: str = "and") -> str:
|
|||
clause matches millions of docs and forces BM25 to rank them all
|
||||
— 13s/shard observed pre-cap).
|
||||
|
||||
The ``extra_or_tokens`` kwarg accepts synonym-expanded tokens
|
||||
(e.g. "telepathy" / "neurotechnology" expanded from the query token
|
||||
"thoughts"). These join the top-N-longest OR pool — long synonym
|
||||
tokens like "neurotechnology" (15 chars) outrank short query tokens
|
||||
like "thoughts" (8) by length and surface the right titles. Pure
|
||||
quality win at OR-mode-fallback time without extra retrieval cost
|
||||
since the top-N cap still applies to the merged pool.
|
||||
|
||||
All-stopword queries fall back to OR over the raw tokens so they
|
||||
still find something instead of crashing FTS5 with an empty MATCH.
|
||||
"""
|
||||
|
|
@ -83,8 +96,20 @@ def _escape_fts5(query: str, *, mode: str = "and") -> str:
|
|||
# corpus). Bounds the per-clause cost so OR-mode terminates
|
||||
# quickly instead of scanning the corpus.
|
||||
sep = " OR "
|
||||
if len(tokens) > _OR_FALLBACK_MAX_TOKENS:
|
||||
tokens = sorted(tokens, key=len, reverse=True)[:_OR_FALLBACK_MAX_TOKENS]
|
||||
candidate_pool = list(tokens)
|
||||
if extra_or_tokens:
|
||||
# Synonyms join the OR pool; dedupe lowercase.
|
||||
seen_lower = {t.lower() for t in candidate_pool}
|
||||
for t in extra_or_tokens:
|
||||
tl = t.lower()
|
||||
if tl and tl not in seen_lower and tl not in _FTS5_STOPWORDS:
|
||||
candidate_pool.append(tl)
|
||||
seen_lower.add(tl)
|
||||
if len(candidate_pool) > _OR_FALLBACK_MAX_TOKENS:
|
||||
candidate_pool = sorted(candidate_pool, key=len, reverse=True)[
|
||||
:_OR_FALLBACK_MAX_TOKENS
|
||||
]
|
||||
tokens = candidate_pool
|
||||
return sep.join(_quote(t) for t in tokens)
|
||||
|
||||
|
||||
|
|
@ -153,12 +178,31 @@ class FTS5Backend(SearchBackend):
|
|||
name = "fts5"
|
||||
audit_mode = AuditMode.UNGROUNDED
|
||||
|
||||
def search(self, query: str, limit: int = 20) -> list[Hit]:
|
||||
def search(
|
||||
self,
|
||||
query: str,
|
||||
limit: int = 20,
|
||||
extra_or_tokens: set[str] | None = None,
|
||||
) -> list[Hit]:
|
||||
"""Run FTS5 BM25 over chunk content.
|
||||
|
||||
``extra_or_tokens`` (synonym-expanded set) is passed through to
|
||||
the OR-mode fallback only. AND mode stays on original query
|
||||
tokens (adding synonyms there would relax the AND constraint
|
||||
and pull in noise). The intended caller is the retrieval
|
||||
pipeline that has already computed ``synonym_expand(qtokens)``
|
||||
— passing it here saves the OR-mode pool from missing topical
|
||||
synonym terms.
|
||||
"""
|
||||
if not query.strip():
|
||||
return []
|
||||
# Try strict AND first; fall back to OR if it returns nothing.
|
||||
for mode in ("and", "or"):
|
||||
fts_query = _escape_fts5(query, mode=mode)
|
||||
fts_query = _escape_fts5(
|
||||
query,
|
||||
mode=mode,
|
||||
extra_or_tokens=extra_or_tokens if mode == "or" else None,
|
||||
)
|
||||
try:
|
||||
rows = self.conn.execute(
|
||||
"""
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue