arborist/aborist/search
russell@unturf.com 2f7132b9ec
qa(retrieval): synonym-aware FTS5 OR-fallback restores brain-tech surfacing
The previous fix capped OR-mode at top-5 longest tokens which made
the long brain-tech query fast (118s → 13s) but lost the synonym
retrieval that surfaced Telepathy / Neurotechnology — those titles
weren't reachable via the original query tokens alone.

Fix: pass synonym-expanded tokens to the OR-mode fallback as
``extra_or_tokens``. The merged pool keeps the top-5-longest cap so
retrieval cost is unchanged, but long synonym tokens like
"neurotechnology" (15 chars) and "neuroimaging" (12 chars) now beat
shorter query tokens like "thoughts" (8) by length & surface the
right titles.

OR pool example for the brain-tech query:
  Pre-synonym:  reconstruct, technology, understand, available, thoughts
  Post-synonym: consciousness, neuroimaging, clairvoyance, compensation, reconstruct
  → 3/5 brain-tech terms surface naturally in OR-mode.

AND mode stays unchanged (synonyms in AND would relax the strict-
relevance constraint & pull in noise — wrong tradeoff). The
synonym signal flows ONLY into OR-mode-fallback.

Live verified on the 19-token brain-tech query:
  - 13s total (Hermes 6-8s + retrieval ~1s + verify ~2s)
  - 2/2 verified pairs
  - Cited Functional neuroimaging E5 + FreeSurfer E9 (both real
    brain-tech corpus sources, not Universal-pragmatics nonsense)

`_search_corpus` recomputes synonym_expand(qtokens) once into
`or_synonym_pool` (caches in concepts.query._CACHE after first
shard); each shard's FTS5Backend.search() receives the same set
through `extra_or_tokens=`. Cost stays sub-second per shard.

Tests: 641 passed (no regression).
2026-05-01 22:56:50 -04:00
..
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
base.py verify: layered strategies + entity policies, rename VISUAL → UNGROUNDED 2026-04-28 16:58:31 -04:00
fts5.py qa(retrieval): synonym-aware FTS5 OR-fallback restores brain-tech surfacing 2026-05-01 22:56:50 -04:00