arborist/aborist
russell@unturf.com 2f7132b9ec
qa(retrieval): synonym-aware FTS5 OR-fallback restores brain-tech surfacing
The previous fix capped OR-mode at top-5 longest tokens which made
the long brain-tech query fast (118s → 13s) but lost the synonym
retrieval that surfaced Telepathy / Neurotechnology — those titles
weren't reachable via the original query tokens alone.

Fix: pass synonym-expanded tokens to the OR-mode fallback as
``extra_or_tokens``. The merged pool keeps the top-5-longest cap so
retrieval cost is unchanged, but long synonym tokens like
"neurotechnology" (15 chars) and "neuroimaging" (12 chars) now beat
shorter query tokens like "thoughts" (8) by length & surface the
right titles.

OR pool example for the brain-tech query:
  Pre-synonym:  reconstruct, technology, understand, available, thoughts
  Post-synonym: consciousness, neuroimaging, clairvoyance, compensation, reconstruct
  → 3/5 brain-tech terms surface naturally in OR-mode.

AND mode stays unchanged (synonyms in AND would relax the strict-
relevance constraint & pull in noise — wrong tradeoff). The
synonym signal flows ONLY into OR-mode-fallback.

Live verified on the 19-token brain-tech query:
  - 13s total (Hermes 6-8s + retrieval ~1s + verify ~2s)
  - 2/2 verified pairs
  - Cited Functional neuroimaging E5 + FreeSurfer E9 (both real
    brain-tech corpus sources, not Universal-pragmatics nonsense)

`_search_corpus` recomputes synonym_expand(qtokens) once into
`or_synonym_pool` (caches in concepts.query._CACHE after first
shard); each shard's FTS5Backend.search() receives the same set
through `extra_or_tokens=`. Cost stays sub-second per shard.

Tests: 641 passed (no regression).
2026-05-01 22:56:50 -04:00
..
concepts qa(concepts): direct-neighbor expansion + degree caps + manual/derived split 2026-05-01 22:34:23 -04:00
distill storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
mesh mesh: per-peer audit chain-of-claims tracking on the wire 2026-04-28 19:41:25 -04:00
qa qa(retrieval): synonym-aware FTS5 OR-fallback restores brain-tech surfacing 2026-05-01 22:56:50 -04:00
search qa(retrieval): synonym-aware FTS5 OR-fallback restores brain-tech surfacing 2026-05-01 22:56:50 -04:00
sources qa: self-reference thought chains — STRICT-as-fact substrate 2026-05-01 10:16:47 -04:00
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
cli.py qa(verify): Rule 8 — title-relevance promoted to hard verifier check 2026-05-01 20:59:14 -04:00
compress.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
document.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
evict.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
ingest.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
merkle.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
progress.py progress reporter + structured benchmark 2026-04-27 11:37:20 -04:00
snapshot.py add corpus-level snapshots: single-hash identity for the forest 2026-04-27 21:29:10 -04:00
source.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
store.py qa(concepts): corpus-derived concept_relations table replaces frozensets 2026-05-01 21:14:53 -04:00
wikitext.py verify: strip wikitext from context before substring matching 2026-04-28 15:48:07 -04:00