arborist/aborist/qa
russell@unturf.com 2f7132b9ec
qa(retrieval): synonym-aware FTS5 OR-fallback restores brain-tech surfacing
The previous fix capped OR-mode at top-5 longest tokens which made
the long brain-tech query fast (118s → 13s) but lost the synonym
retrieval that surfaced Telepathy / Neurotechnology — those titles
weren't reachable via the original query tokens alone.

Fix: pass synonym-expanded tokens to the OR-mode fallback as
``extra_or_tokens``. The merged pool keeps the top-5-longest cap so
retrieval cost is unchanged, but long synonym tokens like
"neurotechnology" (15 chars) and "neuroimaging" (12 chars) now beat
shorter query tokens like "thoughts" (8) by length & surface the
right titles.

OR pool example for the brain-tech query:
  Pre-synonym:  reconstruct, technology, understand, available, thoughts
  Post-synonym: consciousness, neuroimaging, clairvoyance, compensation, reconstruct
  → 3/5 brain-tech terms surface naturally in OR-mode.

AND mode stays unchanged (synonyms in AND would relax the strict-
relevance constraint & pull in noise — wrong tradeoff). The
synonym signal flows ONLY into OR-mode-fallback.

Live verified on the 19-token brain-tech query:
  - 13s total (Hermes 6-8s + retrieval ~1s + verify ~2s)
  - 2/2 verified pairs
  - Cited Functional neuroimaging E5 + FreeSurfer E9 (both real
    brain-tech corpus sources, not Universal-pragmatics nonsense)

`_search_corpus` recomputes synonym_expand(qtokens) once into
`or_synonym_pool` (caches in concepts.query._CACHE after first
shard); each shard's FTS5Backend.search() receives the same set
through `extra_or_tokens=`. Cost stays sub-second per shard.

Tests: 641 passed (no regression).
2026-05-01 22:56:50 -04:00
..
__init__.py multi-source corpus query: pose a question, the tree pulls related cached docs 2026-04-27 11:43:57 -04:00
client.py qa: JSON-mode stop-sequence guards against post-brace token runaway 2026-04-30 16:15:07 -04:00
concepts.py qa(concepts): corpus-derived concept_relations table replaces frozensets 2026-05-01 21:14:53 -04:00
dag.py qa(provenance): #000001 land — retrieval_plan_hash binds D4 2026-05-01 19:26:01 -04:00
evidence.py qa: spotlight density rank — pick load-bearing slice, not leading match 2026-05-01 13:37:04 -04:00
frame.py qa(frame): #000002 land — reference-frame polarity contract (D3 → ✓) 2026-05-01 20:18:13 -04:00
inspect.py qa/inspect: title-relevance sidecar — flag retrieval-driven hallucinations 2026-05-01 10:19:44 -04:00
keys.py qa: warrant-lite — relation-question hard check (Ticket H) 2026-05-01 13:23:24 -04:00
parse_claims.py qa: G0 claim-lattice-pointer answer mode — CTI quote-by-pointer 2026-04-29 21:19:19 -04:00
query.py qa(retrieval): synonym-aware FTS5 OR-fallback restores brain-tech surfacing 2026-05-01 22:56:50 -04:00
repair.py qa: chain-segment failure localization + re-prompt repair tier 2026-04-29 19:27:33 -04:00
retrieval_plan.py qa(provenance): #000001 land — retrieval_plan_hash binds D4 2026-05-01 19:26:01 -04:00
runner.py qa: warrant-lite — wire JSON variant of claim_lattice verifier 2026-05-01 13:32:05 -04:00
verify.py qa(verify): Rule 8 — title-relevance promoted to hard verifier check 2026-05-01 20:59:14 -04:00
warrant.py qa(warrant): #000003 land — anchor-class generalization (D6 → ✓) 2026-05-01 19:16:06 -04:00