Phase 2 of the holistic-tuning pair. Replaces the un-indexable
LOWER(title) LIKE '%tok%' title-search with an FTS5 MATCH-based
lookup. The structural fix that was deferred when the >5-token
bypass landed (commit 1d70c4f).
(1) Schema: new `documents_fts` virtual table over the title column.
Contentless mode (same trick as chunks_fts) — stores only the
inverted index, not a copy of the title. Joins back to documents
via rowid for the post-filter caller.
(2) Extractor: backfill_documents_fts (registered as evidence_kind
"documents_fts"). Reset-and-rebuild the index from documents in
one INSERT...SELECT. Idempotent. Cost: ~2.5s per 870k-doc shard.
(3) `_search_titles` rewritten: try FTS5 MATCH first, fall back to
LOWER(title) LIKE only on shards lacking documents_fts data
(legacy ingest pre-this-commit). MATCH expression OR-joins
quoted query tokens; falls through to LIKE form on tokenizer
edge cases.
(4) Removed the `>5 accept_tokens` bypass in `_search_corpus`
that commit 1d70c4f added as a workaround for the LIKE
full-scan cost. With FTS5 the title search is sub-second
regardless of token count, so synonym-expanded title search
is affordable at any query length.
Live verified on the 19-token brain-tech query with 50 accept_tokens
(post-IDF expansion):
Pre-FTS5: ~50s (LIKE '%tok%' × 870k docs × 4 shards)
Post-FTS5: ~0.04s ← 1250× speedup
End-to-end query: 10.4s (was 11.6-13s; the saved title-search
time partially absorbed by Hermes inference variance).
Backfill cost: 10.5s wall-clock across 4 wiki shards + 1 crawl
shard. Storage: ~30 MB per wiki shard for documents_fts (well
inside the 90 MB/shard concept-layer budget). Re-running is
idempotent (DELETE FROM ... INSERT INTO ...).
Tests: 641 passed (no regression).
russell@unturf.com
7887a588f9