Two compounding crashes/hangs in the corpus-derived synonym layer
that the bench surfaced 2026-05-02:
(1) Union-find chained reciprocal-link clusters into one giant
connected component (54,538 tokens for any seed in the 4-shard
Wikipedia corpus). One query token expanded to the entire
synonym alphabet, tripping SQLite's expression-tree-depth=1000
limit on the OR-clause in `_search_titles`. Replaced with
direct-neighbor adjacency only.
(2) Even direct-neighbor expansion was too noisy on generic tokens:
a 19-token query expanded to 419 tokens (every token had ~22
reciprocal-link neighbors averaging out to topic-adjacency
noise). Title-LIKE on 419 patterns × 3.47M docs × 4 shards
hung indefinitely.
Fix: split index by evidence_kind, cap derived expansion only.
- ``manual_index`` — manual_legacy + manual rows. Curated; ALWAYS
expand regardless of per-token degree. The brain-tech / AMD-
family / Mac / Linux / etc. seed groups have legitimately many
members per token after seed.py started writing clique edges.
- ``derived_index`` — link_reciprocity & corpus-extracted edges.
Subject to MAX_NEIGHBORS_PER_TOKEN=8 cap. Generic tokens
("person", "thoughts") have huge degree from Wikipedia link
noise; specific named entities have small focused neighborhoods
that pass the cap.
- ``MAX_TOTAL_TOKENS=50`` overall cap on expanded set. Bounds
the SQL clause count so title-LIKE search stays tractable.
seed.py: write CLIQUE edges within each legacy group (every
member-pair, not just anchor→member). Preserves the legacy
frozenset semantic where any member retrieves every other
member. Quadratic in group size but groups stay small (largest
is the 30-member brain-tech → 435 pairs).
Live shard 000 re-seeded: 80 → 747 manual_legacy rows. Idempotent
re-seed via INSERT OR IGNORE — re-running adds nothing new.
Verified:
- 19-token query: 419 → 50 expanded (cap saturated)
- thoughts: 30+ tokens incl. brain-tech members preserved
- athlon: full AMD-family clique (amd, duron, opteron, ryzen, …)
Tests: 14/14 concept tests pass.
russell@unturf.com
50a1bbdc0e