Phase 2 of the concepts/ layer. The Apr 27 commit (c6182ae) shipped
hand-curated frozensets in aborist/qa/concepts.py with a TODO to
"derive from Wikipedia's category graph or 'See also' sections" —
that's this commit. Fox's 2026-05-01 critique landed it: a 7-entry
list of arbitrary frozensets won't scale to a 3.47M-doc corpus, &
adding domains shouldn't require a Python edit + commit + redeploy.
Architecture:
(1) Per-shard concept_relations SQLite table. Append-only, with
UNIQUE (source_root, relation_kind, token, target, evidence_kind)
so re-derivation is idempotent. Lives next to documents in each
shard so mesh sync moves relations alongside the docs that
derived them. A SECONDARY index — writes here NEVER affect
document_root / chunk_root / cache_key, so backfilling is safe
across the entire corpus.
(2) aborist/concepts/ package:
- store.py: add_concept_relation, concept_relations_for_token,
purge_by_evidence_kind, list_evidence_kinds
- query.py: cross-shard synonym_expand, rivalry_excluded;
union-find collapse on synonym edges so partial
pairs build full equivalence classes; mtime-keyed
per-process LRU so retrieval doesn't re-walk
shards on hot loops
- seed.py: one-shot migration of legacy frozensets (8 groups
incl. brain-tech) to evidence_kind='manual_legacy'
rows under source_root='__legacy__concepts__'
- extract.py: pluggable extractor registry. Built-in:
link_reciprocity_synonym — for any reciprocal
edge pair (A→B AND B→A) in the existing edges
table, emit synonym edges between the docs'
title-tokens. Works for Wikipedia (See-also
bidirectional), HTML site internal-link clusters
(russell.ballestrini.net pattern), or any link
graph the corpus already encodes — no new
crawler needed; the html_page parser already
populates `edges` rows on ingest.
(3) aborist/qa/concepts.py rewritten as a backwards-compat shim —
same public API (synonym_expand, rivalry_excluded,
has_compare_phrasing) so query.py call sites unchanged.
shards_dir threaded through _filter_by_title_relevance &
_search_corpus' synonym_expand calls. Without shards_dir
(legacy 2-arg call shape), helpers degenerate to no-op —
matches the behavior the frozenset code had when no group hit.
(4) Cross-shard UNION view: concept_relations added to
_SHARDABLE_TABLES in store.py so connect_query() exposes a
unified view across all shards (same pattern as documents,
chunks, providence_cache, etc.).
Live-verified on the 4-shard 3.47M-doc corpus + the
crawl_russell_ballestrini_net.db shard:
Q: what technology are currently or soon available which may
enable one person to reconstruct and understand some or a
portion of another persons thoughts or ideas without speaking
or sign language?
Result: same as the bde1bd6 in-memory frozenset version —
Telepathy E4 cited alongside Videoconferencing E1/E2,
POINTER-LINKED-PARTIAL 6/10. The DB-backed lookup reproduces
the frozenset behavior byte-for-byte.
Tests: 633 passed (was 624). 9 new concept tests covering
DB-backed synonym/rivalry lookup, shards_dir=None degenerate
behavior, brain-tech group seed, cache invalidation. Old tests
that called helpers directly without shards_dir kept as no-op
assertions (synonym_expand({"athlon"}) without shards_dir returns
{"athlon"} unchanged).
Backfill mechanics: existing live shards needed a one-shot
`connect()` to auto-create the new concept_relations table
(SCHEMA_SQL has CREATE TABLE IF NOT EXISTS). Re-derivation never
mutates the Merkle tree — concept_relations is fully orthogonal
to document_root / chunk_root. Cache_key dimensions are
unaffected.
Deferred (follow-on):
- CLI commands: aborist concepts {seed,list,add,derive,purge}
(currently fox runs the helpers via python -c)
- Wikipedia See-also extractor (requires parsing wikitext sections
beyond what's already in edges)
- Wikipedia category extractor (requires reading Category: links
from chunked wikitext)
russell@unturf.com
5fd458aa41