arborist/docs/concept-relations-design.md

Concept relations: corpus-derived synonym & rivalry layer

Status: landed — aborist/concepts/ package, 2026-05-01 (commit 5fd458a) Audience: anyone editing retrieval, storage budget, or planning new concept extractors Hard constraint: writes to concept_relations MUST NOT affect document_root, chunk_root, or cache_key. The layer is a secondary index over the Merkle-committed corpus; backfilling it is safe across the entire corpus without invalidating any cached answer.

Why

Pre-2026-05-01 the synonym & rivalry data lived as hand-curated frozensets in aborist/qa/concepts.py:

SYNONYM_GROUPS = [
    frozenset({"amd", "athlon", "duron", ...}),
    frozenset({"intel", "pentium", ...}),
    ...  # 7 groups total
]
RIVALRIES = [(0, 1), (4, 5)]

The original commit message even flagged the limit:

Phase 1: hand-curated. Phase 2 idea: derive from Wikipedia's category graph or "See also" sections.

Phase 1 didn't scale. Adding a domain meant editing Python source, committing, pushing, redeploying. A 3.47M-doc corpus has thousands of concept families; hand-curating them is a fool's errand. The corpus already encodes the relationships we'd be hand-rebuilding — "See also" sections, category links, internal-link clusters. Phase 2 reads what's already there.

Architecture

A per-shard concept_relations SQLite table (live alongside documents and chunks):

CREATE TABLE concept_relations (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    source_root     TEXT NOT NULL,
    relation_kind   TEXT NOT NULL CHECK (relation_kind IN
                        ('synonym','antonym','rivalry','category')),
    token           TEXT NOT NULL,
    target          TEXT NOT NULL,
    evidence_kind   TEXT NOT NULL,
    confidence      REAL NOT NULL DEFAULT 1.0,
    derived_at      INTEGER NOT NULL,
    derived_from    TEXT,
    UNIQUE (source_root, relation_kind, token, target, evidence_kind)
);

Append-only. UNIQUE on (source_root, relation_kind, token, target, evidence_kind) makes re-derivation idempotent — re-running an extractor adds nothing if no new relations have appeared since last run. INSERT OR IGNORE is the only write path.

Per-shard. Concept relations live in the shard whose document they were derived from. Mesh sync moves shards between peers; relations come along automatically.

Cross-shard lookup. aborist.concepts.query.synonyms_for walks every shard via connect_query's UNION view (the same pattern as cross-shard FTS5 search). Concept relations from shard 003 are visible to a query routed at shard 000.

Public API stable. synonym_expand(tokens, *, shards_dir=...) & rivalry_excluded(tokens, ..., shards_dir=...) keep the legacy shape from aborist/qa/concepts.py. The old module is now a back-compat shim that delegates to aborist.concepts.query. Call sites in qa/query.py thread shards_dir through but did not otherwise change.

Extractors

aborist/concepts/extract.py ships an extractor framework. Each extractor reads existing corpus rows (no new crawling) & emits concept relations under a stable evidence_kind string. An operator can purge --evidence-kind X to revoke a single extractor's output without touching manual or other-extractor rows.

link_reciprocity_synonym (built-in). For any reciprocal edge pair (A→B AND B→A) in the existing edges table, emit a synonym edge between every (title-token-of-A, title-token-of-B) pair. Title-tokens are filtered to ≥4 chars + stopword-stripped to keep generic words like "the" or "and" from generating noise.

The edges table is already populated on ingest. For Wikipedia, the wikitext parser pulls every [[link]] as an edge row. For HTML, aborist/sources/html_page.py:parse_html pulls every <a href>. So the link reciprocity extractor works for any corpus that flows through aborist's standard ingest path: Wikipedia, crawled HTML sites (russell.ballestrini.net pattern), or any other document graph the corpus already encodes.

Storage cost — measured

Backfill on 6 GB of Wikipedia (2003 cur dump, 4 shards, 3.47M docs):

Shard	Docs	Resolved edges	Reciprocal pairs	Synonyms inserted	Backfill time	Storage
000.db	~870k	2,703,287	13,562	71,288	50.6s	23.52 MB
001.db	~870k	2,571,390	14,078	73,351	47.7s	24.20 MB
002.db	~870k	2,772,156	13,708	72,576	1m34s	23.92 MB
003.db	~870k	2,707,627	13,800	72,633	1m03s	23.94 MB
total	3.47M	10,754,460	55,148	289,848	4m16s	95.58 MB

Per-row cost: ~330 bytes. The bulk is the 64-char hex source_root stored both in the table & in the UNIQUE auto-index that enforces the idempotent-re-derivation key. Per-shard breakdown:

Component	Size per shard	Purpose
concept_relations table	~10.0 MB	rows
sqlite_autoindex (UNIQUE)	~9.3 MB	enforces idempotent re-derivation
idx_concept_evid	~1.8 MB	for purge --evidence-kind X
idx_concept_kind	~1.0 MB	filter by relation_kind
idx_concept_target	~1.0 MB	reverse lookup
idx_concept_token	~1.0 MB	forward lookup

95.58 MB on a 6 GB corpus = 1.6% storage tax for the entire denormalization. Full backfill took 4m16s wall-clock across the 4 wiki shards — the cost is paid once.

Storage choice — keep, don't compact

We considered three compactions & rejected all three. Documented here so a future maintainer doesn't reopen the question without a measured reason.

Option A — drop idx_concept_evid (saves ~1.8 MB / shard, ~7 MB total)

idx_concept_evid exists to make purge --evidence-kind X cheap (one index lookup per evidence_kind instead of a full table scan). Without it, purge becomes O(N) over the whole table — acceptable for a one-shot cleanup, painful for a tight extractor-development loop where an operator runs purge then re-derive many times.

Decision: keep. The 1.8 MB saving doesn't justify the painful debugging loop.

Option B — store source_root as 32-byte BLOB instead of 64-char hex TEXT (saves ~5 MB / shard, ~20 MB total)

The source_root column is the largest single contributor to storage (~50% of per-row cost). Storing as BLOB instead of hex TEXT halves it.

Decision: keep TEXT. The rest of the schema (documents.document_root, chunks.leaf_hash, providence_cache.source_root, audit_events.subject_root) all use hex TEXT. Mixing TEXT vs BLOB across tables hurts schema legibility & complicates joins. 5 MB saving doesn't justify the inconsistency.

Option C — normalize source_root into a source_lookup(id, hex) foreign key (saves ~7 MB / shard, ~28 MB total)

Replace the 64-char hex source_root column with a 4-8 byte source_id integer pointing at a source_lookup table that maps id ↔ hex.

Decision: keep flat. The savings are real but every concept lookup adds a JOIN. The lookup is in the retrieval hot path (synonym_expand is called per-query). Adding a JOIN for a 0.5% storage win is the wrong direction.

Final verdict

1.6% storage tax is fine. Concept relations are a query-time performance accelerator over an already-3.5GB-per-shard corpus. The tax is paid once at backfill & every retrieval-time lookup benefits. If a future shard layout pushes total storage to a different cost regime (e.g. compressed columnstore), revisit Option B at that point — the boundary changes the tradeoff math.

How to use

Backfill the existing corpus

make backfill-concepts                        # all extractors × all shards × 4 workers
make backfill-concepts CONCEPTS_WORKERS=2     # tune parallelism

Driven by scripts/backfill_concepts.py — runs each registered extractor in aborist.concepts.extract.EXTRACTORS (link_reciprocity, token_idf, documents_fts) across every numeric-stem shard in parallel. Idempotent — re-running on already-backfilled shards is ~no-op-cost (INSERT OR IGNORE / DELETE-and-rebuild semantics depending on extractor).

(CLI commands aborist concepts {seed,list,add,derive,purge} deferred to a follow-on commit.)

Add a manual relation (e.g. when a domain expert sees a gap)

from aborist.concepts.store import add_concept_relation
from aborist.store import connect

conn = connect('~/.aborist/shards/000.db')
add_concept_relation(
    conn,
    source_root='__manual__',
    relation_kind='synonym',
    token='telepathy',
    target='neurotechnology',
    evidence_kind='manual',
    derived_from='fox 2026-05-01: brain-tech retrieval gap',
)
conn.commit()
conn.close()

Revoke a buggy extractor's output

from aborist.concepts.store import purge_by_evidence_kind
from aborist.store import connect

conn = connect('~/.aborist/shards/000.db')
n = purge_by_evidence_kind(conn, 'broken_extractor_v1')
conn.commit()
print(f'removed {n} rows')

evidence_kind='manual' rows are NOT touched by a purge of any other kind — manual contributions are safe.

Adding new extractors

1. Implement a callable

   def my_extractor(conn, *, derived_from=None) -> dict[str, int]:
       ...
       return {"items_inserted": N, "items_skipped": K}
   

   that walks the shard's existing rows (documents, chunks, edges, derivations, …) & calls aborist.concepts.store.add_concept_relation for each finding.

2. Pick a stable evidence_kind string. Don't reuse an existing one unless your extractor genuinely produces the same kind of output as that one (so purge --evidence-kind X semantics stay clean).

3. Register in aborist/concepts/extract.py:EXTRACTORS.

4. Document the trade-offs in this file alongside link_reciprocity.

Deferred follow-ons

• CLI commands — aborist concepts {seed,list,add,derive,purge}. Today the helpers are accessible via python -c "...".
• Wikipedia See-also extractor — parses ==See also== sections in chunked wikitext beyond what edges already captures (some See-also entries are wikilinks already in edges; some are bullet lists with annotations that aren't).
• Wikipedia category extractor — parses [[Category:X]] tails & emits relation_kind='category' rows.
• Hatnote / disambiguation extractor — parses {{about|...}} & {{not to be confused with|...}} templates as antonym / rivalry signals.