arborist/docs/modules/concepts.md

aborist.concepts

Per-shard concept_relations SQLite table — the corpus-derived synonym, rivalry, antonym & category overlay that replaces the hand-curated frozensets that lived in aborist/qa/concepts.py through April 2026.

Full architecture rationale lives in ../concept-relations-design.md including the 1.6% storage-tax measurement and the three-compactions- considered-and-rejected analysis. This page is the API reference.

Sub-modules

concepts.store — append-only CRUD

from aborist.concepts.store import (
    add_concept_relation,           # idempotent INSERT OR IGNORE
    concept_relations_for_token,    # read all relations for a token
    purge_by_evidence_kind,         # the only DELETE path
    list_evidence_kinds,            # diagnostic
    RELATION_KINDS,                 # ('synonym','antonym','rivalry','category')
)

UNIQUE on (source_root, relation_kind, token, target, evidence_kind) makes re-derivation idempotent. purge_by_evidence_kind lets an operator revoke a single extractor's output without touching manual or other-extractor rows.

aborist/concepts/store.py

concepts.query — cross-shard lookup

from aborist.concepts import (
    synonym_expand,        # query tokens → expanded set
    rivalry_excluded,      # query tokens → tokens to drop from results
    has_compare_phrasing,  # bool — does the query say "vs", "compare", etc.
    invalidate_cache,      # drop the per-process LRU
)

expanded = synonym_expand({"thoughts"}, shards_dir=shards_dir)
# → {"thoughts", "telepathy", "neurotechnology", "mind", "cognition", ...}

Two synonym indices are loaded:

• manual_index — manual_legacy + manual rows. Curated; always expanded regardless of per-token degree. Captures the brain-tech / AMD-family / Mac / Linux / etc. seed groups.
• derived_index — link_reciprocity & other corpus-derived edges. Subject to MAX_NEIGHBORS_PER_TOKEN=8 cap because the Wikipedia link graph carries topic-adjacency noise on generic tokens (person, thoughts, language).

Overall MAX_TOTAL_TOKENS=50 cap on expanded set bounds the SQL clause count downstream so retrieval stays sub-second.

A per-process LRU keyed on shard mtime avoids re-loading the index on every query (290k rows across 4 shards loads in ~1.8s cold).

aborist/concepts/query.py

concepts.extract — pluggable extractor framework

from aborist.concepts.extract import (
    EXTRACTORS,                  # registry: evidence_kind → callable
    link_reciprocity_synonym,    # built-in extractor
)

# Run an extractor against a shard:
result = link_reciprocity_synonym(conn, derived_from="backfill@2026-05-01")
# → {"reciprocal_pairs": N, "synonyms_inserted": M, "synonyms_skipped": K}

Each extractor walks the shard's existing rows (documents, chunks, edges, derivations) — no new crawler needed — & emits concept relations under a stable evidence_kind string that supports targeted purge.

The built-in link_reciprocity_synonym reads the existing edges table for reciprocal A↔B link pairs and emits a synonym edge between every (title-token-of-A, title-token-of-B) pair. Works for Wikipedia (See-also bidirectional), HTML site internal links (russell.ballestrini.net pattern), or any document graph with bidirectional links. Title-tokens are filtered to ≥4 chars + stopword-stripped.

To add a new extractor:

1. Implement (conn, *, derived_from) -> dict[str, int] that calls add_concept_relation for each finding.
2. Pick a stable evidence_kind string.
3. Register in EXTRACTORS.

aborist/concepts/extract.py

concepts.seed — legacy frozenset migration

One-shot migration of the 8 hand-curated frozenset groups (AMD-family, Intel-family, HTTP, FTP, Mac, Windows, Linux, brain-tech) to evidence_kind='manual_legacy' rows.

Writes clique edges within each group — every (a, b) pair — so any member retrieves every other member (preserves the legacy frozenset semantic where lookups didn't depend on which token in the group was the anchor).

from aborist.concepts.seed import seed_legacy_concepts
result = seed_legacy_concepts(conn)
# → {"synonyms_inserted": N, "rivalries_inserted": M, "skipped": K}

Idempotent — re-running adds nothing if all the rows already exist.

aborist/concepts/seed.py

Data model

CREATE TABLE concept_relations (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    source_root     TEXT NOT NULL,
    relation_kind   TEXT NOT NULL CHECK (relation_kind IN
                        ('synonym','antonym','rivalry','category')),
    token           TEXT NOT NULL,
    target          TEXT NOT NULL,
    evidence_kind   TEXT NOT NULL,
    confidence      REAL NOT NULL DEFAULT 1.0,
    derived_at      INTEGER NOT NULL,
    derived_from    TEXT,
    UNIQUE (source_root, relation_kind, token, target, evidence_kind)
);

Append-only by construction. Re-running an extractor adds nothing if every relation already exists. No UPDATE path; only add_concept_relation (insert) and purge_by_evidence_kind (targeted delete).

Per-shard storage. Concept relations live in the shard whose document derived them. Mesh sync moves shards between peers; relations come along.

Orthogonal to Merkle. Writes to concept_relations NEVER affect document_root, chunk_root, or cache_key. Backfilling relations is safe across the entire corpus without invalidating any cached answer or breaking any audit chain.

Storage cost — measured

Backfill on 4 wiki shards (3.47M docs, 10.75M resolved edges):

Shard	Reciprocal pairs	Synonyms	Storage
000.db	13,562	71,288	23.52 MB
001.db	14,078	73,351	24.20 MB
002.db	13,708	72,576	23.92 MB
003.db	13,800	72,633	23.94 MB
total	55,148	289,848	95.58 MB

1.6% storage tax on the 6 GB corpus. Backfill takes ~4 min wall-clock total. Cost is paid once at backfill; every retrieval- time lookup benefits.

Diagrams

Source

• aborist/concepts/ — package
• docs/concept-relations-design.md — full design doc
• Whitepaper §13.4.11 — public-facing summary