# `aborist.concepts`

Per-shard `concept_relations` SQLite table — the corpus-derived
synonym, rivalry, antonym & category overlay that replaces the
hand-curated frozensets that lived in `aborist/qa/concepts.py`
through April 2026.

**Full architecture rationale lives in
[`../concept-relations-design.md`](../concept-relations-design.md)**
including the 1.6% storage-tax measurement and the three-compactions-
considered-and-rejected analysis. This page is the API reference.

## Sub-modules

### `concepts.store` — append-only CRUD

```python
from aborist.concepts.store import (
    add_concept_relation,           # idempotent INSERT OR IGNORE
    concept_relations_for_token,    # read all relations for a token
    purge_by_evidence_kind,         # the only DELETE path
    list_evidence_kinds,            # diagnostic
    RELATION_KINDS,                 # ('synonym','antonym','rivalry','category')
)
```

UNIQUE on `(source_root, relation_kind, token, target, evidence_kind)`
makes re-derivation idempotent. `purge_by_evidence_kind` lets an
operator revoke a single extractor's output without touching manual
or other-extractor rows.

[`aborist/concepts/store.py`](../../aborist/concepts/store.py)

### `concepts.query` — cross-shard lookup

```python
from aborist.concepts import (
    synonym_expand,        # query tokens → expanded set
    rivalry_excluded,      # query tokens → tokens to drop from results
    has_compare_phrasing,  # bool — does the query say "vs", "compare", etc.
    invalidate_cache,      # drop the per-process LRU
)

expanded = synonym_expand({"thoughts"}, shards_dir=shards_dir)
# → {"thoughts", "telepathy", "neurotechnology", "mind", "cognition", ...}
```

Two synonym indices are loaded:

- **`manual_index`** — `manual_legacy` + `manual` rows. Curated;
  always expanded regardless of per-token degree. Captures the
  brain-tech / AMD-family / Mac / Linux / etc. seed groups.
- **`derived_index`** — `link_reciprocity` & other corpus-derived
  edges. Subject to **`MAX_NEIGHBORS_PER_TOKEN=8`** cap because the
  Wikipedia link graph carries topic-adjacency noise on generic
  tokens (person, thoughts, language).

Overall **`MAX_TOTAL_TOKENS=50`** cap on expanded set bounds the
SQL clause count downstream so retrieval stays sub-second.

A per-process LRU keyed on shard mtime avoids re-loading the index
on every query (290k rows across 4 shards loads in ~1.8s cold).

[`aborist/concepts/query.py`](../../aborist/concepts/query.py)

### `concepts.extract` — pluggable extractor framework

```python
from aborist.concepts.extract import (
    EXTRACTORS,                  # registry: evidence_kind → callable
    link_reciprocity_synonym,    # built-in extractor
)

# Run an extractor against a shard:
result = link_reciprocity_synonym(conn, derived_from="backfill@2026-05-01")
# → {"reciprocal_pairs": N, "synonyms_inserted": M, "synonyms_skipped": K}
```

Each extractor walks the shard's existing rows (`documents`,
`chunks`, `edges`, `derivations`) — **no new crawler needed** — &
emits concept relations under a stable `evidence_kind` string that
supports targeted purge.

The built-in `link_reciprocity_synonym` reads the existing `edges`
table for reciprocal A↔B link pairs and emits a synonym edge between
every (title-token-of-A, title-token-of-B) pair. Works for
Wikipedia (See-also bidirectional), HTML site internal links
(russell.ballestrini.net pattern), or any document graph with
bidirectional links. Title-tokens are filtered to ≥4 chars +
stopword-stripped.

To add a new extractor:

1. Implement `(conn, *, derived_from) -> dict[str, int]` that calls
   `add_concept_relation` for each finding.
2. Pick a stable `evidence_kind` string.
3. Register in `EXTRACTORS`.

[`aborist/concepts/extract.py`](../../aborist/concepts/extract.py)

### `concepts.seed` — legacy frozenset migration

One-shot migration of the 8 hand-curated frozenset groups
(AMD-family, Intel-family, HTTP, FTP, Mac, Windows, Linux,
brain-tech) to `evidence_kind='manual_legacy'` rows.

Writes **clique edges** within each group — every `(a, b)` pair —
so any member retrieves every other member (preserves the legacy
frozenset semantic where lookups didn't depend on which token in
the group was the anchor).

```python
from aborist.concepts.seed import seed_legacy_concepts
result = seed_legacy_concepts(conn)
# → {"synonyms_inserted": N, "rivalries_inserted": M, "skipped": K}
```

Idempotent — re-running adds nothing if all the rows already exist.

[`aborist/concepts/seed.py`](../../aborist/concepts/seed.py)

## Data model

```sql
CREATE TABLE concept_relations (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    source_root     TEXT NOT NULL,
    relation_kind   TEXT NOT NULL CHECK (relation_kind IN
                        ('synonym','antonym','rivalry','category')),
    token           TEXT NOT NULL,
    target          TEXT NOT NULL,
    evidence_kind   TEXT NOT NULL,
    confidence      REAL NOT NULL DEFAULT 1.0,
    derived_at      INTEGER NOT NULL,
    derived_from    TEXT,
    UNIQUE (source_root, relation_kind, token, target, evidence_kind)
);
```

**Append-only by construction.** Re-running an extractor adds
nothing if every relation already exists. No `UPDATE` path; only
`add_concept_relation` (insert) and `purge_by_evidence_kind`
(targeted delete).

**Per-shard storage.** Concept relations live in the shard whose
document derived them. Mesh sync moves shards between peers;
relations come along.

**Orthogonal to Merkle.** Writes to `concept_relations` NEVER
affect `document_root`, `chunk_root`, or `cache_key`. Backfilling
relations is safe across the entire corpus without invalidating
any cached answer or breaking any audit chain.

## Storage cost — measured

Backfill on 4 wiki shards (3.47M docs, 10.75M resolved edges):

| Shard | Reciprocal pairs | Synonyms | Storage |
|---|---:|---:|---:|
| 000.db | 13,562 | 71,288 | 23.52 MB |
| 001.db | 14,078 | 73,351 | 24.20 MB |
| 002.db | 13,708 | 72,576 | 23.92 MB |
| 003.db | 13,800 | 72,633 | 23.94 MB |
| **total** | **55,148** | **289,848** | **95.58 MB** |

**1.6% storage tax** on the 6 GB corpus. Backfill takes ~4 min
wall-clock total. Cost is paid once at backfill; every retrieval-
time lookup benefits.

## Diagrams

![module graph](../diagrams/aborist-modules.svg)
![query pipeline](../diagrams/query-pipeline.svg)

## Source

- [`aborist/concepts/`](../../aborist/concepts/) — package
- [`docs/concept-relations-design.md`](../concept-relations-design.md) — full design doc
- Whitepaper §13.4.11 — public-facing summary