# Concept relations: corpus-derived synonym & rivalry layer

**Status:** landed — `aborist/concepts/` package, 2026-05-01 (commit
`5fd458a`)
**Audience:** anyone editing retrieval, storage budget, or planning
new concept extractors
**Hard constraint:** writes to `concept_relations` MUST NOT affect
`document_root`, `chunk_root`, or `cache_key`. The layer is a
secondary index over the Merkle-committed corpus; backfilling it is
safe across the entire corpus without invalidating any cached answer.

## Why

Pre-2026-05-01 the synonym & rivalry data lived as hand-curated
`frozenset`s in `aborist/qa/concepts.py`:

```python
SYNONYM_GROUPS = [
    frozenset({"amd", "athlon", "duron", ...}),
    frozenset({"intel", "pentium", ...}),
    ...  # 7 groups total
]
RIVALRIES = [(0, 1), (4, 5)]
```

The original commit message even flagged the limit:

> *Phase 1: hand-curated. Phase 2 idea: derive from Wikipedia's
> category graph or "See also" sections.*

Phase 1 didn't scale. Adding a domain meant editing Python source,
committing, pushing, redeploying. A 3.47M-doc corpus has thousands
of concept families; hand-curating them is a fool's errand. The
corpus already encodes the relationships we'd be hand-rebuilding —
"See also" sections, category links, internal-link clusters.
Phase 2 reads what's already there.

## Architecture

A per-shard `concept_relations` SQLite table (live alongside
`documents` and `chunks`):

```sql
CREATE TABLE concept_relations (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    source_root     TEXT NOT NULL,
    relation_kind   TEXT NOT NULL CHECK (relation_kind IN
                        ('synonym','antonym','rivalry','category')),
    token           TEXT NOT NULL,
    target          TEXT NOT NULL,
    evidence_kind   TEXT NOT NULL,
    confidence      REAL NOT NULL DEFAULT 1.0,
    derived_at      INTEGER NOT NULL,
    derived_from    TEXT,
    UNIQUE (source_root, relation_kind, token, target, evidence_kind)
);
```

**Append-only.** UNIQUE on `(source_root, relation_kind, token,
target, evidence_kind)` makes re-derivation idempotent — re-running
an extractor adds nothing if no new relations have appeared since
last run. `INSERT OR IGNORE` is the only write path.

**Per-shard.** Concept relations live in the shard whose document
they were derived from. Mesh sync moves shards between peers;
relations come along automatically.

**Cross-shard lookup.** `aborist.concepts.query.synonyms_for` walks
every shard via `connect_query`'s UNION view (the same pattern as
cross-shard FTS5 search). Concept relations from shard 003 are
visible to a query routed at shard 000.

**Public API stable.** `synonym_expand(tokens, *, shards_dir=...)`
& `rivalry_excluded(tokens, ..., shards_dir=...)` keep the legacy
shape from `aborist/qa/concepts.py`. The old module is now a
back-compat shim that delegates to `aborist.concepts.query`. Call
sites in `qa/query.py` thread `shards_dir` through but did not
otherwise change.

## Extractors

`aborist/concepts/extract.py` ships an extractor framework. Each
extractor reads existing corpus rows (no new crawling) & emits
concept relations under a stable `evidence_kind` string. An operator
can `purge --evidence-kind X` to revoke a single extractor's output
without touching manual or other-extractor rows.

**`link_reciprocity_synonym`** (built-in). For any reciprocal edge
pair (A→B AND B→A) in the existing `edges` table, emit a synonym
edge between every (title-token-of-A, title-token-of-B) pair.
Title-tokens are filtered to ≥4 chars + stopword-stripped to keep
generic words like "the" or "and" from generating noise.

The `edges` table is already populated on ingest. For Wikipedia,
the wikitext parser pulls every `[[link]]` as an edge row. For HTML,
`aborist/sources/html_page.py:parse_html` pulls every `<a href>`.
So the link reciprocity extractor works for any corpus that flows
through aborist's standard ingest path: Wikipedia, crawled HTML
sites (russell.ballestrini.net pattern), or any other document
graph the corpus already encodes.

## Storage cost — measured

Backfill on 6 GB of Wikipedia (2003 cur dump, 4 shards, 3.47M docs):

| Shard      | Docs       | Resolved edges | Reciprocal pairs | Synonyms inserted | Backfill time | Storage |
|------------|------------|----------------|------------------|-------------------|---------------|---------|
| `000.db`   | ~870k      | 2,703,287      | 13,562           | 71,288            | 50.6s         | 23.52 MB |
| `001.db`   | ~870k      | 2,571,390      | 14,078           | 73,351            | 47.7s         | 24.20 MB |
| `002.db`   | ~870k      | 2,772,156      | 13,708           | 72,576            | 1m34s         | 23.92 MB |
| `003.db`   | ~870k      | 2,707,627      | 13,800           | 72,633            | 1m03s         | 23.94 MB |
| **total**  | **3.47M**  | **10,754,460** | **55,148**       | **289,848**       | **4m16s**     | **95.58 MB** |

**Per-row cost: ~330 bytes.** The bulk is the 64-char hex
`source_root` stored both in the table & in the UNIQUE auto-index
that enforces the idempotent-re-derivation key. Per-shard breakdown:

| Component                      | Size per shard | Purpose |
|--------------------------------|----------------|---------|
| `concept_relations` table      | ~10.0 MB       | rows |
| `sqlite_autoindex` (UNIQUE)    | ~9.3 MB        | enforces idempotent re-derivation |
| `idx_concept_evid`             | ~1.8 MB        | for `purge --evidence-kind X` |
| `idx_concept_kind`             | ~1.0 MB        | filter by relation_kind |
| `idx_concept_target`           | ~1.0 MB        | reverse lookup |
| `idx_concept_token`            | ~1.0 MB        | forward lookup |

**95.58 MB on a 6 GB corpus = 1.6% storage tax for the entire
denormalization.** Full backfill took 4m16s wall-clock across the
4 wiki shards — the cost is paid once.

## Storage choice — keep, don't compact

We considered three compactions & rejected all three. Documented
here so a future maintainer doesn't reopen the question without a
measured reason.

### Option A — drop `idx_concept_evid` (saves ~1.8 MB / shard, ~7 MB total)

`idx_concept_evid` exists to make `purge --evidence-kind X` cheap
(one index lookup per evidence_kind instead of a full table scan).
Without it, purge becomes O(N) over the whole table — acceptable
for a one-shot cleanup, painful for a tight extractor-development
loop where an operator runs purge then re-derive many times.

**Decision:** keep. The 1.8 MB saving doesn't justify the painful
debugging loop.

### Option B — store `source_root` as 32-byte BLOB instead of 64-char hex TEXT (saves ~5 MB / shard, ~20 MB total)

The `source_root` column is the largest single contributor to
storage (~50% of per-row cost). Storing as BLOB instead of hex TEXT
halves it.

**Decision:** keep TEXT. The rest of the schema (documents.document_root,
chunks.leaf_hash, providence_cache.source_root, audit_events.subject_root)
all use hex TEXT. Mixing TEXT vs BLOB across tables hurts schema
legibility & complicates joins. 5 MB saving doesn't justify the
inconsistency.

### Option C — normalize `source_root` into a `source_lookup(id, hex)` foreign key (saves ~7 MB / shard, ~28 MB total)

Replace the 64-char hex `source_root` column with a 4-8 byte
`source_id` integer pointing at a `source_lookup` table that maps
id ↔ hex.

**Decision:** keep flat. The savings are real but every concept
lookup adds a JOIN. The lookup is in the retrieval hot path
(`synonym_expand` is called per-query). Adding a JOIN for a 0.5%
storage win is the wrong direction.

### Final verdict

**1.6% storage tax is fine.** Concept relations are a query-time
performance accelerator over an already-3.5GB-per-shard corpus.
The tax is paid once at backfill & every retrieval-time lookup
benefits. If a future shard layout pushes total storage to a
different cost regime (e.g. compressed columnstore), revisit Option
B at that point — the boundary changes the tradeoff math.

## How to use

### Backfill the existing corpus

```bash
# Manual one-shot backfill against all wiki shards
python -c "
from aborist.concepts.extract import link_reciprocity_synonym
from aborist.store import connect, discover_shards
from pathlib import Path
for shard in discover_shards(Path('~/.aborist/shards').expanduser()):
    if not shard.stem.isdigit():
        continue
    conn = connect(shard)
    print(shard.name, link_reciprocity_synonym(conn, derived_from=f'backfill@{shard.name}'))
    conn.close()
"
```

(CLI commands `aborist concepts {seed,list,add,derive,purge}`
deferred to a follow-on commit.)

### Add a manual relation (e.g. when a domain expert sees a gap)

```python
from aborist.concepts.store import add_concept_relation
from aborist.store import connect

conn = connect('~/.aborist/shards/000.db')
add_concept_relation(
    conn,
    source_root='__manual__',
    relation_kind='synonym',
    token='telepathy',
    target='neurotechnology',
    evidence_kind='manual',
    derived_from='fox 2026-05-01: brain-tech retrieval gap',
)
conn.commit()
conn.close()
```

### Revoke a buggy extractor's output

```python
from aborist.concepts.store import purge_by_evidence_kind
from aborist.store import connect

conn = connect('~/.aborist/shards/000.db')
n = purge_by_evidence_kind(conn, 'broken_extractor_v1')
conn.commit()
print(f'removed {n} rows')
```

`evidence_kind='manual'` rows are NOT touched by a purge of any
other kind — manual contributions are safe.

## Adding new extractors

1. Implement a callable
   ```python
   def my_extractor(conn, *, derived_from=None) -> dict[str, int]:
       ...
       return {"items_inserted": N, "items_skipped": K}
   ```
   that walks the shard's existing rows (`documents`, `chunks`,
   `edges`, `derivations`, …) & calls
   `aborist.concepts.store.add_concept_relation` for each finding.

2. Pick a stable `evidence_kind` string. Don't reuse an existing
   one unless your extractor genuinely produces the same kind of
   output as that one (so `purge --evidence-kind X` semantics
   stay clean).

3. Register in `aborist/concepts/extract.py:EXTRACTORS`.

4. Document the trade-offs in this file alongside `link_reciprocity`.

## Deferred follow-ons

- **CLI commands** — `aborist concepts {seed,list,add,derive,purge}`.
  Today the helpers are accessible via `python -c "..."`.
- **Wikipedia See-also extractor** — parses `==See also==` sections
  in chunked wikitext beyond what `edges` already captures (some
  See-also entries are wikilinks already in edges; some are bullet
  lists with annotations that aren't).
- **Wikipedia category extractor** — parses `[[Category:X]]` tails
  & emits `relation_kind='category'` rows.
- **Hatnote / disambiguation extractor** — parses `{{about|...}}`
  & `{{not to be confused with|...}}` templates as antonym /
  rivalry signals.
