docs(concepts): design reference + 1.6% storage-tax rationale
New `docs/concept-relations-design.md`: architecture reference for
the per-shard concept_relations layer that replaced the legacy
frozenset module (commit 5fd458a). Covers:
- Why phase 1 (hand-curated frozensets) didn't scale.
- Append-only schema + the three by-construction properties (idempotent
re-derivation via UNIQUE, per-shard storage, Merkle-orthogonal).
- Built-in `link_reciprocity_synonym` extractor reading the existing
`edges` table — no new crawler, works for Wikipedia AND HTML sites.
- Measured storage: 95.58 MB across 4 wiki shards (3.47M docs,
10.75M resolved edges, 55,148 reciprocal pairs, 289,848 synonyms),
4m16s wall-clock backfill. 1.6% tax on the 6 GB corpus.
- Three storage compactions considered & rejected, each with the
specific trade-off it loses on (drop idx_concept_evid → painful
purge debugging; BLOB source_root → schema inconsistency; FK
normalization → JOIN in retrieval hot path).
- How-to: backfill, manual add, purge.
- Adding new extractors.
- Deferred follow-ons (CLI commands, Wikipedia See-also extractor,
category extractor, hatnote extractor).
CLAUDE.md item 5 in the retrieval-pipeline list updated to point at
the new module path (aborist/concepts/) and the design doc.
TICKETS.md reference list updated to mention the new design doc.
This commit is contained in:
parent
5fd458aa41
commit
5247d8e282
3 changed files with 279 additions and 2 deletions
|
|
@ -204,8 +204,12 @@ enough; revert at your peril. Order:
|
|||
4. **`_filter_by_title_relevance` — four accept paths**: title-token
|
||||
overlap, TF-IDF core match, body density, **phrase match**
|
||||
(accept-path 4 lets phrase-route hits with no title overlap survive).
|
||||
5. **Rivalry exclusion** (`qa/concepts.py`) — Intel-titled docs drop
|
||||
from AMD queries; reverse holds.
|
||||
5. **Rivalry exclusion + synonym expansion** (`aborist/concepts/`) —
|
||||
Intel-titled docs drop from AMD queries; reverse holds. Backed
|
||||
by the per-shard `concept_relations` SQLite table (corpus-derived,
|
||||
not hand-curated). 1.6% storage tax measured at backfill on 6 GB
|
||||
wiki — kept flat, no further compaction. See
|
||||
`docs/concept-relations-design.md` for the storage choice rationale.
|
||||
6. **Stem-aware token matching** — possessive / plural collapse
|
||||
(`superman's → supermans → superman`).
|
||||
7. **Per-source context cap** — `max_context_chars / top_k`. Prevents
|
||||
|
|
|
|||
|
|
@ -45,6 +45,9 @@ proposing change. Examples that stay un-numbered:
|
|||
`self-reference-thought-chains-design.md` (older design docs that
|
||||
pre-date the ticket convention; leave as-is unless retroactively
|
||||
promoting one to a ticket adds value)
|
||||
- `concept-relations-design.md` (architecture reference for the
|
||||
per-shard `concept_relations` synonym/rivalry layer + the 1.6%
|
||||
storage-tax rationale; landed 2026-05-01)
|
||||
|
||||
If a doc proposes change AND awaits a decision AND has scoped
|
||||
implementation cost, it's a ticket. Otherwise it's reference.
|
||||
|
|
|
|||
270
docs/concept-relations-design.md
Normal file
270
docs/concept-relations-design.md
Normal file
|
|
@ -0,0 +1,270 @@
|
|||
# Concept relations: corpus-derived synonym & rivalry layer
|
||||
|
||||
**Status:** landed — `aborist/concepts/` package, 2026-05-01 (commit
|
||||
`5fd458a`)
|
||||
**Audience:** anyone editing retrieval, storage budget, or planning
|
||||
new concept extractors
|
||||
**Hard constraint:** writes to `concept_relations` MUST NOT affect
|
||||
`document_root`, `chunk_root`, or `cache_key`. The layer is a
|
||||
secondary index over the Merkle-committed corpus; backfilling it is
|
||||
safe across the entire corpus without invalidating any cached answer.
|
||||
|
||||
## Why
|
||||
|
||||
Pre-2026-05-01 the synonym & rivalry data lived as hand-curated
|
||||
`frozenset`s in `aborist/qa/concepts.py`:
|
||||
|
||||
```python
|
||||
SYNONYM_GROUPS = [
|
||||
frozenset({"amd", "athlon", "duron", ...}),
|
||||
frozenset({"intel", "pentium", ...}),
|
||||
... # 7 groups total
|
||||
]
|
||||
RIVALRIES = [(0, 1), (4, 5)]
|
||||
```
|
||||
|
||||
The original commit message even flagged the limit:
|
||||
|
||||
> *Phase 1: hand-curated. Phase 2 idea: derive from Wikipedia's
|
||||
> category graph or "See also" sections.*
|
||||
|
||||
Phase 1 didn't scale. Adding a domain meant editing Python source,
|
||||
committing, pushing, redeploying. A 3.47M-doc corpus has thousands
|
||||
of concept families; hand-curating them is a fool's errand. The
|
||||
corpus already encodes the relationships we'd be hand-rebuilding —
|
||||
"See also" sections, category links, internal-link clusters.
|
||||
Phase 2 reads what's already there.
|
||||
|
||||
## Architecture
|
||||
|
||||
A per-shard `concept_relations` SQLite table (live alongside
|
||||
`documents` and `chunks`):
|
||||
|
||||
```sql
|
||||
CREATE TABLE concept_relations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_root TEXT NOT NULL,
|
||||
relation_kind TEXT NOT NULL CHECK (relation_kind IN
|
||||
('synonym','antonym','rivalry','category')),
|
||||
token TEXT NOT NULL,
|
||||
target TEXT NOT NULL,
|
||||
evidence_kind TEXT NOT NULL,
|
||||
confidence REAL NOT NULL DEFAULT 1.0,
|
||||
derived_at INTEGER NOT NULL,
|
||||
derived_from TEXT,
|
||||
UNIQUE (source_root, relation_kind, token, target, evidence_kind)
|
||||
);
|
||||
```
|
||||
|
||||
**Append-only.** UNIQUE on `(source_root, relation_kind, token,
|
||||
target, evidence_kind)` makes re-derivation idempotent — re-running
|
||||
an extractor adds nothing if no new relations have appeared since
|
||||
last run. `INSERT OR IGNORE` is the only write path.
|
||||
|
||||
**Per-shard.** Concept relations live in the shard whose document
|
||||
they were derived from. Mesh sync moves shards between peers;
|
||||
relations come along automatically.
|
||||
|
||||
**Cross-shard lookup.** `aborist.concepts.query.synonyms_for` walks
|
||||
every shard via `connect_query`'s UNION view (the same pattern as
|
||||
cross-shard FTS5 search). Concept relations from shard 003 are
|
||||
visible to a query routed at shard 000.
|
||||
|
||||
**Public API stable.** `synonym_expand(tokens, *, shards_dir=...)`
|
||||
& `rivalry_excluded(tokens, ..., shards_dir=...)` keep the legacy
|
||||
shape from `aborist/qa/concepts.py`. The old module is now a
|
||||
back-compat shim that delegates to `aborist.concepts.query`. Call
|
||||
sites in `qa/query.py` thread `shards_dir` through but did not
|
||||
otherwise change.
|
||||
|
||||
## Extractors
|
||||
|
||||
`aborist/concepts/extract.py` ships an extractor framework. Each
|
||||
extractor reads existing corpus rows (no new crawling) & emits
|
||||
concept relations under a stable `evidence_kind` string. An operator
|
||||
can `purge --evidence-kind X` to revoke a single extractor's output
|
||||
without touching manual or other-extractor rows.
|
||||
|
||||
**`link_reciprocity_synonym`** (built-in). For any reciprocal edge
|
||||
pair (A→B AND B→A) in the existing `edges` table, emit a synonym
|
||||
edge between every (title-token-of-A, title-token-of-B) pair.
|
||||
Title-tokens are filtered to ≥4 chars + stopword-stripped to keep
|
||||
generic words like "the" or "and" from generating noise.
|
||||
|
||||
The `edges` table is already populated on ingest. For Wikipedia,
|
||||
the wikitext parser pulls every `[[link]]` as an edge row. For HTML,
|
||||
`aborist/sources/html_page.py:parse_html` pulls every `<a href>`.
|
||||
So the link reciprocity extractor works for any corpus that flows
|
||||
through aborist's standard ingest path: Wikipedia, crawled HTML
|
||||
sites (russell.ballestrini.net pattern), or any other document
|
||||
graph the corpus already encodes.
|
||||
|
||||
## Storage cost — measured
|
||||
|
||||
Backfill on 6 GB of Wikipedia (2003 cur dump, 4 shards, 3.47M docs):
|
||||
|
||||
| Shard | Docs | Resolved edges | Reciprocal pairs | Synonyms inserted | Backfill time | Storage |
|
||||
|------------|------------|----------------|------------------|-------------------|---------------|---------|
|
||||
| `000.db` | ~870k | 2,703,287 | 13,562 | 71,288 | 50.6s | 23.52 MB |
|
||||
| `001.db` | ~870k | 2,571,390 | 14,078 | 73,351 | 47.7s | 24.20 MB |
|
||||
| `002.db` | ~870k | 2,772,156 | 13,708 | 72,576 | 1m34s | 23.92 MB |
|
||||
| `003.db` | ~870k | 2,707,627 | 13,800 | 72,633 | 1m03s | 23.94 MB |
|
||||
| **total** | **3.47M** | **10,754,460** | **55,148** | **289,848** | **4m16s** | **95.58 MB** |
|
||||
|
||||
**Per-row cost: ~330 bytes.** The bulk is the 64-char hex
|
||||
`source_root` stored both in the table & in the UNIQUE auto-index
|
||||
that enforces the idempotent-re-derivation key. Per-shard breakdown:
|
||||
|
||||
| Component | Size per shard | Purpose |
|
||||
|--------------------------------|----------------|---------|
|
||||
| `concept_relations` table | ~10.0 MB | rows |
|
||||
| `sqlite_autoindex` (UNIQUE) | ~9.3 MB | enforces idempotent re-derivation |
|
||||
| `idx_concept_evid` | ~1.8 MB | for `purge --evidence-kind X` |
|
||||
| `idx_concept_kind` | ~1.0 MB | filter by relation_kind |
|
||||
| `idx_concept_target` | ~1.0 MB | reverse lookup |
|
||||
| `idx_concept_token` | ~1.0 MB | forward lookup |
|
||||
|
||||
**95.58 MB on a 6 GB corpus = 1.6% storage tax for the entire
|
||||
denormalization.** Full backfill took 4m16s wall-clock across the
|
||||
4 wiki shards — the cost is paid once.
|
||||
|
||||
## Storage choice — keep, don't compact
|
||||
|
||||
We considered three compactions & rejected all three. Documented
|
||||
here so a future maintainer doesn't reopen the question without a
|
||||
measured reason.
|
||||
|
||||
### Option A — drop `idx_concept_evid` (saves ~1.8 MB / shard, ~7 MB total)
|
||||
|
||||
`idx_concept_evid` exists to make `purge --evidence-kind X` cheap
|
||||
(one index lookup per evidence_kind instead of a full table scan).
|
||||
Without it, purge becomes O(N) over the whole table — acceptable
|
||||
for a one-shot cleanup, painful for a tight extractor-development
|
||||
loop where an operator runs purge then re-derive many times.
|
||||
|
||||
**Decision:** keep. The 1.8 MB saving doesn't justify the painful
|
||||
debugging loop.
|
||||
|
||||
### Option B — store `source_root` as 32-byte BLOB instead of 64-char hex TEXT (saves ~5 MB / shard, ~20 MB total)
|
||||
|
||||
The `source_root` column is the largest single contributor to
|
||||
storage (~50% of per-row cost). Storing as BLOB instead of hex TEXT
|
||||
halves it.
|
||||
|
||||
**Decision:** keep TEXT. The rest of the schema (documents.document_root,
|
||||
chunks.leaf_hash, providence_cache.source_root, audit_events.subject_root)
|
||||
all use hex TEXT. Mixing TEXT vs BLOB across tables hurts schema
|
||||
legibility & complicates joins. 5 MB saving doesn't justify the
|
||||
inconsistency.
|
||||
|
||||
### Option C — normalize `source_root` into a `source_lookup(id, hex)` foreign key (saves ~7 MB / shard, ~28 MB total)
|
||||
|
||||
Replace the 64-char hex `source_root` column with a 4-8 byte
|
||||
`source_id` integer pointing at a `source_lookup` table that maps
|
||||
id ↔ hex.
|
||||
|
||||
**Decision:** keep flat. The savings are real but every concept
|
||||
lookup adds a JOIN. The lookup is in the retrieval hot path
|
||||
(`synonym_expand` is called per-query). Adding a JOIN for a 0.5%
|
||||
storage win is the wrong direction.
|
||||
|
||||
### Final verdict
|
||||
|
||||
**1.6% storage tax is fine.** Concept relations are a query-time
|
||||
performance accelerator over an already-3.5GB-per-shard corpus.
|
||||
The tax is paid once at backfill & every retrieval-time lookup
|
||||
benefits. If a future shard layout pushes total storage to a
|
||||
different cost regime (e.g. compressed columnstore), revisit Option
|
||||
B at that point — the boundary changes the tradeoff math.
|
||||
|
||||
## How to use
|
||||
|
||||
### Backfill the existing corpus
|
||||
|
||||
```bash
|
||||
# Manual one-shot backfill against all wiki shards
|
||||
python -c "
|
||||
from aborist.concepts.extract import link_reciprocity_synonym
|
||||
from aborist.store import connect, discover_shards
|
||||
from pathlib import Path
|
||||
for shard in discover_shards(Path('~/.aborist/shards').expanduser()):
|
||||
if not shard.stem.isdigit():
|
||||
continue
|
||||
conn = connect(shard)
|
||||
print(shard.name, link_reciprocity_synonym(conn, derived_from=f'backfill@{shard.name}'))
|
||||
conn.close()
|
||||
"
|
||||
```
|
||||
|
||||
(CLI commands `aborist concepts {seed,list,add,derive,purge}`
|
||||
deferred to a follow-on commit.)
|
||||
|
||||
### Add a manual relation (e.g. when a domain expert sees a gap)
|
||||
|
||||
```python
|
||||
from aborist.concepts.store import add_concept_relation
|
||||
from aborist.store import connect
|
||||
|
||||
conn = connect('~/.aborist/shards/000.db')
|
||||
add_concept_relation(
|
||||
conn,
|
||||
source_root='__manual__',
|
||||
relation_kind='synonym',
|
||||
token='telepathy',
|
||||
target='neurotechnology',
|
||||
evidence_kind='manual',
|
||||
derived_from='fox 2026-05-01: brain-tech retrieval gap',
|
||||
)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
```
|
||||
|
||||
### Revoke a buggy extractor's output
|
||||
|
||||
```python
|
||||
from aborist.concepts.store import purge_by_evidence_kind
|
||||
from aborist.store import connect
|
||||
|
||||
conn = connect('~/.aborist/shards/000.db')
|
||||
n = purge_by_evidence_kind(conn, 'broken_extractor_v1')
|
||||
conn.commit()
|
||||
print(f'removed {n} rows')
|
||||
```
|
||||
|
||||
`evidence_kind='manual'` rows are NOT touched by a purge of any
|
||||
other kind — manual contributions are safe.
|
||||
|
||||
## Adding new extractors
|
||||
|
||||
1. Implement a callable
|
||||
```python
|
||||
def my_extractor(conn, *, derived_from=None) -> dict[str, int]:
|
||||
...
|
||||
return {"items_inserted": N, "items_skipped": K}
|
||||
```
|
||||
that walks the shard's existing rows (`documents`, `chunks`,
|
||||
`edges`, `derivations`, …) & calls
|
||||
`aborist.concepts.store.add_concept_relation` for each finding.
|
||||
|
||||
2. Pick a stable `evidence_kind` string. Don't reuse an existing
|
||||
one unless your extractor genuinely produces the same kind of
|
||||
output as that one (so `purge --evidence-kind X` semantics
|
||||
stay clean).
|
||||
|
||||
3. Register in `aborist/concepts/extract.py:EXTRACTORS`.
|
||||
|
||||
4. Document the trade-offs in this file alongside `link_reciprocity`.
|
||||
|
||||
## Deferred follow-ons
|
||||
|
||||
- **CLI commands** — `aborist concepts {seed,list,add,derive,purge}`.
|
||||
Today the helpers are accessible via `python -c "..."`.
|
||||
- **Wikipedia See-also extractor** — parses `==See also==` sections
|
||||
in chunked wikitext beyond what `edges` already captures (some
|
||||
See-also entries are wikilinks already in edges; some are bullet
|
||||
lists with annotations that aren't).
|
||||
- **Wikipedia category extractor** — parses `[[Category:X]]` tails
|
||||
& emits `relation_kind='category'` rows.
|
||||
- **Hatnote / disambiguation extractor** — parses `{{about|...}}`
|
||||
& `{{not to be confused with|...}}` templates as antonym /
|
||||
rivalry signals.
|
||||
Loading…
Add table
Add a link
Reference in a new issue