diff --git a/CLAUDE.md b/CLAUDE.md index 34134ee..c3d7fee 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -204,8 +204,12 @@ enough; revert at your peril. Order: 4. **`_filter_by_title_relevance` — four accept paths**: title-token overlap, TF-IDF core match, body density, **phrase match** (accept-path 4 lets phrase-route hits with no title overlap survive). -5. **Rivalry exclusion** (`qa/concepts.py`) — Intel-titled docs drop - from AMD queries; reverse holds. +5. **Rivalry exclusion + synonym expansion** (`aborist/concepts/`) — + Intel-titled docs drop from AMD queries; reverse holds. Backed + by the per-shard `concept_relations` SQLite table (corpus-derived, + not hand-curated). 1.6% storage tax measured at backfill on 6 GB + wiki — kept flat, no further compaction. See + `docs/concept-relations-design.md` for the storage choice rationale. 6. **Stem-aware token matching** — possessive / plural collapse (`superman's → supermans → superman`). 7. **Per-source context cap** — `max_context_chars / top_k`. Prevents diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 8953d30..b9c040a 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -45,6 +45,9 @@ proposing change. Examples that stay un-numbered: `self-reference-thought-chains-design.md` (older design docs that pre-date the ticket convention; leave as-is unless retroactively promoting one to a ticket adds value) +- `concept-relations-design.md` (architecture reference for the + per-shard `concept_relations` synonym/rivalry layer + the 1.6% + storage-tax rationale; landed 2026-05-01) If a doc proposes change AND awaits a decision AND has scoped implementation cost, it's a ticket. Otherwise it's reference. diff --git a/docs/concept-relations-design.md b/docs/concept-relations-design.md new file mode 100644 index 0000000..0a51f7c --- /dev/null +++ b/docs/concept-relations-design.md @@ -0,0 +1,270 @@ +# Concept relations: corpus-derived synonym & rivalry layer + +**Status:** landed — `aborist/concepts/` package, 2026-05-01 (commit +`5fd458a`) +**Audience:** anyone editing retrieval, storage budget, or planning +new concept extractors +**Hard constraint:** writes to `concept_relations` MUST NOT affect +`document_root`, `chunk_root`, or `cache_key`. The layer is a +secondary index over the Merkle-committed corpus; backfilling it is +safe across the entire corpus without invalidating any cached answer. + +## Why + +Pre-2026-05-01 the synonym & rivalry data lived as hand-curated +`frozenset`s in `aborist/qa/concepts.py`: + +```python +SYNONYM_GROUPS = [ + frozenset({"amd", "athlon", "duron", ...}), + frozenset({"intel", "pentium", ...}), + ... # 7 groups total +] +RIVALRIES = [(0, 1), (4, 5)] +``` + +The original commit message even flagged the limit: + +> *Phase 1: hand-curated. Phase 2 idea: derive from Wikipedia's +> category graph or "See also" sections.* + +Phase 1 didn't scale. Adding a domain meant editing Python source, +committing, pushing, redeploying. A 3.47M-doc corpus has thousands +of concept families; hand-curating them is a fool's errand. The +corpus already encodes the relationships we'd be hand-rebuilding — +"See also" sections, category links, internal-link clusters. +Phase 2 reads what's already there. + +## Architecture + +A per-shard `concept_relations` SQLite table (live alongside +`documents` and `chunks`): + +```sql +CREATE TABLE concept_relations ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + source_root TEXT NOT NULL, + relation_kind TEXT NOT NULL CHECK (relation_kind IN + ('synonym','antonym','rivalry','category')), + token TEXT NOT NULL, + target TEXT NOT NULL, + evidence_kind TEXT NOT NULL, + confidence REAL NOT NULL DEFAULT 1.0, + derived_at INTEGER NOT NULL, + derived_from TEXT, + UNIQUE (source_root, relation_kind, token, target, evidence_kind) +); +``` + +**Append-only.** UNIQUE on `(source_root, relation_kind, token, +target, evidence_kind)` makes re-derivation idempotent — re-running +an extractor adds nothing if no new relations have appeared since +last run. `INSERT OR IGNORE` is the only write path. + +**Per-shard.** Concept relations live in the shard whose document +they were derived from. Mesh sync moves shards between peers; +relations come along automatically. + +**Cross-shard lookup.** `aborist.concepts.query.synonyms_for` walks +every shard via `connect_query`'s UNION view (the same pattern as +cross-shard FTS5 search). Concept relations from shard 003 are +visible to a query routed at shard 000. + +**Public API stable.** `synonym_expand(tokens, *, shards_dir=...)` +& `rivalry_excluded(tokens, ..., shards_dir=...)` keep the legacy +shape from `aborist/qa/concepts.py`. The old module is now a +back-compat shim that delegates to `aborist.concepts.query`. Call +sites in `qa/query.py` thread `shards_dir` through but did not +otherwise change. + +## Extractors + +`aborist/concepts/extract.py` ships an extractor framework. Each +extractor reads existing corpus rows (no new crawling) & emits +concept relations under a stable `evidence_kind` string. An operator +can `purge --evidence-kind X` to revoke a single extractor's output +without touching manual or other-extractor rows. + +**`link_reciprocity_synonym`** (built-in). For any reciprocal edge +pair (A→B AND B→A) in the existing `edges` table, emit a synonym +edge between every (title-token-of-A, title-token-of-B) pair. +Title-tokens are filtered to ≥4 chars + stopword-stripped to keep +generic words like "the" or "and" from generating noise. + +The `edges` table is already populated on ingest. For Wikipedia, +the wikitext parser pulls every `[[link]]` as an edge row. For HTML, +`aborist/sources/html_page.py:parse_html` pulls every ``. +So the link reciprocity extractor works for any corpus that flows +through aborist's standard ingest path: Wikipedia, crawled HTML +sites (russell.ballestrini.net pattern), or any other document +graph the corpus already encodes. + +## Storage cost — measured + +Backfill on 6 GB of Wikipedia (2003 cur dump, 4 shards, 3.47M docs): + +| Shard | Docs | Resolved edges | Reciprocal pairs | Synonyms inserted | Backfill time | Storage | +|------------|------------|----------------|------------------|-------------------|---------------|---------| +| `000.db` | ~870k | 2,703,287 | 13,562 | 71,288 | 50.6s | 23.52 MB | +| `001.db` | ~870k | 2,571,390 | 14,078 | 73,351 | 47.7s | 24.20 MB | +| `002.db` | ~870k | 2,772,156 | 13,708 | 72,576 | 1m34s | 23.92 MB | +| `003.db` | ~870k | 2,707,627 | 13,800 | 72,633 | 1m03s | 23.94 MB | +| **total** | **3.47M** | **10,754,460** | **55,148** | **289,848** | **4m16s** | **95.58 MB** | + +**Per-row cost: ~330 bytes.** The bulk is the 64-char hex +`source_root` stored both in the table & in the UNIQUE auto-index +that enforces the idempotent-re-derivation key. Per-shard breakdown: + +| Component | Size per shard | Purpose | +|--------------------------------|----------------|---------| +| `concept_relations` table | ~10.0 MB | rows | +| `sqlite_autoindex` (UNIQUE) | ~9.3 MB | enforces idempotent re-derivation | +| `idx_concept_evid` | ~1.8 MB | for `purge --evidence-kind X` | +| `idx_concept_kind` | ~1.0 MB | filter by relation_kind | +| `idx_concept_target` | ~1.0 MB | reverse lookup | +| `idx_concept_token` | ~1.0 MB | forward lookup | + +**95.58 MB on a 6 GB corpus = 1.6% storage tax for the entire +denormalization.** Full backfill took 4m16s wall-clock across the +4 wiki shards — the cost is paid once. + +## Storage choice — keep, don't compact + +We considered three compactions & rejected all three. Documented +here so a future maintainer doesn't reopen the question without a +measured reason. + +### Option A — drop `idx_concept_evid` (saves ~1.8 MB / shard, ~7 MB total) + +`idx_concept_evid` exists to make `purge --evidence-kind X` cheap +(one index lookup per evidence_kind instead of a full table scan). +Without it, purge becomes O(N) over the whole table — acceptable +for a one-shot cleanup, painful for a tight extractor-development +loop where an operator runs purge then re-derive many times. + +**Decision:** keep. The 1.8 MB saving doesn't justify the painful +debugging loop. + +### Option B — store `source_root` as 32-byte BLOB instead of 64-char hex TEXT (saves ~5 MB / shard, ~20 MB total) + +The `source_root` column is the largest single contributor to +storage (~50% of per-row cost). Storing as BLOB instead of hex TEXT +halves it. + +**Decision:** keep TEXT. The rest of the schema (documents.document_root, +chunks.leaf_hash, providence_cache.source_root, audit_events.subject_root) +all use hex TEXT. Mixing TEXT vs BLOB across tables hurts schema +legibility & complicates joins. 5 MB saving doesn't justify the +inconsistency. + +### Option C — normalize `source_root` into a `source_lookup(id, hex)` foreign key (saves ~7 MB / shard, ~28 MB total) + +Replace the 64-char hex `source_root` column with a 4-8 byte +`source_id` integer pointing at a `source_lookup` table that maps +id ↔ hex. + +**Decision:** keep flat. The savings are real but every concept +lookup adds a JOIN. The lookup is in the retrieval hot path +(`synonym_expand` is called per-query). Adding a JOIN for a 0.5% +storage win is the wrong direction. + +### Final verdict + +**1.6% storage tax is fine.** Concept relations are a query-time +performance accelerator over an already-3.5GB-per-shard corpus. +The tax is paid once at backfill & every retrieval-time lookup +benefits. If a future shard layout pushes total storage to a +different cost regime (e.g. compressed columnstore), revisit Option +B at that point — the boundary changes the tradeoff math. + +## How to use + +### Backfill the existing corpus + +```bash +# Manual one-shot backfill against all wiki shards +python -c " +from aborist.concepts.extract import link_reciprocity_synonym +from aborist.store import connect, discover_shards +from pathlib import Path +for shard in discover_shards(Path('~/.aborist/shards').expanduser()): + if not shard.stem.isdigit(): + continue + conn = connect(shard) + print(shard.name, link_reciprocity_synonym(conn, derived_from=f'backfill@{shard.name}')) + conn.close() +" +``` + +(CLI commands `aborist concepts {seed,list,add,derive,purge}` +deferred to a follow-on commit.) + +### Add a manual relation (e.g. when a domain expert sees a gap) + +```python +from aborist.concepts.store import add_concept_relation +from aborist.store import connect + +conn = connect('~/.aborist/shards/000.db') +add_concept_relation( + conn, + source_root='__manual__', + relation_kind='synonym', + token='telepathy', + target='neurotechnology', + evidence_kind='manual', + derived_from='fox 2026-05-01: brain-tech retrieval gap', +) +conn.commit() +conn.close() +``` + +### Revoke a buggy extractor's output + +```python +from aborist.concepts.store import purge_by_evidence_kind +from aborist.store import connect + +conn = connect('~/.aborist/shards/000.db') +n = purge_by_evidence_kind(conn, 'broken_extractor_v1') +conn.commit() +print(f'removed {n} rows') +``` + +`evidence_kind='manual'` rows are NOT touched by a purge of any +other kind — manual contributions are safe. + +## Adding new extractors + +1. Implement a callable + ```python + def my_extractor(conn, *, derived_from=None) -> dict[str, int]: + ... + return {"items_inserted": N, "items_skipped": K} + ``` + that walks the shard's existing rows (`documents`, `chunks`, + `edges`, `derivations`, …) & calls + `aborist.concepts.store.add_concept_relation` for each finding. + +2. Pick a stable `evidence_kind` string. Don't reuse an existing + one unless your extractor genuinely produces the same kind of + output as that one (so `purge --evidence-kind X` semantics + stay clean). + +3. Register in `aborist/concepts/extract.py:EXTRACTORS`. + +4. Document the trade-offs in this file alongside `link_reciprocity`. + +## Deferred follow-ons + +- **CLI commands** — `aborist concepts {seed,list,add,derive,purge}`. + Today the helpers are accessible via `python -c "..."`. +- **Wikipedia See-also extractor** — parses `==See also==` sections + in chunked wikitext beyond what `edges` already captures (some + See-also entries are wikilinks already in edges; some are bullet + lists with annotations that aren't). +- **Wikipedia category extractor** — parses `[[Category:X]]` tails + & emits `relation_kind='category'` rows. +- **Hatnote / disambiguation extractor** — parses `{{about|...}}` + & `{{not to be confused with|...}}` templates as antonym / + rivalry signals.