russell@unturf.com
326badf6d8
Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).
(1) README — label refresh:
- Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
- Verifier section spells out both layers (schema trichotomy +
display ladder), the seven hard checks of claim_lattice, and
the five anchor classes of warrant.
- Architecture tree updated: concepts/ package added, qa/
sub-modules expanded (warrant, evidence, parse_claims, dag),
verify.py described as quote/span/entity/paraphrase + claim_lattice.
- Concept overlay description updated for corpus-derived layer
(concept_relations table, link_reciprocity extractor, 1.6%
tax cite).
- Test count: 326+ → 641+.
(2) docs/diagrams/ — Graphviz dot sources:
- aborist-modules.dot — top-level package graph (substrate /
storage / sources / retrieval / qa / mesh / cli)
- query-pipeline.dot — question → cache → retrieval → LLM →
verify → render → cache write, with phase budgets
- ingest-pipeline.dot — source doc → canonicalize → chunk →
Merkle → upsert (+ optional distill)
- verifier-ladder.dot — (audit_mode, violations) → display rung
decision tree
Existing mesh-*.dot kept as-is. Makefile `make docs` target
extended to also emit .svg alongside the existing .png so the
diagrams render in markdown viewers.
(3) docs/modules/ — per-module reference pages:
- index.md (links to every diagram + every module page)
- merkle.md, document.md, store.md, ingest.md, evict.md,
sources.md, search.md, concepts.md, qa.md, distill.md,
wikitext.md
Each page is a one-screenful concise reference: what the
module is for, public API, key invariants, embedded diagrams
where useful, link to source. Mesh stays at the existing
docs/mesh.md + docs/mesh-deploy.md (already comprehensive).
Tests: 641 passed (no code change).
179 lines
6.5 KiB
Markdown
179 lines
6.5 KiB
Markdown
# `aborist.concepts`
|
|
|
|
Per-shard `concept_relations` SQLite table — the corpus-derived
|
|
synonym, rivalry, antonym & category overlay that replaces the
|
|
hand-curated frozensets that lived in `aborist/qa/concepts.py`
|
|
through April 2026.
|
|
|
|
**Full architecture rationale lives in
|
|
[`../concept-relations-design.md`](../concept-relations-design.md)**
|
|
including the 1.6% storage-tax measurement and the three-compactions-
|
|
considered-and-rejected analysis. This page is the API reference.
|
|
|
|
## Sub-modules
|
|
|
|
### `concepts.store` — append-only CRUD
|
|
|
|
```python
|
|
from aborist.concepts.store import (
|
|
add_concept_relation, # idempotent INSERT OR IGNORE
|
|
concept_relations_for_token, # read all relations for a token
|
|
purge_by_evidence_kind, # the only DELETE path
|
|
list_evidence_kinds, # diagnostic
|
|
RELATION_KINDS, # ('synonym','antonym','rivalry','category')
|
|
)
|
|
```
|
|
|
|
UNIQUE on `(source_root, relation_kind, token, target, evidence_kind)`
|
|
makes re-derivation idempotent. `purge_by_evidence_kind` lets an
|
|
operator revoke a single extractor's output without touching manual
|
|
or other-extractor rows.
|
|
|
|
[`aborist/concepts/store.py`](../../aborist/concepts/store.py)
|
|
|
|
### `concepts.query` — cross-shard lookup
|
|
|
|
```python
|
|
from aborist.concepts import (
|
|
synonym_expand, # query tokens → expanded set
|
|
rivalry_excluded, # query tokens → tokens to drop from results
|
|
has_compare_phrasing, # bool — does the query say "vs", "compare", etc.
|
|
invalidate_cache, # drop the per-process LRU
|
|
)
|
|
|
|
expanded = synonym_expand({"thoughts"}, shards_dir=shards_dir)
|
|
# → {"thoughts", "telepathy", "neurotechnology", "mind", "cognition", ...}
|
|
```
|
|
|
|
Two synonym indices are loaded:
|
|
|
|
- **`manual_index`** — `manual_legacy` + `manual` rows. Curated;
|
|
always expanded regardless of per-token degree. Captures the
|
|
brain-tech / AMD-family / Mac / Linux / etc. seed groups.
|
|
- **`derived_index`** — `link_reciprocity` & other corpus-derived
|
|
edges. Subject to **`MAX_NEIGHBORS_PER_TOKEN=8`** cap because the
|
|
Wikipedia link graph carries topic-adjacency noise on generic
|
|
tokens (person, thoughts, language).
|
|
|
|
Overall **`MAX_TOTAL_TOKENS=50`** cap on expanded set bounds the
|
|
SQL clause count downstream so retrieval stays sub-second.
|
|
|
|
A per-process LRU keyed on shard mtime avoids re-loading the index
|
|
on every query (290k rows across 4 shards loads in ~1.8s cold).
|
|
|
|
[`aborist/concepts/query.py`](../../aborist/concepts/query.py)
|
|
|
|
### `concepts.extract` — pluggable extractor framework
|
|
|
|
```python
|
|
from aborist.concepts.extract import (
|
|
EXTRACTORS, # registry: evidence_kind → callable
|
|
link_reciprocity_synonym, # built-in extractor
|
|
)
|
|
|
|
# Run an extractor against a shard:
|
|
result = link_reciprocity_synonym(conn, derived_from="backfill@2026-05-01")
|
|
# → {"reciprocal_pairs": N, "synonyms_inserted": M, "synonyms_skipped": K}
|
|
```
|
|
|
|
Each extractor walks the shard's existing rows (`documents`,
|
|
`chunks`, `edges`, `derivations`) — **no new crawler needed** — &
|
|
emits concept relations under a stable `evidence_kind` string that
|
|
supports targeted purge.
|
|
|
|
The built-in `link_reciprocity_synonym` reads the existing `edges`
|
|
table for reciprocal A↔B link pairs and emits a synonym edge between
|
|
every (title-token-of-A, title-token-of-B) pair. Works for
|
|
Wikipedia (See-also bidirectional), HTML site internal links
|
|
(russell.ballestrini.net pattern), or any document graph with
|
|
bidirectional links. Title-tokens are filtered to ≥4 chars +
|
|
stopword-stripped.
|
|
|
|
To add a new extractor:
|
|
|
|
1. Implement `(conn, *, derived_from) -> dict[str, int]` that calls
|
|
`add_concept_relation` for each finding.
|
|
2. Pick a stable `evidence_kind` string.
|
|
3. Register in `EXTRACTORS`.
|
|
|
|
[`aborist/concepts/extract.py`](../../aborist/concepts/extract.py)
|
|
|
|
### `concepts.seed` — legacy frozenset migration
|
|
|
|
One-shot migration of the 8 hand-curated frozenset groups
|
|
(AMD-family, Intel-family, HTTP, FTP, Mac, Windows, Linux,
|
|
brain-tech) to `evidence_kind='manual_legacy'` rows.
|
|
|
|
Writes **clique edges** within each group — every `(a, b)` pair —
|
|
so any member retrieves every other member (preserves the legacy
|
|
frozenset semantic where lookups didn't depend on which token in
|
|
the group was the anchor).
|
|
|
|
```python
|
|
from aborist.concepts.seed import seed_legacy_concepts
|
|
result = seed_legacy_concepts(conn)
|
|
# → {"synonyms_inserted": N, "rivalries_inserted": M, "skipped": K}
|
|
```
|
|
|
|
Idempotent — re-running adds nothing if all the rows already exist.
|
|
|
|
[`aborist/concepts/seed.py`](../../aborist/concepts/seed.py)
|
|
|
|
## Data model
|
|
|
|
```sql
|
|
CREATE TABLE concept_relations (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
source_root TEXT NOT NULL,
|
|
relation_kind TEXT NOT NULL CHECK (relation_kind IN
|
|
('synonym','antonym','rivalry','category')),
|
|
token TEXT NOT NULL,
|
|
target TEXT NOT NULL,
|
|
evidence_kind TEXT NOT NULL,
|
|
confidence REAL NOT NULL DEFAULT 1.0,
|
|
derived_at INTEGER NOT NULL,
|
|
derived_from TEXT,
|
|
UNIQUE (source_root, relation_kind, token, target, evidence_kind)
|
|
);
|
|
```
|
|
|
|
**Append-only by construction.** Re-running an extractor adds
|
|
nothing if every relation already exists. No `UPDATE` path; only
|
|
`add_concept_relation` (insert) and `purge_by_evidence_kind`
|
|
(targeted delete).
|
|
|
|
**Per-shard storage.** Concept relations live in the shard whose
|
|
document derived them. Mesh sync moves shards between peers;
|
|
relations come along.
|
|
|
|
**Orthogonal to Merkle.** Writes to `concept_relations` NEVER
|
|
affect `document_root`, `chunk_root`, or `cache_key`. Backfilling
|
|
relations is safe across the entire corpus without invalidating
|
|
any cached answer or breaking any audit chain.
|
|
|
|
## Storage cost — measured
|
|
|
|
Backfill on 4 wiki shards (3.47M docs, 10.75M resolved edges):
|
|
|
|
| Shard | Reciprocal pairs | Synonyms | Storage |
|
|
|---|---:|---:|---:|
|
|
| 000.db | 13,562 | 71,288 | 23.52 MB |
|
|
| 001.db | 14,078 | 73,351 | 24.20 MB |
|
|
| 002.db | 13,708 | 72,576 | 23.92 MB |
|
|
| 003.db | 13,800 | 72,633 | 23.94 MB |
|
|
| **total** | **55,148** | **289,848** | **95.58 MB** |
|
|
|
|
**1.6% storage tax** on the 6 GB corpus. Backfill takes ~4 min
|
|
wall-clock total. Cost is paid once at backfill; every retrieval-
|
|
time lookup benefits.
|
|
|
|
## Diagrams
|
|
|
|
|
|
|
|
|
|
## Source
|
|
|
|
- [`aborist/concepts/`](../../aborist/concepts/) — package
|
|
- [`docs/concept-relations-design.md`](../concept-relations-design.md) — full design doc
|
|
- Whitepaper §13.4.11 — public-facing summary
|