# Aborist module reference
_Single-file reference for every top-level package + diagrams._
## Diagrams index
### Diagrams
| Diagram | What it shows | File |
|---|---|---|
| **Module graph** | Top-level packages & how they import each other | [`aborist-modules.svg`](diagrams/aborist-modules.svg) ([dot](diagrams/aborist-modules.dot)) |
| **Query pipeline** | Question → cache → retrieval → LLM → verify → render | [`query-pipeline.svg`](diagrams/query-pipeline.svg) ([dot](diagrams/query-pipeline.dot)) |
| **Ingest pipeline** | Source document → Merkle-committed shard | [`ingest-pipeline.svg`](diagrams/ingest-pipeline.svg) ([dot](diagrams/ingest-pipeline.dot)) |
| **Verifier ladder** | (audit_mode, violations) → display rung | [`verifier-ladder.svg`](diagrams/verifier-ladder.svg) ([dot](diagrams/verifier-ladder.dot)) |
| **Mesh data flow** | Federation: roster, gossip, AEAD envelope | [`mesh-data-flow.svg`](diagrams/mesh-data-flow.svg) |
| **Mesh epoch lifecycle** | Epoch advance via add/kick/rotate | [`mesh-epoch-lifecycle.svg`](diagrams/mesh-epoch-lifecycle.svg) |
| **Mesh identity stack** | Ed25519 sign + X25519 DH key derivation | [`mesh-identity-stack.svg`](diagrams/mesh-identity-stack.svg) |
| **Mesh secret envelope** | AEAD-wrapped epoch secret per peer | [`mesh-secret-envelope.svg`](diagrams/mesh-secret-envelope.svg) |
| **Mesh group decisions** | Membership change voting & quorum | [`mesh-group-decisions.svg`](diagrams/mesh-group-decisions.svg) |
Render diagrams locally:
```
make docs # runs `dot -Tsvg` and `-Tpng` on every docs/diagrams/*.dot
```
### Substrate (no SQL, pure data structures)
| Module | One-line role | Doc |
|---|---|---|
| [`merkle.py`](../aborist/merkle.py) | Merkle tree + proof — Python port of `proxy.unturf.com/pkg/verified/merkle.go` | [↓](#merkle-py) |
| [`document.py`](../aborist/document.py) | `Document`, `Edge`, `Chunker` (default `tok-512-v1`) | [↓](#document-py) |
| [`wikitext.py`](../aborist/wikitext.py) | `to_base()` — wikitext → plain prose, BASE_VERSION-pinned | [↓](#wikitext-py) |
### Storage
| Module | One-line role | Doc |
|---|---|---|
| [`store.py`](../aborist/store.py) | v9.8 SQLite schema + audit chain helpers | [↓](#store-py) |
| [`ingest.py`](../aborist/ingest.py) | normalize → chunk → merkle → upsert (bulk-batched) | [↓](#ingest-py) |
| [`evict.py`](../aborist/evict.py) | hot ↔ cold tier transitions; rehydrate via source | [↓](#evict-py) |
### Sources (corpus producers)
| Module | One-line role | Doc |
|---|---|---|
| [`sources/wikipedia.py`](../aborist/sources/wikipedia.py) | Wikipedia 2003 cur + old SQL dumps (bz2-streamed) | [↓](#sources-py) |
| [`sources/wikipedia_xml.py`](../aborist/sources/wikipedia_xml.py) | Phase IV XML dumps (iterparse, page + history) | [↓](#sources-py) |
| [`sources/html_page.py`](../aborist/sources/html_page.py) | URL list + selectolax + httpx (robots-aware) | [↓](#sources-py) |
| [`sources/crawler/`](../aborist/sources/crawler/) | verbatim AsyncWebFetcher lift + ingest bridge | [↓](#sources-py) |
| [`sources/grok.py`](../aborist/sources/grok.py) | xAI data export (conversations + media prompts) | [↓](#sources-py) |
| [`sources/vcs.py`](../aborist/sources/vcs.py) | git + Mercurial repos (HEAD walk, supersedes chain) | [↓](#sources-py) |
### Search & retrieval
| Module | One-line role | Doc |
|---|---|---|
| [`search/`](../aborist/search/) | FTS5 backend + `SearchBackend` ABC + `AuditMode` enum | [↓](#search-py) |
| [`concepts/`](../aborist/concepts/) | Per-shard `concept_relations` synonym/rivalry overlay | [↓](#concepts-py) |
### Q&A pipeline
| Module | One-line role | Doc |
|---|---|---|
| [`qa/keys.py`](../aborist/qa/keys.py) | 8-dim cache_key + `question_hash` | [↓](#qa-py) |
| [`qa/client.py`](../aborist/qa/client.py) | `ChatClient` + `StubClient` + `OpenAICompatibleClient` | [↓](#qa-py) |
| [`qa/runner.py`](../aborist/qa/runner.py) | `ask()`: single-doc Q&A + cache + verify | [↓](#qa-py) |
| [`qa/query.py`](../aborist/qa/query.py) | `query()`: multi-source RAG + concept overlay | [↓](#qa-py) |
| [`qa/verify.py`](../aborist/qa/verify.py) | quote/span/entity/paraphrase + claim_lattice (7 hard checks) | [↓](#qa-py) |
| [`qa/warrant.py`](../aborist/qa/warrant.py) | 5 anchor classes (proper-noun · date · count · entity-list · cause) | [↓](#qa-py) |
| [`qa/evidence.py`](../aborist/qa/evidence.py) | `EvidenceObject` + spotlight excerpt (density rank) | [↓](#qa-py) |
| [`qa/parse_claims.py`](../aborist/qa/parse_claims.py) | pointer-line parser (`claim. [E1,E2]`) | [↓](#qa-py) |
| [`qa/dag.py`](../aborist/qa/dag.py) | per-run Merkle-DAG (7-stage quote / 9-stage CTI) | [↓](#qa-py) |
| [`qa/inspect.py`](../aborist/qa/inspect.py) | sidecar diagnostic (read-only span classifier) | [↓](#qa-py) |
### Distillation
| Module | One-line role | Doc |
|---|---|---|
| [`distill/`](../aborist/distill/) | `Distiller` ABC + `first_sentence` + `tfidf` + runner | [↓](#distill-py) |
### Federation (off by default)
| Module | One-line role | Doc |
|---|---|---|
| [`mesh/`](../aborist/mesh/) | identity (Ed25519/X25519), per-epoch roster, AEAD envelope, gossip wire | [../mesh.md](mesh.md), [../mesh-deploy.md](mesh-deploy.md) |
### Entry point
| Module | One-line role | Doc |
|---|---|---|
| [`cli.py`](../aborist/cli.py) | `argparse` entrypoint — every `make` target dispatches here | run `aborist --help` or any `make help` target |
### Tickets, design docs, journals
See [`../TICKETS.md`](TICKETS.md) for the ticket index and the
list of design-reference docs that aren't tickets.
## `aborist.merkle`
Pure Merkle tree + proof primitives. Python port of
`proxy.unturf.com/pkg/verified/merkle.go` — convention-identical.
Used everywhere a content-addressable handle is needed: per-chunk
leaves, document_root, evidence_map_root, run_dag_root, snapshots.
### Conventions (do not silently change)
These match the Go reference & are load-bearing for cross-language
verification (Go peer ↔ Python peer compute bit-identical roots):
- **Leaf hash:** `sha256(0x00 || canonical_chunk_bytes)`. The `0x00`
prefix domain-separates leaves from internal nodes.
- **Internal hash:** `sha256(0x03 || left || right)`. The `0x03`
prefix is the **non-commutative** combine — `H(L,R) ≠ H(R,L)`.
Order matters.
- **Odd-element rule:** when a level has an odd count, the last
leaf is **self-duplicated** before pairing. NOT zero-padded.
- **Proof path:** each step carries an explicit `is_left: bool`
alongside the sibling hash so a verifier knows which side to put
the sibling on. Never sort siblings lexically — the order tells
the verifier the tree topology.
### API surface
```python
from aborist.merkle import MerkleTree, MerkleProof
tree = MerkleTree.build([b"chunk_0_bytes", b"chunk_1_bytes", ...])
tree.root # bytes(32) — sha256 of the whole tree
tree.leaves # list[bytes(32)] — leaf hashes in input order
proof = tree.proof_for(leaf_index=2)
proof.siblings # list[(sibling_hash, is_left)]
proof.verify(leaf_hash=tree.leaves[2], root=tree.root) # bool
```
### When to read the source
- Adding a new content-addressable artifact (cores, evidence maps,
snapshots, run-DAGs all touch this).
- Cross-language verification debugging (Go peer says one root,
Python peer says another — the difference is always in canonical
encoding, ordering, or one of the three prefix bytes above).
- Performance work — the Python build is ~3× slower than the Go
reference; if it ever shows up in profiling, that's the file.
### Diagrams
The module graph shows what depends on `merkle.py` (a lot — it's
substrate):
The ingest pipeline shows where leaf & root hashes get computed:
### Source
[`aborist/merkle.py`](../aborist/merkle.py) ·
Reference: [`proxy.unturf.com/pkg/verified/merkle.go`](https://git.unturf.com/engineering/unturf/proxy.unturf.com/-/blob/main/pkg/verified/merkle.go)
## `aborist.document`
The data structures every source produces and every storage layer
consumes. Three core types: `Document`, `Edge`, `Chunker`.
### `Document`
A single ingest unit: a Wikipedia article, an HTML page, a Grok
conversation, a git commit message, etc. Carries both the raw
content AND the version tags that determine its identity:
```python
@dataclass(frozen=True)
class Document:
document_uri: str # canonical URI (or stable surrogate for non-URI sources)
raw_content: str # source-of-truth bytes pre-canonicalization
kind: str # 'surface' / 'core' / 'visual' / etc.
chunking_version: str # e.g. 'tok-512-v1' — pinned by the Chunker
canonicalization_version: str # e.g. 'norm-v1' — pinned by canonicalize()
schema_version: str # e.g. 'v9.8.0' — store schema generation
title: str | None = None
edges: list[Edge] = () # outbound link graph
metadata: dict = ... # source-specific opaque payload
```
`document_root` is computed at ingest time as the Merkle root over
the canonicalized chunks. Two peers ingesting the same source +
running the same `chunking_version` + `canonicalization_version`
get bit-identical `document_root`s — the v9.8 admissibility property.
### `Edge`
One outbound link. `aborist/sources/wikipedia.py` emits one Edge
per `[[wikilink]]`; `aborist/sources/html_page.py` emits one per
``. The link graph IS the corpus topology — `concepts/extract.py`
later reads `edges` rows to derive synonym relations from
reciprocal links (no separate crawler needed).
```python
@dataclass(frozen=True)
class Edge:
src_root: str # source document_root
dst_uri: str # always present
dst_root: str # '' (unresolved) until the dst doc is also ingested
edge_type: str # 'wikilink' / 'href' / 'citation' / 'derived_from' / ...
anchor: str # chunk index or fragment, '' if N/A
```
### `Chunker`
ABC with one method `chunk(text: str) -> list[str]`. Default impl
is `TokenChunker` (`name='tok-512-v1'`) — splits on token-rough
windows so the resulting chunks are predictable for downstream FTS5
indexing & for the LLM context budget.
**Changing the chunker bumps `chunking_version` AND stales every
prior cache record** (chunking is one of the 8 cache_key dimensions).
Don't redefine `tok-512-v1`; add a new chunker as a new `name`
instead.
### Diagrams
### Source
[`aborist/document.py`](../aborist/document.py)
## `aborist.wikitext`
A single function: `to_base(raw)`. Converts MediaWiki wikitext to
plain prose deterministically.
```python
from aborist.wikitext import to_base, BASE_VERSION
prose = to_base("[[The Beatles]] are an [[English rock band]] from [[Liverpool]].")
## → "The Beatles are an English rock band from Liverpool."
```
### Why it exists
The corpus stores raw wikitext (so the link graph is recoverable
on demand) but the LLM and verifier both want plain prose. Reasons:
1. **Token efficiency.** Wikipedia chunks ship to Hermes with ~43%
fewer tokens after wikitext-strip — bigger context window for
the same chars budget.
2. **Verbatim citation.** The model can quote source paragraphs
verbatim instead of escaping `[[wikilinks]]`. The verifier's
substring test then matches cleanly.
3. **Pinned identity.** `BASE_VERSION='wikitext-base-v1'` lives in
`policy["base_version"]`, which folds into
`governance_policy_hash`. Bumping `BASE_VERSION` invalidates
every prior cache record on next lookup — same discipline as
`chunking_version` and `canonicalization_version`.
### Hot-path discipline
`to_base()` runs on the assembled context **before** the LLM call
in `aborist/qa/runner.py` and `aborist/qa/query.py`, **and again
inside `verify_quotes`** so the verifier compares like-against-like.
Both sides see prose.
### Optional dependency
Backed by `mwparserfromhell`. Install via `pip install '.[wikitext]'`
to enable. Without the dep, `_wikitext_to_base = None` and
`policy["base_version"] = None` — graceful fallback leaves raw
wikitext in both context and verifier (works, just less efficient).
### Source
[`aborist/wikitext.py`](../aborist/wikitext.py)
## `aborist.store`
The v9.8 SQLite schema, the audit chain, and the cross-shard
read-only view. Every table that holds runtime state lives here.
### Schema overview (per shard)
```
documents – one row per source document, keyed on document_root
chunks – per-document chunk content + tier (hot/cold)
chunks_fts – FTS5 contentless index, rowid = chunks.chunk_id
merkle_nodes – internal-node hashes for proof reconstruction
edges – src_root → dst_root link graph (wikilink, href, …)
derivations – core_root ← src_root with proof_blob (Merkle)
providence_cache – Q&A records keyed on the v9.8 8-dim cache_key
audit_events – linear chain; event_hash = sha256(prev || canonical(body))
falsifications – record_id → state transition + reason + actor
snapshots – named corpus roots (one hash names a forest)
document_http_meta – ETag + Last-Modified for crawler conditional fetches
concept_relations – per-shard synonym/rivalry/category/antonym overlay
mesh_* – federation tables (off by default)
```
### v9.8 invariants (do not break)
- **Aborist is a v9.8 store.** Every providence record carries the
full **8-dim cache_key**: `source_root | question_hash |
model_profile_hash | conversation_hash | governance_policy_hash |
schema_version | canonicalization_version | chunking_version`.
Bumping any one invalidates prior records on lookup.
- **`falsification_state ∈ {live, failed, stale, quarantined}`.**
Cache lookups must filter on `state='live'`. Drift detection
flips to `stale`.
- **Audit chain.** Every state-changing op writes one row in
`audit_events` with `event_hash = sha256(prev_event_hash ||
canonical(body))`. Chain integrity is verified in
`make analyze-shards`. **Never insert into `audit_events`
directly — use `aborist.store.append_audit`.**
- **Cores never evict.** `evict_to_cold` only touches `kind='surface'`.
- **Idempotent re-ingest.** Same content → same `document_root` →
no-op insert. Same URI + different content → new doc + `supersedes`
edge linking new → old (lossless history).
### API surface
```python
from aborist.store import (
connect, # writable connection to a single shard
connect_query, # read-only UNION view across all shards
discover_shards, # list *.db files in a shards_dir
transaction, # BEGIN IMMEDIATE / COMMIT / ROLLBACK context manager
get_meta, set_meta,
append_audit, # the ONLY way to write audit_events
)
```
Cross-shard reads use `connect_query(shards_dir=...)` which ATTACHes
every `*.db` and creates UNION views over the shardable tables
(`documents`, `chunks`, `merkle_nodes`, `edges`, `derivations`,
`providence_cache`, `audit_events`, `falsifications`,
`concept_relations`).
### Performance pragmas
`connect()` applies these per-connection:
- `journal_mode=WAL` (set in SCHEMA_SQL once at first ingest)
- `synchronous=NORMAL` (skip per-commit fsync; safe under WAL)
- `cache_size=-65536` (64 MB page cache)
- `temp_store=MEMORY` (no /tmp churn for temp tables)
Don't downgrade to `synchronous=FULL` without a measured reason —
costs ~5× throughput.
### Diagrams
### Source
[`aborist/store.py`](../aborist/store.py)
## `aborist.ingest`
The bulk-batched pipeline that turns documents from a source into
Merkle-committed shard storage. Source-agnostic: anything that
implements `Source.iter_documents()` flows through here.
### Public API
```python
from aborist.ingest import ingest_source
from aborist.sources.wikipedia import WikipediaSqlDump
source = WikipediaSqlDump("/path/to/cur.sql.bz2")
ingest_source(
source,
db_path=Path("~/.aborist/shards/000.db"),
batch_size=200, # docs per transaction
progress_every=1000,
)
```
### What happens per document
1. **Canonicalize** — `canonicalize(text)`: NFC + ws-collapse +
strip ends. Pinned by `canonicalization_version='norm-v1'`.
2. **Chunk** — `Chunker.chunk(canonical_text)` → list of token-
bounded substrings. Default `tok-512-v1` chunker.
3. **Hash leaves** — `sha256(0x00 || canonical_chunk_bytes)` per
chunk.
4. **Merkle tree** — `MerkleTree.build(leaves).root` →
`document_root`. Two peers running the same chunker on the same
canonicalized content compute bit-identical roots.
5. **Upsert** — `documents` row keyed on `document_root` (idempotent
re-ingest), `chunks` rows with leaf hashes, `merkle_nodes` for
proof reconstruction, `edges` per outbound link.
6. **FTS5** — `chunks_fts` insert with rowid = `chunks.chunk_id`
so the search-time JOIN lines up.
7. **Audit event** — one row per ingest batch in `audit_events`,
chained on `prev_event_hash`.
### Batching discipline
Default `batch_size=200`: balances Python GIL overhead vs SQLite
transaction commit cost. Lower it (e.g. 50) only to bound peak
memory on a low-RAM host. Higher (e.g. 1000) for ETL throughput on
SSD storage when memory isn't tight.
`progress_every` prints a stderr line every N docs so long ingests
are observable. Use `PYTHONUNBUFFERED=1` for tail-able output.
### Resumability
Idempotent re-ingest: same content + same chunker + same canonicalize
= same `document_root` = no-op insert. So a crashed ingest can be
restarted from the source's beginning without duplicating rows.
Different content at the same URI gets a new `document_root` AND a
`supersedes` edge linking new → old (lossless history).
### Source
[`aborist/ingest.py`](../aborist/ingest.py)
## `aborist.evict`
Hot ↔ cold tier transitions. The corpus is large (3.47M Wikipedia
docs); not every chunk fits in working memory. `evict.py` is the
mechanism that moves rarely-touched chunks to a cold tier (still
indexed, just stored separately) and rehydrates them on demand
from the original source.
### API surface
```python
from aborist.evict import evict_to_cold, rehydrate
## Move chunks unused for >threshold days to cold tier
evict_to_cold(conn, max_age_days=90, max_evictions=10000)
## Pull a cold chunk back to hot from its original source
rehydrate(conn, document_root="abc123...")
```
### Invariant: cores never evict
`evict_to_cold` filters `WHERE kind='surface'`. Cores are always
hot — they're the long-tail-friendly compression layer that justifies
evicting their underlying surfaces. Evicting cores would defeat the
purpose.
### v9.8 falsification on drift
When `rehydrate()` re-fetches a document and the recomputed
`document_root` differs from the stored one, the source has
changed since ingest (Wikipedia article was edited, HTML page was
republished, etc.). The cache record's `falsification_state` flips
from `live` to `stale` — every providence record keyed on that
`source_root` is no longer admissible to lookups.
This is the **drift-detection-as-falsification** discipline: cache
hits don't blindly trust historical answers; they trust answers
that the SAME source still grounds.
### Tier values
`chunks.tier ∈ {'hot', 'cold'}`. Hot chunks live in `chunks.content`;
cold chunks live with NULL `content` and a `cold_uri` pointing at
the source. The QA pipeline's chunk-fetch path checks `tier`; on
'cold', it triggers `rehydrate` before continuing.
### Source
[`aborist/evict.py`](../aborist/evict.py)
## `aborist.sources`
Corpus producers. Each is a `Source` ABC implementation that yields
`Document` instances; the standard `ingest.ingest_source(source, db)`
pipeline takes them from there.
The `Source` ABC lives in [`aborist/source.py`](../aborist/source.py):
```python
class Source(ABC):
@abstractmethod
def iter_documents(self) -> Iterator[Document]: ...
```
### Built-in sources
#### `wikipedia.py` — Phase III SQL dumps (the canonical bootstrap)
Streams the Wikipedia 2003 `cur` (current revisions) and `old`
(revision history) SQL dumps. Hand-rolled escape-aware parser
(no `sqlite3` import — the dump is MySQL syntax). 4× speedup vs
char-by-char loops via `str.find` + slicing. cProfile any change.
Default Wikipedia 2003-05-16 dump source:
`https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`.
robots.txt returned 404 → no rules.
[`aborist/sources/wikipedia.py`](../aborist/sources/wikipedia.py)
#### `wikipedia_xml.py` — Phase IV XML dumps
Modern Wikipedia dump format (`enwiki-YYYYMMDD-pages-articles.xml.bz2`,
`enwiki-YYYYMMDD-pages-meta-history*.xml.bz2`). Uses `xml.etree.ElementTree.iterparse`
to stream-parse without loading the whole tree.
[`aborist/sources/wikipedia_xml.py`](../aborist/sources/wikipedia_xml.py)
#### `html_page.py` — single-URL or URL-list HTML ingest
Robots-aware (`urllib.robotparser`). Uses `selectolax` for fast
HTML parsing (CSS-selector based; ~10× faster than `lxml`). Pulls
the main body text + every `` as an `Edge` row.
The `edges` rows are what the corpus-derived synonym extractor
later reads — no separate crawler needed for site-internal link
graphs.
Optional dep: `pip install '.[html]'` for `selectolax` + `httpx`.
[`aborist/sources/html_page.py`](../aborist/sources/html_page.py)
#### `crawler/` — async BFS web crawl
Verbatim lift of an `AsyncWebFetcher` implementation + an `ingest`
bridge. BFS-discovers same-domain URLs from a seed, respecting
`robots.txt` + crawl delays. Captures ETag + Last-Modified per URL
into `document_http_meta` so a future recrawl can send conditional
HEAD requests.
[`aborist/sources/crawler/bridge.py`](../aborist/sources/crawler/bridge.py)
#### `grok.py` — xAI Grok export
Reads the `xAI-conversations.json` data export shape. Each
conversation becomes one `Document`; media prompts are kept
inline.
[`aborist/sources/grok.py`](../aborist/sources/grok.py)
#### `vcs.py` — git + Mercurial repositories
HEAD walk. Each commit becomes a `Document` (commit message + diff
stat). The supersedes chain captures commit ancestry as edges.
[`aborist/sources/vcs.py`](../aborist/sources/vcs.py)
### Source
[`aborist/sources/`](../aborist/sources/) ·
[`aborist/source.py`](../aborist/source.py) (ABC)
## `aborist.search`
The retrieval primitive. Today's only backend is FTS5 over the
chunks table; the `SearchBackend` ABC is in place so additional
backends (BM25 over titles, embedding-based vector search) can be
added without touching the rest of the QA pipeline.
### `SearchBackend` ABC
```python
from aborist.search import SearchBackend, AuditMode, Hit
class SearchBackend(ABC):
@abstractmethod
def search(self, query: str, limit: int = 20) -> list[Hit]: ...
```
Each `Hit` carries `(document_root, document_uri, chunk_idx,
snippet, score, audit_mode, title)`. `audit_mode` is the sticky
provenance label that tracks how the chunk made it into the index;
FTS5 backend always sets `UNGROUNDED` (search itself doesn't verify
anything — that's the QA pipeline's job).
### `FTS5Backend`
Wraps the contentless `chunks_fts` virtual table. Two-mode query:
- **AND-mode (strict, primary):** every content token must appear
in the doc. Keeps unrelated docs out of the context window.
- **OR-mode (fallback):** when AND returns 0 hits, fall back to OR
but **capped to top-5 longest tokens** (proxy for rarity). Long
topical synonyms fed via `extra_or_tokens` join the pool —
`neurotechnology` (15 chars) outranks `thoughts` (8) by length
and surfaces brain-tech titles for vocabulary-mismatch queries.
```python
from aborist.search import FTS5Backend
backend = FTS5Backend(conn)
## Plain search
hits = backend.search("permacomputer", limit=32)
## Search with synonym pool injection (used by qa.query._search_corpus)
hits = backend.search(
long_query,
limit=32,
extra_or_tokens=synonym_expand(qtokens, shards_dir=shards_dir),
)
```
### Stopword & stopword-cap discipline
`_FTS5_STOPWORDS` filters question words (`what`, `tell`, `please`)
+ generic connectors (`one`, `some`, `another`, `without`, `soon`,
`currently`) before AND/OR construction. Two principles:
- **Stay in sync with `_TITLE_STOPWORDS`** in `qa/query.py`. A token
filtered at retrieval time but kept at title-relevance check (or
vice versa) creates ranking incoherence.
- **`_OR_FALLBACK_MAX_TOKENS=5`** caps the OR-mode pool. Without
this, a 19-token OR clause matches millions of docs and forces
BM25 to rank them all — 13s/shard observed pre-cap. Now 0.25s/shard.
### Snippet building
FTS5 contentless mode means SQLite's built-in `snippet()` and
`highlight()` return empty. Aborist builds snippets in Python by
joining `chunks_fts.rowid = chunks.chunk_id`, decompressing the
chunk content, and locating query tokens locally
(`_build_snippet`).
### Source
[`aborist/search/fts5.py`](../aborist/search/fts5.py) ·
[`aborist/search/__init__.py`](../aborist/search/__init__.py)
## `aborist.concepts`
Per-shard `concept_relations` SQLite table — the corpus-derived
synonym, rivalry, antonym & category overlay that replaces the
hand-curated frozensets that lived in `aborist/qa/concepts.py`
through April 2026.
**Full architecture rationale lives in
[`../concept-relations-design.md`](../concept-relations-design.md)**
including the 1.6% storage-tax measurement and the three-compactions-
considered-and-rejected analysis. This page is the API reference.
### Sub-modules
#### `concepts.store` — append-only CRUD
```python
from aborist.concepts.store import (
add_concept_relation, # idempotent INSERT OR IGNORE
concept_relations_for_token, # read all relations for a token
purge_by_evidence_kind, # the only DELETE path
list_evidence_kinds, # diagnostic
RELATION_KINDS, # ('synonym','antonym','rivalry','category')
)
```
UNIQUE on `(source_root, relation_kind, token, target, evidence_kind)`
makes re-derivation idempotent. `purge_by_evidence_kind` lets an
operator revoke a single extractor's output without touching manual
or other-extractor rows.
[`aborist/concepts/store.py`](../aborist/concepts/store.py)
#### `concepts.query` — cross-shard lookup
```python
from aborist.concepts import (
synonym_expand, # query tokens → expanded set
rivalry_excluded, # query tokens → tokens to drop from results
has_compare_phrasing, # bool — does the query say "vs", "compare", etc.
invalidate_cache, # drop the per-process LRU
)
expanded = synonym_expand({"thoughts"}, shards_dir=shards_dir)
## → {"thoughts", "telepathy", "neurotechnology", "mind", "cognition", ...}
```
Two synonym indices are loaded:
- **`manual_index`** — `manual_legacy` + `manual` rows. Curated;
always expanded regardless of per-token degree. Captures the
brain-tech / AMD-family / Mac / Linux / etc. seed groups.
- **`derived_index`** — `link_reciprocity` & other corpus-derived
edges. Subject to **`MAX_NEIGHBORS_PER_TOKEN=8`** cap because the
Wikipedia link graph carries topic-adjacency noise on generic
tokens (person, thoughts, language).
Overall **`MAX_TOTAL_TOKENS=50`** cap on expanded set bounds the
SQL clause count downstream so retrieval stays sub-second.
A per-process LRU keyed on shard mtime avoids re-loading the index
on every query (290k rows across 4 shards loads in ~1.8s cold).
[`aborist/concepts/query.py`](../aborist/concepts/query.py)
#### `concepts.extract` — pluggable extractor framework
```python
from aborist.concepts.extract import (
EXTRACTORS, # registry: evidence_kind → callable
link_reciprocity_synonym, # built-in extractor
)
## Run an extractor against a shard:
result = link_reciprocity_synonym(conn, derived_from="backfill@2026-05-01")
## → {"reciprocal_pairs": N, "synonyms_inserted": M, "synonyms_skipped": K}
```
Each extractor walks the shard's existing rows (`documents`,
`chunks`, `edges`, `derivations`) — **no new crawler needed** — &
emits concept relations under a stable `evidence_kind` string that
supports targeted purge.
The built-in `link_reciprocity_synonym` reads the existing `edges`
table for reciprocal A↔B link pairs and emits a synonym edge between
every (title-token-of-A, title-token-of-B) pair. Works for
Wikipedia (See-also bidirectional), HTML site internal links
(russell.ballestrini.net pattern), or any document graph with
bidirectional links. Title-tokens are filtered to ≥4 chars +
stopword-stripped.
To add a new extractor:
1. Implement `(conn, *, derived_from) -> dict[str, int]` that calls
`add_concept_relation` for each finding.
2. Pick a stable `evidence_kind` string.
3. Register in `EXTRACTORS`.
[`aborist/concepts/extract.py`](../aborist/concepts/extract.py)
#### `concepts.seed` — legacy frozenset migration
One-shot migration of the 8 hand-curated frozenset groups
(AMD-family, Intel-family, HTTP, FTP, Mac, Windows, Linux,
brain-tech) to `evidence_kind='manual_legacy'` rows.
Writes **clique edges** within each group — every `(a, b)` pair —
so any member retrieves every other member (preserves the legacy
frozenset semantic where lookups didn't depend on which token in
the group was the anchor).
```python
from aborist.concepts.seed import seed_legacy_concepts
result = seed_legacy_concepts(conn)
## → {"synonyms_inserted": N, "rivalries_inserted": M, "skipped": K}
```
Idempotent — re-running adds nothing if all the rows already exist.
[`aborist/concepts/seed.py`](../aborist/concepts/seed.py)
### Data model
```sql
CREATE TABLE concept_relations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source_root TEXT NOT NULL,
relation_kind TEXT NOT NULL CHECK (relation_kind IN
('synonym','antonym','rivalry','category')),
token TEXT NOT NULL,
target TEXT NOT NULL,
evidence_kind TEXT NOT NULL,
confidence REAL NOT NULL DEFAULT 1.0,
derived_at INTEGER NOT NULL,
derived_from TEXT,
UNIQUE (source_root, relation_kind, token, target, evidence_kind)
);
```
**Append-only by construction.** Re-running an extractor adds
nothing if every relation already exists. No `UPDATE` path; only
`add_concept_relation` (insert) and `purge_by_evidence_kind`
(targeted delete).
**Per-shard storage.** Concept relations live in the shard whose
document derived them. Mesh sync moves shards between peers;
relations come along.
**Orthogonal to Merkle.** Writes to `concept_relations` NEVER
affect `document_root`, `chunk_root`, or `cache_key`. Backfilling
relations is safe across the entire corpus without invalidating
any cached answer or breaking any audit chain.
### Storage cost — measured
Backfill on 4 wiki shards (3.47M docs, 10.75M resolved edges):
| Shard | Reciprocal pairs | Synonyms | Storage |
|---|---:|---:|---:|
| 000.db | 13,562 | 71,288 | 23.52 MB |
| 001.db | 14,078 | 73,351 | 24.20 MB |
| 002.db | 13,708 | 72,576 | 23.92 MB |
| 003.db | 13,800 | 72,633 | 23.94 MB |
| **total** | **55,148** | **289,848** | **95.58 MB** |
**1.6% storage tax** on the 6 GB corpus. Backfill takes ~4 min
wall-clock total. Cost is paid once at backfill; every retrieval-
time lookup benefits.
### Diagrams
### Source
- [`aborist/concepts/`](../aborist/concepts/) — package
- [`docs/concept-relations-design.md`](../concept-relations-design.md) — full design doc
- Whitepaper §13.4.11 — public-facing summary
## `aborist.qa`
The Q&A pipeline. Question → cache → retrieval → LLM → verify →
render → cache write. Lives in 9 sub-modules; this page is the
map.
### Sub-modules
#### `qa.client` — LLM transport
`ChatClient` ABC with three concrete implementations:
- `StubClient` — deterministic test fixture; returns canned
responses keyed on the input. Used in unit tests to avoid network.
- `OpenAICompatibleClient` — talks to any OpenAI-shape `/v1/chat/completions`
endpoint (Hermes-3 on vLLM by default). Includes HTTP retry layer
(3× exponential backoff on 5xx).
- (Future) `AnthropicClient` — Claude API direct.
[`aborist/qa/client.py`](../aborist/qa/client.py)
#### `qa.keys` — the 8-dim cache_key
```
cache_key = sha256(
source_root | question_hash | model_profile_hash |
conversation_hash | governance_policy_hash |
schema_version | canonicalization_version | chunking_version
)
```
Two question-hash modes (`strict` vs `equivalence_class`) live here.
Bumping any of these eight dimensions invalidates prior records on
lookup. The `verifier_policy_hash` (v9.9 9th dim) is also
implemented here.
[`aborist/qa/keys.py`](../aborist/qa/keys.py)
#### `qa.runner` — `ask()` for single-doc Q&A
The simplest entry point. Take one document, ask one question, get
back an answer + audit_mode + cache record. Used by the CLI for
focused queries against one URI.
[`aborist/qa/runner.py`](../aborist/qa/runner.py)
#### `qa.query` — `query()` for multi-source RAG
The main retrieval entry point. Walks shards, runs FTS5 BM25 with
AND→OR fallback (with synonym-pool injection in OR mode), filters
by title relevance with 4 accept paths, reranks by body coverage +
title boost + source role + title purity, assembles a 60 KB context
budget, calls the LLM, runs the verifier, persists to
`providence_cache`.
[`aborist/qa/query.py`](../aborist/qa/query.py)
#### `qa.verify` — the layered verifier
Five strategies run in sequence; first to find evidence classifies:
1. `quote` — `"..."`-wrapped claims tested verbatim
2. `span` — bullet/sentence units substring-tested
3. `entity` — multi-word proper nouns with proximity gating
4. `paraphrase` — token coverage on prose-shaped spans (≥85%)
5. `claim_lattice` — pointer-line `[E1,E2]` or JSON; runs **seven
deterministic hard checks**:
1. parser succeeded
2. evidence_id resolves
3. source_role allowed
4. claim text non-empty
5. citation coverage threshold
6. pointer count cap (trim-and-verify)
7. anchor-class warrant (see `qa.warrant`)
The classifier output rolls up into the v9.8 trichotomy
`audit_mode ∈ {STRICT, HYBRID, UNGROUNDED}`. Display layer (in
`cli.py`) maps `(audit_mode, violations) → four-rung ladder`.
[`aborist/qa/verify.py`](../aborist/qa/verify.py)
#### `qa.warrant` — anchor-class warrant
Five lexical anchor classes the verifier composes:
- **Proper-noun** — relation-question shape; at least one
Title-Case anchor must appear in some cited span
- **Date** — claim has a 4-digit year + month name; ALL components
required in some cited span
- **Entity-list** — entity-list-shape question; ≥1 named entity
must anchor (demote-don't-reject)
- **Count** — count-shape question; count token must appear in
word OR digit form (digit↔word equivalence)
- **Cause** — why-shape question; ≥1 cause anchor (proper noun OR
≥5-char common noun outside stopword pool)
The warrant layer earns proof-path entry by staying **lexical** —
no NLI, no embeddings. Substring tests over already-canonicalized
spans. See `docs/concept-relations-design.md` (sibling section)
for the relationship to retrieval-time synonym expansion.
[`aborist/qa/warrant.py`](../aborist/qa/warrant.py)
#### `qa.evidence` — EvidenceObject + spotlight
Builds the runtime evidence map for claim-lattice modes. Each
chunk becomes one `EvidenceObject` carrying TWO ids:
- `pointer_id` — short prompt-facing tag (`E1`, `E2`, …)
- `evidence_id` — content-addressed `E########` (sha256-derived)
The model sees only `pointer_id`s in the prompt; the runtime maps
to `evidence_id` for the cache & run-DAG (run-stable identity).
The spotlight excerpt picks the load-bearing slice via **density
rank** — find ALL match positions for ALL claim content tokens,
pick the position with maximum distinct-token cluster within
±half-window. Replaces the older first-match-of-longest-token
approach which lost the load-bearing slice on noisy chunks.
[`aborist/qa/evidence.py`](../aborist/qa/evidence.py)
#### `qa.parse_claims` — pointer-line parser
Walks lines of the model output, pulls every `[E\d+]` and
`[E\d+,E\d+,…]` bracket payload, returns
`(claim_text, pointer_ids[])` per line. Lines without a tag get
`parse_status='NO_EVIDENCE_POINTER'` & count toward the denominator
so unsourced prose can't smuggle past the verifier.
[`aborist/qa/parse_claims.py`](../aborist/qa/parse_claims.py)
#### `qa.dag` — per-run Merkle DAG
Commits each provenance step independently as a stage hash. Two
shapes:
- **7-stage (quote mode):** question / retrieval / context /
prompt / answer / verify / final_label
- **9-stage (claim-lattice / CTI):** question / retrieval /
evidence_map / prompt / raw_answer / parsed_claim_lattice /
verify / render / final_label
The `run_dag_root` is persisted alongside every providence record;
`run_dag_blob` carries the full `{root, nodes}` JSON so an auditor
can recompute & verify any step.
[`aborist/qa/dag.py`](../aborist/qa/dag.py)
#### `qa.inspect` — read-only sidecar
Pulls source chunks for a given cache_key & classifies each
unverified span: `verbatim_in_base` / `verbatim_in_raw_only` /
`trailing_artifact` / `paraphrase` / `partial_paraphrase` /
`no_overlap`. Also includes the deflection-detection sidecar
(subject-anchor heuristic for adversarial-premise topic shift).
**Sidecars never write to `providence_cache` or `audit_events`** —
they're diagnostic only. That invariant is what keeps `audit_mode`
a binary classification rather than a soft score.
[`aborist/qa/inspect.py`](../aborist/qa/inspect.py)
#### `qa.concepts` — backwards-compat shim
Delegates to `aborist.concepts` (the corpus-derived synonym/rivalry
layer). Pre-2026-05-01 the data lived as hand-curated frozensets in
this file; now it's a per-shard SQLite table. The shim preserves
the legacy public API (`synonym_expand`, `rivalry_excluded`,
`has_compare_phrasing`) so call sites in `qa/query.py` didn't have
to change.
[`aborist/qa/concepts.py`](../aborist/qa/concepts.py) →
[`aborist/concepts/`](../aborist/concepts/)
### Source papers
- Whitepaper §13.8 covers the layered verifier in depth
- Whitepaper §13.9 covers claim-lattice / CTI mode
- `docs/cti-architecture.md` is the architecture reference
- `docs/seven-point-program.md` enumerates the seven hard checks
- `docs/concept-relations-design.md` covers the synonym layer
## `aborist.distill`
Surface → core distillation. Takes a set of surface documents
(the original ingest layer) and produces "core" documents — shorter,
more focused, Merkle-bound back to their contributing surface
chunks via per-chunk inclusion proofs.
The "trees and forests of cross-linked information" tagline aborist
takes its name from comes from this layer: planet-toward-center
compression where each layer of cores derives from the previous,
recursively.
### Layered design
```
distill/
├── base.py Distiller ABC + DistillationResult dataclass
├── first_sentence.py no-ML stub: take the first sentence of each doc
├── tfidf.py pure-Python TF-IDF top-keyword extraction
└── runner.py batched distillation + per-contrib-chunk proofs
```
### `Distiller` ABC
```python
from aborist.distill import Distiller, DistillationResult
class Distiller(ABC):
@abstractmethod
def distill(self, docs: list[Document]) -> DistillationResult: ...
```
Each `DistillationResult` carries the new core's content +
references to every contributing surface chunk by `(document_root,
chunk_root)`. The runner writes one `derivations` row per core,
with `proof_blob = json.dumps(per_chunk_inclusion_proofs)`.
### Built-in distillers
#### `FirstSentenceDistiller` (no-ML stub)
Take the first sentence of each input doc, concatenate. Used as a
sanity-check for the pipeline + a baseline for measuring the
benefit of richer distillers.
#### `TfidfKeywordDistiller`
Pure-Python TF-IDF. Computes term frequencies across the input
doc set + inverse document frequencies; emits the top-K terms per
doc as the core's content. The "permacomputer" neologism case
(every Grok conversation has the word, no Wikipedia article does)
is the canonical TF-IDF win — surfaces the topic that title-search
can't catch.
### Why distill
Three use cases:
1. **Retrieval signal.** Cores feed the third accept path in
`_filter_by_title_relevance` — `core_match_roots` (TF-IDF top-
keywords contain a query token). Closes the gap for neologisms
that never make Wikipedia titles but ARE distinctive.
2. **Hot/cold tier discipline.** `evict_to_cold` only touches
`kind='surface'` — cores never evict. Distilling surface to
cores then evicting surfaces gives a "long tail keeps small
cache" pattern with full provenance preserved.
3. **Recursive abstraction.** Cores can themselves be distilled
into shorter cores. Each generation Merkle-binds back to the
previous via `derivations.proof_blob` — the audit chain stays
intact across an arbitrary distillation depth.
### Source
[`aborist/distill/`](../aborist/distill/)