arborist/docs/modules.md

Aborist module reference

Single-file reference for every top-level package + diagrams.

Diagrams index

Diagrams

Diagram	What it shows	File
Module graph	Top-level packages & how they import each other	aborist-modules.svg (dot)
Query pipeline	Question → cache → retrieval → LLM → verify → render	query-pipeline.svg (dot)
Ingest pipeline	Source document → Merkle-committed shard	ingest-pipeline.svg (dot)
Verifier ladder	(audit_mode, violations) → display rung	verifier-ladder.svg (dot)
Mesh data flow	Federation: roster, gossip, AEAD envelope	mesh-data-flow.svg
Mesh epoch lifecycle	Epoch advance via add/kick/rotate	mesh-epoch-lifecycle.svg
Mesh identity stack	Ed25519 sign + X25519 DH key derivation	mesh-identity-stack.svg
Mesh secret envelope	AEAD-wrapped epoch secret per peer	mesh-secret-envelope.svg
Mesh group decisions	Membership change voting & quorum	mesh-group-decisions.svg

Render diagrams locally:

make docs   # runs `dot -Tsvg` and `-Tpng` on every docs/diagrams/*.dot

Substrate (no SQL, pure data structures)

Module	One-line role	Doc
merkle.py	Merkle tree + proof — Python port of proxy.unturf.com/pkg/verified/merkle.go	↓
document.py	Document, Edge, Chunker (default tok-512-v1)	↓
wikitext.py	to_base() — wikitext → plain prose, BASE_VERSION-pinned	↓

Storage

Module	One-line role	Doc
store.py	v9.8 SQLite schema + audit chain helpers	↓
ingest.py	normalize → chunk → merkle → upsert (bulk-batched)	↓
evict.py	hot ↔ cold tier transitions; rehydrate via source	↓

Sources (corpus producers)

Module	One-line role	Doc
sources/wikipedia.py	Wikipedia 2003 cur + old SQL dumps (bz2-streamed)	↓
sources/wikipedia_xml.py	Phase IV XML dumps (iterparse, page + history)	↓
sources/html_page.py	URL list + selectolax + httpx (robots-aware)	↓
sources/crawler/	verbatim AsyncWebFetcher lift + ingest bridge	↓
sources/grok.py	xAI data export (conversations + media prompts)	↓
sources/vcs.py	git + Mercurial repos (HEAD walk, supersedes chain)	↓

Search & retrieval

Module	One-line role	Doc
search/	FTS5 backend + SearchBackend ABC + AuditMode enum	↓
concepts/	Per-shard concept_relations synonym/rivalry overlay	↓

Q&A pipeline

Module	One-line role	Doc
qa/keys.py	8-dim cache_key + question_hash	↓
qa/client.py	ChatClient + StubClient + OpenAICompatibleClient	↓
qa/runner.py	ask(): single-doc Q&A + cache + verify	↓
qa/query.py	query(): multi-source RAG + concept overlay	↓
qa/verify.py	quote/span/entity/paraphrase + claim_lattice (7 hard checks)	↓
qa/warrant.py	5 anchor classes (proper-noun · date · count · entity-list · cause)	↓
qa/evidence.py	EvidenceObject + spotlight excerpt (density rank)	↓
qa/parse_claims.py	pointer-line parser (claim. [E1,E2])	↓
qa/dag.py	per-run Merkle-DAG (7-stage quote / 9-stage CTI)	↓
qa/inspect.py	sidecar diagnostic (read-only span classifier)	↓

Distillation

Module	One-line role	Doc
distill/	Distiller ABC + first_sentence + tfidf + runner	↓

Federation (off by default)

Module	One-line role	Doc
mesh/	identity (Ed25519/X25519), per-epoch roster, AEAD envelope, gossip wire	../mesh.md, ../mesh-deploy.md

Entry point

Module	One-line role	Doc
cli.py	argparse entrypoint — every make target dispatches here	run aborist --help or any make help target

Tickets, design docs, journals

See ../TICKETS.md for the ticket index and the list of design-reference docs that aren't tickets.

aborist.merkle

Pure Merkle tree + proof primitives. Python port of proxy.unturf.com/pkg/verified/merkle.go — convention-identical. Used everywhere a content-addressable handle is needed: per-chunk leaves, document_root, evidence_map_root, run_dag_root, snapshots.

Conventions (do not silently change)

These match the Go reference & are load-bearing for cross-language verification (Go peer ↔ Python peer compute bit-identical roots):

• Leaf hash: sha256(0x00 || canonical_chunk_bytes). The 0x00 prefix domain-separates leaves from internal nodes.
• Internal hash: sha256(0x03 || left || right). The 0x03 prefix is the non-commutative combine — H(L,R) ≠ H(R,L). Order matters.
• Odd-element rule: when a level has an odd count, the last leaf is self-duplicated before pairing. NOT zero-padded.
• Proof path: each step carries an explicit is_left: bool alongside the sibling hash so a verifier knows which side to put the sibling on. Never sort siblings lexically — the order tells the verifier the tree topology.

API surface

from aborist.merkle import MerkleTree, MerkleProof

tree = MerkleTree.build([b"chunk_0_bytes", b"chunk_1_bytes", ...])
tree.root            # bytes(32) — sha256 of the whole tree
tree.leaves          # list[bytes(32)] — leaf hashes in input order

proof = tree.proof_for(leaf_index=2)
proof.siblings       # list[(sibling_hash, is_left)]
proof.verify(leaf_hash=tree.leaves[2], root=tree.root)  # bool

When to read the source

• Adding a new content-addressable artifact (cores, evidence maps, snapshots, run-DAGs all touch this).
• Cross-language verification debugging (Go peer says one root, Python peer says another — the difference is always in canonical encoding, ordering, or one of the three prefix bytes above).
• Performance work — the Python build is ~3× slower than the Go reference; if it ever shows up in profiling, that's the file.

Diagrams

The module graph shows what depends on merkle.py (a lot — it's substrate):

The ingest pipeline shows where leaf & root hashes get computed:

Source

aborist/merkle.py · Reference: proxy.unturf.com/pkg/verified/merkle.go

aborist.document

The data structures every source produces and every storage layer consumes. Three core types: Document, Edge, Chunker.

Document

A single ingest unit: a Wikipedia article, an HTML page, a Grok conversation, a git commit message, etc. Carries both the raw content AND the version tags that determine its identity:

@dataclass(frozen=True)
class Document:
    document_uri: str           # canonical URI (or stable surrogate for non-URI sources)
    raw_content: str            # source-of-truth bytes pre-canonicalization
    kind: str                   # 'surface' / 'core' / 'visual' / etc.
    chunking_version: str       # e.g. 'tok-512-v1' — pinned by the Chunker
    canonicalization_version: str  # e.g. 'norm-v1' — pinned by canonicalize()
    schema_version: str         # e.g. 'v9.8.0' — store schema generation
    title: str | None = None
    edges: list[Edge] = ()      # outbound link graph
    metadata: dict = ...        # source-specific opaque payload

document_root is computed at ingest time as the Merkle root over the canonicalized chunks. Two peers ingesting the same source + running the same chunking_version + canonicalization_version get bit-identical document_roots — the v9.8 admissibility property.

Edge

One outbound link. aborist/sources/wikipedia.py emits one Edge per [[wikilink]]; aborist/sources/html_page.py emits one per <a href>. The link graph IS the corpus topology — concepts/extract.py later reads edges rows to derive synonym relations from reciprocal links (no separate crawler needed).

@dataclass(frozen=True)
class Edge:
    src_root: str   # source document_root
    dst_uri: str    # always present
    dst_root: str   # '' (unresolved) until the dst doc is also ingested
    edge_type: str  # 'wikilink' / 'href' / 'citation' / 'derived_from' / ...
    anchor: str     # chunk index or fragment, '' if N/A

Chunker

ABC with one method chunk(text: str) -> list[str]. Default impl is TokenChunker (name='tok-512-v1') — splits on token-rough windows so the resulting chunks are predictable for downstream FTS5 indexing & for the LLM context budget.

Changing the chunker bumps chunking_version AND stales every prior cache record (chunking is one of the 8 cache_key dimensions). Don't redefine tok-512-v1; add a new chunker as a new name instead.

Diagrams

Source

aborist/document.py

aborist.wikitext

A single function: to_base(raw). Converts MediaWiki wikitext to plain prose deterministically.

from aborist.wikitext import to_base, BASE_VERSION

prose = to_base("[[The Beatles]] are an [[English rock band]] from [[Liverpool]].")
## → "The Beatles are an English rock band from Liverpool."

Why it exists

The corpus stores raw wikitext (so the link graph is recoverable on demand) but the LLM and verifier both want plain prose. Reasons:

1. Token efficiency. Wikipedia chunks ship to Hermes with ~43% fewer tokens after wikitext-strip — bigger context window for the same chars budget.
2. Verbatim citation. The model can quote source paragraphs verbatim instead of escaping [[wikilinks]]. The verifier's substring test then matches cleanly.
3. Pinned identity. BASE_VERSION='wikitext-base-v1' lives in policy["base_version"], which folds into governance_policy_hash. Bumping BASE_VERSION invalidates every prior cache record on next lookup — same discipline as chunking_version and canonicalization_version.

Hot-path discipline

to_base() runs on the assembled context before the LLM call in aborist/qa/runner.py and aborist/qa/query.py, and again inside verify_quotes so the verifier compares like-against-like. Both sides see prose.

Optional dependency

Backed by mwparserfromhell. Install via pip install '.[wikitext]' to enable. Without the dep, _wikitext_to_base = None and policy["base_version"] = None — graceful fallback leaves raw wikitext in both context and verifier (works, just less efficient).

Source

aborist/wikitext.py

aborist.store

The v9.8 SQLite schema, the audit chain, and the cross-shard read-only view. Every table that holds runtime state lives here.

Schema overview (per shard)

documents              – one row per source document, keyed on document_root
chunks                 – per-document chunk content + tier (hot/cold)
chunks_fts             – FTS5 contentless index, rowid = chunks.chunk_id
merkle_nodes           – internal-node hashes for proof reconstruction
edges                  – src_root → dst_root link graph (wikilink, href, …)
derivations            – core_root ← src_root with proof_blob (Merkle)
providence_cache       – Q&A records keyed on the v9.8 8-dim cache_key
audit_events           – linear chain; event_hash = sha256(prev || canonical(body))
falsifications         – record_id → state transition + reason + actor
snapshots              – named corpus roots (one hash names a forest)
document_http_meta     – ETag + Last-Modified for crawler conditional fetches
concept_relations      – per-shard synonym/rivalry/category/antonym overlay
mesh_*                 – federation tables (off by default)

v9.8 invariants (do not break)

• Aborist is a v9.8 store. Every providence record carries the full 8-dim cache_key: source_root | question_hash | model_profile_hash | conversation_hash | governance_policy_hash | schema_version | canonicalization_version | chunking_version. Bumping any one invalidates prior records on lookup.
• falsification_state ∈ {live, failed, stale, quarantined}. Cache lookups must filter on state='live'. Drift detection flips to stale.
• Audit chain. Every state-changing op writes one row in audit_events with event_hash = sha256(prev_event_hash || canonical(body)). Chain integrity is verified in make analyze-shards. Never insert into audit_events directly — use aborist.store.append_audit.
• Cores never evict. evict_to_cold only touches kind='surface'.
• Idempotent re-ingest. Same content → same document_root → no-op insert. Same URI + different content → new doc + supersedes edge linking new → old (lossless history).

API surface

from aborist.store import (
    connect,           # writable connection to a single shard
    connect_query,     # read-only UNION view across all shards
    discover_shards,   # list *.db files in a shards_dir
    transaction,       # BEGIN IMMEDIATE / COMMIT / ROLLBACK context manager
    get_meta, set_meta,
    append_audit,      # the ONLY way to write audit_events
)

Cross-shard reads use connect_query(shards_dir=...) which ATTACHes every *.db and creates UNION views over the shardable tables (documents, chunks, merkle_nodes, edges, derivations, providence_cache, audit_events, falsifications, concept_relations).

Performance pragmas

connect() applies these per-connection:

• journal_mode=WAL (set in SCHEMA_SQL once at first ingest)
• synchronous=NORMAL (skip per-commit fsync; safe under WAL)
• cache_size=-65536 (64 MB page cache)
• temp_store=MEMORY (no /tmp churn for temp tables)

Don't downgrade to synchronous=FULL without a measured reason — costs ~5× throughput.

Diagrams

Source

aborist/store.py

aborist.ingest

The bulk-batched pipeline that turns documents from a source into Merkle-committed shard storage. Source-agnostic: anything that implements Source.iter_documents() flows through here.

Public API

from aborist.ingest import ingest_source
from aborist.sources.wikipedia import WikipediaSqlDump

source = WikipediaSqlDump("/path/to/cur.sql.bz2")
ingest_source(
    source,
    db_path=Path("~/.aborist/shards/000.db"),
    batch_size=200,        # docs per transaction
    progress_every=1000,
)

What happens per document

1. Canonicalize — canonicalize(text): NFC + ws-collapse + strip ends. Pinned by canonicalization_version='norm-v1'.
2. Chunk — Chunker.chunk(canonical_text) → list of token- bounded substrings. Default tok-512-v1 chunker.
3. Hash leaves — sha256(0x00 || canonical_chunk_bytes) per chunk.
4. Merkle tree — MerkleTree.build(leaves).root → document_root. Two peers running the same chunker on the same canonicalized content compute bit-identical roots.
5. Upsert — documents row keyed on document_root (idempotent re-ingest), chunks rows with leaf hashes, merkle_nodes for proof reconstruction, edges per outbound link.
6. FTS5 — chunks_fts insert with rowid = chunks.chunk_id so the search-time JOIN lines up.
7. Audit event — one row per ingest batch in audit_events, chained on prev_event_hash.

Batching discipline

Default batch_size=200: balances Python GIL overhead vs SQLite transaction commit cost. Lower it (e.g. 50) only to bound peak memory on a low-RAM host. Higher (e.g. 1000) for ETL throughput on SSD storage when memory isn't tight.

progress_every prints a stderr line every N docs so long ingests are observable. Use PYTHONUNBUFFERED=1 for tail-able output.

Resumability

Idempotent re-ingest: same content + same chunker + same canonicalize = same document_root = no-op insert. So a crashed ingest can be restarted from the source's beginning without duplicating rows.

Different content at the same URI gets a new document_root AND a supersedes edge linking new → old (lossless history).

Source

aborist/ingest.py

aborist.evict

Hot ↔ cold tier transitions. The corpus is large (3.47M Wikipedia docs); not every chunk fits in working memory. evict.py is the mechanism that moves rarely-touched chunks to a cold tier (still indexed, just stored separately) and rehydrates them on demand from the original source.

API surface

from aborist.evict import evict_to_cold, rehydrate

## Move chunks unused for >threshold days to cold tier
evict_to_cold(conn, max_age_days=90, max_evictions=10000)

## Pull a cold chunk back to hot from its original source
rehydrate(conn, document_root="abc123...")

Invariant: cores never evict

evict_to_cold filters WHERE kind='surface'. Cores are always hot — they're the long-tail-friendly compression layer that justifies evicting their underlying surfaces. Evicting cores would defeat the purpose.

v9.8 falsification on drift

When rehydrate() re-fetches a document and the recomputed document_root differs from the stored one, the source has changed since ingest (Wikipedia article was edited, HTML page was republished, etc.). The cache record's falsification_state flips from live to stale — every providence record keyed on that source_root is no longer admissible to lookups.

This is the drift-detection-as-falsification discipline: cache hits don't blindly trust historical answers; they trust answers that the SAME source still grounds.

Tier values

chunks.tier ∈ {'hot', 'cold'}. Hot chunks live in chunks.content; cold chunks live with NULL content and a cold_uri pointing at the source. The QA pipeline's chunk-fetch path checks tier; on 'cold', it triggers rehydrate before continuing.

Source

aborist/evict.py

aborist.sources

Corpus producers. Each is a Source ABC implementation that yields Document instances; the standard ingest.ingest_source(source, db) pipeline takes them from there.

The Source ABC lives in aborist/source.py:

class Source(ABC):
    @abstractmethod
    def iter_documents(self) -> Iterator[Document]: ...

Built-in sources

wikipedia.py — Phase III SQL dumps (the canonical bootstrap)

Streams the Wikipedia 2003 cur (current revisions) and old (revision history) SQL dumps. Hand-rolled escape-aware parser (no sqlite3 import — the dump is MySQL syntax). 4× speedup vs char-by-char loops via str.find + slicing. cProfile any change.

Default Wikipedia 2003-05-16 dump source: https://dumps.wikimedia.org/archive/2003/2003-05-16/en/. robots.txt returned 404 → no rules.

aborist/sources/wikipedia.py

wikipedia_xml.py — Phase IV XML dumps

Modern Wikipedia dump format (enwiki-YYYYMMDD-pages-articles.xml.bz2, enwiki-YYYYMMDD-pages-meta-history*.xml.bz2). Uses xml.etree.ElementTree.iterparse to stream-parse without loading the whole tree.

aborist/sources/wikipedia_xml.py

html_page.py — single-URL or URL-list HTML ingest

Robots-aware (urllib.robotparser). Uses selectolax for fast HTML parsing (CSS-selector based; ~10× faster than lxml). Pulls the main body text + every <a href> as an Edge row.

The edges rows are what the corpus-derived synonym extractor later reads — no separate crawler needed for site-internal link graphs.

Optional dep: pip install '.[html]' for selectolax + httpx.

aborist/sources/html_page.py

crawler/ — async BFS web crawl

Verbatim lift of an AsyncWebFetcher implementation + an ingest bridge. BFS-discovers same-domain URLs from a seed, respecting robots.txt + crawl delays. Captures ETag + Last-Modified per URL into document_http_meta so a future recrawl can send conditional HEAD requests.

aborist/sources/crawler/bridge.py

grok.py — xAI Grok export

Reads the xAI-conversations.json data export shape. Each conversation becomes one Document; media prompts are kept inline.

aborist/sources/grok.py

vcs.py — git + Mercurial repositories

HEAD walk. Each commit becomes a Document (commit message + diff stat). The supersedes chain captures commit ancestry as edges.

aborist/sources/vcs.py

Source

aborist/sources/ · aborist/source.py (ABC)

aborist.search

The retrieval primitive. Today's only backend is FTS5 over the chunks table; the SearchBackend ABC is in place so additional backends (BM25 over titles, embedding-based vector search) can be added without touching the rest of the QA pipeline.

SearchBackend ABC

from aborist.search import SearchBackend, AuditMode, Hit

class SearchBackend(ABC):
    @abstractmethod
    def search(self, query: str, limit: int = 20) -> list[Hit]: ...

Each Hit carries (document_root, document_uri, chunk_idx, snippet, score, audit_mode, title). audit_mode is the sticky provenance label that tracks how the chunk made it into the index; FTS5 backend always sets UNGROUNDED (search itself doesn't verify anything — that's the QA pipeline's job).

FTS5Backend

Wraps the contentless chunks_fts virtual table. Two-mode query:

• AND-mode (strict, primary): every content token must appear in the doc. Keeps unrelated docs out of the context window.
• OR-mode (fallback): when AND returns 0 hits, fall back to OR but capped to top-5 longest tokens (proxy for rarity). Long topical synonyms fed via extra_or_tokens join the pool — neurotechnology (15 chars) outranks thoughts (8) by length and surfaces brain-tech titles for vocabulary-mismatch queries.

from aborist.search import FTS5Backend
backend = FTS5Backend(conn)

## Plain search
hits = backend.search("permacomputer", limit=32)

## Search with synonym pool injection (used by qa.query._search_corpus)
hits = backend.search(
    long_query,
    limit=32,
    extra_or_tokens=synonym_expand(qtokens, shards_dir=shards_dir),
)

Stopword & stopword-cap discipline

_FTS5_STOPWORDS filters question words (what, tell, please)

• generic connectors (one, some, another, without, soon, currently) before AND/OR construction. Two principles:

• Stay in sync with _TITLE_STOPWORDS in qa/query.py. A token filtered at retrieval time but kept at title-relevance check (or vice versa) creates ranking incoherence.
• _OR_FALLBACK_MAX_TOKENS=5 caps the OR-mode pool. Without this, a 19-token OR clause matches millions of docs and forces BM25 to rank them all — 13s/shard observed pre-cap. Now 0.25s/shard.

Snippet building

FTS5 contentless mode means SQLite's built-in snippet() and highlight() return empty. Aborist builds snippets in Python by joining chunks_fts.rowid = chunks.chunk_id, decompressing the chunk content, and locating query tokens locally (_build_snippet).

Source

aborist/search/fts5.py · aborist/search/__init__.py

aborist.concepts

Per-shard concept_relations SQLite table — the corpus-derived synonym, rivalry, antonym & category overlay that replaces the hand-curated frozensets that lived in aborist/qa/concepts.py through April 2026.

Full architecture rationale lives in ../concept-relations-design.md including the 1.6% storage-tax measurement and the three-compactions- considered-and-rejected analysis. This page is the API reference.

Sub-modules

concepts.store — append-only CRUD

from aborist.concepts.store import (
    add_concept_relation,           # idempotent INSERT OR IGNORE
    concept_relations_for_token,    # read all relations for a token
    purge_by_evidence_kind,         # the only DELETE path
    list_evidence_kinds,            # diagnostic
    RELATION_KINDS,                 # ('synonym','antonym','rivalry','category')
)

UNIQUE on (source_root, relation_kind, token, target, evidence_kind) makes re-derivation idempotent. purge_by_evidence_kind lets an operator revoke a single extractor's output without touching manual or other-extractor rows.

aborist/concepts/store.py

concepts.query — cross-shard lookup

from aborist.concepts import (
    synonym_expand,        # query tokens → expanded set
    rivalry_excluded,      # query tokens → tokens to drop from results
    has_compare_phrasing,  # bool — does the query say "vs", "compare", etc.
    invalidate_cache,      # drop the per-process LRU
)

expanded = synonym_expand({"thoughts"}, shards_dir=shards_dir)
## → {"thoughts", "telepathy", "neurotechnology", "mind", "cognition", ...}

Two synonym indices are loaded:

• manual_index — manual_legacy + manual rows. Curated; always expanded regardless of per-token degree. Captures the brain-tech / AMD-family / Mac / Linux / etc. seed groups.
• derived_index — link_reciprocity & other corpus-derived edges. Subject to MAX_NEIGHBORS_PER_TOKEN=8 cap because the Wikipedia link graph carries topic-adjacency noise on generic tokens (person, thoughts, language).

Overall MAX_TOTAL_TOKENS=50 cap on expanded set bounds the SQL clause count downstream so retrieval stays sub-second.

A per-process LRU keyed on shard mtime avoids re-loading the index on every query (290k rows across 4 shards loads in ~1.8s cold).

aborist/concepts/query.py

concepts.extract — pluggable extractor framework

from aborist.concepts.extract import (
    EXTRACTORS,                  # registry: evidence_kind → callable
    link_reciprocity_synonym,    # built-in extractor
)

## Run an extractor against a shard:
result = link_reciprocity_synonym(conn, derived_from="backfill@2026-05-01")
## → {"reciprocal_pairs": N, "synonyms_inserted": M, "synonyms_skipped": K}

Each extractor walks the shard's existing rows (documents, chunks, edges, derivations) — no new crawler needed — & emits concept relations under a stable evidence_kind string that supports targeted purge.

The built-in link_reciprocity_synonym reads the existing edges table for reciprocal A↔B link pairs and emits a synonym edge between every (title-token-of-A, title-token-of-B) pair. Works for Wikipedia (See-also bidirectional), HTML site internal links (russell.ballestrini.net pattern), or any document graph with bidirectional links. Title-tokens are filtered to ≥4 chars + stopword-stripped.

To add a new extractor:

1. Implement (conn, *, derived_from) -> dict[str, int] that calls add_concept_relation for each finding.
2. Pick a stable evidence_kind string.
3. Register in EXTRACTORS.

aborist/concepts/extract.py

concepts.seed — legacy frozenset migration

One-shot migration of the 8 hand-curated frozenset groups (AMD-family, Intel-family, HTTP, FTP, Mac, Windows, Linux, brain-tech) to evidence_kind='manual_legacy' rows.

Writes clique edges within each group — every (a, b) pair — so any member retrieves every other member (preserves the legacy frozenset semantic where lookups didn't depend on which token in the group was the anchor).

from aborist.concepts.seed import seed_legacy_concepts
result = seed_legacy_concepts(conn)
## → {"synonyms_inserted": N, "rivalries_inserted": M, "skipped": K}

Idempotent — re-running adds nothing if all the rows already exist.

aborist/concepts/seed.py

Data model

CREATE TABLE concept_relations (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    source_root     TEXT NOT NULL,
    relation_kind   TEXT NOT NULL CHECK (relation_kind IN
                        ('synonym','antonym','rivalry','category')),
    token           TEXT NOT NULL,
    target          TEXT NOT NULL,
    evidence_kind   TEXT NOT NULL,
    confidence      REAL NOT NULL DEFAULT 1.0,
    derived_at      INTEGER NOT NULL,
    derived_from    TEXT,
    UNIQUE (source_root, relation_kind, token, target, evidence_kind)
);

Append-only by construction. Re-running an extractor adds nothing if every relation already exists. No UPDATE path; only add_concept_relation (insert) and purge_by_evidence_kind (targeted delete).

Per-shard storage. Concept relations live in the shard whose document derived them. Mesh sync moves shards between peers; relations come along.

Orthogonal to Merkle. Writes to concept_relations NEVER affect document_root, chunk_root, or cache_key. Backfilling relations is safe across the entire corpus without invalidating any cached answer or breaking any audit chain.

Storage cost — measured

Backfill on 4 wiki shards (3.47M docs, 10.75M resolved edges):

Shard	Reciprocal pairs	Synonyms	Storage
000.db	13,562	71,288	23.52 MB
001.db	14,078	73,351	24.20 MB
002.db	13,708	72,576	23.92 MB
003.db	13,800	72,633	23.94 MB
total	55,148	289,848	95.58 MB

1.6% storage tax on the 6 GB corpus. Backfill takes ~4 min wall-clock total. Cost is paid once at backfill; every retrieval- time lookup benefits.

Diagrams

Source

• aborist/concepts/ — package
• docs/concept-relations-design.md — full design doc
• Whitepaper §13.4.11 — public-facing summary

aborist.qa

The Q&A pipeline. Question → cache → retrieval → LLM → verify → render → cache write. Lives in 9 sub-modules; this page is the map.

Sub-modules

qa.client — LLM transport

ChatClient ABC with three concrete implementations:

• StubClient — deterministic test fixture; returns canned responses keyed on the input. Used in unit tests to avoid network.
• OpenAICompatibleClient — talks to any OpenAI-shape /v1/chat/completions endpoint (Hermes-3 on vLLM by default). Includes HTTP retry layer (3× exponential backoff on 5xx).
• (Future) AnthropicClient — Claude API direct.

aborist/qa/client.py

qa.keys — the 8-dim cache_key

cache_key = sha256(
    source_root | question_hash | model_profile_hash |
    conversation_hash | governance_policy_hash |
    schema_version | canonicalization_version | chunking_version
)

Two question-hash modes (strict vs equivalence_class) live here. Bumping any of these eight dimensions invalidates prior records on lookup. The verifier_policy_hash (v9.9 9th dim) is also implemented here.

aborist/qa/keys.py

qa.runner — ask() for single-doc Q&A

The simplest entry point. Take one document, ask one question, get back an answer + audit_mode + cache record. Used by the CLI for focused queries against one URI.

aborist/qa/runner.py

qa.query — query() for multi-source RAG

The main retrieval entry point. Walks shards, runs FTS5 BM25 with AND→OR fallback (with synonym-pool injection in OR mode), filters by title relevance with 4 accept paths, reranks by body coverage + title boost + source role + title purity, assembles a 60 KB context budget, calls the LLM, runs the verifier, persists to providence_cache.

aborist/qa/query.py

qa.verify — the layered verifier

Five strategies run in sequence; first to find evidence classifies:

1. quote — "..."-wrapped claims tested verbatim
2. span — bullet/sentence units substring-tested
3. entity — multi-word proper nouns with proximity gating
4. paraphrase — token coverage on prose-shaped spans (≥85%)
5. claim_lattice — pointer-line [E1,E2] or JSON; runs seven deterministic hard checks:
   1. parser succeeded
   2. evidence_id resolves
   3. source_role allowed
   4. claim text non-empty
   5. citation coverage threshold
   6. pointer count cap (trim-and-verify)
   7. anchor-class warrant (see qa.warrant)

The classifier output rolls up into the v9.8 trichotomy audit_mode ∈ {STRICT, HYBRID, UNGROUNDED}. Display layer (in cli.py) maps (audit_mode, violations) → four-rung ladder.

aborist/qa/verify.py

qa.warrant — anchor-class warrant

Five lexical anchor classes the verifier composes:

• Proper-noun — relation-question shape; at least one Title-Case anchor must appear in some cited span
• Date — claim has a 4-digit year + month name; ALL components required in some cited span
• Entity-list — entity-list-shape question; ≥1 named entity must anchor (demote-don't-reject)
• Count — count-shape question; count token must appear in word OR digit form (digit↔word equivalence)
• Cause — why-shape question; ≥1 cause anchor (proper noun OR ≥5-char common noun outside stopword pool)

The warrant layer earns proof-path entry by staying lexical — no NLI, no embeddings. Substring tests over already-canonicalized spans. See docs/concept-relations-design.md (sibling section) for the relationship to retrieval-time synonym expansion.

aborist/qa/warrant.py

qa.evidence — EvidenceObject + spotlight

Builds the runtime evidence map for claim-lattice modes. Each chunk becomes one EvidenceObject carrying TWO ids:

• pointer_id — short prompt-facing tag (E1, E2, …)
• evidence_id — content-addressed E######## (sha256-derived)

The model sees only pointer_ids in the prompt; the runtime maps to evidence_id for the cache & run-DAG (run-stable identity).

The spotlight excerpt picks the load-bearing slice via density rank — find ALL match positions for ALL claim content tokens, pick the position with maximum distinct-token cluster within ±half-window. Replaces the older first-match-of-longest-token approach which lost the load-bearing slice on noisy chunks.

aborist/qa/evidence.py

qa.parse_claims — pointer-line parser

Walks lines of the model output, pulls every [E\d+] and [E\d+,E\d+,…] bracket payload, returns (claim_text, pointer_ids[]) per line. Lines without a tag get parse_status='NO_EVIDENCE_POINTER' & count toward the denominator so unsourced prose can't smuggle past the verifier.

aborist/qa/parse_claims.py

qa.dag — per-run Merkle DAG

Commits each provenance step independently as a stage hash. Two shapes:

• 7-stage (quote mode): question / retrieval / context / prompt / answer / verify / final_label
• 9-stage (claim-lattice / CTI): question / retrieval / evidence_map / prompt / raw_answer / parsed_claim_lattice / verify / render / final_label

The run_dag_root is persisted alongside every providence record; run_dag_blob carries the full {root, nodes} JSON so an auditor can recompute & verify any step.

aborist/qa/dag.py

qa.inspect — read-only sidecar

Pulls source chunks for a given cache_key & classifies each unverified span: verbatim_in_base / verbatim_in_raw_only / trailing_artifact / paraphrase / partial_paraphrase / no_overlap. Also includes the deflection-detection sidecar (subject-anchor heuristic for adversarial-premise topic shift).

Sidecars never write to providence_cache or audit_events — they're diagnostic only. That invariant is what keeps audit_mode a binary classification rather than a soft score.

aborist/qa/inspect.py

qa.concepts — backwards-compat shim

Delegates to aborist.concepts (the corpus-derived synonym/rivalry layer). Pre-2026-05-01 the data lived as hand-curated frozensets in this file; now it's a per-shard SQLite table. The shim preserves the legacy public API (synonym_expand, rivalry_excluded, has_compare_phrasing) so call sites in qa/query.py didn't have to change.

aborist/qa/concepts.py → aborist/concepts/

Source papers

• Whitepaper §13.8 covers the layered verifier in depth
• Whitepaper §13.9 covers claim-lattice / CTI mode
• docs/cti-architecture.md is the architecture reference
• docs/seven-point-program.md enumerates the seven hard checks
• docs/concept-relations-design.md covers the synonym layer

aborist.distill

Surface → core distillation. Takes a set of surface documents (the original ingest layer) and produces "core" documents — shorter, more focused, Merkle-bound back to their contributing surface chunks via per-chunk inclusion proofs.

The "trees and forests of cross-linked information" tagline aborist takes its name from comes from this layer: planet-toward-center compression where each layer of cores derives from the previous, recursively.

Layered design

distill/
├── base.py            Distiller ABC + DistillationResult dataclass
├── first_sentence.py  no-ML stub: take the first sentence of each doc
├── tfidf.py           pure-Python TF-IDF top-keyword extraction
└── runner.py          batched distillation + per-contrib-chunk proofs

Distiller ABC

from aborist.distill import Distiller, DistillationResult

class Distiller(ABC):
    @abstractmethod
    def distill(self, docs: list[Document]) -> DistillationResult: ...

Each DistillationResult carries the new core's content + references to every contributing surface chunk by (document_root, chunk_root). The runner writes one derivations row per core, with proof_blob = json.dumps(per_chunk_inclusion_proofs).

Built-in distillers

FirstSentenceDistiller (no-ML stub)

Take the first sentence of each input doc, concatenate. Used as a sanity-check for the pipeline + a baseline for measuring the benefit of richer distillers.

TfidfKeywordDistiller

Pure-Python TF-IDF. Computes term frequencies across the input doc set + inverse document frequencies; emits the top-K terms per doc as the core's content. The "permacomputer" neologism case (every Grok conversation has the word, no Wikipedia article does) is the canonical TF-IDF win — surfaces the topic that title-search can't catch.

Why distill

Three use cases:

1. Retrieval signal. Cores feed the third accept path in _filter_by_title_relevance — core_match_roots (TF-IDF top- keywords contain a query token). Closes the gap for neologisms that never make Wikipedia titles but ARE distinctive.

2. Hot/cold tier discipline. evict_to_cold only touches kind='surface' — cores never evict. Distilling surface to cores then evicting surfaces gives a "long tail keeps small cache" pattern with full provenance preserved.

3. Recursive abstraction. Cores can themselves be distilled into shorter cores. Each generation Merkle-binds back to the previous via derivations.proof_blob — the audit chain stays intact across an arbitrary distillation depth.

Source

aborist/distill/