diff --git a/README.md b/README.md
index 944aeaf..19765c7 100644
--- a/README.md
+++ b/README.md
@@ -414,7 +414,7 @@ make activity ACTIVITY_LIMIT=20
-For per-module reference docs see [`docs/modules/`](docs/modules/index.md).
+For per-module reference docs see [`docs/modules.md`](docs/modules.md).
Full pipeline diagrams live in [`docs/diagrams/`](docs/diagrams/) — query,
ingest, verifier-ladder, plus the mesh sub-diagrams. Render any change
with `make docs`.
@@ -478,7 +478,7 @@ aborist/
└── cli.py argparse entrypoint (the make targets call into here).
```
-For diagrams of how these modules wire together, see `docs/diagrams/` (rendered SVGs of the module graph, query pipeline, ingest pipeline, mesh wire, & verifier ladder). For per-module reference docs see `docs/modules/` — every top-level package has its own page.
+For diagrams of how these modules wire together, see `docs/diagrams/` (rendered SVGs of the module graph, query pipeline, ingest pipeline, mesh wire, & verifier ladder). For per-module reference docs see `docs/modules.md` — every top-level package has its own page.
Source papers (read first if confused):
diff --git a/docs/modules.md b/docs/modules.md
new file mode 100644
index 0000000..3524208
--- /dev/null
+++ b/docs/modules.md
@@ -0,0 +1,1124 @@
+# Aborist module reference
+
+_Single-file reference for every top-level package + diagrams._
+
+
+
+
+
+
+## Diagrams index
+
+### Diagrams
+
+| Diagram | What it shows | File |
+|---|---|---|
+| **Module graph** | Top-level packages & how they import each other | [`aborist-modules.svg`](diagrams/aborist-modules.svg) ([dot](diagrams/aborist-modules.dot)) |
+| **Query pipeline** | Question → cache → retrieval → LLM → verify → render | [`query-pipeline.svg`](diagrams/query-pipeline.svg) ([dot](diagrams/query-pipeline.dot)) |
+| **Ingest pipeline** | Source document → Merkle-committed shard | [`ingest-pipeline.svg`](diagrams/ingest-pipeline.svg) ([dot](diagrams/ingest-pipeline.dot)) |
+| **Verifier ladder** | (audit_mode, violations) → display rung | [`verifier-ladder.svg`](diagrams/verifier-ladder.svg) ([dot](diagrams/verifier-ladder.dot)) |
+| **Mesh data flow** | Federation: roster, gossip, AEAD envelope | [`mesh-data-flow.svg`](diagrams/mesh-data-flow.svg) |
+| **Mesh epoch lifecycle** | Epoch advance via add/kick/rotate | [`mesh-epoch-lifecycle.svg`](diagrams/mesh-epoch-lifecycle.svg) |
+| **Mesh identity stack** | Ed25519 sign + X25519 DH key derivation | [`mesh-identity-stack.svg`](diagrams/mesh-identity-stack.svg) |
+| **Mesh secret envelope** | AEAD-wrapped epoch secret per peer | [`mesh-secret-envelope.svg`](diagrams/mesh-secret-envelope.svg) |
+| **Mesh group decisions** | Membership change voting & quorum | [`mesh-group-decisions.svg`](diagrams/mesh-group-decisions.svg) |
+
+Render diagrams locally:
+
+```
+make docs # runs `dot -Tsvg` and `-Tpng` on every docs/diagrams/*.dot
+```
+
+### Substrate (no SQL, pure data structures)
+
+| Module | One-line role | Doc |
+|---|---|---|
+| [`merkle.py`](../aborist/merkle.py) | Merkle tree + proof — Python port of `proxy.unturf.com/pkg/verified/merkle.go` | [↓](#merkle-py) |
+| [`document.py`](../aborist/document.py) | `Document`, `Edge`, `Chunker` (default `tok-512-v1`) | [↓](#document-py) |
+| [`wikitext.py`](../aborist/wikitext.py) | `to_base()` — wikitext → plain prose, BASE_VERSION-pinned | [↓](#wikitext-py) |
+
+### Storage
+
+| Module | One-line role | Doc |
+|---|---|---|
+| [`store.py`](../aborist/store.py) | v9.8 SQLite schema + audit chain helpers | [↓](#store-py) |
+| [`ingest.py`](../aborist/ingest.py) | normalize → chunk → merkle → upsert (bulk-batched) | [↓](#ingest-py) |
+| [`evict.py`](../aborist/evict.py) | hot ↔ cold tier transitions; rehydrate via source | [↓](#evict-py) |
+
+### Sources (corpus producers)
+
+| Module | One-line role | Doc |
+|---|---|---|
+| [`sources/wikipedia.py`](../aborist/sources/wikipedia.py) | Wikipedia 2003 cur + old SQL dumps (bz2-streamed) | [↓](#sources-py) |
+| [`sources/wikipedia_xml.py`](../aborist/sources/wikipedia_xml.py) | Phase IV XML dumps (iterparse, page + history) | [↓](#sources-py) |
+| [`sources/html_page.py`](../aborist/sources/html_page.py) | URL list + selectolax + httpx (robots-aware) | [↓](#sources-py) |
+| [`sources/crawler/`](../aborist/sources/crawler/) | verbatim AsyncWebFetcher lift + ingest bridge | [↓](#sources-py) |
+| [`sources/grok.py`](../aborist/sources/grok.py) | xAI data export (conversations + media prompts) | [↓](#sources-py) |
+| [`sources/vcs.py`](../aborist/sources/vcs.py) | git + Mercurial repos (HEAD walk, supersedes chain) | [↓](#sources-py) |
+
+### Search & retrieval
+
+| Module | One-line role | Doc |
+|---|---|---|
+| [`search/`](../aborist/search/) | FTS5 backend + `SearchBackend` ABC + `AuditMode` enum | [↓](#search-py) |
+| [`concepts/`](../aborist/concepts/) | Per-shard `concept_relations` synonym/rivalry overlay | [↓](#concepts-py) |
+
+### Q&A pipeline
+
+| Module | One-line role | Doc |
+|---|---|---|
+| [`qa/keys.py`](../aborist/qa/keys.py) | 8-dim cache_key + `question_hash` | [↓](#qa-py) |
+| [`qa/client.py`](../aborist/qa/client.py) | `ChatClient` + `StubClient` + `OpenAICompatibleClient` | [↓](#qa-py) |
+| [`qa/runner.py`](../aborist/qa/runner.py) | `ask()`: single-doc Q&A + cache + verify | [↓](#qa-py) |
+| [`qa/query.py`](../aborist/qa/query.py) | `query()`: multi-source RAG + concept overlay | [↓](#qa-py) |
+| [`qa/verify.py`](../aborist/qa/verify.py) | quote/span/entity/paraphrase + claim_lattice (7 hard checks) | [↓](#qa-py) |
+| [`qa/warrant.py`](../aborist/qa/warrant.py) | 5 anchor classes (proper-noun · date · count · entity-list · cause) | [↓](#qa-py) |
+| [`qa/evidence.py`](../aborist/qa/evidence.py) | `EvidenceObject` + spotlight excerpt (density rank) | [↓](#qa-py) |
+| [`qa/parse_claims.py`](../aborist/qa/parse_claims.py) | pointer-line parser (`claim. [E1,E2]`) | [↓](#qa-py) |
+| [`qa/dag.py`](../aborist/qa/dag.py) | per-run Merkle-DAG (7-stage quote / 9-stage CTI) | [↓](#qa-py) |
+| [`qa/inspect.py`](../aborist/qa/inspect.py) | sidecar diagnostic (read-only span classifier) | [↓](#qa-py) |
+
+### Distillation
+
+| Module | One-line role | Doc |
+|---|---|---|
+| [`distill/`](../aborist/distill/) | `Distiller` ABC + `first_sentence` + `tfidf` + runner | [↓](#distill-py) |
+
+### Federation (off by default)
+
+| Module | One-line role | Doc |
+|---|---|---|
+| [`mesh/`](../aborist/mesh/) | identity (Ed25519/X25519), per-epoch roster, AEAD envelope, gossip wire | [../mesh.md](mesh.md), [../mesh-deploy.md](mesh-deploy.md) |
+
+### Entry point
+
+| Module | One-line role | Doc |
+|---|---|---|
+| [`cli.py`](../aborist/cli.py) | `argparse` entrypoint — every `make` target dispatches here | run `aborist --help` or any `make help` target |
+
+### Tickets, design docs, journals
+
+See [`../TICKETS.md`](TICKETS.md) for the ticket index and the
+list of design-reference docs that aren't tickets.
+
+
+
+
+
+## `aborist.merkle`
+
+Pure Merkle tree + proof primitives. Python port of
+`proxy.unturf.com/pkg/verified/merkle.go` — convention-identical.
+Used everywhere a content-addressable handle is needed: per-chunk
+leaves, document_root, evidence_map_root, run_dag_root, snapshots.
+
+### Conventions (do not silently change)
+
+These match the Go reference & are load-bearing for cross-language
+verification (Go peer ↔ Python peer compute bit-identical roots):
+
+- **Leaf hash:** `sha256(0x00 || canonical_chunk_bytes)`. The `0x00`
+ prefix domain-separates leaves from internal nodes.
+- **Internal hash:** `sha256(0x03 || left || right)`. The `0x03`
+ prefix is the **non-commutative** combine — `H(L,R) ≠ H(R,L)`.
+ Order matters.
+- **Odd-element rule:** when a level has an odd count, the last
+ leaf is **self-duplicated** before pairing. NOT zero-padded.
+- **Proof path:** each step carries an explicit `is_left: bool`
+ alongside the sibling hash so a verifier knows which side to put
+ the sibling on. Never sort siblings lexically — the order tells
+ the verifier the tree topology.
+
+### API surface
+
+```python
+from aborist.merkle import MerkleTree, MerkleProof
+
+tree = MerkleTree.build([b"chunk_0_bytes", b"chunk_1_bytes", ...])
+tree.root # bytes(32) — sha256 of the whole tree
+tree.leaves # list[bytes(32)] — leaf hashes in input order
+
+proof = tree.proof_for(leaf_index=2)
+proof.siblings # list[(sibling_hash, is_left)]
+proof.verify(leaf_hash=tree.leaves[2], root=tree.root) # bool
+```
+
+### When to read the source
+
+- Adding a new content-addressable artifact (cores, evidence maps,
+ snapshots, run-DAGs all touch this).
+- Cross-language verification debugging (Go peer says one root,
+ Python peer says another — the difference is always in canonical
+ encoding, ordering, or one of the three prefix bytes above).
+- Performance work — the Python build is ~3× slower than the Go
+ reference; if it ever shows up in profiling, that's the file.
+
+### Diagrams
+
+The module graph shows what depends on `merkle.py` (a lot — it's
+substrate):
+
+
+
+The ingest pipeline shows where leaf & root hashes get computed:
+
+
+
+### Source
+
+[`aborist/merkle.py`](../aborist/merkle.py) ·
+Reference: [`proxy.unturf.com/pkg/verified/merkle.go`](https://git.unturf.com/engineering/unturf/proxy.unturf.com/-/blob/main/pkg/verified/merkle.go)
+
+
+
+
+
+## `aborist.document`
+
+The data structures every source produces and every storage layer
+consumes. Three core types: `Document`, `Edge`, `Chunker`.
+
+### `Document`
+
+A single ingest unit: a Wikipedia article, an HTML page, a Grok
+conversation, a git commit message, etc. Carries both the raw
+content AND the version tags that determine its identity:
+
+```python
+@dataclass(frozen=True)
+class Document:
+ document_uri: str # canonical URI (or stable surrogate for non-URI sources)
+ raw_content: str # source-of-truth bytes pre-canonicalization
+ kind: str # 'surface' / 'core' / 'visual' / etc.
+ chunking_version: str # e.g. 'tok-512-v1' — pinned by the Chunker
+ canonicalization_version: str # e.g. 'norm-v1' — pinned by canonicalize()
+ schema_version: str # e.g. 'v9.8.0' — store schema generation
+ title: str | None = None
+ edges: list[Edge] = () # outbound link graph
+ metadata: dict = ... # source-specific opaque payload
+```
+
+`document_root` is computed at ingest time as the Merkle root over
+the canonicalized chunks. Two peers ingesting the same source +
+running the same `chunking_version` + `canonicalization_version`
+get bit-identical `document_root`s — the v9.8 admissibility property.
+
+### `Edge`
+
+One outbound link. `aborist/sources/wikipedia.py` emits one Edge
+per `[[wikilink]]`; `aborist/sources/html_page.py` emits one per
+``. The link graph IS the corpus topology — `concepts/extract.py`
+later reads `edges` rows to derive synonym relations from
+reciprocal links (no separate crawler needed).
+
+```python
+@dataclass(frozen=True)
+class Edge:
+ src_root: str # source document_root
+ dst_uri: str # always present
+ dst_root: str # '' (unresolved) until the dst doc is also ingested
+ edge_type: str # 'wikilink' / 'href' / 'citation' / 'derived_from' / ...
+ anchor: str # chunk index or fragment, '' if N/A
+```
+
+### `Chunker`
+
+ABC with one method `chunk(text: str) -> list[str]`. Default impl
+is `TokenChunker` (`name='tok-512-v1'`) — splits on token-rough
+windows so the resulting chunks are predictable for downstream FTS5
+indexing & for the LLM context budget.
+
+**Changing the chunker bumps `chunking_version` AND stales every
+prior cache record** (chunking is one of the 8 cache_key dimensions).
+Don't redefine `tok-512-v1`; add a new chunker as a new `name`
+instead.
+
+### Diagrams
+
+
+
+
+### Source
+
+[`aborist/document.py`](../aborist/document.py)
+
+
+
+
+
+## `aborist.wikitext`
+
+A single function: `to_base(raw)`. Converts MediaWiki wikitext to
+plain prose deterministically.
+
+```python
+from aborist.wikitext import to_base, BASE_VERSION
+
+prose = to_base("[[The Beatles]] are an [[English rock band]] from [[Liverpool]].")
+## → "The Beatles are an English rock band from Liverpool."
+```
+
+### Why it exists
+
+The corpus stores raw wikitext (so the link graph is recoverable
+on demand) but the LLM and verifier both want plain prose. Reasons:
+
+1. **Token efficiency.** Wikipedia chunks ship to Hermes with ~43%
+ fewer tokens after wikitext-strip — bigger context window for
+ the same chars budget.
+2. **Verbatim citation.** The model can quote source paragraphs
+ verbatim instead of escaping `[[wikilinks]]`. The verifier's
+ substring test then matches cleanly.
+3. **Pinned identity.** `BASE_VERSION='wikitext-base-v1'` lives in
+ `policy["base_version"]`, which folds into
+ `governance_policy_hash`. Bumping `BASE_VERSION` invalidates
+ every prior cache record on next lookup — same discipline as
+ `chunking_version` and `canonicalization_version`.
+
+### Hot-path discipline
+
+`to_base()` runs on the assembled context **before** the LLM call
+in `aborist/qa/runner.py` and `aborist/qa/query.py`, **and again
+inside `verify_quotes`** so the verifier compares like-against-like.
+Both sides see prose.
+
+### Optional dependency
+
+Backed by `mwparserfromhell`. Install via `pip install '.[wikitext]'`
+to enable. Without the dep, `_wikitext_to_base = None` and
+`policy["base_version"] = None` — graceful fallback leaves raw
+wikitext in both context and verifier (works, just less efficient).
+
+### Source
+
+[`aborist/wikitext.py`](../aborist/wikitext.py)
+
+
+
+
+
+## `aborist.store`
+
+The v9.8 SQLite schema, the audit chain, and the cross-shard
+read-only view. Every table that holds runtime state lives here.
+
+### Schema overview (per shard)
+
+```
+documents – one row per source document, keyed on document_root
+chunks – per-document chunk content + tier (hot/cold)
+chunks_fts – FTS5 contentless index, rowid = chunks.chunk_id
+merkle_nodes – internal-node hashes for proof reconstruction
+edges – src_root → dst_root link graph (wikilink, href, …)
+derivations – core_root ← src_root with proof_blob (Merkle)
+providence_cache – Q&A records keyed on the v9.8 8-dim cache_key
+audit_events – linear chain; event_hash = sha256(prev || canonical(body))
+falsifications – record_id → state transition + reason + actor
+snapshots – named corpus roots (one hash names a forest)
+document_http_meta – ETag + Last-Modified for crawler conditional fetches
+concept_relations – per-shard synonym/rivalry/category/antonym overlay
+mesh_* – federation tables (off by default)
+```
+
+### v9.8 invariants (do not break)
+
+- **Aborist is a v9.8 store.** Every providence record carries the
+ full **8-dim cache_key**: `source_root | question_hash |
+ model_profile_hash | conversation_hash | governance_policy_hash |
+ schema_version | canonicalization_version | chunking_version`.
+ Bumping any one invalidates prior records on lookup.
+- **`falsification_state ∈ {live, failed, stale, quarantined}`.**
+ Cache lookups must filter on `state='live'`. Drift detection
+ flips to `stale`.
+- **Audit chain.** Every state-changing op writes one row in
+ `audit_events` with `event_hash = sha256(prev_event_hash ||
+ canonical(body))`. Chain integrity is verified in
+ `make analyze-shards`. **Never insert into `audit_events`
+ directly — use `aborist.store.append_audit`.**
+- **Cores never evict.** `evict_to_cold` only touches `kind='surface'`.
+- **Idempotent re-ingest.** Same content → same `document_root` →
+ no-op insert. Same URI + different content → new doc + `supersedes`
+ edge linking new → old (lossless history).
+
+### API surface
+
+```python
+from aborist.store import (
+ connect, # writable connection to a single shard
+ connect_query, # read-only UNION view across all shards
+ discover_shards, # list *.db files in a shards_dir
+ transaction, # BEGIN IMMEDIATE / COMMIT / ROLLBACK context manager
+ get_meta, set_meta,
+ append_audit, # the ONLY way to write audit_events
+)
+```
+
+Cross-shard reads use `connect_query(shards_dir=...)` which ATTACHes
+every `*.db` and creates UNION views over the shardable tables
+(`documents`, `chunks`, `merkle_nodes`, `edges`, `derivations`,
+`providence_cache`, `audit_events`, `falsifications`,
+`concept_relations`).
+
+### Performance pragmas
+
+`connect()` applies these per-connection:
+- `journal_mode=WAL` (set in SCHEMA_SQL once at first ingest)
+- `synchronous=NORMAL` (skip per-commit fsync; safe under WAL)
+- `cache_size=-65536` (64 MB page cache)
+- `temp_store=MEMORY` (no /tmp churn for temp tables)
+
+Don't downgrade to `synchronous=FULL` without a measured reason —
+costs ~5× throughput.
+
+### Diagrams
+
+
+
+
+### Source
+
+[`aborist/store.py`](../aborist/store.py)
+
+
+
+
+
+## `aborist.ingest`
+
+The bulk-batched pipeline that turns documents from a source into
+Merkle-committed shard storage. Source-agnostic: anything that
+implements `Source.iter_documents()` flows through here.
+
+
+
+### Public API
+
+```python
+from aborist.ingest import ingest_source
+from aborist.sources.wikipedia import WikipediaSqlDump
+
+source = WikipediaSqlDump("/path/to/cur.sql.bz2")
+ingest_source(
+ source,
+ db_path=Path("~/.aborist/shards/000.db"),
+ batch_size=200, # docs per transaction
+ progress_every=1000,
+)
+```
+
+### What happens per document
+
+1. **Canonicalize** — `canonicalize(text)`: NFC + ws-collapse +
+ strip ends. Pinned by `canonicalization_version='norm-v1'`.
+2. **Chunk** — `Chunker.chunk(canonical_text)` → list of token-
+ bounded substrings. Default `tok-512-v1` chunker.
+3. **Hash leaves** — `sha256(0x00 || canonical_chunk_bytes)` per
+ chunk.
+4. **Merkle tree** — `MerkleTree.build(leaves).root` →
+ `document_root`. Two peers running the same chunker on the same
+ canonicalized content compute bit-identical roots.
+5. **Upsert** — `documents` row keyed on `document_root` (idempotent
+ re-ingest), `chunks` rows with leaf hashes, `merkle_nodes` for
+ proof reconstruction, `edges` per outbound link.
+6. **FTS5** — `chunks_fts` insert with rowid = `chunks.chunk_id`
+ so the search-time JOIN lines up.
+7. **Audit event** — one row per ingest batch in `audit_events`,
+ chained on `prev_event_hash`.
+
+### Batching discipline
+
+Default `batch_size=200`: balances Python GIL overhead vs SQLite
+transaction commit cost. Lower it (e.g. 50) only to bound peak
+memory on a low-RAM host. Higher (e.g. 1000) for ETL throughput on
+SSD storage when memory isn't tight.
+
+`progress_every` prints a stderr line every N docs so long ingests
+are observable. Use `PYTHONUNBUFFERED=1` for tail-able output.
+
+### Resumability
+
+Idempotent re-ingest: same content + same chunker + same canonicalize
+= same `document_root` = no-op insert. So a crashed ingest can be
+restarted from the source's beginning without duplicating rows.
+
+Different content at the same URI gets a new `document_root` AND a
+`supersedes` edge linking new → old (lossless history).
+
+### Source
+
+[`aborist/ingest.py`](../aborist/ingest.py)
+
+
+
+
+
+## `aborist.evict`
+
+Hot ↔ cold tier transitions. The corpus is large (3.47M Wikipedia
+docs); not every chunk fits in working memory. `evict.py` is the
+mechanism that moves rarely-touched chunks to a cold tier (still
+indexed, just stored separately) and rehydrates them on demand
+from the original source.
+
+### API surface
+
+```python
+from aborist.evict import evict_to_cold, rehydrate
+
+## Move chunks unused for >threshold days to cold tier
+evict_to_cold(conn, max_age_days=90, max_evictions=10000)
+
+## Pull a cold chunk back to hot from its original source
+rehydrate(conn, document_root="abc123...")
+```
+
+### Invariant: cores never evict
+
+`evict_to_cold` filters `WHERE kind='surface'`. Cores are always
+hot — they're the long-tail-friendly compression layer that justifies
+evicting their underlying surfaces. Evicting cores would defeat the
+purpose.
+
+### v9.8 falsification on drift
+
+When `rehydrate()` re-fetches a document and the recomputed
+`document_root` differs from the stored one, the source has
+changed since ingest (Wikipedia article was edited, HTML page was
+republished, etc.). The cache record's `falsification_state` flips
+from `live` to `stale` — every providence record keyed on that
+`source_root` is no longer admissible to lookups.
+
+This is the **drift-detection-as-falsification** discipline: cache
+hits don't blindly trust historical answers; they trust answers
+that the SAME source still grounds.
+
+### Tier values
+
+`chunks.tier ∈ {'hot', 'cold'}`. Hot chunks live in `chunks.content`;
+cold chunks live with NULL `content` and a `cold_uri` pointing at
+the source. The QA pipeline's chunk-fetch path checks `tier`; on
+'cold', it triggers `rehydrate` before continuing.
+
+### Source
+
+[`aborist/evict.py`](../aborist/evict.py)
+
+
+
+
+
+## `aborist.sources`
+
+Corpus producers. Each is a `Source` ABC implementation that yields
+`Document` instances; the standard `ingest.ingest_source(source, db)`
+pipeline takes them from there.
+
+The `Source` ABC lives in [`aborist/source.py`](../aborist/source.py):
+
+```python
+class Source(ABC):
+ @abstractmethod
+ def iter_documents(self) -> Iterator[Document]: ...
+```
+
+### Built-in sources
+
+#### `wikipedia.py` — Phase III SQL dumps (the canonical bootstrap)
+
+Streams the Wikipedia 2003 `cur` (current revisions) and `old`
+(revision history) SQL dumps. Hand-rolled escape-aware parser
+(no `sqlite3` import — the dump is MySQL syntax). 4× speedup vs
+char-by-char loops via `str.find` + slicing. cProfile any change.
+
+Default Wikipedia 2003-05-16 dump source:
+`https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`.
+robots.txt returned 404 → no rules.
+
+[`aborist/sources/wikipedia.py`](../aborist/sources/wikipedia.py)
+
+#### `wikipedia_xml.py` — Phase IV XML dumps
+
+Modern Wikipedia dump format (`enwiki-YYYYMMDD-pages-articles.xml.bz2`,
+`enwiki-YYYYMMDD-pages-meta-history*.xml.bz2`). Uses `xml.etree.ElementTree.iterparse`
+to stream-parse without loading the whole tree.
+
+[`aborist/sources/wikipedia_xml.py`](../aborist/sources/wikipedia_xml.py)
+
+#### `html_page.py` — single-URL or URL-list HTML ingest
+
+Robots-aware (`urllib.robotparser`). Uses `selectolax` for fast
+HTML parsing (CSS-selector based; ~10× faster than `lxml`). Pulls
+the main body text + every `` as an `Edge` row.
+
+The `edges` rows are what the corpus-derived synonym extractor
+later reads — no separate crawler needed for site-internal link
+graphs.
+
+Optional dep: `pip install '.[html]'` for `selectolax` + `httpx`.
+
+[`aborist/sources/html_page.py`](../aborist/sources/html_page.py)
+
+#### `crawler/` — async BFS web crawl
+
+Verbatim lift of an `AsyncWebFetcher` implementation + an `ingest`
+bridge. BFS-discovers same-domain URLs from a seed, respecting
+`robots.txt` + crawl delays. Captures ETag + Last-Modified per URL
+into `document_http_meta` so a future recrawl can send conditional
+HEAD requests.
+
+[`aborist/sources/crawler/bridge.py`](../aborist/sources/crawler/bridge.py)
+
+#### `grok.py` — xAI Grok export
+
+Reads the `xAI-conversations.json` data export shape. Each
+conversation becomes one `Document`; media prompts are kept
+inline.
+
+[`aborist/sources/grok.py`](../aborist/sources/grok.py)
+
+#### `vcs.py` — git + Mercurial repositories
+
+HEAD walk. Each commit becomes a `Document` (commit message + diff
+stat). The supersedes chain captures commit ancestry as edges.
+
+[`aborist/sources/vcs.py`](../aborist/sources/vcs.py)
+
+### Source
+
+[`aborist/sources/`](../aborist/sources/) ·
+[`aborist/source.py`](../aborist/source.py) (ABC)
+
+
+
+
+
+## `aborist.search`
+
+The retrieval primitive. Today's only backend is FTS5 over the
+chunks table; the `SearchBackend` ABC is in place so additional
+backends (BM25 over titles, embedding-based vector search) can be
+added without touching the rest of the QA pipeline.
+
+### `SearchBackend` ABC
+
+```python
+from aborist.search import SearchBackend, AuditMode, Hit
+
+class SearchBackend(ABC):
+ @abstractmethod
+ def search(self, query: str, limit: int = 20) -> list[Hit]: ...
+```
+
+Each `Hit` carries `(document_root, document_uri, chunk_idx,
+snippet, score, audit_mode, title)`. `audit_mode` is the sticky
+provenance label that tracks how the chunk made it into the index;
+FTS5 backend always sets `UNGROUNDED` (search itself doesn't verify
+anything — that's the QA pipeline's job).
+
+### `FTS5Backend`
+
+Wraps the contentless `chunks_fts` virtual table. Two-mode query:
+
+- **AND-mode (strict, primary):** every content token must appear
+ in the doc. Keeps unrelated docs out of the context window.
+- **OR-mode (fallback):** when AND returns 0 hits, fall back to OR
+ but **capped to top-5 longest tokens** (proxy for rarity). Long
+ topical synonyms fed via `extra_or_tokens` join the pool —
+ `neurotechnology` (15 chars) outranks `thoughts` (8) by length
+ and surfaces brain-tech titles for vocabulary-mismatch queries.
+
+```python
+from aborist.search import FTS5Backend
+backend = FTS5Backend(conn)
+
+## Plain search
+hits = backend.search("permacomputer", limit=32)
+
+## Search with synonym pool injection (used by qa.query._search_corpus)
+hits = backend.search(
+ long_query,
+ limit=32,
+ extra_or_tokens=synonym_expand(qtokens, shards_dir=shards_dir),
+)
+```
+
+### Stopword & stopword-cap discipline
+
+`_FTS5_STOPWORDS` filters question words (`what`, `tell`, `please`)
++ generic connectors (`one`, `some`, `another`, `without`, `soon`,
+`currently`) before AND/OR construction. Two principles:
+
+- **Stay in sync with `_TITLE_STOPWORDS`** in `qa/query.py`. A token
+ filtered at retrieval time but kept at title-relevance check (or
+ vice versa) creates ranking incoherence.
+- **`_OR_FALLBACK_MAX_TOKENS=5`** caps the OR-mode pool. Without
+ this, a 19-token OR clause matches millions of docs and forces
+ BM25 to rank them all — 13s/shard observed pre-cap. Now 0.25s/shard.
+
+### Snippet building
+
+FTS5 contentless mode means SQLite's built-in `snippet()` and
+`highlight()` return empty. Aborist builds snippets in Python by
+joining `chunks_fts.rowid = chunks.chunk_id`, decompressing the
+chunk content, and locating query tokens locally
+(`_build_snippet`).
+
+### Source
+
+[`aborist/search/fts5.py`](../aborist/search/fts5.py) ·
+[`aborist/search/__init__.py`](../aborist/search/__init__.py)
+
+
+
+
+
+## `aborist.concepts`
+
+Per-shard `concept_relations` SQLite table — the corpus-derived
+synonym, rivalry, antonym & category overlay that replaces the
+hand-curated frozensets that lived in `aborist/qa/concepts.py`
+through April 2026.
+
+**Full architecture rationale lives in
+[`../concept-relations-design.md`](../concept-relations-design.md)**
+including the 1.6% storage-tax measurement and the three-compactions-
+considered-and-rejected analysis. This page is the API reference.
+
+### Sub-modules
+
+#### `concepts.store` — append-only CRUD
+
+```python
+from aborist.concepts.store import (
+ add_concept_relation, # idempotent INSERT OR IGNORE
+ concept_relations_for_token, # read all relations for a token
+ purge_by_evidence_kind, # the only DELETE path
+ list_evidence_kinds, # diagnostic
+ RELATION_KINDS, # ('synonym','antonym','rivalry','category')
+)
+```
+
+UNIQUE on `(source_root, relation_kind, token, target, evidence_kind)`
+makes re-derivation idempotent. `purge_by_evidence_kind` lets an
+operator revoke a single extractor's output without touching manual
+or other-extractor rows.
+
+[`aborist/concepts/store.py`](../aborist/concepts/store.py)
+
+#### `concepts.query` — cross-shard lookup
+
+```python
+from aborist.concepts import (
+ synonym_expand, # query tokens → expanded set
+ rivalry_excluded, # query tokens → tokens to drop from results
+ has_compare_phrasing, # bool — does the query say "vs", "compare", etc.
+ invalidate_cache, # drop the per-process LRU
+)
+
+expanded = synonym_expand({"thoughts"}, shards_dir=shards_dir)
+## → {"thoughts", "telepathy", "neurotechnology", "mind", "cognition", ...}
+```
+
+Two synonym indices are loaded:
+
+- **`manual_index`** — `manual_legacy` + `manual` rows. Curated;
+ always expanded regardless of per-token degree. Captures the
+ brain-tech / AMD-family / Mac / Linux / etc. seed groups.
+- **`derived_index`** — `link_reciprocity` & other corpus-derived
+ edges. Subject to **`MAX_NEIGHBORS_PER_TOKEN=8`** cap because the
+ Wikipedia link graph carries topic-adjacency noise on generic
+ tokens (person, thoughts, language).
+
+Overall **`MAX_TOTAL_TOKENS=50`** cap on expanded set bounds the
+SQL clause count downstream so retrieval stays sub-second.
+
+A per-process LRU keyed on shard mtime avoids re-loading the index
+on every query (290k rows across 4 shards loads in ~1.8s cold).
+
+[`aborist/concepts/query.py`](../aborist/concepts/query.py)
+
+#### `concepts.extract` — pluggable extractor framework
+
+```python
+from aborist.concepts.extract import (
+ EXTRACTORS, # registry: evidence_kind → callable
+ link_reciprocity_synonym, # built-in extractor
+)
+
+## Run an extractor against a shard:
+result = link_reciprocity_synonym(conn, derived_from="backfill@2026-05-01")
+## → {"reciprocal_pairs": N, "synonyms_inserted": M, "synonyms_skipped": K}
+```
+
+Each extractor walks the shard's existing rows (`documents`,
+`chunks`, `edges`, `derivations`) — **no new crawler needed** — &
+emits concept relations under a stable `evidence_kind` string that
+supports targeted purge.
+
+The built-in `link_reciprocity_synonym` reads the existing `edges`
+table for reciprocal A↔B link pairs and emits a synonym edge between
+every (title-token-of-A, title-token-of-B) pair. Works for
+Wikipedia (See-also bidirectional), HTML site internal links
+(russell.ballestrini.net pattern), or any document graph with
+bidirectional links. Title-tokens are filtered to ≥4 chars +
+stopword-stripped.
+
+To add a new extractor:
+
+1. Implement `(conn, *, derived_from) -> dict[str, int]` that calls
+ `add_concept_relation` for each finding.
+2. Pick a stable `evidence_kind` string.
+3. Register in `EXTRACTORS`.
+
+[`aborist/concepts/extract.py`](../aborist/concepts/extract.py)
+
+#### `concepts.seed` — legacy frozenset migration
+
+One-shot migration of the 8 hand-curated frozenset groups
+(AMD-family, Intel-family, HTTP, FTP, Mac, Windows, Linux,
+brain-tech) to `evidence_kind='manual_legacy'` rows.
+
+Writes **clique edges** within each group — every `(a, b)` pair —
+so any member retrieves every other member (preserves the legacy
+frozenset semantic where lookups didn't depend on which token in
+the group was the anchor).
+
+```python
+from aborist.concepts.seed import seed_legacy_concepts
+result = seed_legacy_concepts(conn)
+## → {"synonyms_inserted": N, "rivalries_inserted": M, "skipped": K}
+```
+
+Idempotent — re-running adds nothing if all the rows already exist.
+
+[`aborist/concepts/seed.py`](../aborist/concepts/seed.py)
+
+### Data model
+
+```sql
+CREATE TABLE concept_relations (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ source_root TEXT NOT NULL,
+ relation_kind TEXT NOT NULL CHECK (relation_kind IN
+ ('synonym','antonym','rivalry','category')),
+ token TEXT NOT NULL,
+ target TEXT NOT NULL,
+ evidence_kind TEXT NOT NULL,
+ confidence REAL NOT NULL DEFAULT 1.0,
+ derived_at INTEGER NOT NULL,
+ derived_from TEXT,
+ UNIQUE (source_root, relation_kind, token, target, evidence_kind)
+);
+```
+
+**Append-only by construction.** Re-running an extractor adds
+nothing if every relation already exists. No `UPDATE` path; only
+`add_concept_relation` (insert) and `purge_by_evidence_kind`
+(targeted delete).
+
+**Per-shard storage.** Concept relations live in the shard whose
+document derived them. Mesh sync moves shards between peers;
+relations come along.
+
+**Orthogonal to Merkle.** Writes to `concept_relations` NEVER
+affect `document_root`, `chunk_root`, or `cache_key`. Backfilling
+relations is safe across the entire corpus without invalidating
+any cached answer or breaking any audit chain.
+
+### Storage cost — measured
+
+Backfill on 4 wiki shards (3.47M docs, 10.75M resolved edges):
+
+| Shard | Reciprocal pairs | Synonyms | Storage |
+|---|---:|---:|---:|
+| 000.db | 13,562 | 71,288 | 23.52 MB |
+| 001.db | 14,078 | 73,351 | 24.20 MB |
+| 002.db | 13,708 | 72,576 | 23.92 MB |
+| 003.db | 13,800 | 72,633 | 23.94 MB |
+| **total** | **55,148** | **289,848** | **95.58 MB** |
+
+**1.6% storage tax** on the 6 GB corpus. Backfill takes ~4 min
+wall-clock total. Cost is paid once at backfill; every retrieval-
+time lookup benefits.
+
+### Diagrams
+
+
+
+
+### Source
+
+- [`aborist/concepts/`](../aborist/concepts/) — package
+- [`docs/concept-relations-design.md`](../concept-relations-design.md) — full design doc
+- Whitepaper §13.4.11 — public-facing summary
+
+
+
+
+
+## `aborist.qa`
+
+The Q&A pipeline. Question → cache → retrieval → LLM → verify →
+render → cache write. Lives in 9 sub-modules; this page is the
+map.
+
+
+
+### Sub-modules
+
+#### `qa.client` — LLM transport
+
+`ChatClient` ABC with three concrete implementations:
+
+- `StubClient` — deterministic test fixture; returns canned
+ responses keyed on the input. Used in unit tests to avoid network.
+- `OpenAICompatibleClient` — talks to any OpenAI-shape `/v1/chat/completions`
+ endpoint (Hermes-3 on vLLM by default). Includes HTTP retry layer
+ (3× exponential backoff on 5xx).
+- (Future) `AnthropicClient` — Claude API direct.
+
+[`aborist/qa/client.py`](../aborist/qa/client.py)
+
+#### `qa.keys` — the 8-dim cache_key
+
+```
+cache_key = sha256(
+ source_root | question_hash | model_profile_hash |
+ conversation_hash | governance_policy_hash |
+ schema_version | canonicalization_version | chunking_version
+)
+```
+
+Two question-hash modes (`strict` vs `equivalence_class`) live here.
+Bumping any of these eight dimensions invalidates prior records on
+lookup. The `verifier_policy_hash` (v9.9 9th dim) is also
+implemented here.
+
+[`aborist/qa/keys.py`](../aborist/qa/keys.py)
+
+#### `qa.runner` — `ask()` for single-doc Q&A
+
+The simplest entry point. Take one document, ask one question, get
+back an answer + audit_mode + cache record. Used by the CLI for
+focused queries against one URI.
+
+[`aborist/qa/runner.py`](../aborist/qa/runner.py)
+
+#### `qa.query` — `query()` for multi-source RAG
+
+The main retrieval entry point. Walks shards, runs FTS5 BM25 with
+AND→OR fallback (with synonym-pool injection in OR mode), filters
+by title relevance with 4 accept paths, reranks by body coverage +
+title boost + source role + title purity, assembles a 60 KB context
+budget, calls the LLM, runs the verifier, persists to
+`providence_cache`.
+
+[`aborist/qa/query.py`](../aborist/qa/query.py)
+
+#### `qa.verify` — the layered verifier
+
+Five strategies run in sequence; first to find evidence classifies:
+1. `quote` — `"..."`-wrapped claims tested verbatim
+2. `span` — bullet/sentence units substring-tested
+3. `entity` — multi-word proper nouns with proximity gating
+4. `paraphrase` — token coverage on prose-shaped spans (≥85%)
+5. `claim_lattice` — pointer-line `[E1,E2]` or JSON; runs **seven
+ deterministic hard checks**:
+ 1. parser succeeded
+ 2. evidence_id resolves
+ 3. source_role allowed
+ 4. claim text non-empty
+ 5. citation coverage threshold
+ 6. pointer count cap (trim-and-verify)
+ 7. anchor-class warrant (see `qa.warrant`)
+
+The classifier output rolls up into the v9.8 trichotomy
+`audit_mode ∈ {STRICT, HYBRID, UNGROUNDED}`. Display layer (in
+`cli.py`) maps `(audit_mode, violations) → four-rung ladder`.
+
+
+
+[`aborist/qa/verify.py`](../aborist/qa/verify.py)
+
+#### `qa.warrant` — anchor-class warrant
+
+Five lexical anchor classes the verifier composes:
+
+- **Proper-noun** — relation-question shape; at least one
+ Title-Case anchor must appear in some cited span
+- **Date** — claim has a 4-digit year + month name; ALL components
+ required in some cited span
+- **Entity-list** — entity-list-shape question; ≥1 named entity
+ must anchor (demote-don't-reject)
+- **Count** — count-shape question; count token must appear in
+ word OR digit form (digit↔word equivalence)
+- **Cause** — why-shape question; ≥1 cause anchor (proper noun OR
+ ≥5-char common noun outside stopword pool)
+
+The warrant layer earns proof-path entry by staying **lexical** —
+no NLI, no embeddings. Substring tests over already-canonicalized
+spans. See `docs/concept-relations-design.md` (sibling section)
+for the relationship to retrieval-time synonym expansion.
+
+[`aborist/qa/warrant.py`](../aborist/qa/warrant.py)
+
+#### `qa.evidence` — EvidenceObject + spotlight
+
+Builds the runtime evidence map for claim-lattice modes. Each
+chunk becomes one `EvidenceObject` carrying TWO ids:
+
+- `pointer_id` — short prompt-facing tag (`E1`, `E2`, …)
+- `evidence_id` — content-addressed `E########` (sha256-derived)
+
+The model sees only `pointer_id`s in the prompt; the runtime maps
+to `evidence_id` for the cache & run-DAG (run-stable identity).
+
+The spotlight excerpt picks the load-bearing slice via **density
+rank** — find ALL match positions for ALL claim content tokens,
+pick the position with maximum distinct-token cluster within
+±half-window. Replaces the older first-match-of-longest-token
+approach which lost the load-bearing slice on noisy chunks.
+
+[`aborist/qa/evidence.py`](../aborist/qa/evidence.py)
+
+#### `qa.parse_claims` — pointer-line parser
+
+Walks lines of the model output, pulls every `[E\d+]` and
+`[E\d+,E\d+,…]` bracket payload, returns
+`(claim_text, pointer_ids[])` per line. Lines without a tag get
+`parse_status='NO_EVIDENCE_POINTER'` & count toward the denominator
+so unsourced prose can't smuggle past the verifier.
+
+[`aborist/qa/parse_claims.py`](../aborist/qa/parse_claims.py)
+
+#### `qa.dag` — per-run Merkle DAG
+
+Commits each provenance step independently as a stage hash. Two
+shapes:
+
+- **7-stage (quote mode):** question / retrieval / context /
+ prompt / answer / verify / final_label
+- **9-stage (claim-lattice / CTI):** question / retrieval /
+ evidence_map / prompt / raw_answer / parsed_claim_lattice /
+ verify / render / final_label
+
+The `run_dag_root` is persisted alongside every providence record;
+`run_dag_blob` carries the full `{root, nodes}` JSON so an auditor
+can recompute & verify any step.
+
+[`aborist/qa/dag.py`](../aborist/qa/dag.py)
+
+#### `qa.inspect` — read-only sidecar
+
+Pulls source chunks for a given cache_key & classifies each
+unverified span: `verbatim_in_base` / `verbatim_in_raw_only` /
+`trailing_artifact` / `paraphrase` / `partial_paraphrase` /
+`no_overlap`. Also includes the deflection-detection sidecar
+(subject-anchor heuristic for adversarial-premise topic shift).
+
+**Sidecars never write to `providence_cache` or `audit_events`** —
+they're diagnostic only. That invariant is what keeps `audit_mode`
+a binary classification rather than a soft score.
+
+[`aborist/qa/inspect.py`](../aborist/qa/inspect.py)
+
+#### `qa.concepts` — backwards-compat shim
+
+Delegates to `aborist.concepts` (the corpus-derived synonym/rivalry
+layer). Pre-2026-05-01 the data lived as hand-curated frozensets in
+this file; now it's a per-shard SQLite table. The shim preserves
+the legacy public API (`synonym_expand`, `rivalry_excluded`,
+`has_compare_phrasing`) so call sites in `qa/query.py` didn't have
+to change.
+
+[`aborist/qa/concepts.py`](../aborist/qa/concepts.py) →
+[`aborist/concepts/`](../aborist/concepts/)
+
+### Source papers
+
+- Whitepaper §13.8 covers the layered verifier in depth
+- Whitepaper §13.9 covers claim-lattice / CTI mode
+- `docs/cti-architecture.md` is the architecture reference
+- `docs/seven-point-program.md` enumerates the seven hard checks
+- `docs/concept-relations-design.md` covers the synonym layer
+
+
+
+
+
+## `aborist.distill`
+
+Surface → core distillation. Takes a set of surface documents
+(the original ingest layer) and produces "core" documents — shorter,
+more focused, Merkle-bound back to their contributing surface
+chunks via per-chunk inclusion proofs.
+
+The "trees and forests of cross-linked information" tagline aborist
+takes its name from comes from this layer: planet-toward-center
+compression where each layer of cores derives from the previous,
+recursively.
+
+### Layered design
+
+```
+distill/
+├── base.py Distiller ABC + DistillationResult dataclass
+├── first_sentence.py no-ML stub: take the first sentence of each doc
+├── tfidf.py pure-Python TF-IDF top-keyword extraction
+└── runner.py batched distillation + per-contrib-chunk proofs
+```
+
+### `Distiller` ABC
+
+```python
+from aborist.distill import Distiller, DistillationResult
+
+class Distiller(ABC):
+ @abstractmethod
+ def distill(self, docs: list[Document]) -> DistillationResult: ...
+```
+
+Each `DistillationResult` carries the new core's content +
+references to every contributing surface chunk by `(document_root,
+chunk_root)`. The runner writes one `derivations` row per core,
+with `proof_blob = json.dumps(per_chunk_inclusion_proofs)`.
+
+### Built-in distillers
+
+#### `FirstSentenceDistiller` (no-ML stub)
+
+Take the first sentence of each input doc, concatenate. Used as a
+sanity-check for the pipeline + a baseline for measuring the
+benefit of richer distillers.
+
+#### `TfidfKeywordDistiller`
+
+Pure-Python TF-IDF. Computes term frequencies across the input
+doc set + inverse document frequencies; emits the top-K terms per
+doc as the core's content. The "permacomputer" neologism case
+(every Grok conversation has the word, no Wikipedia article does)
+is the canonical TF-IDF win — surfaces the topic that title-search
+can't catch.
+
+### Why distill
+
+Three use cases:
+
+1. **Retrieval signal.** Cores feed the third accept path in
+ `_filter_by_title_relevance` — `core_match_roots` (TF-IDF top-
+ keywords contain a query token). Closes the gap for neologisms
+ that never make Wikipedia titles but ARE distinctive.
+
+2. **Hot/cold tier discipline.** `evict_to_cold` only touches
+ `kind='surface'` — cores never evict. Distilling surface to
+ cores then evicting surfaces gives a "long tail keeps small
+ cache" pattern with full provenance preserved.
+
+3. **Recursive abstraction.** Cores can themselves be distilled
+ into shorter cores. Each generation Merkle-binds back to the
+ previous via `derivations.proof_blob` — the audit chain stays
+ intact across an arbitrary distillation depth.
+
+### Source
+
+[`aborist/distill/`](../aborist/distill/)
diff --git a/docs/modules/concepts.md b/docs/modules/concepts.md
deleted file mode 100644
index 9838714..0000000
--- a/docs/modules/concepts.md
+++ /dev/null
@@ -1,179 +0,0 @@
-# `aborist.concepts`
-
-Per-shard `concept_relations` SQLite table — the corpus-derived
-synonym, rivalry, antonym & category overlay that replaces the
-hand-curated frozensets that lived in `aborist/qa/concepts.py`
-through April 2026.
-
-**Full architecture rationale lives in
-[`../concept-relations-design.md`](../concept-relations-design.md)**
-including the 1.6% storage-tax measurement and the three-compactions-
-considered-and-rejected analysis. This page is the API reference.
-
-## Sub-modules
-
-### `concepts.store` — append-only CRUD
-
-```python
-from aborist.concepts.store import (
- add_concept_relation, # idempotent INSERT OR IGNORE
- concept_relations_for_token, # read all relations for a token
- purge_by_evidence_kind, # the only DELETE path
- list_evidence_kinds, # diagnostic
- RELATION_KINDS, # ('synonym','antonym','rivalry','category')
-)
-```
-
-UNIQUE on `(source_root, relation_kind, token, target, evidence_kind)`
-makes re-derivation idempotent. `purge_by_evidence_kind` lets an
-operator revoke a single extractor's output without touching manual
-or other-extractor rows.
-
-[`aborist/concepts/store.py`](../../aborist/concepts/store.py)
-
-### `concepts.query` — cross-shard lookup
-
-```python
-from aborist.concepts import (
- synonym_expand, # query tokens → expanded set
- rivalry_excluded, # query tokens → tokens to drop from results
- has_compare_phrasing, # bool — does the query say "vs", "compare", etc.
- invalidate_cache, # drop the per-process LRU
-)
-
-expanded = synonym_expand({"thoughts"}, shards_dir=shards_dir)
-# → {"thoughts", "telepathy", "neurotechnology", "mind", "cognition", ...}
-```
-
-Two synonym indices are loaded:
-
-- **`manual_index`** — `manual_legacy` + `manual` rows. Curated;
- always expanded regardless of per-token degree. Captures the
- brain-tech / AMD-family / Mac / Linux / etc. seed groups.
-- **`derived_index`** — `link_reciprocity` & other corpus-derived
- edges. Subject to **`MAX_NEIGHBORS_PER_TOKEN=8`** cap because the
- Wikipedia link graph carries topic-adjacency noise on generic
- tokens (person, thoughts, language).
-
-Overall **`MAX_TOTAL_TOKENS=50`** cap on expanded set bounds the
-SQL clause count downstream so retrieval stays sub-second.
-
-A per-process LRU keyed on shard mtime avoids re-loading the index
-on every query (290k rows across 4 shards loads in ~1.8s cold).
-
-[`aborist/concepts/query.py`](../../aborist/concepts/query.py)
-
-### `concepts.extract` — pluggable extractor framework
-
-```python
-from aborist.concepts.extract import (
- EXTRACTORS, # registry: evidence_kind → callable
- link_reciprocity_synonym, # built-in extractor
-)
-
-# Run an extractor against a shard:
-result = link_reciprocity_synonym(conn, derived_from="backfill@2026-05-01")
-# → {"reciprocal_pairs": N, "synonyms_inserted": M, "synonyms_skipped": K}
-```
-
-Each extractor walks the shard's existing rows (`documents`,
-`chunks`, `edges`, `derivations`) — **no new crawler needed** — &
-emits concept relations under a stable `evidence_kind` string that
-supports targeted purge.
-
-The built-in `link_reciprocity_synonym` reads the existing `edges`
-table for reciprocal A↔B link pairs and emits a synonym edge between
-every (title-token-of-A, title-token-of-B) pair. Works for
-Wikipedia (See-also bidirectional), HTML site internal links
-(russell.ballestrini.net pattern), or any document graph with
-bidirectional links. Title-tokens are filtered to ≥4 chars +
-stopword-stripped.
-
-To add a new extractor:
-
-1. Implement `(conn, *, derived_from) -> dict[str, int]` that calls
- `add_concept_relation` for each finding.
-2. Pick a stable `evidence_kind` string.
-3. Register in `EXTRACTORS`.
-
-[`aborist/concepts/extract.py`](../../aborist/concepts/extract.py)
-
-### `concepts.seed` — legacy frozenset migration
-
-One-shot migration of the 8 hand-curated frozenset groups
-(AMD-family, Intel-family, HTTP, FTP, Mac, Windows, Linux,
-brain-tech) to `evidence_kind='manual_legacy'` rows.
-
-Writes **clique edges** within each group — every `(a, b)` pair —
-so any member retrieves every other member (preserves the legacy
-frozenset semantic where lookups didn't depend on which token in
-the group was the anchor).
-
-```python
-from aborist.concepts.seed import seed_legacy_concepts
-result = seed_legacy_concepts(conn)
-# → {"synonyms_inserted": N, "rivalries_inserted": M, "skipped": K}
-```
-
-Idempotent — re-running adds nothing if all the rows already exist.
-
-[`aborist/concepts/seed.py`](../../aborist/concepts/seed.py)
-
-## Data model
-
-```sql
-CREATE TABLE concept_relations (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- source_root TEXT NOT NULL,
- relation_kind TEXT NOT NULL CHECK (relation_kind IN
- ('synonym','antonym','rivalry','category')),
- token TEXT NOT NULL,
- target TEXT NOT NULL,
- evidence_kind TEXT NOT NULL,
- confidence REAL NOT NULL DEFAULT 1.0,
- derived_at INTEGER NOT NULL,
- derived_from TEXT,
- UNIQUE (source_root, relation_kind, token, target, evidence_kind)
-);
-```
-
-**Append-only by construction.** Re-running an extractor adds
-nothing if every relation already exists. No `UPDATE` path; only
-`add_concept_relation` (insert) and `purge_by_evidence_kind`
-(targeted delete).
-
-**Per-shard storage.** Concept relations live in the shard whose
-document derived them. Mesh sync moves shards between peers;
-relations come along.
-
-**Orthogonal to Merkle.** Writes to `concept_relations` NEVER
-affect `document_root`, `chunk_root`, or `cache_key`. Backfilling
-relations is safe across the entire corpus without invalidating
-any cached answer or breaking any audit chain.
-
-## Storage cost — measured
-
-Backfill on 4 wiki shards (3.47M docs, 10.75M resolved edges):
-
-| Shard | Reciprocal pairs | Synonyms | Storage |
-|---|---:|---:|---:|
-| 000.db | 13,562 | 71,288 | 23.52 MB |
-| 001.db | 14,078 | 73,351 | 24.20 MB |
-| 002.db | 13,708 | 72,576 | 23.92 MB |
-| 003.db | 13,800 | 72,633 | 23.94 MB |
-| **total** | **55,148** | **289,848** | **95.58 MB** |
-
-**1.6% storage tax** on the 6 GB corpus. Backfill takes ~4 min
-wall-clock total. Cost is paid once at backfill; every retrieval-
-time lookup benefits.
-
-## Diagrams
-
-
-
-
-## Source
-
-- [`aborist/concepts/`](../../aborist/concepts/) — package
-- [`docs/concept-relations-design.md`](../concept-relations-design.md) — full design doc
-- Whitepaper §13.4.11 — public-facing summary
diff --git a/docs/modules/distill.md b/docs/modules/distill.md
deleted file mode 100644
index 774a124..0000000
--- a/docs/modules/distill.md
+++ /dev/null
@@ -1,76 +0,0 @@
-# `aborist.distill`
-
-Surface → core distillation. Takes a set of surface documents
-(the original ingest layer) and produces "core" documents — shorter,
-more focused, Merkle-bound back to their contributing surface
-chunks via per-chunk inclusion proofs.
-
-The "trees and forests of cross-linked information" tagline aborist
-takes its name from comes from this layer: planet-toward-center
-compression where each layer of cores derives from the previous,
-recursively.
-
-## Layered design
-
-```
-distill/
-├── base.py Distiller ABC + DistillationResult dataclass
-├── first_sentence.py no-ML stub: take the first sentence of each doc
-├── tfidf.py pure-Python TF-IDF top-keyword extraction
-└── runner.py batched distillation + per-contrib-chunk proofs
-```
-
-## `Distiller` ABC
-
-```python
-from aborist.distill import Distiller, DistillationResult
-
-class Distiller(ABC):
- @abstractmethod
- def distill(self, docs: list[Document]) -> DistillationResult: ...
-```
-
-Each `DistillationResult` carries the new core's content +
-references to every contributing surface chunk by `(document_root,
-chunk_root)`. The runner writes one `derivations` row per core,
-with `proof_blob = json.dumps(per_chunk_inclusion_proofs)`.
-
-## Built-in distillers
-
-### `FirstSentenceDistiller` (no-ML stub)
-
-Take the first sentence of each input doc, concatenate. Used as a
-sanity-check for the pipeline + a baseline for measuring the
-benefit of richer distillers.
-
-### `TfidfKeywordDistiller`
-
-Pure-Python TF-IDF. Computes term frequencies across the input
-doc set + inverse document frequencies; emits the top-K terms per
-doc as the core's content. The "permacomputer" neologism case
-(every Grok conversation has the word, no Wikipedia article does)
-is the canonical TF-IDF win — surfaces the topic that title-search
-can't catch.
-
-## Why distill
-
-Three use cases:
-
-1. **Retrieval signal.** Cores feed the third accept path in
- `_filter_by_title_relevance` — `core_match_roots` (TF-IDF top-
- keywords contain a query token). Closes the gap for neologisms
- that never make Wikipedia titles but ARE distinctive.
-
-2. **Hot/cold tier discipline.** `evict_to_cold` only touches
- `kind='surface'` — cores never evict. Distilling surface to
- cores then evicting surfaces gives a "long tail keeps small
- cache" pattern with full provenance preserved.
-
-3. **Recursive abstraction.** Cores can themselves be distilled
- into shorter cores. Each generation Merkle-binds back to the
- previous via `derivations.proof_blob` — the audit chain stays
- intact across an arbitrary distillation depth.
-
-## Source
-
-[`aborist/distill/`](../../aborist/distill/)
diff --git a/docs/modules/document.md b/docs/modules/document.md
deleted file mode 100644
index db0e76c..0000000
--- a/docs/modules/document.md
+++ /dev/null
@@ -1,68 +0,0 @@
-# `aborist.document`
-
-The data structures every source produces and every storage layer
-consumes. Three core types: `Document`, `Edge`, `Chunker`.
-
-## `Document`
-
-A single ingest unit: a Wikipedia article, an HTML page, a Grok
-conversation, a git commit message, etc. Carries both the raw
-content AND the version tags that determine its identity:
-
-```python
-@dataclass(frozen=True)
-class Document:
- document_uri: str # canonical URI (or stable surrogate for non-URI sources)
- raw_content: str # source-of-truth bytes pre-canonicalization
- kind: str # 'surface' / 'core' / 'visual' / etc.
- chunking_version: str # e.g. 'tok-512-v1' — pinned by the Chunker
- canonicalization_version: str # e.g. 'norm-v1' — pinned by canonicalize()
- schema_version: str # e.g. 'v9.8.0' — store schema generation
- title: str | None = None
- edges: list[Edge] = () # outbound link graph
- metadata: dict = ... # source-specific opaque payload
-```
-
-`document_root` is computed at ingest time as the Merkle root over
-the canonicalized chunks. Two peers ingesting the same source +
-running the same `chunking_version` + `canonicalization_version`
-get bit-identical `document_root`s — the v9.8 admissibility property.
-
-## `Edge`
-
-One outbound link. `aborist/sources/wikipedia.py` emits one Edge
-per `[[wikilink]]`; `aborist/sources/html_page.py` emits one per
-``. The link graph IS the corpus topology — `concepts/extract.py`
-later reads `edges` rows to derive synonym relations from
-reciprocal links (no separate crawler needed).
-
-```python
-@dataclass(frozen=True)
-class Edge:
- src_root: str # source document_root
- dst_uri: str # always present
- dst_root: str # '' (unresolved) until the dst doc is also ingested
- edge_type: str # 'wikilink' / 'href' / 'citation' / 'derived_from' / ...
- anchor: str # chunk index or fragment, '' if N/A
-```
-
-## `Chunker`
-
-ABC with one method `chunk(text: str) -> list[str]`. Default impl
-is `TokenChunker` (`name='tok-512-v1'`) — splits on token-rough
-windows so the resulting chunks are predictable for downstream FTS5
-indexing & for the LLM context budget.
-
-**Changing the chunker bumps `chunking_version` AND stales every
-prior cache record** (chunking is one of the 8 cache_key dimensions).
-Don't redefine `tok-512-v1`; add a new chunker as a new `name`
-instead.
-
-## Diagrams
-
-
-
-
-## Source
-
-[`aborist/document.py`](../../aborist/document.py)
diff --git a/docs/modules/evict.md b/docs/modules/evict.md
deleted file mode 100644
index c5c596e..0000000
--- a/docs/modules/evict.md
+++ /dev/null
@@ -1,50 +0,0 @@
-# `aborist.evict`
-
-Hot ↔ cold tier transitions. The corpus is large (3.47M Wikipedia
-docs); not every chunk fits in working memory. `evict.py` is the
-mechanism that moves rarely-touched chunks to a cold tier (still
-indexed, just stored separately) and rehydrates them on demand
-from the original source.
-
-## API surface
-
-```python
-from aborist.evict import evict_to_cold, rehydrate
-
-# Move chunks unused for >threshold days to cold tier
-evict_to_cold(conn, max_age_days=90, max_evictions=10000)
-
-# Pull a cold chunk back to hot from its original source
-rehydrate(conn, document_root="abc123...")
-```
-
-## Invariant: cores never evict
-
-`evict_to_cold` filters `WHERE kind='surface'`. Cores are always
-hot — they're the long-tail-friendly compression layer that justifies
-evicting their underlying surfaces. Evicting cores would defeat the
-purpose.
-
-## v9.8 falsification on drift
-
-When `rehydrate()` re-fetches a document and the recomputed
-`document_root` differs from the stored one, the source has
-changed since ingest (Wikipedia article was edited, HTML page was
-republished, etc.). The cache record's `falsification_state` flips
-from `live` to `stale` — every providence record keyed on that
-`source_root` is no longer admissible to lookups.
-
-This is the **drift-detection-as-falsification** discipline: cache
-hits don't blindly trust historical answers; they trust answers
-that the SAME source still grounds.
-
-## Tier values
-
-`chunks.tier ∈ {'hot', 'cold'}`. Hot chunks live in `chunks.content`;
-cold chunks live with NULL `content` and a `cold_uri` pointing at
-the source. The QA pipeline's chunk-fetch path checks `tier`; on
-'cold', it triggers `rehydrate` before continuing.
-
-## Source
-
-[`aborist/evict.py`](../../aborist/evict.py)
diff --git a/docs/modules/index.md b/docs/modules/index.md
deleted file mode 100644
index f12c26a..0000000
--- a/docs/modules/index.md
+++ /dev/null
@@ -1,97 +0,0 @@
-# Aborist module reference
-
-One page per top-level package + one diagram per major data flow.
-This index is the entry point — every module page links back here
-and to the relevant diagrams.
-
-## Diagrams
-
-| Diagram | What it shows | File |
-|---|---|---|
-| **Module graph** | Top-level packages & how they import each other | [`aborist-modules.svg`](../diagrams/aborist-modules.svg) ([dot](../diagrams/aborist-modules.dot)) |
-| **Query pipeline** | Question → cache → retrieval → LLM → verify → render | [`query-pipeline.svg`](../diagrams/query-pipeline.svg) ([dot](../diagrams/query-pipeline.dot)) |
-| **Ingest pipeline** | Source document → Merkle-committed shard | [`ingest-pipeline.svg`](../diagrams/ingest-pipeline.svg) ([dot](../diagrams/ingest-pipeline.dot)) |
-| **Verifier ladder** | (audit_mode, violations) → display rung | [`verifier-ladder.svg`](../diagrams/verifier-ladder.svg) ([dot](../diagrams/verifier-ladder.dot)) |
-| **Mesh data flow** | Federation: roster, gossip, AEAD envelope | [`mesh-data-flow.svg`](../diagrams/mesh-data-flow.svg) |
-| **Mesh epoch lifecycle** | Epoch advance via add/kick/rotate | [`mesh-epoch-lifecycle.svg`](../diagrams/mesh-epoch-lifecycle.svg) |
-| **Mesh identity stack** | Ed25519 sign + X25519 DH key derivation | [`mesh-identity-stack.svg`](../diagrams/mesh-identity-stack.svg) |
-| **Mesh secret envelope** | AEAD-wrapped epoch secret per peer | [`mesh-secret-envelope.svg`](../diagrams/mesh-secret-envelope.svg) |
-| **Mesh group decisions** | Membership change voting & quorum | [`mesh-group-decisions.svg`](../diagrams/mesh-group-decisions.svg) |
-
-Render diagrams locally:
-
-```
-make docs # runs `dot -Tsvg` and `-Tpng` on every docs/diagrams/*.dot
-```
-
-## Substrate (no SQL, pure data structures)
-
-| Module | One-line role | Doc |
-|---|---|---|
-| [`merkle.py`](../../aborist/merkle.py) | Merkle tree + proof — Python port of `proxy.unturf.com/pkg/verified/merkle.go` | [merkle.md](./merkle.md) |
-| [`document.py`](../../aborist/document.py) | `Document`, `Edge`, `Chunker` (default `tok-512-v1`) | [document.md](./document.md) |
-| [`wikitext.py`](../../aborist/wikitext.py) | `to_base()` — wikitext → plain prose, BASE_VERSION-pinned | [wikitext.md](./wikitext.md) |
-
-## Storage
-
-| Module | One-line role | Doc |
-|---|---|---|
-| [`store.py`](../../aborist/store.py) | v9.8 SQLite schema + audit chain helpers | [store.md](./store.md) |
-| [`ingest.py`](../../aborist/ingest.py) | normalize → chunk → merkle → upsert (bulk-batched) | [ingest.md](./ingest.md) |
-| [`evict.py`](../../aborist/evict.py) | hot ↔ cold tier transitions; rehydrate via source | [evict.md](./evict.md) |
-
-## Sources (corpus producers)
-
-| Module | One-line role | Doc |
-|---|---|---|
-| [`sources/wikipedia.py`](../../aborist/sources/wikipedia.py) | Wikipedia 2003 cur + old SQL dumps (bz2-streamed) | [sources.md](./sources.md) |
-| [`sources/wikipedia_xml.py`](../../aborist/sources/wikipedia_xml.py) | Phase IV XML dumps (iterparse, page + history) | [sources.md](./sources.md) |
-| [`sources/html_page.py`](../../aborist/sources/html_page.py) | URL list + selectolax + httpx (robots-aware) | [sources.md](./sources.md) |
-| [`sources/crawler/`](../../aborist/sources/crawler/) | verbatim AsyncWebFetcher lift + ingest bridge | [sources.md](./sources.md) |
-| [`sources/grok.py`](../../aborist/sources/grok.py) | xAI data export (conversations + media prompts) | [sources.md](./sources.md) |
-| [`sources/vcs.py`](../../aborist/sources/vcs.py) | git + Mercurial repos (HEAD walk, supersedes chain) | [sources.md](./sources.md) |
-
-## Search & retrieval
-
-| Module | One-line role | Doc |
-|---|---|---|
-| [`search/`](../../aborist/search/) | FTS5 backend + `SearchBackend` ABC + `AuditMode` enum | [search.md](./search.md) |
-| [`concepts/`](../../aborist/concepts/) | Per-shard `concept_relations` synonym/rivalry overlay | [concepts.md](./concepts.md) |
-
-## Q&A pipeline
-
-| Module | One-line role | Doc |
-|---|---|---|
-| [`qa/keys.py`](../../aborist/qa/keys.py) | 8-dim cache_key + `question_hash` | [qa.md](./qa.md) |
-| [`qa/client.py`](../../aborist/qa/client.py) | `ChatClient` + `StubClient` + `OpenAICompatibleClient` | [qa.md](./qa.md) |
-| [`qa/runner.py`](../../aborist/qa/runner.py) | `ask()`: single-doc Q&A + cache + verify | [qa.md](./qa.md) |
-| [`qa/query.py`](../../aborist/qa/query.py) | `query()`: multi-source RAG + concept overlay | [qa.md](./qa.md) |
-| [`qa/verify.py`](../../aborist/qa/verify.py) | quote/span/entity/paraphrase + claim_lattice (7 hard checks) | [qa.md](./qa.md) |
-| [`qa/warrant.py`](../../aborist/qa/warrant.py) | 5 anchor classes (proper-noun · date · count · entity-list · cause) | [qa.md](./qa.md) |
-| [`qa/evidence.py`](../../aborist/qa/evidence.py) | `EvidenceObject` + spotlight excerpt (density rank) | [qa.md](./qa.md) |
-| [`qa/parse_claims.py`](../../aborist/qa/parse_claims.py) | pointer-line parser (`claim. [E1,E2]`) | [qa.md](./qa.md) |
-| [`qa/dag.py`](../../aborist/qa/dag.py) | per-run Merkle-DAG (7-stage quote / 9-stage CTI) | [qa.md](./qa.md) |
-| [`qa/inspect.py`](../../aborist/qa/inspect.py) | sidecar diagnostic (read-only span classifier) | [qa.md](./qa.md) |
-
-## Distillation
-
-| Module | One-line role | Doc |
-|---|---|---|
-| [`distill/`](../../aborist/distill/) | `Distiller` ABC + `first_sentence` + `tfidf` + runner | [distill.md](./distill.md) |
-
-## Federation (off by default)
-
-| Module | One-line role | Doc |
-|---|---|---|
-| [`mesh/`](../../aborist/mesh/) | identity (Ed25519/X25519), per-epoch roster, AEAD envelope, gossip wire | [../mesh.md](../mesh.md), [../mesh-deploy.md](../mesh-deploy.md) |
-
-## Entry point
-
-| Module | One-line role | Doc |
-|---|---|---|
-| [`cli.py`](../../aborist/cli.py) | `argparse` entrypoint — every `make` target dispatches here | run `aborist --help` or any `make help` target |
-
-## Tickets, design docs, journals
-
-See [`../TICKETS.md`](../TICKETS.md) for the ticket index and the
-list of design-reference docs that aren't tickets.
diff --git a/docs/modules/ingest.md b/docs/modules/ingest.md
deleted file mode 100644
index 916edb6..0000000
--- a/docs/modules/ingest.md
+++ /dev/null
@@ -1,64 +0,0 @@
-# `aborist.ingest`
-
-The bulk-batched pipeline that turns documents from a source into
-Merkle-committed shard storage. Source-agnostic: anything that
-implements `Source.iter_documents()` flows through here.
-
-
-
-## Public API
-
-```python
-from aborist.ingest import ingest_source
-from aborist.sources.wikipedia import WikipediaSqlDump
-
-source = WikipediaSqlDump("/path/to/cur.sql.bz2")
-ingest_source(
- source,
- db_path=Path("~/.aborist/shards/000.db"),
- batch_size=200, # docs per transaction
- progress_every=1000,
-)
-```
-
-## What happens per document
-
-1. **Canonicalize** — `canonicalize(text)`: NFC + ws-collapse +
- strip ends. Pinned by `canonicalization_version='norm-v1'`.
-2. **Chunk** — `Chunker.chunk(canonical_text)` → list of token-
- bounded substrings. Default `tok-512-v1` chunker.
-3. **Hash leaves** — `sha256(0x00 || canonical_chunk_bytes)` per
- chunk.
-4. **Merkle tree** — `MerkleTree.build(leaves).root` →
- `document_root`. Two peers running the same chunker on the same
- canonicalized content compute bit-identical roots.
-5. **Upsert** — `documents` row keyed on `document_root` (idempotent
- re-ingest), `chunks` rows with leaf hashes, `merkle_nodes` for
- proof reconstruction, `edges` per outbound link.
-6. **FTS5** — `chunks_fts` insert with rowid = `chunks.chunk_id`
- so the search-time JOIN lines up.
-7. **Audit event** — one row per ingest batch in `audit_events`,
- chained on `prev_event_hash`.
-
-## Batching discipline
-
-Default `batch_size=200`: balances Python GIL overhead vs SQLite
-transaction commit cost. Lower it (e.g. 50) only to bound peak
-memory on a low-RAM host. Higher (e.g. 1000) for ETL throughput on
-SSD storage when memory isn't tight.
-
-`progress_every` prints a stderr line every N docs so long ingests
-are observable. Use `PYTHONUNBUFFERED=1` for tail-able output.
-
-## Resumability
-
-Idempotent re-ingest: same content + same chunker + same canonicalize
-= same `document_root` = no-op insert. So a crashed ingest can be
-restarted from the source's beginning without duplicating rows.
-
-Different content at the same URI gets a new `document_root` AND a
-`supersedes` edge linking new → old (lossless history).
-
-## Source
-
-[`aborist/ingest.py`](../../aborist/ingest.py)
diff --git a/docs/modules/merkle.md b/docs/modules/merkle.md
deleted file mode 100644
index 28edaaa..0000000
--- a/docs/modules/merkle.md
+++ /dev/null
@@ -1,63 +0,0 @@
-# `aborist.merkle`
-
-Pure Merkle tree + proof primitives. Python port of
-`proxy.unturf.com/pkg/verified/merkle.go` — convention-identical.
-Used everywhere a content-addressable handle is needed: per-chunk
-leaves, document_root, evidence_map_root, run_dag_root, snapshots.
-
-## Conventions (do not silently change)
-
-These match the Go reference & are load-bearing for cross-language
-verification (Go peer ↔ Python peer compute bit-identical roots):
-
-- **Leaf hash:** `sha256(0x00 || canonical_chunk_bytes)`. The `0x00`
- prefix domain-separates leaves from internal nodes.
-- **Internal hash:** `sha256(0x03 || left || right)`. The `0x03`
- prefix is the **non-commutative** combine — `H(L,R) ≠ H(R,L)`.
- Order matters.
-- **Odd-element rule:** when a level has an odd count, the last
- leaf is **self-duplicated** before pairing. NOT zero-padded.
-- **Proof path:** each step carries an explicit `is_left: bool`
- alongside the sibling hash so a verifier knows which side to put
- the sibling on. Never sort siblings lexically — the order tells
- the verifier the tree topology.
-
-## API surface
-
-```python
-from aborist.merkle import MerkleTree, MerkleProof
-
-tree = MerkleTree.build([b"chunk_0_bytes", b"chunk_1_bytes", ...])
-tree.root # bytes(32) — sha256 of the whole tree
-tree.leaves # list[bytes(32)] — leaf hashes in input order
-
-proof = tree.proof_for(leaf_index=2)
-proof.siblings # list[(sibling_hash, is_left)]
-proof.verify(leaf_hash=tree.leaves[2], root=tree.root) # bool
-```
-
-## When to read the source
-
-- Adding a new content-addressable artifact (cores, evidence maps,
- snapshots, run-DAGs all touch this).
-- Cross-language verification debugging (Go peer says one root,
- Python peer says another — the difference is always in canonical
- encoding, ordering, or one of the three prefix bytes above).
-- Performance work — the Python build is ~3× slower than the Go
- reference; if it ever shows up in profiling, that's the file.
-
-## Diagrams
-
-The module graph shows what depends on `merkle.py` (a lot — it's
-substrate):
-
-
-
-The ingest pipeline shows where leaf & root hashes get computed:
-
-
-
-## Source
-
-[`aborist/merkle.py`](../../aborist/merkle.py) ·
-Reference: [`proxy.unturf.com/pkg/verified/merkle.go`](https://git.unturf.com/engineering/unturf/proxy.unturf.com/-/blob/main/pkg/verified/merkle.go)
diff --git a/docs/modules/qa.md b/docs/modules/qa.md
deleted file mode 100644
index 129ad63..0000000
--- a/docs/modules/qa.md
+++ /dev/null
@@ -1,185 +0,0 @@
-# `aborist.qa`
-
-The Q&A pipeline. Question → cache → retrieval → LLM → verify →
-render → cache write. Lives in 9 sub-modules; this page is the
-map.
-
-
-
-## Sub-modules
-
-### `qa.client` — LLM transport
-
-`ChatClient` ABC with three concrete implementations:
-
-- `StubClient` — deterministic test fixture; returns canned
- responses keyed on the input. Used in unit tests to avoid network.
-- `OpenAICompatibleClient` — talks to any OpenAI-shape `/v1/chat/completions`
- endpoint (Hermes-3 on vLLM by default). Includes HTTP retry layer
- (3× exponential backoff on 5xx).
-- (Future) `AnthropicClient` — Claude API direct.
-
-[`aborist/qa/client.py`](../../aborist/qa/client.py)
-
-### `qa.keys` — the 8-dim cache_key
-
-```
-cache_key = sha256(
- source_root | question_hash | model_profile_hash |
- conversation_hash | governance_policy_hash |
- schema_version | canonicalization_version | chunking_version
-)
-```
-
-Two question-hash modes (`strict` vs `equivalence_class`) live here.
-Bumping any of these eight dimensions invalidates prior records on
-lookup. The `verifier_policy_hash` (v9.9 9th dim) is also
-implemented here.
-
-[`aborist/qa/keys.py`](../../aborist/qa/keys.py)
-
-### `qa.runner` — `ask()` for single-doc Q&A
-
-The simplest entry point. Take one document, ask one question, get
-back an answer + audit_mode + cache record. Used by the CLI for
-focused queries against one URI.
-
-[`aborist/qa/runner.py`](../../aborist/qa/runner.py)
-
-### `qa.query` — `query()` for multi-source RAG
-
-The main retrieval entry point. Walks shards, runs FTS5 BM25 with
-AND→OR fallback (with synonym-pool injection in OR mode), filters
-by title relevance with 4 accept paths, reranks by body coverage +
-title boost + source role + title purity, assembles a 60 KB context
-budget, calls the LLM, runs the verifier, persists to
-`providence_cache`.
-
-[`aborist/qa/query.py`](../../aborist/qa/query.py)
-
-### `qa.verify` — the layered verifier
-
-Five strategies run in sequence; first to find evidence classifies:
-1. `quote` — `"..."`-wrapped claims tested verbatim
-2. `span` — bullet/sentence units substring-tested
-3. `entity` — multi-word proper nouns with proximity gating
-4. `paraphrase` — token coverage on prose-shaped spans (≥85%)
-5. `claim_lattice` — pointer-line `[E1,E2]` or JSON; runs **seven
- deterministic hard checks**:
- 1. parser succeeded
- 2. evidence_id resolves
- 3. source_role allowed
- 4. claim text non-empty
- 5. citation coverage threshold
- 6. pointer count cap (trim-and-verify)
- 7. anchor-class warrant (see `qa.warrant`)
-
-The classifier output rolls up into the v9.8 trichotomy
-`audit_mode ∈ {STRICT, HYBRID, UNGROUNDED}`. Display layer (in
-`cli.py`) maps `(audit_mode, violations) → four-rung ladder`.
-
-
-
-[`aborist/qa/verify.py`](../../aborist/qa/verify.py)
-
-### `qa.warrant` — anchor-class warrant
-
-Five lexical anchor classes the verifier composes:
-
-- **Proper-noun** — relation-question shape; at least one
- Title-Case anchor must appear in some cited span
-- **Date** — claim has a 4-digit year + month name; ALL components
- required in some cited span
-- **Entity-list** — entity-list-shape question; ≥1 named entity
- must anchor (demote-don't-reject)
-- **Count** — count-shape question; count token must appear in
- word OR digit form (digit↔word equivalence)
-- **Cause** — why-shape question; ≥1 cause anchor (proper noun OR
- ≥5-char common noun outside stopword pool)
-
-The warrant layer earns proof-path entry by staying **lexical** —
-no NLI, no embeddings. Substring tests over already-canonicalized
-spans. See `docs/concept-relations-design.md` (sibling section)
-for the relationship to retrieval-time synonym expansion.
-
-[`aborist/qa/warrant.py`](../../aborist/qa/warrant.py)
-
-### `qa.evidence` — EvidenceObject + spotlight
-
-Builds the runtime evidence map for claim-lattice modes. Each
-chunk becomes one `EvidenceObject` carrying TWO ids:
-
-- `pointer_id` — short prompt-facing tag (`E1`, `E2`, …)
-- `evidence_id` — content-addressed `E########` (sha256-derived)
-
-The model sees only `pointer_id`s in the prompt; the runtime maps
-to `evidence_id` for the cache & run-DAG (run-stable identity).
-
-The spotlight excerpt picks the load-bearing slice via **density
-rank** — find ALL match positions for ALL claim content tokens,
-pick the position with maximum distinct-token cluster within
-±half-window. Replaces the older first-match-of-longest-token
-approach which lost the load-bearing slice on noisy chunks.
-
-[`aborist/qa/evidence.py`](../../aborist/qa/evidence.py)
-
-### `qa.parse_claims` — pointer-line parser
-
-Walks lines of the model output, pulls every `[E\d+]` and
-`[E\d+,E\d+,…]` bracket payload, returns
-`(claim_text, pointer_ids[])` per line. Lines without a tag get
-`parse_status='NO_EVIDENCE_POINTER'` & count toward the denominator
-so unsourced prose can't smuggle past the verifier.
-
-[`aborist/qa/parse_claims.py`](../../aborist/qa/parse_claims.py)
-
-### `qa.dag` — per-run Merkle DAG
-
-Commits each provenance step independently as a stage hash. Two
-shapes:
-
-- **7-stage (quote mode):** question / retrieval / context /
- prompt / answer / verify / final_label
-- **9-stage (claim-lattice / CTI):** question / retrieval /
- evidence_map / prompt / raw_answer / parsed_claim_lattice /
- verify / render / final_label
-
-The `run_dag_root` is persisted alongside every providence record;
-`run_dag_blob` carries the full `{root, nodes}` JSON so an auditor
-can recompute & verify any step.
-
-[`aborist/qa/dag.py`](../../aborist/qa/dag.py)
-
-### `qa.inspect` — read-only sidecar
-
-Pulls source chunks for a given cache_key & classifies each
-unverified span: `verbatim_in_base` / `verbatim_in_raw_only` /
-`trailing_artifact` / `paraphrase` / `partial_paraphrase` /
-`no_overlap`. Also includes the deflection-detection sidecar
-(subject-anchor heuristic for adversarial-premise topic shift).
-
-**Sidecars never write to `providence_cache` or `audit_events`** —
-they're diagnostic only. That invariant is what keeps `audit_mode`
-a binary classification rather than a soft score.
-
-[`aborist/qa/inspect.py`](../../aborist/qa/inspect.py)
-
-### `qa.concepts` — backwards-compat shim
-
-Delegates to `aborist.concepts` (the corpus-derived synonym/rivalry
-layer). Pre-2026-05-01 the data lived as hand-curated frozensets in
-this file; now it's a per-shard SQLite table. The shim preserves
-the legacy public API (`synonym_expand`, `rivalry_excluded`,
-`has_compare_phrasing`) so call sites in `qa/query.py` didn't have
-to change.
-
-[`aborist/qa/concepts.py`](../../aborist/qa/concepts.py) →
-[`aborist/concepts/`](../../aborist/concepts/)
-
-## Source papers
-
-- Whitepaper §13.8 covers the layered verifier in depth
-- Whitepaper §13.9 covers claim-lattice / CTI mode
-- `docs/cti-architecture.md` is the architecture reference
-- `docs/seven-point-program.md` enumerates the seven hard checks
-- `docs/concept-relations-design.md` covers the synonym layer
diff --git a/docs/modules/search.md b/docs/modules/search.md
deleted file mode 100644
index 4dfe5c9..0000000
--- a/docs/modules/search.md
+++ /dev/null
@@ -1,75 +0,0 @@
-# `aborist.search`
-
-The retrieval primitive. Today's only backend is FTS5 over the
-chunks table; the `SearchBackend` ABC is in place so additional
-backends (BM25 over titles, embedding-based vector search) can be
-added without touching the rest of the QA pipeline.
-
-## `SearchBackend` ABC
-
-```python
-from aborist.search import SearchBackend, AuditMode, Hit
-
-class SearchBackend(ABC):
- @abstractmethod
- def search(self, query: str, limit: int = 20) -> list[Hit]: ...
-```
-
-Each `Hit` carries `(document_root, document_uri, chunk_idx,
-snippet, score, audit_mode, title)`. `audit_mode` is the sticky
-provenance label that tracks how the chunk made it into the index;
-FTS5 backend always sets `UNGROUNDED` (search itself doesn't verify
-anything — that's the QA pipeline's job).
-
-## `FTS5Backend`
-
-Wraps the contentless `chunks_fts` virtual table. Two-mode query:
-
-- **AND-mode (strict, primary):** every content token must appear
- in the doc. Keeps unrelated docs out of the context window.
-- **OR-mode (fallback):** when AND returns 0 hits, fall back to OR
- but **capped to top-5 longest tokens** (proxy for rarity). Long
- topical synonyms fed via `extra_or_tokens` join the pool —
- `neurotechnology` (15 chars) outranks `thoughts` (8) by length
- and surfaces brain-tech titles for vocabulary-mismatch queries.
-
-```python
-from aborist.search import FTS5Backend
-backend = FTS5Backend(conn)
-
-# Plain search
-hits = backend.search("permacomputer", limit=32)
-
-# Search with synonym pool injection (used by qa.query._search_corpus)
-hits = backend.search(
- long_query,
- limit=32,
- extra_or_tokens=synonym_expand(qtokens, shards_dir=shards_dir),
-)
-```
-
-## Stopword & stopword-cap discipline
-
-`_FTS5_STOPWORDS` filters question words (`what`, `tell`, `please`)
-+ generic connectors (`one`, `some`, `another`, `without`, `soon`,
-`currently`) before AND/OR construction. Two principles:
-
-- **Stay in sync with `_TITLE_STOPWORDS`** in `qa/query.py`. A token
- filtered at retrieval time but kept at title-relevance check (or
- vice versa) creates ranking incoherence.
-- **`_OR_FALLBACK_MAX_TOKENS=5`** caps the OR-mode pool. Without
- this, a 19-token OR clause matches millions of docs and forces
- BM25 to rank them all — 13s/shard observed pre-cap. Now 0.25s/shard.
-
-## Snippet building
-
-FTS5 contentless mode means SQLite's built-in `snippet()` and
-`highlight()` return empty. Aborist builds snippets in Python by
-joining `chunks_fts.rowid = chunks.chunk_id`, decompressing the
-chunk content, and locating query tokens locally
-(`_build_snippet`).
-
-## Source
-
-[`aborist/search/fts5.py`](../../aborist/search/fts5.py) ·
-[`aborist/search/__init__.py`](../../aborist/search/__init__.py)
diff --git a/docs/modules/sources.md b/docs/modules/sources.md
deleted file mode 100644
index e8ff2c4..0000000
--- a/docs/modules/sources.md
+++ /dev/null
@@ -1,80 +0,0 @@
-# `aborist.sources`
-
-Corpus producers. Each is a `Source` ABC implementation that yields
-`Document` instances; the standard `ingest.ingest_source(source, db)`
-pipeline takes them from there.
-
-The `Source` ABC lives in [`aborist/source.py`](../../aborist/source.py):
-
-```python
-class Source(ABC):
- @abstractmethod
- def iter_documents(self) -> Iterator[Document]: ...
-```
-
-## Built-in sources
-
-### `wikipedia.py` — Phase III SQL dumps (the canonical bootstrap)
-
-Streams the Wikipedia 2003 `cur` (current revisions) and `old`
-(revision history) SQL dumps. Hand-rolled escape-aware parser
-(no `sqlite3` import — the dump is MySQL syntax). 4× speedup vs
-char-by-char loops via `str.find` + slicing. cProfile any change.
-
-Default Wikipedia 2003-05-16 dump source:
-`https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`.
-robots.txt returned 404 → no rules.
-
-[`aborist/sources/wikipedia.py`](../../aborist/sources/wikipedia.py)
-
-### `wikipedia_xml.py` — Phase IV XML dumps
-
-Modern Wikipedia dump format (`enwiki-YYYYMMDD-pages-articles.xml.bz2`,
-`enwiki-YYYYMMDD-pages-meta-history*.xml.bz2`). Uses `xml.etree.ElementTree.iterparse`
-to stream-parse without loading the whole tree.
-
-[`aborist/sources/wikipedia_xml.py`](../../aborist/sources/wikipedia_xml.py)
-
-### `html_page.py` — single-URL or URL-list HTML ingest
-
-Robots-aware (`urllib.robotparser`). Uses `selectolax` for fast
-HTML parsing (CSS-selector based; ~10× faster than `lxml`). Pulls
-the main body text + every `` as an `Edge` row.
-
-The `edges` rows are what the corpus-derived synonym extractor
-later reads — no separate crawler needed for site-internal link
-graphs.
-
-Optional dep: `pip install '.[html]'` for `selectolax` + `httpx`.
-
-[`aborist/sources/html_page.py`](../../aborist/sources/html_page.py)
-
-### `crawler/` — async BFS web crawl
-
-Verbatim lift of an `AsyncWebFetcher` implementation + an `ingest`
-bridge. BFS-discovers same-domain URLs from a seed, respecting
-`robots.txt` + crawl delays. Captures ETag + Last-Modified per URL
-into `document_http_meta` so a future recrawl can send conditional
-HEAD requests.
-
-[`aborist/sources/crawler/bridge.py`](../../aborist/sources/crawler/bridge.py)
-
-### `grok.py` — xAI Grok export
-
-Reads the `xAI-conversations.json` data export shape. Each
-conversation becomes one `Document`; media prompts are kept
-inline.
-
-[`aborist/sources/grok.py`](../../aborist/sources/grok.py)
-
-### `vcs.py` — git + Mercurial repositories
-
-HEAD walk. Each commit becomes a `Document` (commit message + diff
-stat). The supersedes chain captures commit ancestry as edges.
-
-[`aborist/sources/vcs.py`](../../aborist/sources/vcs.py)
-
-## Source
-
-[`aborist/sources/`](../../aborist/sources/) ·
-[`aborist/source.py`](../../aborist/source.py) (ABC)
diff --git a/docs/modules/store.md b/docs/modules/store.md
deleted file mode 100644
index 188a2f7..0000000
--- a/docs/modules/store.md
+++ /dev/null
@@ -1,81 +0,0 @@
-# `aborist.store`
-
-The v9.8 SQLite schema, the audit chain, and the cross-shard
-read-only view. Every table that holds runtime state lives here.
-
-## Schema overview (per shard)
-
-```
-documents – one row per source document, keyed on document_root
-chunks – per-document chunk content + tier (hot/cold)
-chunks_fts – FTS5 contentless index, rowid = chunks.chunk_id
-merkle_nodes – internal-node hashes for proof reconstruction
-edges – src_root → dst_root link graph (wikilink, href, …)
-derivations – core_root ← src_root with proof_blob (Merkle)
-providence_cache – Q&A records keyed on the v9.8 8-dim cache_key
-audit_events – linear chain; event_hash = sha256(prev || canonical(body))
-falsifications – record_id → state transition + reason + actor
-snapshots – named corpus roots (one hash names a forest)
-document_http_meta – ETag + Last-Modified for crawler conditional fetches
-concept_relations – per-shard synonym/rivalry/category/antonym overlay
-mesh_* – federation tables (off by default)
-```
-
-## v9.8 invariants (do not break)
-
-- **Aborist is a v9.8 store.** Every providence record carries the
- full **8-dim cache_key**: `source_root | question_hash |
- model_profile_hash | conversation_hash | governance_policy_hash |
- schema_version | canonicalization_version | chunking_version`.
- Bumping any one invalidates prior records on lookup.
-- **`falsification_state ∈ {live, failed, stale, quarantined}`.**
- Cache lookups must filter on `state='live'`. Drift detection
- flips to `stale`.
-- **Audit chain.** Every state-changing op writes one row in
- `audit_events` with `event_hash = sha256(prev_event_hash ||
- canonical(body))`. Chain integrity is verified in
- `make analyze-shards`. **Never insert into `audit_events`
- directly — use `aborist.store.append_audit`.**
-- **Cores never evict.** `evict_to_cold` only touches `kind='surface'`.
-- **Idempotent re-ingest.** Same content → same `document_root` →
- no-op insert. Same URI + different content → new doc + `supersedes`
- edge linking new → old (lossless history).
-
-## API surface
-
-```python
-from aborist.store import (
- connect, # writable connection to a single shard
- connect_query, # read-only UNION view across all shards
- discover_shards, # list *.db files in a shards_dir
- transaction, # BEGIN IMMEDIATE / COMMIT / ROLLBACK context manager
- get_meta, set_meta,
- append_audit, # the ONLY way to write audit_events
-)
-```
-
-Cross-shard reads use `connect_query(shards_dir=...)` which ATTACHes
-every `*.db` and creates UNION views over the shardable tables
-(`documents`, `chunks`, `merkle_nodes`, `edges`, `derivations`,
-`providence_cache`, `audit_events`, `falsifications`,
-`concept_relations`).
-
-## Performance pragmas
-
-`connect()` applies these per-connection:
-- `journal_mode=WAL` (set in SCHEMA_SQL once at first ingest)
-- `synchronous=NORMAL` (skip per-commit fsync; safe under WAL)
-- `cache_size=-65536` (64 MB page cache)
-- `temp_store=MEMORY` (no /tmp churn for temp tables)
-
-Don't downgrade to `synchronous=FULL` without a measured reason —
-costs ~5× throughput.
-
-## Diagrams
-
-
-
-
-## Source
-
-[`aborist/store.py`](../../aborist/store.py)
diff --git a/docs/modules/wikitext.md b/docs/modules/wikitext.md
deleted file mode 100644
index f4f5e3c..0000000
--- a/docs/modules/wikitext.md
+++ /dev/null
@@ -1,46 +0,0 @@
-# `aborist.wikitext`
-
-A single function: `to_base(raw)`. Converts MediaWiki wikitext to
-plain prose deterministically.
-
-```python
-from aborist.wikitext import to_base, BASE_VERSION
-
-prose = to_base("[[The Beatles]] are an [[English rock band]] from [[Liverpool]].")
-# → "The Beatles are an English rock band from Liverpool."
-```
-
-## Why it exists
-
-The corpus stores raw wikitext (so the link graph is recoverable
-on demand) but the LLM and verifier both want plain prose. Reasons:
-
-1. **Token efficiency.** Wikipedia chunks ship to Hermes with ~43%
- fewer tokens after wikitext-strip — bigger context window for
- the same chars budget.
-2. **Verbatim citation.** The model can quote source paragraphs
- verbatim instead of escaping `[[wikilinks]]`. The verifier's
- substring test then matches cleanly.
-3. **Pinned identity.** `BASE_VERSION='wikitext-base-v1'` lives in
- `policy["base_version"]`, which folds into
- `governance_policy_hash`. Bumping `BASE_VERSION` invalidates
- every prior cache record on next lookup — same discipline as
- `chunking_version` and `canonicalization_version`.
-
-## Hot-path discipline
-
-`to_base()` runs on the assembled context **before** the LLM call
-in `aborist/qa/runner.py` and `aborist/qa/query.py`, **and again
-inside `verify_quotes`** so the verifier compares like-against-like.
-Both sides see prose.
-
-## Optional dependency
-
-Backed by `mwparserfromhell`. Install via `pip install '.[wikitext]'`
-to enable. Without the dep, `_wikitext_to_base = None` and
-`policy["base_version"] = None` — graceful fallback leaves raw
-wikitext in both context and verifier (works, just less efficient).
-
-## Source
-
-[`aborist/wikitext.py`](../../aborist/wikitext.py)