# Aborist module reference

One page per top-level package + one diagram per major data flow.
This index is the entry point — every module page links back here
and to the relevant diagrams.

## Diagrams

| Diagram | What it shows | File |
|---|---|---|
| **Module graph** | Top-level packages & how they import each other | [`aborist-modules.svg`](../diagrams/aborist-modules.svg) ([dot](../diagrams/aborist-modules.dot)) |
| **Query pipeline** | Question → cache → retrieval → LLM → verify → render | [`query-pipeline.svg`](../diagrams/query-pipeline.svg) ([dot](../diagrams/query-pipeline.dot)) |
| **Ingest pipeline** | Source document → Merkle-committed shard | [`ingest-pipeline.svg`](../diagrams/ingest-pipeline.svg) ([dot](../diagrams/ingest-pipeline.dot)) |
| **Verifier ladder** | (audit_mode, violations) → display rung | [`verifier-ladder.svg`](../diagrams/verifier-ladder.svg) ([dot](../diagrams/verifier-ladder.dot)) |
| **Mesh data flow** | Federation: roster, gossip, AEAD envelope | [`mesh-data-flow.svg`](../diagrams/mesh-data-flow.svg) |
| **Mesh epoch lifecycle** | Epoch advance via add/kick/rotate | [`mesh-epoch-lifecycle.svg`](../diagrams/mesh-epoch-lifecycle.svg) |
| **Mesh identity stack** | Ed25519 sign + X25519 DH key derivation | [`mesh-identity-stack.svg`](../diagrams/mesh-identity-stack.svg) |
| **Mesh secret envelope** | AEAD-wrapped epoch secret per peer | [`mesh-secret-envelope.svg`](../diagrams/mesh-secret-envelope.svg) |
| **Mesh group decisions** | Membership change voting & quorum | [`mesh-group-decisions.svg`](../diagrams/mesh-group-decisions.svg) |

Render diagrams locally:

```
make docs   # runs `dot -Tsvg` and `-Tpng` on every docs/diagrams/*.dot
```

## Substrate (no SQL, pure data structures)

| Module | One-line role | Doc |
|---|---|---|
| [`merkle.py`](../../aborist/merkle.py) | Merkle tree + proof — Python port of `proxy.unturf.com/pkg/verified/merkle.go` | [merkle.md](./merkle.md) |
| [`document.py`](../../aborist/document.py) | `Document`, `Edge`, `Chunker` (default `tok-512-v1`) | [document.md](./document.md) |
| [`wikitext.py`](../../aborist/wikitext.py) | `to_base()` — wikitext → plain prose, BASE_VERSION-pinned | [wikitext.md](./wikitext.md) |

## Storage

| Module | One-line role | Doc |
|---|---|---|
| [`store.py`](../../aborist/store.py) | v9.8 SQLite schema + audit chain helpers | [store.md](./store.md) |
| [`ingest.py`](../../aborist/ingest.py) | normalize → chunk → merkle → upsert (bulk-batched) | [ingest.md](./ingest.md) |
| [`evict.py`](../../aborist/evict.py) | hot ↔ cold tier transitions; rehydrate via source | [evict.md](./evict.md) |

## Sources (corpus producers)

| Module | One-line role | Doc |
|---|---|---|
| [`sources/wikipedia.py`](../../aborist/sources/wikipedia.py) | Wikipedia 2003 cur + old SQL dumps (bz2-streamed) | [sources.md](./sources.md) |
| [`sources/wikipedia_xml.py`](../../aborist/sources/wikipedia_xml.py) | Phase IV XML dumps (iterparse, page + history) | [sources.md](./sources.md) |
| [`sources/html_page.py`](../../aborist/sources/html_page.py) | URL list + selectolax + httpx (robots-aware) | [sources.md](./sources.md) |
| [`sources/crawler/`](../../aborist/sources/crawler/) | verbatim AsyncWebFetcher lift + ingest bridge | [sources.md](./sources.md) |
| [`sources/grok.py`](../../aborist/sources/grok.py) | xAI data export (conversations + media prompts) | [sources.md](./sources.md) |
| [`sources/vcs.py`](../../aborist/sources/vcs.py) | git + Mercurial repos (HEAD walk, supersedes chain) | [sources.md](./sources.md) |

## Search & retrieval

| Module | One-line role | Doc |
|---|---|---|
| [`search/`](../../aborist/search/) | FTS5 backend + `SearchBackend` ABC + `AuditMode` enum | [search.md](./search.md) |
| [`concepts/`](../../aborist/concepts/) | Per-shard `concept_relations` synonym/rivalry overlay | [concepts.md](./concepts.md) |

## Q&A pipeline

| Module | One-line role | Doc |
|---|---|---|
| [`qa/keys.py`](../../aborist/qa/keys.py) | 8-dim cache_key + `question_hash` | [qa.md](./qa.md) |
| [`qa/client.py`](../../aborist/qa/client.py) | `ChatClient` + `StubClient` + `OpenAICompatibleClient` | [qa.md](./qa.md) |
| [`qa/runner.py`](../../aborist/qa/runner.py) | `ask()`: single-doc Q&A + cache + verify | [qa.md](./qa.md) |
| [`qa/query.py`](../../aborist/qa/query.py) | `query()`: multi-source RAG + concept overlay | [qa.md](./qa.md) |
| [`qa/verify.py`](../../aborist/qa/verify.py) | quote/span/entity/paraphrase + claim_lattice (7 hard checks) | [qa.md](./qa.md) |
| [`qa/warrant.py`](../../aborist/qa/warrant.py) | 5 anchor classes (proper-noun · date · count · entity-list · cause) | [qa.md](./qa.md) |
| [`qa/evidence.py`](../../aborist/qa/evidence.py) | `EvidenceObject` + spotlight excerpt (density rank) | [qa.md](./qa.md) |
| [`qa/parse_claims.py`](../../aborist/qa/parse_claims.py) | pointer-line parser (`claim. [E1,E2]`) | [qa.md](./qa.md) |
| [`qa/dag.py`](../../aborist/qa/dag.py) | per-run Merkle-DAG (7-stage quote / 9-stage CTI) | [qa.md](./qa.md) |
| [`qa/inspect.py`](../../aborist/qa/inspect.py) | sidecar diagnostic (read-only span classifier) | [qa.md](./qa.md) |

## Distillation

| Module | One-line role | Doc |
|---|---|---|
| [`distill/`](../../aborist/distill/) | `Distiller` ABC + `first_sentence` + `tfidf` + runner | [distill.md](./distill.md) |

## Federation (off by default)

| Module | One-line role | Doc |
|---|---|---|
| [`mesh/`](../../aborist/mesh/) | identity (Ed25519/X25519), per-epoch roster, AEAD envelope, gossip wire | [../mesh.md](../mesh.md), [../mesh-deploy.md](../mesh-deploy.md) |

## Entry point

| Module | One-line role | Doc |
|---|---|---|
| [`cli.py`](../../aborist/cli.py) | `argparse` entrypoint — every `make` target dispatches here | run `aborist --help` or any `make help` target |

## Tickets, design docs, journals

See [`../TICKETS.md`](../TICKETS.md) for the ticket index and the
list of design-reference docs that aren't tickets.
