engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: consolidate docs/modules/* → docs/modules.md (single file)

Browse source 
12 per-module files + index = 13 files of 50-185 lines each =
1,064 lines of API reference scattered across a directory.
Each per-module file had real meat (API examples, ASCII tree,
conventions) but the cognitive cost of 'which file is this in?'
outweighed the navigation benefit.

Built via concatenation + patch-fix:

  - cat index.md + per-module files in topological order
  - rewrite ../diagrams/  -> diagrams/  (relative to docs/modules.md)
  - rewrite ../../aborist/ -> ../aborist/
  - rewrite ../TICKETS.md  -> TICKETS.md, ../mesh.md -> mesh.md, etc.
  - inter-module links (./<name>.md, <name>.md) -> #<name>-py anchors
  - demote per-module H1 -> H2, H2 -> H3, etc., so the wrapper H1
    is the only top-level heading
  - de-dup the index.md's (now-H2) 'Aborist module reference'
    header against the wrapper, replace with 'Diagrams index'
  - inject explicit <a id="<name>-py"></a> anchors after each
    module's H2 so the TOC links resolve regardless of GitHub's
    auto-slug rules
  - polish TOC link text: '[merkle.md](#merkle-py)' -> '[↓](#...)'
    (the '.md' suffix made no sense once it's an in-doc anchor)

References updated:
  - README.md (×2)

Net: 1,124 single-file lines vs 1,064 across 13 files. Slightly
larger because of the patch-fix scaffolding (anchors + section
markers), but one Cmd-F covers everything.

751/34 tests still pass.
This commit is contained in:
russell@unturf.com 2026-05-02 13:23:43 -04:00
parent 7444f106fc
commit ea82e4288c
No known key found for this signature in database
14 changed files with 1126 additions and 1066 deletions
Download patch file Download diff file Expand all files Collapse all files

4
README.md
View file

@ -414,7 +414,7 @@ make activity ACTIVITY_LIMIT=20
![Aborist module graph](docs/diagrams/aborist-modules.svg)
For per-module reference docs see [`docs/modules/`](docs/modules/index.md).
For per-module reference docs see [`docs/modules.md`](docs/modules.md).
Full pipeline diagrams live in [`docs/diagrams/`](docs/diagrams/) — query,
ingest, verifier-ladder, plus the mesh sub-diagrams. Render any change
with `make docs`.
@ -478,7 +478,7 @@ aborist/
└── cli.py argparse entrypoint (the make targets call into here).
```
For diagrams of how these modules wire together, see `docs/diagrams/` (rendered SVGs of the module graph, query pipeline, ingest pipeline, mesh wire, & verifier ladder). For per-module reference docs see `docs/modules/` — every top-level package has its own page.
For diagrams of how these modules wire together, see `docs/diagrams/` (rendered SVGs of the module graph, query pipeline, ingest pipeline, mesh wire, & verifier ladder). For per-module reference docs see `docs/modules.md` — every top-level package has its own page.
Source papers (read first if confused):

1124
docs/modules.md Normal file
View file

File diff suppressed because it is too large Load diff

179
docs/modules/concepts.md
View file

@ -1,179 +0,0 @@
# `aborist.concepts`
Per-shard `concept_relations` SQLite table — the corpus-derived
synonym, rivalry, antonym & category overlay that replaces the
hand-curated frozensets that lived in `aborist/qa/concepts.py`
through April 2026.
**Full architecture rationale lives in
[`../concept-relations-design.md`](../concept-relations-design.md)**
including the 1.6% storage-tax measurement and the three-compactions-
considered-and-rejected analysis. This page is the API reference.
## Sub-modules
### `concepts.store` — append-only CRUD
```python
from aborist.concepts.store import (
add_concept_relation, # idempotent INSERT OR IGNORE
concept_relations_for_token, # read all relations for a token
purge_by_evidence_kind, # the only DELETE path
list_evidence_kinds, # diagnostic
RELATION_KINDS, # ('synonym','antonym','rivalry','category')
)
```
UNIQUE on `(source_root, relation_kind, token, target, evidence_kind)`
makes re-derivation idempotent. `purge_by_evidence_kind` lets an
operator revoke a single extractor's output without touching manual
or other-extractor rows.
[`aborist/concepts/store.py`](../../aborist/concepts/store.py)
### `concepts.query` — cross-shard lookup
```python
from aborist.concepts import (
synonym_expand, # query tokens → expanded set
rivalry_excluded, # query tokens → tokens to drop from results
has_compare_phrasing, # bool — does the query say "vs", "compare", etc.
invalidate_cache, # drop the per-process LRU
)
expanded = synonym_expand({"thoughts"}, shards_dir=shards_dir)
# → {"thoughts", "telepathy", "neurotechnology", "mind", "cognition", ...}
```
Two synonym indices are loaded:
- **`manual_index`** — `manual_legacy` + `manual` rows. Curated;
always expanded regardless of per-token degree. Captures the
brain-tech / AMD-family / Mac / Linux / etc. seed groups.
- **`derived_index`** — `link_reciprocity` & other corpus-derived
edges. Subject to **`MAX_NEIGHBORS_PER_TOKEN=8`** cap because the
Wikipedia link graph carries topic-adjacency noise on generic
tokens (person, thoughts, language).
Overall **`MAX_TOTAL_TOKENS=50`** cap on expanded set bounds the
SQL clause count downstream so retrieval stays sub-second.
A per-process LRU keyed on shard mtime avoids re-loading the index
on every query (290k rows across 4 shards loads in ~1.8s cold).
[`aborist/concepts/query.py`](../../aborist/concepts/query.py)
### `concepts.extract` — pluggable extractor framework
```python
from aborist.concepts.extract import (
EXTRACTORS, # registry: evidence_kind → callable
link_reciprocity_synonym, # built-in extractor
)
# Run an extractor against a shard:
result = link_reciprocity_synonym(conn, derived_from="backfill@2026-05-01")
# → {"reciprocal_pairs": N, "synonyms_inserted": M, "synonyms_skipped": K}
```
Each extractor walks the shard's existing rows (`documents`,
`chunks`, `edges`, `derivations`) — **no new crawler needed** — &
emits concept relations under a stable `evidence_kind` string that
supports targeted purge.
The built-in `link_reciprocity_synonym` reads the existing `edges`
table for reciprocal A↔B link pairs and emits a synonym edge between
every (title-token-of-A, title-token-of-B) pair. Works for
Wikipedia (See-also bidirectional), HTML site internal links
(russell.ballestrini.net pattern), or any document graph with
bidirectional links. Title-tokens are filtered to ≥4 chars +
stopword-stripped.
To add a new extractor:
1. Implement `(conn, *, derived_from) -> dict[str, int]` that calls
`add_concept_relation` for each finding.
2. Pick a stable `evidence_kind` string.
3. Register in `EXTRACTORS`.
[`aborist/concepts/extract.py`](../../aborist/concepts/extract.py)
### `concepts.seed` — legacy frozenset migration
One-shot migration of the 8 hand-curated frozenset groups
(AMD-family, Intel-family, HTTP, FTP, Mac, Windows, Linux,
brain-tech) to `evidence_kind='manual_legacy'` rows.
Writes **clique edges** within each group — every `(a, b)` pair —
so any member retrieves every other member (preserves the legacy
frozenset semantic where lookups didn't depend on which token in
the group was the anchor).
```python
from aborist.concepts.seed import seed_legacy_concepts
result = seed_legacy_concepts(conn)
# → {"synonyms_inserted": N, "rivalries_inserted": M, "skipped": K}
```
Idempotent — re-running adds nothing if all the rows already exist.
[`aborist/concepts/seed.py`](../../aborist/concepts/seed.py)
## Data model
```sql
CREATE TABLE concept_relations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source_root TEXT NOT NULL,
relation_kind TEXT NOT NULL CHECK (relation_kind IN
('synonym','antonym','rivalry','category')),
token TEXT NOT NULL,
target TEXT NOT NULL,
evidence_kind TEXT NOT NULL,
confidence REAL NOT NULL DEFAULT 1.0,
derived_at INTEGER NOT NULL,
derived_from TEXT,
UNIQUE (source_root, relation_kind, token, target, evidence_kind)
);
```
**Append-only by construction.** Re-running an extractor adds
nothing if every relation already exists. No `UPDATE` path; only
`add_concept_relation` (insert) and `purge_by_evidence_kind`
(targeted delete).
**Per-shard storage.** Concept relations live in the shard whose
document derived them. Mesh sync moves shards between peers;
relations come along.
**Orthogonal to Merkle.** Writes to `concept_relations` NEVER
affect `document_root`, `chunk_root`, or `cache_key`. Backfilling
relations is safe across the entire corpus without invalidating
any cached answer or breaking any audit chain.
## Storage cost — measured
Backfill on 4 wiki shards (3.47M docs, 10.75M resolved edges):
| Shard | Reciprocal pairs | Synonyms | Storage |
|---|---:|---:|---:|
| 000.db | 13,562 | 71,288 | 23.52 MB |
| 001.db | 14,078 | 73,351 | 24.20 MB |
| 002.db | 13,708 | 72,576 | 23.92 MB |
| 003.db | 13,800 | 72,633 | 23.94 MB |
| **total** | **55,148** | **289,848** | **95.58 MB** |
**1.6% storage tax** on the 6 GB corpus. Backfill takes ~4 min
wall-clock total. Cost is paid once at backfill; every retrieval-
time lookup benefits.
## Diagrams
![module graph](../diagrams/aborist-modules.svg)
![query pipeline](../diagrams/query-pipeline.svg)
## Source
- [`aborist/concepts/`](../../aborist/concepts/) — package
- [`docs/concept-relations-design.md`](../concept-relations-design.md) — full design doc
- Whitepaper §13.4.11 — public-facing summary

76
docs/modules/distill.md
View file

@ -1,76 +0,0 @@
# `aborist.distill`
Surface → core distillation. Takes a set of surface documents
(the original ingest layer) and produces "core" documents — shorter,
more focused, Merkle-bound back to their contributing surface
chunks via per-chunk inclusion proofs.
The "trees and forests of cross-linked information" tagline aborist
takes its name from comes from this layer: planet-toward-center
compression where each layer of cores derives from the previous,
recursively.
## Layered design
```
distill/
├── base.py Distiller ABC + DistillationResult dataclass
├── first_sentence.py no-ML stub: take the first sentence of each doc
├── tfidf.py pure-Python TF-IDF top-keyword extraction
└── runner.py batched distillation + per-contrib-chunk proofs
```
## `Distiller` ABC
```python
from aborist.distill import Distiller, DistillationResult
class Distiller(ABC):
@abstractmethod
def distill(self, docs: list[Document]) -> DistillationResult: ...
```
Each `DistillationResult` carries the new core's content +
references to every contributing surface chunk by `(document_root,
chunk_root)`. The runner writes one `derivations` row per core,
with `proof_blob = json.dumps(per_chunk_inclusion_proofs)`.
## Built-in distillers
### `FirstSentenceDistiller` (no-ML stub)
Take the first sentence of each input doc, concatenate. Used as a
sanity-check for the pipeline + a baseline for measuring the
benefit of richer distillers.
### `TfidfKeywordDistiller`
Pure-Python TF-IDF. Computes term frequencies across the input
doc set + inverse document frequencies; emits the top-K terms per
doc as the core's content. The "permacomputer" neologism case
(every Grok conversation has the word, no Wikipedia article does)
is the canonical TF-IDF win — surfaces the topic that title-search
can't catch.
## Why distill
Three use cases:
1. **Retrieval signal.** Cores feed the third accept path in
`_filter_by_title_relevance` — `core_match_roots` (TF-IDF top-
keywords contain a query token). Closes the gap for neologisms
that never make Wikipedia titles but ARE distinctive.
2. **Hot/cold tier discipline.** `evict_to_cold` only touches
`kind='surface'` — cores never evict. Distilling surface to
cores then evicting surfaces gives a "long tail keeps small
cache" pattern with full provenance preserved.
3. **Recursive abstraction.** Cores can themselves be distilled
into shorter cores. Each generation Merkle-binds back to the
previous via `derivations.proof_blob` — the audit chain stays
intact across an arbitrary distillation depth.
## Source
[`aborist/distill/`](../../aborist/distill/)

68
docs/modules/document.md
View file

@ -1,68 +0,0 @@
# `aborist.document`
The data structures every source produces and every storage layer
consumes. Three core types: `Document`, `Edge`, `Chunker`.
## `Document`
A single ingest unit: a Wikipedia article, an HTML page, a Grok
conversation, a git commit message, etc. Carries both the raw
content AND the version tags that determine its identity:
```python
@dataclass(frozen=True)
class Document:
document_uri: str # canonical URI (or stable surrogate for non-URI sources)
raw_content: str # source-of-truth bytes pre-canonicalization
kind: str # 'surface' / 'core' / 'visual' / etc.
chunking_version: str # e.g. 'tok-512-v1' — pinned by the Chunker
canonicalization_version: str # e.g. 'norm-v1' — pinned by canonicalize()
schema_version: str # e.g. 'v9.8.0' — store schema generation
title: str | None = None
edges: list[Edge] = () # outbound link graph
metadata: dict = ... # source-specific opaque payload
```
`document_root` is computed at ingest time as the Merkle root over
the canonicalized chunks. Two peers ingesting the same source +
running the same `chunking_version` + `canonicalization_version`
get bit-identical `document_root`s — the v9.8 admissibility property.
## `Edge`
One outbound link. `aborist/sources/wikipedia.py` emits one Edge
per `[[wikilink]]`; `aborist/sources/html_page.py` emits one per
`<a href>`. The link graph IS the corpus topology — `concepts/extract.py`
later reads `edges` rows to derive synonym relations from
reciprocal links (no separate crawler needed).
```python
@dataclass(frozen=True)
class Edge:
src_root: str # source document_root
dst_uri: str # always present
dst_root: str # '' (unresolved) until the dst doc is also ingested
edge_type: str # 'wikilink' / 'href' / 'citation' / 'derived_from' / ...
anchor: str # chunk index or fragment, '' if N/A
```
## `Chunker`
ABC with one method `chunk(text: str) -> list[str]`. Default impl
is `TokenChunker` (`name='tok-512-v1'`) — splits on token-rough
windows so the resulting chunks are predictable for downstream FTS5
indexing & for the LLM context budget.
**Changing the chunker bumps `chunking_version` AND stales every
prior cache record** (chunking is one of the 8 cache_key dimensions).
Don't redefine `tok-512-v1`; add a new chunker as a new `name`
instead.
## Diagrams
![module graph](../diagrams/aborist-modules.svg)
![ingest pipeline](../diagrams/ingest-pipeline.svg)
## Source
[`aborist/document.py`](../../aborist/document.py)

50
docs/modules/evict.md
View file

@ -1,50 +0,0 @@
# `aborist.evict`
Hot ↔ cold tier transitions. The corpus is large (3.47M Wikipedia
docs); not every chunk fits in working memory. `evict.py` is the
mechanism that moves rarely-touched chunks to a cold tier (still
indexed, just stored separately) and rehydrates them on demand
from the original source.
## API surface
```python
from aborist.evict import evict_to_cold, rehydrate
# Move chunks unused for >threshold days to cold tier
evict_to_cold(conn, max_age_days=90, max_evictions=10000)
# Pull a cold chunk back to hot from its original source
rehydrate(conn, document_root="abc123...")
```
## Invariant: cores never evict
`evict_to_cold` filters `WHERE kind='surface'`. Cores are always
hot — they're the long-tail-friendly compression layer that justifies
evicting their underlying surfaces. Evicting cores would defeat the
purpose.
## v9.8 falsification on drift
When `rehydrate()` re-fetches a document and the recomputed
`document_root` differs from the stored one, the source has
changed since ingest (Wikipedia article was edited, HTML page was
republished, etc.). The cache record's `falsification_state` flips
from `live` to `stale` — every providence record keyed on that
`source_root` is no longer admissible to lookups.
This is the **drift-detection-as-falsification** discipline: cache
hits don't blindly trust historical answers; they trust answers
that the SAME source still grounds.
## Tier values
`chunks.tier ∈ {'hot', 'cold'}`. Hot chunks live in `chunks.content`;
cold chunks live with NULL `content` and a `cold_uri` pointing at
the source. The QA pipeline's chunk-fetch path checks `tier`; on
'cold', it triggers `rehydrate` before continuing.
## Source
[`aborist/evict.py`](../../aborist/evict.py)

97
docs/modules/index.md
View file

@ -1,97 +0,0 @@
# Aborist module reference
One page per top-level package + one diagram per major data flow.
This index is the entry point — every module page links back here
and to the relevant diagrams.
## Diagrams
| Diagram | What it shows | File |
|---|---|---|
| **Module graph** | Top-level packages & how they import each other | [`aborist-modules.svg`](../diagrams/aborist-modules.svg) ([dot](../diagrams/aborist-modules.dot)) |
| **Query pipeline** | Question → cache → retrieval → LLM → verify → render | [`query-pipeline.svg`](../diagrams/query-pipeline.svg) ([dot](../diagrams/query-pipeline.dot)) |
| **Ingest pipeline** | Source document → Merkle-committed shard | [`ingest-pipeline.svg`](../diagrams/ingest-pipeline.svg) ([dot](../diagrams/ingest-pipeline.dot)) |
| **Verifier ladder** | (audit_mode, violations) → display rung | [`verifier-ladder.svg`](../diagrams/verifier-ladder.svg) ([dot](../diagrams/verifier-ladder.dot)) |
| **Mesh data flow** | Federation: roster, gossip, AEAD envelope | [`mesh-data-flow.svg`](../diagrams/mesh-data-flow.svg) |
| **Mesh epoch lifecycle** | Epoch advance via add/kick/rotate | [`mesh-epoch-lifecycle.svg`](../diagrams/mesh-epoch-lifecycle.svg) |
| **Mesh identity stack** | Ed25519 sign + X25519 DH key derivation | [`mesh-identity-stack.svg`](../diagrams/mesh-identity-stack.svg) |
| **Mesh secret envelope** | AEAD-wrapped epoch secret per peer | [`mesh-secret-envelope.svg`](../diagrams/mesh-secret-envelope.svg) |
| **Mesh group decisions** | Membership change voting & quorum | [`mesh-group-decisions.svg`](../diagrams/mesh-group-decisions.svg) |
Render diagrams locally:
```
make docs # runs `dot -Tsvg` and `-Tpng` on every docs/diagrams/*.dot
```
## Substrate (no SQL, pure data structures)
| Module | One-line role | Doc |
|---|---|---|
| [`merkle.py`](../../aborist/merkle.py) | Merkle tree + proof — Python port of `proxy.unturf.com/pkg/verified/merkle.go` | [merkle.md](./merkle.md) |
| [`document.py`](../../aborist/document.py) | `Document`, `Edge`, `Chunker` (default `tok-512-v1`) | [document.md](./document.md) |
| [`wikitext.py`](../../aborist/wikitext.py) | `to_base()` — wikitext → plain prose, BASE_VERSION-pinned | [wikitext.md](./wikitext.md) |
## Storage
| Module | One-line role | Doc |
|---|---|---|
| [`store.py`](../../aborist/store.py) | v9.8 SQLite schema + audit chain helpers | [store.md](./store.md) |
| [`ingest.py`](../../aborist/ingest.py) | normalize → chunk → merkle → upsert (bulk-batched) | [ingest.md](./ingest.md) |
| [`evict.py`](../../aborist/evict.py) | hot ↔ cold tier transitions; rehydrate via source | [evict.md](./evict.md) |
## Sources (corpus producers)
| Module | One-line role | Doc |
|---|---|---|
| [`sources/wikipedia.py`](../../aborist/sources/wikipedia.py) | Wikipedia 2003 cur + old SQL dumps (bz2-streamed) | [sources.md](./sources.md) |
| [`sources/wikipedia_xml.py`](../../aborist/sources/wikipedia_xml.py) | Phase IV XML dumps (iterparse, page + history) | [sources.md](./sources.md) |
| [`sources/html_page.py`](../../aborist/sources/html_page.py) | URL list + selectolax + httpx (robots-aware) | [sources.md](./sources.md) |
| [`sources/crawler/`](../../aborist/sources/crawler/) | verbatim AsyncWebFetcher lift + ingest bridge | [sources.md](./sources.md) |
| [`sources/grok.py`](../../aborist/sources/grok.py) | xAI data export (conversations + media prompts) | [sources.md](./sources.md) |
| [`sources/vcs.py`](../../aborist/sources/vcs.py) | git + Mercurial repos (HEAD walk, supersedes chain) | [sources.md](./sources.md) |
## Search & retrieval
| Module | One-line role | Doc |
|---|---|---|
| [`search/`](../../aborist/search/) | FTS5 backend + `SearchBackend` ABC + `AuditMode` enum | [search.md](./search.md) |
| [`concepts/`](../../aborist/concepts/) | Per-shard `concept_relations` synonym/rivalry overlay | [concepts.md](./concepts.md) |
## Q&A pipeline
| Module | One-line role | Doc |
|---|---|---|
| [`qa/keys.py`](../../aborist/qa/keys.py) | 8-dim cache_key + `question_hash` | [qa.md](./qa.md) |
| [`qa/client.py`](../../aborist/qa/client.py) | `ChatClient` + `StubClient` + `OpenAICompatibleClient` | [qa.md](./qa.md) |
| [`qa/runner.py`](../../aborist/qa/runner.py) | `ask()`: single-doc Q&A + cache + verify | [qa.md](./qa.md) |
| [`qa/query.py`](../../aborist/qa/query.py) | `query()`: multi-source RAG + concept overlay | [qa.md](./qa.md) |
| [`qa/verify.py`](../../aborist/qa/verify.py) | quote/span/entity/paraphrase + claim_lattice (7 hard checks) | [qa.md](./qa.md) |
| [`qa/warrant.py`](../../aborist/qa/warrant.py) | 5 anchor classes (proper-noun · date · count · entity-list · cause) | [qa.md](./qa.md) |
| [`qa/evidence.py`](../../aborist/qa/evidence.py) | `EvidenceObject` + spotlight excerpt (density rank) | [qa.md](./qa.md) |
| [`qa/parse_claims.py`](../../aborist/qa/parse_claims.py) | pointer-line parser (`claim. [E1,E2]`) | [qa.md](./qa.md) |
| [`qa/dag.py`](../../aborist/qa/dag.py) | per-run Merkle-DAG (7-stage quote / 9-stage CTI) | [qa.md](./qa.md) |
| [`qa/inspect.py`](../../aborist/qa/inspect.py) | sidecar diagnostic (read-only span classifier) | [qa.md](./qa.md) |
## Distillation
| Module | One-line role | Doc |
|---|---|---|
| [`distill/`](../../aborist/distill/) | `Distiller` ABC + `first_sentence` + `tfidf` + runner | [distill.md](./distill.md) |
## Federation (off by default)
| Module | One-line role | Doc |
|---|---|---|
| [`mesh/`](../../aborist/mesh/) | identity (Ed25519/X25519), per-epoch roster, AEAD envelope, gossip wire | [../mesh.md](../mesh.md), [../mesh-deploy.md](../mesh-deploy.md) |
## Entry point
| Module | One-line role | Doc |
|---|---|---|
| [`cli.py`](../../aborist/cli.py) | `argparse` entrypoint — every `make` target dispatches here | run `aborist --help` or any `make help` target |
## Tickets, design docs, journals
See [`../TICKETS.md`](../TICKETS.md) for the ticket index and the
list of design-reference docs that aren't tickets.

64
docs/modules/ingest.md
View file

@ -1,64 +0,0 @@
# `aborist.ingest`
The bulk-batched pipeline that turns documents from a source into
Merkle-committed shard storage. Source-agnostic: anything that
implements `Source.iter_documents()` flows through here.
![ingest pipeline](../diagrams/ingest-pipeline.svg)
## Public API
```python
from aborist.ingest import ingest_source
from aborist.sources.wikipedia import WikipediaSqlDump
source = WikipediaSqlDump("/path/to/cur.sql.bz2")
ingest_source(
source,
db_path=Path("~/.aborist/shards/000.db"),
batch_size=200, # docs per transaction
progress_every=1000,
)
```
## What happens per document
1. **Canonicalize** — `canonicalize(text)`: NFC + ws-collapse +
strip ends. Pinned by `canonicalization_version='norm-v1'`.
2. **Chunk** — `Chunker.chunk(canonical_text)` → list of token-
bounded substrings. Default `tok-512-v1` chunker.
3. **Hash leaves** — `sha256(0x00 || canonical_chunk_bytes)` per
chunk.
4. **Merkle tree** — `MerkleTree.build(leaves).root` →
`document_root`. Two peers running the same chunker on the same
canonicalized content compute bit-identical roots.
5. **Upsert** — `documents` row keyed on `document_root` (idempotent
re-ingest), `chunks` rows with leaf hashes, `merkle_nodes` for
proof reconstruction, `edges` per outbound link.
6. **FTS5** — `chunks_fts` insert with rowid = `chunks.chunk_id`
so the search-time JOIN lines up.
7. **Audit event** — one row per ingest batch in `audit_events`,
chained on `prev_event_hash`.
## Batching discipline
Default `batch_size=200`: balances Python GIL overhead vs SQLite
transaction commit cost. Lower it (e.g. 50) only to bound peak
memory on a low-RAM host. Higher (e.g. 1000) for ETL throughput on
SSD storage when memory isn't tight.
`progress_every` prints a stderr line every N docs so long ingests
are observable. Use `PYTHONUNBUFFERED=1` for tail-able output.
## Resumability
Idempotent re-ingest: same content + same chunker + same canonicalize
= same `document_root` = no-op insert. So a crashed ingest can be
restarted from the source's beginning without duplicating rows.
Different content at the same URI gets a new `document_root` AND a
`supersedes` edge linking new → old (lossless history).
## Source
[`aborist/ingest.py`](../../aborist/ingest.py)

63
docs/modules/merkle.md
View file

@ -1,63 +0,0 @@
# `aborist.merkle`
Pure Merkle tree + proof primitives. Python port of
`proxy.unturf.com/pkg/verified/merkle.go` — convention-identical.
Used everywhere a content-addressable handle is needed: per-chunk
leaves, document_root, evidence_map_root, run_dag_root, snapshots.
## Conventions (do not silently change)
These match the Go reference & are load-bearing for cross-language
verification (Go peer ↔ Python peer compute bit-identical roots):
- **Leaf hash:** `sha256(0x00 || canonical_chunk_bytes)`. The `0x00`
prefix domain-separates leaves from internal nodes.
- **Internal hash:** `sha256(0x03 || left || right)`. The `0x03`
prefix is the **non-commutative** combine — `H(L,R) ≠ H(R,L)`.
Order matters.
- **Odd-element rule:** when a level has an odd count, the last
leaf is **self-duplicated** before pairing. NOT zero-padded.
- **Proof path:** each step carries an explicit `is_left: bool`
alongside the sibling hash so a verifier knows which side to put
the sibling on. Never sort siblings lexically — the order tells
the verifier the tree topology.
## API surface
```python
from aborist.merkle import MerkleTree, MerkleProof
tree = MerkleTree.build([b"chunk_0_bytes", b"chunk_1_bytes", ...])
tree.root # bytes(32) — sha256 of the whole tree
tree.leaves # list[bytes(32)] — leaf hashes in input order
proof = tree.proof_for(leaf_index=2)
proof.siblings # list[(sibling_hash, is_left)]
proof.verify(leaf_hash=tree.leaves[2], root=tree.root) # bool
```
## When to read the source
- Adding a new content-addressable artifact (cores, evidence maps,
snapshots, run-DAGs all touch this).
- Cross-language verification debugging (Go peer says one root,
Python peer says another — the difference is always in canonical
encoding, ordering, or one of the three prefix bytes above).
- Performance work — the Python build is ~3× slower than the Go
reference; if it ever shows up in profiling, that's the file.
## Diagrams
The module graph shows what depends on `merkle.py` (a lot — it's
substrate):
![module graph](../diagrams/aborist-modules.svg)
The ingest pipeline shows where leaf & root hashes get computed:
![ingest pipeline](../diagrams/ingest-pipeline.svg)
## Source
[`aborist/merkle.py`](../../aborist/merkle.py) ·
Reference: [`proxy.unturf.com/pkg/verified/merkle.go`](https://git.unturf.com/engineering/unturf/proxy.unturf.com/-/blob/main/pkg/verified/merkle.go)

185
docs/modules/qa.md
View file

@ -1,185 +0,0 @@
# `aborist.qa`
The Q&A pipeline. Question → cache → retrieval → LLM → verify →
render → cache write. Lives in 9 sub-modules; this page is the
map.
![query pipeline](../diagrams/query-pipeline.svg)
## Sub-modules
### `qa.client` — LLM transport
`ChatClient` ABC with three concrete implementations:
- `StubClient` — deterministic test fixture; returns canned
responses keyed on the input. Used in unit tests to avoid network.
- `OpenAICompatibleClient` — talks to any OpenAI-shape `/v1/chat/completions`
endpoint (Hermes-3 on vLLM by default). Includes HTTP retry layer
(3× exponential backoff on 5xx).
- (Future) `AnthropicClient` — Claude API direct.
[`aborist/qa/client.py`](../../aborist/qa/client.py)
### `qa.keys` — the 8-dim cache_key
```
cache_key = sha256(
source_root | question_hash | model_profile_hash |
conversation_hash | governance_policy_hash |
schema_version | canonicalization_version | chunking_version
)
```
Two question-hash modes (`strict` vs `equivalence_class`) live here.
Bumping any of these eight dimensions invalidates prior records on
lookup. The `verifier_policy_hash` (v9.9 9th dim) is also
implemented here.
[`aborist/qa/keys.py`](../../aborist/qa/keys.py)
### `qa.runner` — `ask()` for single-doc Q&A
The simplest entry point. Take one document, ask one question, get
back an answer + audit_mode + cache record. Used by the CLI for
focused queries against one URI.
[`aborist/qa/runner.py`](../../aborist/qa/runner.py)
### `qa.query` — `query()` for multi-source RAG
The main retrieval entry point. Walks shards, runs FTS5 BM25 with
AND→OR fallback (with synonym-pool injection in OR mode), filters
by title relevance with 4 accept paths, reranks by body coverage +
title boost + source role + title purity, assembles a 60 KB context
budget, calls the LLM, runs the verifier, persists to
`providence_cache`.
[`aborist/qa/query.py`](../../aborist/qa/query.py)
### `qa.verify` — the layered verifier
Five strategies run in sequence; first to find evidence classifies:
1. `quote` — `"..."`-wrapped claims tested verbatim
2. `span` — bullet/sentence units substring-tested
3. `entity` — multi-word proper nouns with proximity gating
4. `paraphrase` — token coverage on prose-shaped spans (≥85%)
5. `claim_lattice` — pointer-line `[E1,E2]` or JSON; runs **seven
deterministic hard checks**:
1. parser succeeded
2. evidence_id resolves
3. source_role allowed
4. claim text non-empty
5. citation coverage threshold
6. pointer count cap (trim-and-verify)
7. anchor-class warrant (see `qa.warrant`)
The classifier output rolls up into the v9.8 trichotomy
`audit_mode ∈ {STRICT, HYBRID, UNGROUNDED}`. Display layer (in
`cli.py`) maps `(audit_mode, violations) → four-rung ladder`.
![verifier ladder](../diagrams/verifier-ladder.svg)
[`aborist/qa/verify.py`](../../aborist/qa/verify.py)
### `qa.warrant` — anchor-class warrant
Five lexical anchor classes the verifier composes:
- **Proper-noun** — relation-question shape; at least one
Title-Case anchor must appear in some cited span
- **Date** — claim has a 4-digit year + month name; ALL components
required in some cited span
- **Entity-list** — entity-list-shape question; ≥1 named entity
must anchor (demote-don't-reject)
- **Count** — count-shape question; count token must appear in
word OR digit form (digit↔word equivalence)
- **Cause** — why-shape question; ≥1 cause anchor (proper noun OR
≥5-char common noun outside stopword pool)
The warrant layer earns proof-path entry by staying **lexical** —
no NLI, no embeddings. Substring tests over already-canonicalized
spans. See `docs/concept-relations-design.md` (sibling section)
for the relationship to retrieval-time synonym expansion.
[`aborist/qa/warrant.py`](../../aborist/qa/warrant.py)
### `qa.evidence` — EvidenceObject + spotlight
Builds the runtime evidence map for claim-lattice modes. Each
chunk becomes one `EvidenceObject` carrying TWO ids:
- `pointer_id` — short prompt-facing tag (`E1`, `E2`, …)
- `evidence_id` — content-addressed `E########` (sha256-derived)
The model sees only `pointer_id`s in the prompt; the runtime maps
to `evidence_id` for the cache & run-DAG (run-stable identity).
The spotlight excerpt picks the load-bearing slice via **density
rank** — find ALL match positions for ALL claim content tokens,
pick the position with maximum distinct-token cluster within
±half-window. Replaces the older first-match-of-longest-token
approach which lost the load-bearing slice on noisy chunks.
[`aborist/qa/evidence.py`](../../aborist/qa/evidence.py)
### `qa.parse_claims` — pointer-line parser
Walks lines of the model output, pulls every `[E\d+]` and
`[E\d+,E\d+,…]` bracket payload, returns
`(claim_text, pointer_ids[])` per line. Lines without a tag get
`parse_status='NO_EVIDENCE_POINTER'` & count toward the denominator
so unsourced prose can't smuggle past the verifier.
[`aborist/qa/parse_claims.py`](../../aborist/qa/parse_claims.py)
### `qa.dag` — per-run Merkle DAG
Commits each provenance step independently as a stage hash. Two
shapes:
- **7-stage (quote mode):** question / retrieval / context /
prompt / answer / verify / final_label
- **9-stage (claim-lattice / CTI):** question / retrieval /
evidence_map / prompt / raw_answer / parsed_claim_lattice /
verify / render / final_label
The `run_dag_root` is persisted alongside every providence record;
`run_dag_blob` carries the full `{root, nodes}` JSON so an auditor
can recompute & verify any step.
[`aborist/qa/dag.py`](../../aborist/qa/dag.py)
### `qa.inspect` — read-only sidecar
Pulls source chunks for a given cache_key & classifies each
unverified span: `verbatim_in_base` / `verbatim_in_raw_only` /
`trailing_artifact` / `paraphrase` / `partial_paraphrase` /
`no_overlap`. Also includes the deflection-detection sidecar
(subject-anchor heuristic for adversarial-premise topic shift).
**Sidecars never write to `providence_cache` or `audit_events`** —
they're diagnostic only. That invariant is what keeps `audit_mode`
a binary classification rather than a soft score.
[`aborist/qa/inspect.py`](../../aborist/qa/inspect.py)
### `qa.concepts` — backwards-compat shim
Delegates to `aborist.concepts` (the corpus-derived synonym/rivalry
layer). Pre-2026-05-01 the data lived as hand-curated frozensets in
this file; now it's a per-shard SQLite table. The shim preserves
the legacy public API (`synonym_expand`, `rivalry_excluded`,
`has_compare_phrasing`) so call sites in `qa/query.py` didn't have
to change.
[`aborist/qa/concepts.py`](../../aborist/qa/concepts.py) →
[`aborist/concepts/`](../../aborist/concepts/)
## Source papers
- Whitepaper §13.8 covers the layered verifier in depth
- Whitepaper §13.9 covers claim-lattice / CTI mode
- `docs/cti-architecture.md` is the architecture reference
- `docs/seven-point-program.md` enumerates the seven hard checks
- `docs/concept-relations-design.md` covers the synonym layer

75
docs/modules/search.md
View file

@ -1,75 +0,0 @@
# `aborist.search`
The retrieval primitive. Today's only backend is FTS5 over the
chunks table; the `SearchBackend` ABC is in place so additional
backends (BM25 over titles, embedding-based vector search) can be
added without touching the rest of the QA pipeline.
## `SearchBackend` ABC
```python
from aborist.search import SearchBackend, AuditMode, Hit
class SearchBackend(ABC):
@abstractmethod
def search(self, query: str, limit: int = 20) -> list[Hit]: ...
```
Each `Hit` carries `(document_root, document_uri, chunk_idx,
snippet, score, audit_mode, title)`. `audit_mode` is the sticky
provenance label that tracks how the chunk made it into the index;
FTS5 backend always sets `UNGROUNDED` (search itself doesn't verify
anything — that's the QA pipeline's job).
## `FTS5Backend`
Wraps the contentless `chunks_fts` virtual table. Two-mode query:
- **AND-mode (strict, primary):** every content token must appear
in the doc. Keeps unrelated docs out of the context window.
- **OR-mode (fallback):** when AND returns 0 hits, fall back to OR
but **capped to top-5 longest tokens** (proxy for rarity). Long
topical synonyms fed via `extra_or_tokens` join the pool —
`neurotechnology` (15 chars) outranks `thoughts` (8) by length
and surfaces brain-tech titles for vocabulary-mismatch queries.
```python
from aborist.search import FTS5Backend
backend = FTS5Backend(conn)
# Plain search
hits = backend.search("permacomputer", limit=32)
# Search with synonym pool injection (used by qa.query._search_corpus)
hits = backend.search(
long_query,
limit=32,
extra_or_tokens=synonym_expand(qtokens, shards_dir=shards_dir),
)
```
## Stopword & stopword-cap discipline
`_FTS5_STOPWORDS` filters question words (`what`, `tell`, `please`)
+ generic connectors (`one`, `some`, `another`, `without`, `soon`,
`currently`) before AND/OR construction. Two principles:
- **Stay in sync with `_TITLE_STOPWORDS`** in `qa/query.py`. A token
filtered at retrieval time but kept at title-relevance check (or
vice versa) creates ranking incoherence.
- **`_OR_FALLBACK_MAX_TOKENS=5`** caps the OR-mode pool. Without
this, a 19-token OR clause matches millions of docs and forces
BM25 to rank them all — 13s/shard observed pre-cap. Now 0.25s/shard.
## Snippet building
FTS5 contentless mode means SQLite's built-in `snippet()` and
`highlight()` return empty. Aborist builds snippets in Python by
joining `chunks_fts.rowid = chunks.chunk_id`, decompressing the
chunk content, and locating query tokens locally
(`_build_snippet`).
## Source
[`aborist/search/fts5.py`](../../aborist/search/fts5.py) ·
[`aborist/search/__init__.py`](../../aborist/search/__init__.py)

80
docs/modules/sources.md
View file

@ -1,80 +0,0 @@
# `aborist.sources`
Corpus producers. Each is a `Source` ABC implementation that yields
`Document` instances; the standard `ingest.ingest_source(source, db)`
pipeline takes them from there.
The `Source` ABC lives in [`aborist/source.py`](../../aborist/source.py):
```python
class Source(ABC):
@abstractmethod
def iter_documents(self) -> Iterator[Document]: ...
```
## Built-in sources
### `wikipedia.py` — Phase III SQL dumps (the canonical bootstrap)
Streams the Wikipedia 2003 `cur` (current revisions) and `old`
(revision history) SQL dumps. Hand-rolled escape-aware parser
(no `sqlite3` import — the dump is MySQL syntax). 4× speedup vs
char-by-char loops via `str.find` + slicing. cProfile any change.
Default Wikipedia 2003-05-16 dump source:
`https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`.
robots.txt returned 404 → no rules.
[`aborist/sources/wikipedia.py`](../../aborist/sources/wikipedia.py)
### `wikipedia_xml.py` — Phase IV XML dumps
Modern Wikipedia dump format (`enwiki-YYYYMMDD-pages-articles.xml.bz2`,
`enwiki-YYYYMMDD-pages-meta-history*.xml.bz2`). Uses `xml.etree.ElementTree.iterparse`
to stream-parse without loading the whole tree.
[`aborist/sources/wikipedia_xml.py`](../../aborist/sources/wikipedia_xml.py)
### `html_page.py` — single-URL or URL-list HTML ingest
Robots-aware (`urllib.robotparser`). Uses `selectolax` for fast
HTML parsing (CSS-selector based; ~10× faster than `lxml`). Pulls
the main body text + every `<a href>` as an `Edge` row.
The `edges` rows are what the corpus-derived synonym extractor
later reads — no separate crawler needed for site-internal link
graphs.
Optional dep: `pip install '.[html]'` for `selectolax` + `httpx`.
[`aborist/sources/html_page.py`](../../aborist/sources/html_page.py)
### `crawler/` — async BFS web crawl
Verbatim lift of an `AsyncWebFetcher` implementation + an `ingest`
bridge. BFS-discovers same-domain URLs from a seed, respecting
`robots.txt` + crawl delays. Captures ETag + Last-Modified per URL
into `document_http_meta` so a future recrawl can send conditional
HEAD requests.
[`aborist/sources/crawler/bridge.py`](../../aborist/sources/crawler/bridge.py)
### `grok.py` — xAI Grok export
Reads the `xAI-conversations.json` data export shape. Each
conversation becomes one `Document`; media prompts are kept
inline.
[`aborist/sources/grok.py`](../../aborist/sources/grok.py)
### `vcs.py` — git + Mercurial repositories
HEAD walk. Each commit becomes a `Document` (commit message + diff
stat). The supersedes chain captures commit ancestry as edges.
[`aborist/sources/vcs.py`](../../aborist/sources/vcs.py)
## Source
[`aborist/sources/`](../../aborist/sources/) ·
[`aborist/source.py`](../../aborist/source.py) (ABC)

81
docs/modules/store.md
View file

@ -1,81 +0,0 @@
# `aborist.store`
The v9.8 SQLite schema, the audit chain, and the cross-shard
read-only view. Every table that holds runtime state lives here.
## Schema overview (per shard)
```
documents – one row per source document, keyed on document_root
chunks – per-document chunk content + tier (hot/cold)
chunks_fts – FTS5 contentless index, rowid = chunks.chunk_id
merkle_nodes – internal-node hashes for proof reconstruction
edges – src_root → dst_root link graph (wikilink, href, …)
derivations – core_root ← src_root with proof_blob (Merkle)
providence_cache – Q&A records keyed on the v9.8 8-dim cache_key
audit_events – linear chain; event_hash = sha256(prev || canonical(body))
falsifications – record_id → state transition + reason + actor
snapshots – named corpus roots (one hash names a forest)
document_http_meta – ETag + Last-Modified for crawler conditional fetches
concept_relations – per-shard synonym/rivalry/category/antonym overlay
mesh_* – federation tables (off by default)
```
## v9.8 invariants (do not break)
- **Aborist is a v9.8 store.** Every providence record carries the
full **8-dim cache_key**: `source_root | question_hash |
model_profile_hash | conversation_hash | governance_policy_hash |
schema_version | canonicalization_version | chunking_version`.
Bumping any one invalidates prior records on lookup.
- **`falsification_state ∈ {live, failed, stale, quarantined}`.**
Cache lookups must filter on `state='live'`. Drift detection
flips to `stale`.
- **Audit chain.** Every state-changing op writes one row in
`audit_events` with `event_hash = sha256(prev_event_hash ||
canonical(body))`. Chain integrity is verified in
`make analyze-shards`. **Never insert into `audit_events`
directly — use `aborist.store.append_audit`.**
- **Cores never evict.** `evict_to_cold` only touches `kind='surface'`.
- **Idempotent re-ingest.** Same content → same `document_root` →
no-op insert. Same URI + different content → new doc + `supersedes`
edge linking new → old (lossless history).
## API surface
```python
from aborist.store import (
connect, # writable connection to a single shard
connect_query, # read-only UNION view across all shards
discover_shards, # list *.db files in a shards_dir
transaction, # BEGIN IMMEDIATE / COMMIT / ROLLBACK context manager
get_meta, set_meta,
append_audit, # the ONLY way to write audit_events
)
```
Cross-shard reads use `connect_query(shards_dir=...)` which ATTACHes
every `*.db` and creates UNION views over the shardable tables
(`documents`, `chunks`, `merkle_nodes`, `edges`, `derivations`,
`providence_cache`, `audit_events`, `falsifications`,
`concept_relations`).
## Performance pragmas
`connect()` applies these per-connection:
- `journal_mode=WAL` (set in SCHEMA_SQL once at first ingest)
- `synchronous=NORMAL` (skip per-commit fsync; safe under WAL)
- `cache_size=-65536` (64 MB page cache)
- `temp_store=MEMORY` (no /tmp churn for temp tables)
Don't downgrade to `synchronous=FULL` without a measured reason —
costs ~5× throughput.
## Diagrams
![module graph](../diagrams/aborist-modules.svg)
![ingest pipeline](../diagrams/ingest-pipeline.svg)
## Source
[`aborist/store.py`](../../aborist/store.py)

46
docs/modules/wikitext.md
View file

@ -1,46 +0,0 @@
# `aborist.wikitext`
A single function: `to_base(raw)`. Converts MediaWiki wikitext to
plain prose deterministically.
```python
from aborist.wikitext import to_base, BASE_VERSION
prose = to_base("[[The Beatles]] are an [[English rock band]] from [[Liverpool]].")
# → "The Beatles are an English rock band from Liverpool."
```
## Why it exists
The corpus stores raw wikitext (so the link graph is recoverable
on demand) but the LLM and verifier both want plain prose. Reasons:
1. **Token efficiency.** Wikipedia chunks ship to Hermes with ~43%
fewer tokens after wikitext-strip — bigger context window for
the same chars budget.
2. **Verbatim citation.** The model can quote source paragraphs
verbatim instead of escaping `[[wikilinks]]`. The verifier's
substring test then matches cleanly.
3. **Pinned identity.** `BASE_VERSION='wikitext-base-v1'` lives in
`policy["base_version"]`, which folds into
`governance_policy_hash`. Bumping `BASE_VERSION` invalidates
every prior cache record on next lookup — same discipline as
`chunking_version` and `canonicalization_version`.
## Hot-path discipline
`to_base()` runs on the assembled context **before** the LLM call
in `aborist/qa/runner.py` and `aborist/qa/query.py`, **and again
inside `verify_quotes`** so the verifier compares like-against-like.
Both sides see prose.
## Optional dependency
Backed by `mwparserfromhell`. Install via `pip install '.[wikitext]'`
to enable. Without the dep, `_wikitext_to_base = None` and
`policy["base_version"] = None` — graceful fallback leaves raw
wikitext in both context and verifier (works, just less efficient).
## Source
[`aborist/wikitext.py`](../../aborist/wikitext.py)