docs: README label refresh + per-module reference + Graphviz diagrams

Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).

(1) README — label refresh:
    - Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
      four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
      EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
    - Verifier section spells out both layers (schema trichotomy +
      display ladder), the seven hard checks of claim_lattice, and
      the five anchor classes of warrant.
    - Architecture tree updated: concepts/ package added, qa/
      sub-modules expanded (warrant, evidence, parse_claims, dag),
      verify.py described as quote/span/entity/paraphrase + claim_lattice.
    - Concept overlay description updated for corpus-derived layer
      (concept_relations table, link_reciprocity extractor, 1.6%
      tax cite).
    - Test count: 326+ → 641+.

(2) docs/diagrams/ — Graphviz dot sources:
    - aborist-modules.dot — top-level package graph (substrate /
      storage / sources / retrieval / qa / mesh / cli)
    - query-pipeline.dot — question → cache → retrieval → LLM →
      verify → render → cache write, with phase budgets
    - ingest-pipeline.dot — source doc → canonicalize → chunk →
      Merkle → upsert (+ optional distill)
    - verifier-ladder.dot — (audit_mode, violations) → display rung
      decision tree
    Existing mesh-*.dot kept as-is. Makefile `make docs` target
    extended to also emit .svg alongside the existing .png so the
    diagrams render in markdown viewers.

(3) docs/modules/ — per-module reference pages:
    - index.md (links to every diagram + every module page)
    - merkle.md, document.md, store.md, ingest.md, evict.md,
      sources.md, search.md, concepts.md, qa.md, distill.md,
      wikitext.md
    Each page is a one-screenful concise reference: what the
    module is for, public API, key invariants, embedded diagrams
    where useful, link to source. Mesh stays at the existing
    docs/mesh.md + docs/mesh-deploy.md (already comprehensive).

Tests: 641 passed (no code change).

2026-05-01 23:19:01 -04:00

5.8 KiB

Raw Blame History

Aborist module reference

One page per top-level package + one diagram per major data flow. This index is the entry point — every module page links back here and to the relevant diagrams.

Diagrams

Diagram	What it shows	File
Module graph	Top-level packages & how they import each other	`aborist-modules.svg` (dot)
Query pipeline	Question → cache → retrieval → LLM → verify → render	`query-pipeline.svg` (dot)
Ingest pipeline	Source document → Merkle-committed shard	`ingest-pipeline.svg` (dot)
Verifier ladder	(audit_mode, violations) → display rung	`verifier-ladder.svg` (dot)
Mesh data flow	Federation: roster, gossip, AEAD envelope	`mesh-data-flow.svg`
Mesh epoch lifecycle	Epoch advance via add/kick/rotate	`mesh-epoch-lifecycle.svg`
Mesh identity stack	Ed25519 sign + X25519 DH key derivation	`mesh-identity-stack.svg`
Mesh secret envelope	AEAD-wrapped epoch secret per peer	`mesh-secret-envelope.svg`
Mesh group decisions	Membership change voting & quorum	`mesh-group-decisions.svg`

Render diagrams locally:

make docs   # runs `dot -Tsvg` and `-Tpng` on every docs/diagrams/*.dot

Substrate (no SQL, pure data structures)

Module	One-line role	Doc
`merkle.py`	Merkle tree + proof — Python port of `proxy.unturf.com/pkg/verified/merkle.go`	merkle.md
`document.py`	`Document`, `Edge`, `Chunker` (default `tok-512-v1`)	document.md
`wikitext.py`	`to_base()` — wikitext → plain prose, BASE_VERSION-pinned	wikitext.md

Storage

Module	One-line role	Doc
`store.py`	v9.8 SQLite schema + audit chain helpers	store.md
`ingest.py`	normalize → chunk → merkle → upsert (bulk-batched)	ingest.md
`evict.py`	hot ↔ cold tier transitions; rehydrate via source	evict.md

Sources (corpus producers)

Module	One-line role	Doc
`sources/wikipedia.py`	Wikipedia 2003 cur + old SQL dumps (bz2-streamed)	sources.md
`sources/wikipedia_xml.py`	Phase IV XML dumps (iterparse, page + history)	sources.md
`sources/html_page.py`	URL list + selectolax + httpx (robots-aware)	sources.md
`sources/crawler/`	verbatim AsyncWebFetcher lift + ingest bridge	sources.md
`sources/grok.py`	xAI data export (conversations + media prompts)	sources.md
`sources/vcs.py`	git + Mercurial repos (HEAD walk, supersedes chain)	sources.md

Search & retrieval

Module	One-line role	Doc
`search/`	FTS5 backend + `SearchBackend` ABC + `AuditMode` enum	search.md
`concepts/`	Per-shard `concept_relations` synonym/rivalry overlay	concepts.md

Q&A pipeline

Module	One-line role	Doc
`qa/keys.py`	8-dim cache_key + `question_hash`	qa.md
`qa/client.py`	`ChatClient` + `StubClient` + `OpenAICompatibleClient`	qa.md
`qa/runner.py`	`ask()`: single-doc Q&A + cache + verify	qa.md
`qa/query.py`	`query()`: multi-source RAG + concept overlay	qa.md
`qa/verify.py`	quote/span/entity/paraphrase + claim_lattice (7 hard checks)	qa.md
`qa/warrant.py`	5 anchor classes (proper-noun · date · count · entity-list · cause)	qa.md
`qa/evidence.py`	`EvidenceObject` + spotlight excerpt (density rank)	qa.md
`qa/parse_claims.py`	pointer-line parser (`claim. [E1,E2]`)	qa.md
`qa/dag.py`	per-run Merkle-DAG (7-stage quote / 9-stage CTI)	qa.md
`qa/inspect.py`	sidecar diagnostic (read-only span classifier)	qa.md

Distillation

Module	One-line role	Doc
`distill/`	`Distiller` ABC + `first_sentence` + `tfidf` + runner	distill.md

Federation (off by default)

Module	One-line role	Doc
`mesh/`	identity (Ed25519/X25519), per-epoch roster, AEAD envelope, gossip wire	../mesh.md, ../mesh-deploy.md

Entry point

Module	One-line role	Doc
`cli.py`	`argparse` entrypoint — every `make` target dispatches here	run `aborist --help` or any `make help` target

Tickets, design docs, journals

See ../TICKETS.md for the ticket index and the list of design-reference docs that aren't tickets.

5.8 KiB Raw Blame History