docs: README label refresh + per-module reference + Graphviz diagrams

Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).

(1) README — label refresh:
    - Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
      four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
      EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
    - Verifier section spells out both layers (schema trichotomy +
      display ladder), the seven hard checks of claim_lattice, and
      the five anchor classes of warrant.
    - Architecture tree updated: concepts/ package added, qa/
      sub-modules expanded (warrant, evidence, parse_claims, dag),
      verify.py described as quote/span/entity/paraphrase + claim_lattice.
    - Concept overlay description updated for corpus-derived layer
      (concept_relations table, link_reciprocity extractor, 1.6%
      tax cite).
    - Test count: 326+ → 641+.

(2) docs/diagrams/ — Graphviz dot sources:
    - aborist-modules.dot — top-level package graph (substrate /
      storage / sources / retrieval / qa / mesh / cli)
    - query-pipeline.dot — question → cache → retrieval → LLM →
      verify → render → cache write, with phase budgets
    - ingest-pipeline.dot — source doc → canonicalize → chunk →
      Merkle → upsert (+ optional distill)
    - verifier-ladder.dot — (audit_mode, violations) → display rung
      decision tree
    Existing mesh-*.dot kept as-is. Makefile `make docs` target
    extended to also emit .svg alongside the existing .png so the
    diagrams render in markdown viewers.

(3) docs/modules/ — per-module reference pages:
    - index.md (links to every diagram + every module page)
    - merkle.md, document.md, store.md, ingest.md, evict.md,
      sources.md, search.md, concepts.md, qa.md, distill.md,
      wikitext.md
    Each page is a one-screenful concise reference: what the
    module is for, public API, key invariants, embedded diagrams
    where useful, link to source. Mesh stays at the existing
    docs/mesh.md + docs/mesh-deploy.md (already comprehensive).

Tests: 641 passed (no code change).

2026-05-01 23:19:01 -04:00

2.3 KiB

Raw Blame History

`aborist.ingest`

The bulk-batched pipeline that turns documents from a source into Merkle-committed shard storage. Source-agnostic: anything that implements Source.iter_documents() flows through here.

Public API

from aborist.ingest import ingest_source
from aborist.sources.wikipedia import WikipediaSqlDump

source = WikipediaSqlDump("/path/to/cur.sql.bz2")
ingest_source(
    source,
    db_path=Path("~/.aborist/shards/000.db"),
    batch_size=200,        # docs per transaction
    progress_every=1000,
)

What happens per document

Canonicalize — canonicalize(text): NFC + ws-collapse + strip ends. Pinned by canonicalization_version='norm-v1'.
Chunk — Chunker.chunk(canonical_text) → list of token- bounded substrings. Default tok-512-v1 chunker.
Hash leaves — sha256(0x00 || canonical_chunk_bytes) per chunk.
Merkle tree — MerkleTree.build(leaves).root → document_root. Two peers running the same chunker on the same canonicalized content compute bit-identical roots.
Upsert — documents row keyed on document_root (idempotent re-ingest), chunks rows with leaf hashes, merkle_nodes for proof reconstruction, edges per outbound link.
FTS5 — chunks_fts insert with rowid = chunks.chunk_id so the search-time JOIN lines up.
Audit event — one row per ingest batch in audit_events, chained on prev_event_hash.

Batching discipline

Default batch_size=200: balances Python GIL overhead vs SQLite transaction commit cost. Lower it (e.g. 50) only to bound peak memory on a low-RAM host. Higher (e.g. 1000) for ETL throughput on SSD storage when memory isn't tight.

progress_every prints a stderr line every N docs so long ingests are observable. Use PYTHONUNBUFFERED=1 for tail-able output.

Resumability

Idempotent re-ingest: same content + same chunker + same canonicalize = same document_root = no-op insert. So a crashed ingest can be restarted from the source's beginning without duplicating rows.

Different content at the same URI gets a new document_root AND a supersedes edge linking new → old (lossless history).

Source

aborist/ingest.py

2.3 KiB Raw Blame History

aborist.ingest