docs: README label refresh + per-module reference + Graphviz diagrams

Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).

(1) README — label refresh:
    - Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
      four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
      EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
    - Verifier section spells out both layers (schema trichotomy +
      display ladder), the seven hard checks of claim_lattice, and
      the five anchor classes of warrant.
    - Architecture tree updated: concepts/ package added, qa/
      sub-modules expanded (warrant, evidence, parse_claims, dag),
      verify.py described as quote/span/entity/paraphrase + claim_lattice.
    - Concept overlay description updated for corpus-derived layer
      (concept_relations table, link_reciprocity extractor, 1.6%
      tax cite).
    - Test count: 326+ → 641+.

(2) docs/diagrams/ — Graphviz dot sources:
    - aborist-modules.dot — top-level package graph (substrate /
      storage / sources / retrieval / qa / mesh / cli)
    - query-pipeline.dot — question → cache → retrieval → LLM →
      verify → render → cache write, with phase budgets
    - ingest-pipeline.dot — source doc → canonicalize → chunk →
      Merkle → upsert (+ optional distill)
    - verifier-ladder.dot — (audit_mode, violations) → display rung
      decision tree
    Existing mesh-*.dot kept as-is. Makefile `make docs` target
    extended to also emit .svg alongside the existing .png so the
    diagrams render in markdown viewers.

(3) docs/modules/ — per-module reference pages:
    - index.md (links to every diagram + every module page)
    - merkle.md, document.md, store.md, ingest.md, evict.md,
      sources.md, search.md, concepts.md, qa.md, distill.md,
      wikitext.md
    Each page is a one-screenful concise reference: what the
    module is for, public API, key invariants, embedded diagrams
    where useful, link to source. Mesh stays at the existing
    docs/mesh.md + docs/mesh-deploy.md (already comprehensive).

Tests: 641 passed (no code change).

2026-05-01 23:19:01 -04:00

2.4 KiB

Raw Blame History

`aborist.merkle`

Pure Merkle tree + proof primitives. Python port of proxy.unturf.com/pkg/verified/merkle.go — convention-identical. Used everywhere a content-addressable handle is needed: per-chunk leaves, document_root, evidence_map_root, run_dag_root, snapshots.

Conventions (do not silently change)

These match the Go reference & are load-bearing for cross-language verification (Go peer ↔ Python peer compute bit-identical roots):

Leaf hash: sha256(0x00 || canonical_chunk_bytes). The 0x00 prefix domain-separates leaves from internal nodes.
Internal hash: sha256(0x03 || left || right). The 0x03 prefix is the non-commutative combine — H(L,R) ≠ H(R,L). Order matters.
Odd-element rule: when a level has an odd count, the last leaf is self-duplicated before pairing. NOT zero-padded.
Proof path: each step carries an explicit is_left: bool alongside the sibling hash so a verifier knows which side to put the sibling on. Never sort siblings lexically — the order tells the verifier the tree topology.

API surface

from aborist.merkle import MerkleTree, MerkleProof

tree = MerkleTree.build([b"chunk_0_bytes", b"chunk_1_bytes", ...])
tree.root            # bytes(32) — sha256 of the whole tree
tree.leaves          # list[bytes(32)] — leaf hashes in input order

proof = tree.proof_for(leaf_index=2)
proof.siblings       # list[(sibling_hash, is_left)]
proof.verify(leaf_hash=tree.leaves[2], root=tree.root)  # bool

When to read the source

Adding a new content-addressable artifact (cores, evidence maps, snapshots, run-DAGs all touch this).
Cross-language verification debugging (Go peer says one root, Python peer says another — the difference is always in canonical encoding, ordering, or one of the three prefix bytes above).
Performance work — the Python build is ~3× slower than the Go reference; if it ever shows up in profiling, that's the file.

Diagrams

The module graph shows what depends on merkle.py (a lot — it's substrate):

The ingest pipeline shows where leaf & root hashes get computed:

Source

aborist/merkle.py · Reference: proxy.unturf.com/pkg/verified/merkle.go

2.4 KiB Raw Blame History Unescape Escape

aborist.merkle