docs: README label refresh + per-module reference + Graphviz diagrams

Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).

(1) README — label refresh:
    - Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
      four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
      EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
    - Verifier section spells out both layers (schema trichotomy +
      display ladder), the seven hard checks of claim_lattice, and
      the five anchor classes of warrant.
    - Architecture tree updated: concepts/ package added, qa/
      sub-modules expanded (warrant, evidence, parse_claims, dag),
      verify.py described as quote/span/entity/paraphrase + claim_lattice.
    - Concept overlay description updated for corpus-derived layer
      (concept_relations table, link_reciprocity extractor, 1.6%
      tax cite).
    - Test count: 326+ → 641+.

(2) docs/diagrams/ — Graphviz dot sources:
    - aborist-modules.dot — top-level package graph (substrate /
      storage / sources / retrieval / qa / mesh / cli)
    - query-pipeline.dot — question → cache → retrieval → LLM →
      verify → render → cache write, with phase budgets
    - ingest-pipeline.dot — source doc → canonicalize → chunk →
      Merkle → upsert (+ optional distill)
    - verifier-ladder.dot — (audit_mode, violations) → display rung
      decision tree
    Existing mesh-*.dot kept as-is. Makefile `make docs` target
    extended to also emit .svg alongside the existing .png so the
    diagrams render in markdown viewers.

(3) docs/modules/ — per-module reference pages:
    - index.md (links to every diagram + every module page)
    - merkle.md, document.md, store.md, ingest.md, evict.md,
      sources.md, search.md, concepts.md, qa.md, distill.md,
      wikitext.md
    Each page is a one-screenful concise reference: what the
    module is for, public API, key invariants, embedded diagrams
    where useful, link to source. Mesh stays at the existing
    docs/mesh.md + docs/mesh-deploy.md (already comprehensive).

Tests: 641 passed (no code change).

2026-05-01 23:19:01 -04:00

2.5 KiB

Raw Blame History

`aborist.document`

The data structures every source produces and every storage layer consumes. Three core types: Document, Edge, Chunker.

`Document`

A single ingest unit: a Wikipedia article, an HTML page, a Grok conversation, a git commit message, etc. Carries both the raw content AND the version tags that determine its identity:

@dataclass(frozen=True)
class Document:
    document_uri: str           # canonical URI (or stable surrogate for non-URI sources)
    raw_content: str            # source-of-truth bytes pre-canonicalization
    kind: str                   # 'surface' / 'core' / 'visual' / etc.
    chunking_version: str       # e.g. 'tok-512-v1' — pinned by the Chunker
    canonicalization_version: str  # e.g. 'norm-v1' — pinned by canonicalize()
    schema_version: str         # e.g. 'v9.8.0' — store schema generation
    title: str | None = None
    edges: list[Edge] = ()      # outbound link graph
    metadata: dict = ...        # source-specific opaque payload

document_root is computed at ingest time as the Merkle root over the canonicalized chunks. Two peers ingesting the same source + running the same chunking_version + canonicalization_version get bit-identical document_roots — the v9.8 admissibility property.

`Edge`

One outbound link. aborist/sources/wikipedia.py emits one Edge per [[wikilink]]; aborist/sources/html_page.py emits one per <a href>. The link graph IS the corpus topology — concepts/extract.py later reads edges rows to derive synonym relations from reciprocal links (no separate crawler needed).

@dataclass(frozen=True)
class Edge:
    src_root: str   # source document_root
    dst_uri: str    # always present
    dst_root: str   # '' (unresolved) until the dst doc is also ingested
    edge_type: str  # 'wikilink' / 'href' / 'citation' / 'derived_from' / ...
    anchor: str     # chunk index or fragment, '' if N/A

`Chunker`

ABC with one method chunk(text: str) -> list[str]. Default impl is TokenChunker (name='tok-512-v1') — splits on token-rough windows so the resulting chunks are predictable for downstream FTS5 indexing & for the LLM context budget.

Changing the chunker bumps chunking_version AND stales every prior cache record (chunking is one of the 8 cache_key dimensions). Don't redefine tok-512-v1; add a new chunker as a new name instead.

Diagrams

Source

aborist/document.py

2.5 KiB Raw Blame History

aborist.document

Document

Edge

Chunker