docs: README label refresh + per-module reference + Graphviz diagrams

Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).

(1) README — label refresh:
    - Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
      four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
      EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
    - Verifier section spells out both layers (schema trichotomy +
      display ladder), the seven hard checks of claim_lattice, and
      the five anchor classes of warrant.
    - Architecture tree updated: concepts/ package added, qa/
      sub-modules expanded (warrant, evidence, parse_claims, dag),
      verify.py described as quote/span/entity/paraphrase + claim_lattice.
    - Concept overlay description updated for corpus-derived layer
      (concept_relations table, link_reciprocity extractor, 1.6%
      tax cite).
    - Test count: 326+ → 641+.

(2) docs/diagrams/ — Graphviz dot sources:
    - aborist-modules.dot — top-level package graph (substrate /
      storage / sources / retrieval / qa / mesh / cli)
    - query-pipeline.dot — question → cache → retrieval → LLM →
      verify → render → cache write, with phase budgets
    - ingest-pipeline.dot — source doc → canonicalize → chunk →
      Merkle → upsert (+ optional distill)
    - verifier-ladder.dot — (audit_mode, violations) → display rung
      decision tree
    Existing mesh-*.dot kept as-is. Makefile `make docs` target
    extended to also emit .svg alongside the existing .png so the
    diagrams render in markdown viewers.

(3) docs/modules/ — per-module reference pages:
    - index.md (links to every diagram + every module page)
    - merkle.md, document.md, store.md, ingest.md, evict.md,
      sources.md, search.md, concepts.md, qa.md, distill.md,
      wikitext.md
    Each page is a one-screenful concise reference: what the
    module is for, public API, key invariants, embedded diagrams
    where useful, link to source. Mesh stays at the existing
    docs/mesh.md + docs/mesh-deploy.md (already comprehensive).

Tests: 641 passed (no code change).

2026-05-01 23:19:01 -04:00

3.4 KiB

Raw Blame History

`aborist.store`

The v9.8 SQLite schema, the audit chain, and the cross-shard read-only view. Every table that holds runtime state lives here.

Schema overview (per shard)

documents              – one row per source document, keyed on document_root
chunks                 – per-document chunk content + tier (hot/cold)
chunks_fts             – FTS5 contentless index, rowid = chunks.chunk_id
merkle_nodes           – internal-node hashes for proof reconstruction
edges                  – src_root → dst_root link graph (wikilink, href, …)
derivations            – core_root ← src_root with proof_blob (Merkle)
providence_cache       – Q&A records keyed on the v9.8 8-dim cache_key
audit_events           – linear chain; event_hash = sha256(prev || canonical(body))
falsifications         – record_id → state transition + reason + actor
snapshots              – named corpus roots (one hash names a forest)
document_http_meta     – ETag + Last-Modified for crawler conditional fetches
concept_relations      – per-shard synonym/rivalry/category/antonym overlay
mesh_*                 – federation tables (off by default)

v9.8 invariants (do not break)

Aborist is a v9.8 store. Every providence record carries the full 8-dim cache_key: source_root | question_hash | model_profile_hash | conversation_hash | governance_policy_hash | schema_version | canonicalization_version | chunking_version. Bumping any one invalidates prior records on lookup.
falsification_state ∈ {live, failed, stale, quarantined}. Cache lookups must filter on state='live'. Drift detection flips to stale.
Audit chain. Every state-changing op writes one row in audit_events with event_hash = sha256(prev_event_hash || canonical(body)). Chain integrity is verified in make analyze-shards. Never insert into audit_events directly — use aborist.store.append_audit.
Cores never evict. evict_to_cold only touches kind='surface'.
Idempotent re-ingest. Same content → same document_root → no-op insert. Same URI + different content → new doc + supersedes edge linking new → old (lossless history).

API surface

from aborist.store import (
    connect,           # writable connection to a single shard
    connect_query,     # read-only UNION view across all shards
    discover_shards,   # list *.db files in a shards_dir
    transaction,       # BEGIN IMMEDIATE / COMMIT / ROLLBACK context manager
    get_meta, set_meta,
    append_audit,      # the ONLY way to write audit_events
)

Cross-shard reads use connect_query(shards_dir=...) which ATTACHes every *.db and creates UNION views over the shardable tables (documents, chunks, merkle_nodes, edges, derivations, providence_cache, audit_events, falsifications, concept_relations).

Performance pragmas

connect() applies these per-connection:

journal_mode=WAL (set in SCHEMA_SQL once at first ingest)
synchronous=NORMAL (skip per-commit fsync; safe under WAL)
cache_size=-65536 (64 MB page cache)
temp_store=MEMORY (no /tmp churn for temp tables)

Don't downgrade to synchronous=FULL without a measured reason — costs ~5× throughput.

Diagrams

Source

aborist/store.py

3.4 KiB Raw Blame History Unescape Escape

aborist.store