docs: README label refresh + per-module reference + Graphviz diagrams

Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).

(1) README — label refresh:
    - Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
      four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
      EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
    - Verifier section spells out both layers (schema trichotomy +
      display ladder), the seven hard checks of claim_lattice, and
      the five anchor classes of warrant.
    - Architecture tree updated: concepts/ package added, qa/
      sub-modules expanded (warrant, evidence, parse_claims, dag),
      verify.py described as quote/span/entity/paraphrase + claim_lattice.
    - Concept overlay description updated for corpus-derived layer
      (concept_relations table, link_reciprocity extractor, 1.6%
      tax cite).
    - Test count: 326+ → 641+.

(2) docs/diagrams/ — Graphviz dot sources:
    - aborist-modules.dot — top-level package graph (substrate /
      storage / sources / retrieval / qa / mesh / cli)
    - query-pipeline.dot — question → cache → retrieval → LLM →
      verify → render → cache write, with phase budgets
    - ingest-pipeline.dot — source doc → canonicalize → chunk →
      Merkle → upsert (+ optional distill)
    - verifier-ladder.dot — (audit_mode, violations) → display rung
      decision tree
    Existing mesh-*.dot kept as-is. Makefile `make docs` target
    extended to also emit .svg alongside the existing .png so the
    diagrams render in markdown viewers.

(3) docs/modules/ — per-module reference pages:
    - index.md (links to every diagram + every module page)
    - merkle.md, document.md, store.md, ingest.md, evict.md,
      sources.md, search.md, concepts.md, qa.md, distill.md,
      wikitext.md
    Each page is a one-screenful concise reference: what the
    module is for, public API, key invariants, embedded diagrams
    where useful, link to source. Mesh stays at the existing
    docs/mesh.md + docs/mesh-deploy.md (already comprehensive).

Tests: 641 passed (no code change).

2026-05-01 23:19:01 -04:00

1.7 KiB

Raw Blame History

`aborist.evict`

Hot ↔ cold tier transitions. The corpus is large (3.47M Wikipedia docs); not every chunk fits in working memory. evict.py is the mechanism that moves rarely-touched chunks to a cold tier (still indexed, just stored separately) and rehydrates them on demand from the original source.

API surface

from aborist.evict import evict_to_cold, rehydrate

# Move chunks unused for >threshold days to cold tier
evict_to_cold(conn, max_age_days=90, max_evictions=10000)

# Pull a cold chunk back to hot from its original source
rehydrate(conn, document_root="abc123...")

Invariant: cores never evict

evict_to_cold filters WHERE kind='surface'. Cores are always hot — they're the long-tail-friendly compression layer that justifies evicting their underlying surfaces. Evicting cores would defeat the purpose.

v9.8 falsification on drift

When rehydrate() re-fetches a document and the recomputed document_root differs from the stored one, the source has changed since ingest (Wikipedia article was edited, HTML page was republished, etc.). The cache record's falsification_state flips from live to stale — every providence record keyed on that source_root is no longer admissible to lookups.

This is the drift-detection-as-falsification discipline: cache hits don't blindly trust historical answers; they trust answers that the SAME source still grounds.

Tier values

chunks.tier ∈ {'hot', 'cold'}. Hot chunks live in chunks.content; cold chunks live with NULL content and a cold_uri pointing at the source. The QA pipeline's chunk-fetch path checks tier; on 'cold', it triggers rehydrate before continuing.

Source

aborist/evict.py

1.7 KiB Raw Blame History

aborist.evict