russell@unturf.com
326badf6d8
Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).
(1) README — label refresh:
- Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
- Verifier section spells out both layers (schema trichotomy +
display ladder), the seven hard checks of claim_lattice, and
the five anchor classes of warrant.
- Architecture tree updated: concepts/ package added, qa/
sub-modules expanded (warrant, evidence, parse_claims, dag),
verify.py described as quote/span/entity/paraphrase + claim_lattice.
- Concept overlay description updated for corpus-derived layer
(concept_relations table, link_reciprocity extractor, 1.6%
tax cite).
- Test count: 326+ → 641+.
(2) docs/diagrams/ — Graphviz dot sources:
- aborist-modules.dot — top-level package graph (substrate /
storage / sources / retrieval / qa / mesh / cli)
- query-pipeline.dot — question → cache → retrieval → LLM →
verify → render → cache write, with phase budgets
- ingest-pipeline.dot — source doc → canonicalize → chunk →
Merkle → upsert (+ optional distill)
- verifier-ladder.dot — (audit_mode, violations) → display rung
decision tree
Existing mesh-*.dot kept as-is. Makefile `make docs` target
extended to also emit .svg alongside the existing .png so the
diagrams render in markdown viewers.
(3) docs/modules/ — per-module reference pages:
- index.md (links to every diagram + every module page)
- merkle.md, document.md, store.md, ingest.md, evict.md,
sources.md, search.md, concepts.md, qa.md, distill.md,
wikitext.md
Each page is a one-screenful concise reference: what the
module is for, public API, key invariants, embedded diagrams
where useful, link to source. Mesh stays at the existing
docs/mesh.md + docs/mesh-deploy.md (already comprehensive).
Tests: 641 passed (no code change).
81 lines
3.4 KiB
Markdown
81 lines
3.4 KiB
Markdown
# `aborist.store`
|
||
|
||
The v9.8 SQLite schema, the audit chain, and the cross-shard
|
||
read-only view. Every table that holds runtime state lives here.
|
||
|
||
## Schema overview (per shard)
|
||
|
||
```
|
||
documents – one row per source document, keyed on document_root
|
||
chunks – per-document chunk content + tier (hot/cold)
|
||
chunks_fts – FTS5 contentless index, rowid = chunks.chunk_id
|
||
merkle_nodes – internal-node hashes for proof reconstruction
|
||
edges – src_root → dst_root link graph (wikilink, href, …)
|
||
derivations – core_root ← src_root with proof_blob (Merkle)
|
||
providence_cache – Q&A records keyed on the v9.8 8-dim cache_key
|
||
audit_events – linear chain; event_hash = sha256(prev || canonical(body))
|
||
falsifications – record_id → state transition + reason + actor
|
||
snapshots – named corpus roots (one hash names a forest)
|
||
document_http_meta – ETag + Last-Modified for crawler conditional fetches
|
||
concept_relations – per-shard synonym/rivalry/category/antonym overlay
|
||
mesh_* – federation tables (off by default)
|
||
```
|
||
|
||
## v9.8 invariants (do not break)
|
||
|
||
- **Aborist is a v9.8 store.** Every providence record carries the
|
||
full **8-dim cache_key**: `source_root | question_hash |
|
||
model_profile_hash | conversation_hash | governance_policy_hash |
|
||
schema_version | canonicalization_version | chunking_version`.
|
||
Bumping any one invalidates prior records on lookup.
|
||
- **`falsification_state ∈ {live, failed, stale, quarantined}`.**
|
||
Cache lookups must filter on `state='live'`. Drift detection
|
||
flips to `stale`.
|
||
- **Audit chain.** Every state-changing op writes one row in
|
||
`audit_events` with `event_hash = sha256(prev_event_hash ||
|
||
canonical(body))`. Chain integrity is verified in
|
||
`make analyze-shards`. **Never insert into `audit_events`
|
||
directly — use `aborist.store.append_audit`.**
|
||
- **Cores never evict.** `evict_to_cold` only touches `kind='surface'`.
|
||
- **Idempotent re-ingest.** Same content → same `document_root` →
|
||
no-op insert. Same URI + different content → new doc + `supersedes`
|
||
edge linking new → old (lossless history).
|
||
|
||
## API surface
|
||
|
||
```python
|
||
from aborist.store import (
|
||
connect, # writable connection to a single shard
|
||
connect_query, # read-only UNION view across all shards
|
||
discover_shards, # list *.db files in a shards_dir
|
||
transaction, # BEGIN IMMEDIATE / COMMIT / ROLLBACK context manager
|
||
get_meta, set_meta,
|
||
append_audit, # the ONLY way to write audit_events
|
||
)
|
||
```
|
||
|
||
Cross-shard reads use `connect_query(shards_dir=...)` which ATTACHes
|
||
every `*.db` and creates UNION views over the shardable tables
|
||
(`documents`, `chunks`, `merkle_nodes`, `edges`, `derivations`,
|
||
`providence_cache`, `audit_events`, `falsifications`,
|
||
`concept_relations`).
|
||
|
||
## Performance pragmas
|
||
|
||
`connect()` applies these per-connection:
|
||
- `journal_mode=WAL` (set in SCHEMA_SQL once at first ingest)
|
||
- `synchronous=NORMAL` (skip per-commit fsync; safe under WAL)
|
||
- `cache_size=-65536` (64 MB page cache)
|
||
- `temp_store=MEMORY` (no /tmp churn for temp tables)
|
||
|
||
Don't downgrade to `synchronous=FULL` without a measured reason —
|
||
costs ~5× throughput.
|
||
|
||
## Diagrams
|
||
|
||
|
||
|
||
|
||
## Source
|
||
|
||
[`aborist/store.py`](../../aborist/store.py)
|