A snapshot_root is MerkleTree.build([sorted DISTINCT document_roots]).root
— one 32-byte hash naming the entire content-addressed forest at a
point in time. Two peers that ingested the same dump compute bit-
identical snapshot_roots, so cross-machine "are we synced?" becomes
an O(1) hash comparison; the same root doubles as the TF-IDF
scope_root we sketched in the mesh design (snapshot_root *is* a
scope), and pins Q&A answers to a verifiable corpus state.
Schema: one new table, additive over existing data.
snapshots(snapshot_root PK, taken_at, audit_event_hash, doc_count,
parent_snapshot, reason)
+ idx_snapshots_taken_at
API:
compute_snapshot_root(conn, *, document_roots=None) -> (root, count)
create_snapshot(conn, *, reason, parent_snapshot=None) -> dict
verify_snapshot(conn, snapshot_root) -> dict (matches: bool)
diff_against_current(conn, snapshot_root) -> dict (added/removed/unchanged)
list_snapshots(conn, *, limit) -> list[dict]
CLI:
aborist snapshot create [--reason "..."] [--parent <hex>]
aborist snapshot list [--limit N]
aborist snapshot verify <snapshot_root>
aborist snapshot diff <snapshot_root>
Cross-shard: with --shards-dir + --db, the snapshot is computed over
the cluster-wide UNION view but persisted into args.db (a dedicated
snapshots store, conventionally ~/.aborist/shards/snapshots.db).
parent_snapshot auto-links to the latest prior snapshot in the
writer DB, giving a chain for free.
Each snapshot creation writes an audit_event of type 'snapshot_create'
with subject_root=snapshot_root, so the corpus's pinned states are
themselves tamper-evidently logged.
Defect caught + fixed during live test on the 2010 enwiki ingest:
verify and diff disagreed on the same connection because compute used
a list (with cross-shard duplicate document_roots) while diff used a
set. Two shards can land identical document_roots when canonicalize()
maps two structurally-similar pages to the same byte stream — rare
but real. Switched the underlying SQL to SELECT DISTINCT so a
membership snapshot is always order- AND multiplicity-independent.
Live test: 2010-11 enwiki corpus snapshotted at
43797e46605de08dbab06cdcaf5be7ad78243b193c56f8580200dee6bcc7e1b9
doc_count: 3,468,134 (after dedup)
verify + diff round-trip both report identical against current state.
11 new tests covering empty corpus, single-doc degenerate, order
independence, drift detection, audit-chain pinning, parent auto-link,
and idempotent creation on unchanged corpus. 136 tests + 1 skipped.
Phase 1 of the federation/gossip layer fox sketched as the natural
extension of v9.8 admissibility's content-addressed identity. Two
peers ingesting the same dump already compute identical document_roots
and identical 8-dim cache_keys; the mesh layer is the wire-and-trust
plumbing that lets them dedup answers, exchange Merkle proofs, and
cleanly distrust an evicted member without a hard fork.
Cryptography (cryptography lib, audited):
Ed25519 — every membership mutation + (future) gossip envelope
is signed by the actor's pubkey.
X25519 ECDH — wraps each epoch's symmetric mesh secret to every
current member's DH pubkey via HKDF-derived AEAD key.
ChaCha20-P1305— AEAD for envelope payloads + per-member secret wrap.
State machine:
mesh_identity — singleton; this peer's keys + group name
mesh_roster — per-epoch (member_id, sign_pub, dh_pub, role)
mesh_epochs — epoch_id -> {started_at, started_event_hash,
secret_envelope JSON, reason}
meta:mesh.enabled flag — off by default; gates everything
Eviction works by rotating to a new epoch whose envelope omits the
kicked member. Their prior signatures stay verifiable (the older
roster row is retained), but any gossip from epoch+1 onward is
opaque to them — the secret was never shared with their pubkey.
Authority gate: only roster members with role='admin' can add or
kick. Self-kick is rejected explicitly. The last admin can't be
kicked. Schedule-rotate (refresh secret, no roster change) is open
to any current member as a session-hygiene op.
Audit-chain integration: every mesh state mutation writes an audit
event (mesh_init, mesh_enable/disable, mesh_epoch_rotate). The
epoch's started_event_hash backfills into mesh_epochs after the
audit row commits, giving each epoch a tamper-evident pin into the
ledger.
CLI subcommands: mesh init, mesh status, mesh enable, mesh disable,
mesh members, mesh add, mesh kick, mesh rotate. All read-only or
local-state-only — no network code paths in this commit.
The HTTP gossip wire (`mesh sync`, `mesh serve`) is the next phase.
Schema, cryptography, and roster state machine are all in place to
support it without further migration.
GitRepoSource walks the working tree at HEAD via `git ls-tree` + `git show`,
yielding one Document per text file. URI shape `git://<repo>/file/<path>`
intentionally omits the commit hash — re-ingest after new commits produces
fresh document_roots that aborist's prior-doc detection chains via
`supersedes` edges, so the audit trail and Merkle tree grow as the repo
grows. Binaries skipped via NUL-byte + UTF-8 decode probes; >5 MB files
filtered out by default. Commit hash + timestamp + subject ride along in
`extra` (informational only; not part of the Merkle commitment).
MercurialRepoSource mirrors via `hg manifest` + `hg cat`. Same supersedes
semantics, same shape.
Makefile adds:
make ingest-self (this repo -> aborist-self.db)
make ingest-git GIT_REPO=/path/to/repo (arbitrary git clone)
make ingest-hg HG_REPO=/path/to/repo (mercurial)
Each lands in its own shard file alongside existing shards/grok.db,
keeping per-shard write paths independent of the wikipedia 4-way ingest's
WAL writer lock.
Three cheats stack to drop on-disk store from ~21 KB to ~6.7 KB per doc on
the 2003 enwiki cur corpus (-67% measured, apples-to-apples reingest with
identical document/edge counts; Merkle proofs round-trip 30/30):
1. zstd-compressed chunks.content (level 3). Magic-byte detection on read
means legacy plaintext rows pass through unchanged. Cores stay plaintext
so qa.query._docs_with_core_keyword_match's SQL LOWER+LIKE keeps working.
2. edges WITHOUT ROWID. The composite PK (src_root, edge_type, dst_root,
dst_uri, anchor) covers every column, so a default rowid-based table
near-doubles row data in the PK index. WITHOUT ROWID makes the table
itself the B-tree. Drops idx_edges_dst_uri too — the only query that
filters on dst_uri alone is gravity_top_inbound, a one-shot analytic.
3. contentless FTS5 (content='', contentless_delete=1) eliminates the
28 MB / 1000 docs of duplicated chunk text the old chunks_fts stored.
chunks gets an explicit chunk_id INTEGER PRIMARY KEY so the FTS5
rowid maps back to chunks.chunk_id at search time. Snippets are
built in Python (search/fts5.py:_build_snippet) since SQL snippet()
returns empty in contentless mode.
TF-IDF retrieval also fixed: the prior LIKE '%intel%' substring match
let "intelligence", "intellectual", "intellivision" drown real hits like
Pentium_4 (whose TF-IDF core has "intel" as an exact keyword). Now uses
word-boundary `LIKE '%, intel, %'` patterns plus a match_count over the
distinct query tokens — multi-token coverage outranks single-token title
boosts. Pentium_4 surfaces #1 for "what is the fastest intel CPU?" with
the canonical 2003 answer (Pentium 4 3.20 GHz) instead of an empty
"insufficient sources" reply.
Schema-level changes affect new DBs only; existing v9.8 DBs keep
working at the old layout. Cross-shard UNION views explicitly list the
intersection of columns so a mixed cluster (legacy + new schema shards
in one --shards-dir) still unions cleanly.
Drops in for any dated snapshot in dumps.wikimedia.org/archive — pages-articles.xml.bz2
and pages-meta-current.xml.bz2 (cur snapshot, single-revision-per-page) plus
pages-meta-history.xml.bz2 (multi_revision=True). abstract.xml feed yields
pre-distilled summaries at ~1/100th the chunk volume. Streams .bz2/.gz directly
via iterparse with bounded memory; same shard / resume contract as the SQL
source. Falls back to title-prefix namespace filtering when older export
schemas omit the per-page <ns> element (e.g. enwiki 20101011).
Makefile defaults target enwiki 20101011 (6.2 GB). Override with WP_XML_YEAR /
WP_XML_MONTH / WP_XML_DATE / WP_XML_LANG to fetch any other archived snapshot.
aborist/sources/grok.py exposes two Source subclasses for ingesting
xAI's user data export:
GrokExportSource walks ttl/<period>/export_data/<user-id>/ and
reads prod-grok-backend.json. Yields one Document
per conversation with:
- URI: grok://conversation/<conversation-id>
- title: the conversation's auto-generated title
- content: full message text in turn order
source_type='grok_export'.
GrokMediaPostsSource same export, but yields per media-generation
post (image/video prompts) under URI
grok://media/<post-id>.
source_type='grok_media'.
Both auto-walk down from the export root so callers can pass the
top-level directory xAI delivered (e.g., ~/Downloads/<user-uuid>/).
Wired into the CLI: aborist ingest --source {grok_export,grok_media}
accepts --path <export-root>. tests/test_grok_source.py covers the
walk + parse + Document shape with a fabricated mini-export fixture.
Once ingested, conversations become normal queryable docs in the
shard cluster — your prior chats become memory the corpus can
consult during RAG.
79 tests passing.
A small knowledge-graph layer that does two distinct jobs:
1. SYNONYM_GROUPS broaden retrieval. A query mentioning "Athlon"
now also matches AMD-titled docs because Athlon IS an AMD product.
Groups currently cover AMD-family, Intel-family, HTTP family,
FTP, Mac, Windows, Linux. Easy to extend.
2. RIVALRIES narrow retrieval. The pair (AMD-group, Intel-group)
means: if the query mentions one side and not the other, drop
docs whose titles contain the OTHER side's tokens. So a "fastest
AMD CPU" question never gets Pentium_4 in the context — even if
FTS5 BM25 ranks it high — because Pentium is in the Intel group
and Intel isn't in the query.
COMPARE_WORDS ("vs", "versus", "compare", "between", ...) suppress
the exclusion. "compare AMD vs Intel" keeps both sides. "what is
the fastest AMD CPU?" does not.
Demos against the 128k Wikipedia 2003-05-16 cur shards:
Q "fastest AMD CPU" → 8 sources, all AMD/CPU titled, NO Intel
Answer: "Athlon XP 3200+" (real AMD chip,
grounded in the AMD article)
Q "compare AMD vs Intel"
→ 8 sources, mix of Intel_8028x, Intel_8048x,
AMD_Duron — both sides preserved
Q "Athlon processor" → AMD, AMD_Duron, AMD_5x86, Athlon all
surfaced via synonym expansion
Phase 1 implementation hand-curates the groups; Phase 2 idea is to
derive them from Wikipedia's link/category graph (dense bidirectional
clusters → synonym groups; same-category-without-cross-links →
rivalry candidates).
66 tests passing.
aborist/qa/query.py exposes query() — the user-facing RAG flow:
1. FTS5 search across all shards (chunks_fts can't be UNION'd as a
view, so each shard's index is queried independently and merged
by score).
2. Top-K distinct documents are selected within a max-context-chars
budget (default 60 KB so a 768-token response fits Hermes-3's
82 K context window comfortably).
3. context_root = Merkle root over the sorted source document_roots.
That's the v9.8 'source' dimension for multi-source answers —
a verifier can recompute it from the listed source roots.
4. 8-dim cache_key over (context_root, question_hash, model_profile,
conversation, governance_policy, schema, canonicalization,
chunking). Hit returns STRICT immediately; miss calls Hermes and
persists.
CLI: aborist [--shards-dir DIR] query "<question>"
Default qa_db is <shards-dir>/qa.db (or ~/.aborist/qa.db). Uses the
same OpenAICompatibleClient/StubClient as `ask`. --dry-run skips the
LLM and returns context-only.
Search escape fix: the prior FTS5 escape ANDed every token including
stopwords + punctuation, so "What is anarcho-capitalism?" required
the doc to literally contain "what" + "is" + "anarcho-capitalism?" —
zero hits. New tokenizer drops stopwords + punctuation and ORs the
remaining content tokens; BM25 ranks the multi-token matches highest.
Live demo against the 122k-doc 4-shard cluster:
Q "What is anarcho-capitalism?" 6.1 s wall miss / 0.45 s cache hit
Q "Who was George Washington?" 10.3 s wall miss
Both answers cite the source URIs Hermes was given.
57 tests passing (4 new query tests covering search → context →
cache → audit chain).
Resume (rsync-style)
Each shard DB gains a `meta` table. After every successful batch
flush, ingest_source persists `source_high_water:<source_type>` ->
the largest row id seen (cur_id or old_id). On --resume, the source
reads it back and skips rows whose id is <= the mark, so an ingest
killed at any point can be re-run cheaply: already-cached docs are
fast-forwarded past without re-hashing or DB writes.
WikipediaSqlDump now exposes `start_id` (skip threshold) and
`last_id` (running max). cur_id is surfaced in Document.extra
alongside old_id so both tables behave the same.
CLI: aborist ingest --resume
Demo on shared DB (cur):
Round 1: --limit 3000 --resume high_water = 5714
Round 2: --limit 5000 --resume skips 1..5714, picks up at 5715
Round 3: --limit 100 --resume skips 1..15362, ingests 100 more
Always idempotent on re-run; no dups, no missing docs.
Per-shard audit chain integrity for sharded analyze
Cross-shard analyze previously reported nonsense breaks counts —
each shard owns its own audit chain (genesis -> ... -> latest), and
the UNION view interleaves them so cross-shard transitions look
like break events. The fix: when --shards-dir is set, open each
shard's DB directly and run _check_audit_chain on it, then
aggregate.
Output now reads:
"audit_chain": {
"events": <total>,
"breaks": 0,
"shards": [{"shard": "000.db", "events": N, "breaks": 0}, ...]
}
53 tests passing. Tests cover: high-water write, skip-on-resume,
idempotency across two resume runs, kill-and-resume continuity.
The 2003-05-16 archive ships three files:
20030516_cur_tablesql.bz2 82 MB current snapshot (single revision/page)
old_tablesqlbz2.1 640 MiB \
old_tablesqlbz2.2 252 MiB / split halves of old (full revision
history). Concatenate before bzcat.
Generalize the parser:
WikipediaSqlDump(table='cur'|'old') — shared statement parser, single
column-position contract for the
first 4 fields (id/ns/title/text)
WikipediaCurDump — back-compat wrapper, table='cur'
WikipediaOldDump — new, table='old'; old has no is_redirect, every
revision is real
Old rows surface old_id and old_timestamp via Document.extra so a
downstream pass can sort revisions chronologically before re-ingesting
through the supersedes-edge path.
Makefile gains:
fetch-cur / fetch-old / fetch (both)
ingest-cur / ingest-old / ingest (cur default)
WP_OLD target concatenates the two split parts
CLI ingest --source now accepts wikipedia_cur or wikipedia_old.
Smoke (real dump): 5 revisions of "AtlasShrugged/Companies" yielded
correctly with old_id=2..10, timestamps from January 2002.
aborist/qa/ implements the cache-first answer flow from the providence
whitepaper, scaled up to v9.8's full 8-dim admissibility invariant.
cache_key = SHA-256 of:
source_root | question_hash | model_profile_hash | conversation_hash
| governance_policy_hash | schema_version | canonicalization_version
| chunking_version
Any drift in any dimension yields a distinct cache_key — prior records
cannot serve. Falsification states (failed/stale/quarantined) gate
every cache hit.
- qa/keys.py — pure hash functions, deterministic & testable
- qa/client.py — ChatClient Protocol + StubClient + OpenAI-compatible
HTTP client (vllm/llama.cpp/uncloseai compatible)
- qa/runner.py — ask(): lookup -> hit (no LLM call, hit_count++)
OR miss (call client, write record, audit event,
proof binds answer to source root)
- cli.py — `aborist ask` and `aborist providence` subcommands
- pyproject.toml — httpx promoted from extras to core (used by both
html and qa); selectolax stays in [html] extras
Smoke (StubClient, no network): cache miss writes record with chunk_0
Merkle proof reconstructing source_root; cache hit returns same record
without calling client; 1085 audit events chained 0 breaks across
ingest/derive/evict/rehydrate/providence_write.
When a re-ingest produces a different document_root for an already-seen
URI, both versions now coexist in the store. A 'supersedes' edge from
the new doc to the prior one keeps the lineage addressable, and the
ingest audit body records the supersedes link.
Idempotent re-ingest is unchanged: identical bytes -> identical root ->
no new doc, no supersedes edge.
Three-version chains test that v3->v2 and v2->v1 edges form a walkable
history. The old version stays queryable (its chunks may later be
evicted to cold for storage savings while remaining provable).
Pure-Python (stdlib only). Uses chunk-level corpus baseline so terms
concentrated in fewer chunks outrank common ones. Default top-K = 16.
Output cores are comma-separated keyword lists — extreme compression
toward the tweet/haiku end of the planet metaphor. Same source can
now carry both a first-sentence-v1 core AND a tfidf-keywords-v1 core,
each derived independently and Merkle-signed back to the same surface.
The 'contributing_chunk_indices' for TF-IDF is every chunk that
contains at least one of the top-K keywords — proof binding remains
honest and cryptographically tight.
Distillers now scan kind='surface' OR kind='core'. Cores generate
deeper cores with compression_depth incremented. Each round of
recursion tightens the planet toward its center.
Audit body for derive events now records src_kind and the resulting
compression_depth so the chain reflects the layer transition.
Smoke against the existing 503 surface + 478 depth=1 core corpus
produced N deeper cores with proofs still binding back to their
depth=1 sources.
