aborist

An arborist for trees and forests of cross-linked information.

Aborist ingests documents into a content-addressed, Merkle-committed SQLite store, distills them into recursive cores, and answers questions over the resulting corpus via an OpenAI-compatible LLM. Every cached answer carries a verifiable Merkle proof tying it back to its source documents — Merkle-AGI v9.8 / Merkle Providence Reverse RAG, runnable end-to-end.

What this gets you

make bootstrap              # one-time: venv + deps
make fetch-cur              # download Wikipedia 2003-05-16 (~82 MB)
make ingest-cur-attached    # ~3 min — 128k articles, 4 parallel shards
make distill-shards-parallel
make distill-shards-tfidf-parallel
make query Q="What is anarcho-capitalism?"

After the last command, a Hermes-3 inference runs against the local corpus, picks 4–8 source articles by Merkle root, and returns an answer with a cryptographic proof of its sources. Repeat the same question and a STRICT-mode cache hit replays in ~100 ms.

Setup

Get the source

git clone ssh://git@git.unturf.com:2222/engineering/unturf/aborist.git
cd aborist

HTTPS variant if SSH isn't set up:

git clone https://git.unturf.com/engineering/unturf/aborist.git

Install prerequisites

Aborist needs Python 3.10+, GNU make, curl, and bzip2. SQLite 3.35+ ships with CPython.

macOS (Homebrew)

xcode-select --install        # if you don't already have CLT
brew install python@3.12 git make

Apple's make is GNU make, no extra step needed. bzip2 and curl are bundled.

Ubuntu / Debian

sudo apt update
sudo apt install -y git python3 python3-venv python3-dev build-essential curl bzip2

22.04 ships Python 3.10; 24.04 ships 3.12 — both work.

Windows

The Makefile uses bash idioms, so the supported path is WSL2 running Ubuntu. From an admin PowerShell:

wsl --install -d Ubuntu-24.04

Then inside the WSL Ubuntu shell, follow the Ubuntu instructions above.

(Native cmd / PowerShell + Git Bash mostly works for the Python parts but several make targets call for i in $(seq…) and bash -c — easier to just use WSL2.)

OpenBSD

pkg_add git python-3.12 gmake curl

OpenBSD's default make is BSD make. Aborist's Makefile uses GNU-make features (?=, conditional functions). Substitute gmake for make in every command, e.g. gmake bootstrap, gmake query Q='…'.

Bootstrap

make bootstrap

Creates .venv/, installs the package in editable mode with the [dev,html] extras, and exposes aborist at .venv/bin/aborist. No system-wide install. Re-running make bootstrap is a no-op if the venv is up to date.

After bootstrap, every workflow lives behind a make target. Run make help to list them.

Data: Wikipedia 2003-05-16 (Phase III SQL dump)

The 2003 dataset lives at https://dumps.wikimedia.org/archive/2003/2003-05-16/en/ as three files. This is a MySQL extended-INSERT format dump; for XML-format dumps from 2006 onward see the Phase IV section below.

file	size	what
20030516_cur_tablesql.bz2	82 MB	one row per article: snapshot of every Wikipedia page on 2003-05-16
old_tablesqlbz2.1	640 MiB	first half of the revision-history table (split bzip2 stream)
old_tablesqlbz2.2	252 MiB	second half — cat them together to decompress

make fetch-cur                # snapshot only (~82 MB on disk)
make fetch-old                # full history (~1.4 GB on disk after concat)
make fetch                    # both

Files land in data/. Re-running is idempotent (curl skips if already present).

Sharded ingest (the canonical path)

Per-shard SQLite files, no write-lock contention. Each shard process owns its own DB; cross-shard reads attach all shards as UNION ALL views.

make ingest-cur-attached SHARDS=4    # cur snapshot, 4 parallel shards (~3 min)
make ingest-old-attached SHARDS=4    # full history, ~30–40 min

Shards land in ~/.aborist/shards/. Override with SHARDS_DIR=/path/to/somewhere.

Resumable

Add --resume (or just re-run the make target — --resume is the default for attached ingests). Each shard tracks its own high-water mark in meta; an interrupted ingest picks up where it left off without re-hashing already-cached docs.

Single-DB ingest (simpler, smaller)

For experiments under a few thousand docs, a single SQLite file is fine:

make ingest-cur INGEST_LIMIT=1000       # one DB at $(DB), default ~/.aborist/aborist.db
make ingest-cur-parallel SHARDS=4       # 4 processes, one shared DB (WAL serialized)

Distillation (cores feed retrieval)

Two distillers ship: first-sentence-v1 (one sentence per chunk) and tfidf-keywords-v1 (top-K distinctive terms per doc). Cores are Merkle-signed back to their source docs and serve as enriched titles for retrieval.

make distill-shards-parallel              # first-sentence cores, per-shard
make distill-shards-tfidf-parallel        # TF-IDF cores, per-shard

Run both — they generate independent cores per source. TF-IDF cores let neologisms (like a personal term that never appears in any title) match retrievals via keyword overlap.

Data: Wikipedia 2010-11 (and other Phase IV snapshots)

In 2006 MediaWiki swapped its dumps from MySQL INSERT INTO cur syntax to XML. Aborist reads both — the SQL path above for 2003-2005 cur dumps, and a streaming XML path for any dated snapshot in https://dumps.wikimedia.org/archive/. The largest single snapshot in that archive is enwiki 2010-11-08:

file	size	what
enwiki-20101011-pages-articles.xml.bz2	6.2 GB	latest revision of every main-namespace article on 2010-11-08 (~3.4M pages, ~1.9M after redirects)
enwiki-20101011-abstract.xml	2.9 GB	first-paragraph abstracts only — pre-distilled summaries at ~1/100th the chunk volume

Other useful dated snapshots in the archive: 2006-07 (1.8 GB), 2006-12 (1.9 GB), 2010-03 (varies by language). All work via the same source class.

# defaults target enwiki 20101011 (2010-11)
make fetch-xml                # ~6.2 GB compressed download
make ingest-xml-attached SHARDS=4    # ~2 hours sharded; ~95 GB on disk after

# pick any other snapshot by overriding the date variables
make fetch-xml WP_XML_YEAR=2006 WP_XML_MONTH=2006-07 WP_XML_DATE=20061104

# abstract feed (one-paragraph summaries, full coverage at ~5-10 GB total)
make fetch-abstract
make ingest-abstract

The XML source streams .xml.bz2 directly via iterparse with bounded memory (each <page> is processed and cleared). Same shard / resume / Merkle contract as the SQL source. Title-prefix namespace filtering kicks in for older export schemas that omit per-page <ns>.

To ingest historical revisions instead of just the current snapshot, point WP_XML at a pages-meta-history.xml.bz2 file and use make ingest-xml-history — the source emits one Document per revision and aborist's prior-doc detection chains them with supersedes edges.

Data: personal Grok export

If you have an xAI data-export bundle, point GROK_EXPORT at its root directory (the one containing ttl/30d/export_data/<user-id>/):

export GROK_EXPORT=$HOME/Downloads/<your-user-uuid>
make ingest-grok-attached            # conversations  -> $(SHARDS_DIR)/grok.db
make ingest-grok-media-attached      # media prompts  -> $(SHARDS_DIR)/grok.db

Both walk the export tree, find prod-grok-backend.json, and yield one Document per conversation (or media-generation post). Conversation titles, full message text, and turn ordering are preserved. Each becomes a normal queryable doc in the cluster — your prior chats become memory the corpus can consult.

--resume is the default for these targets. Re-run any time to pick up new exports.

Data: git and Mercurial repos (self-play)

Aborist can consult itself. Point a source at any local clone and every text file at HEAD becomes a queryable Document; re-ingesting after new commits chains old → new via supersedes edges, so the audit trail grows alongside the repo:

make ingest-self                         # this aborist tree, into ~/.aborist/shards/aborist-self.db
make ingest-git GIT_REPO=/path/to/repo   # any other git clone
make ingest-hg  HG_REPO=/path/to/repo    # mercurial flavor

URI shape: git://<repo-name>/file/<relative-path> (no commit hash — that's what enables the supersedes chain on re-ingest). Binary files are skipped (NUL-byte heuristic + UTF-8 decode probe). extra carries the current commit hash, timestamp, and subject for informational purposes; the cryptographic identity is the content-derived document_root as for every other Document.

Data: OpenAI / ChatGPT export (planned)

Not implemented yet. The shape will be one new Source subclass at aborist/sources/openai.py plus a Makefile target. The OpenAI ChatGPT data export is a .zip containing conversations.json with the mapping/messages tree shape. Adding it follows the same pattern as aborist/sources/grok.py — see that file as the template.

# (placeholder)
make ingest-openai-attached         # OPENAI_EXPORT=$HOME/Downloads/<chatgpt-export>

When this lands, conversations from both Grok and OpenAI will sit in the same shard cluster; queries fan out across all of them.

Asking the corpus

make query Q="What is the philosophy of stoicism?"
make query Q="tell me about permacomputer ?"
make query QUERY_TOP_K=12 Q="…"        # widen the source set
make query-dry Q="…"                    # assemble context but skip the LLM call

The query path:

1. Search — FTS5 (body) + SQL LIKE (title) + JOIN over derivations (TF-IDF core keywords) across every shard. Three accept paths to the relevance filter.
2. Concept overlay — synonym groups (Athlon ↔ AMD) widen retrieval; rivalry pairs (AMD ↔ Intel) narrow it unless the query has comparative phrasing ("compare X vs Y").
3. Context assembly — top-K sources concatenated up to a 60 KB budget, fed to Hermes-3 with strict attribution rules in the system prompt.
4. Cache — the v9.8 8-dim cache_key (source_root | question_hash | model_profile | conversation | governance_policy | schema | canonicalization | chunking) keys the answer in qa.db. Cache hits replay STRICT-mode in ~100 ms. Per-phase timings are returned in every result.

LLM endpoint defaults to https://hermes.ai.unturf.com/v1 (Hermes-3 Llama-3.1-8B, 82K context, no auth). Override:

export ABORIST_LLM_ENDPOINT="https://your-vllm.example/v1"
export ABORIST_LLM_MODEL="meta-llama/Llama-3.1-70B-Instruct"
export ABORIST_LLM_API_KEY="..."

Mark a wrong answer

If the LLM produced something incorrect, mark its cache record stale so the next ask re-runs inference fresh:

make falsify KEY=<cache_key> REASON="why it was wrong"

The original record stays in the database (history matters). A falsifications row + falsify audit event record the act. Future lookups skip records whose falsification_state != 'live'.

Mesh / federation (off by default)

Two machines that ingest the same dump compute bit-identical document_roots — that's the v9.8 admissibility property. The mesh layer is the wire-and-trust scaffolding that lets peers gossip those identities (plus derivations, falsifications, cross-witnesses) and dedup-by-content across instances.

It ships off by default. No code path touches the network unless the mesh.enabled flag is set. Initialization flow:

aborist mesh init --group myteam        # mint Ed25519 + X25519 keys; create epoch 0
aborist mesh status                     # always-safe inspection; shows enabled/false until you flip it
aborist mesh enable                     # flip the gating flag on

Membership is per-epoch. Adding a member, kicking a member, or rotating the secret each bumps the epoch and writes an audit event:

aborist mesh add  --member-id bob --sign-pub <hex> --dh-pub <hex>
aborist mesh kick --member-id bob --reason "..."     # admin-only; bumps epoch, omits bob from new envelope
aborist mesh rotate --reason "..."                   # refresh secret, same roster
aborist mesh members                                  # list current epoch's roster

The kicked member's prior signatures stay verifiable forever (their roster row at older epochs is preserved on disk). They have no entry in the new epoch's secret envelope, so any AEAD-protected gossip from epoch+1 onward is opaque to them — that is the eviction guarantee.

The HTTP gossip wire (mesh sync, mesh serve) is on the roadmap; this commit ships the cryptographic foundation, state machine, and CLI. The crypto is cryptography-backed Ed25519 + X25519 + ChaCha20-Poly1305.

Inspecting

make stats-shards                    # totals across shards
make analyze-shards                  # compression spectrum, depth histogram, audit chain integrity
make verify-shards                   # round-trip Merkle proofs on a random sample
make activity                        # recent Q&A + ingests + derives + falsifications (agent timeline)
make activity ACTIVITY_LIMIT=20

activity is JSON; pipe through jq to drill in.

Architecture (one screenful)

aborist/
├── merkle.py           Python port of proxy.unturf.com/pkg/verified/merkle.go
│                       conventions: non-commutative HashCombine (0x03 prefix),
│                       explicit IsLeft per sibling, self-duplicate odd elements.
├── store.py            SQLite v9.8 schema (8-dim cache key, falsification state,
│                       audit chain, surface/core kinds, hot/warm/cold tier).
├── ingest.py           normalize → chunk → merkle → upsert. Bulk-batched writer.
├── document.py         Document, Edge, Chunker (tok-512-v1 default).
├── source.py           Source ABC: iter_documents() -> Iterator[Document].
├── sources/
│   ├── wikipedia.py    cur + old MediaWiki SQL dumps (bz2-streamed).
│   ├── html_page.py    URL list + selectolax + httpx (robots-aware).
│   └── grok.py         xAI data export (conversations + media prompts).
├── distill/            Distiller ABC + first_sentence + tfidf + recursion runner.
├── search/             FTS5 backend + SearchBackend ABC + AuditMode.
├── qa/                 ask (single-doc) + query (multi-source RAG) + concepts overlay.
├── evict.py            hot ↔ cold tier transitions; rehydrate via source pipeline.
└── cli.py              argparse entrypoint (the make targets call into here).

Source papers (read first if confused):

• ~/Downloads/merkle-providence-reverse-rag-whitepaper.pdf — public spec, AGPL-3.0
• ~/Downloads/merkle-agi-dag_v7.txt — formal substrate (TLV/canonical encoding, theorems T1–T5)

Tests

make test            # 79 tests, all stdlib + pytest
make bench           # ETL throughput across configs (serial / shared-WAL / attached)

License

License: AGPL-3.0-only · This algorithm, its implementation, & all associated code carry the GNU Affero General Public License v3.0 (only). You may use, modify, & distribute under those terms. No proprietary relicensing exists.

(Verbatim from the Merkle Providence Reverse RAG whitepaper, April 2026.)