aborist

An arborist for trees and forests of cross-linked information.

Aborist ingests documents into a content-addressed, Merkle-committed SQLite store, distills them into recursive cores, and answers questions over the resulting corpus via an OpenAI-compatible LLM. Every cached answer carries a verifiable Merkle proof tying it back to its source documents — Merkle-AGI v9.8 / Merkle Providence Reverse RAG, runnable end-to-end.

What this gets you

make bootstrap              # one-time: venv + deps
make fetch-cur              # download Wikipedia 2003-05-16 (~82 MB)
make ingest-cur-attached    # ~3 min — 128k articles, 4 parallel shards
make distill-shards-parallel
make distill-shards-tfidf-parallel
make query Q="What is anarcho-capitalism?"

After the last command, a Hermes-3 inference runs against the local corpus, picks 4–8 source articles by Merkle root, and returns an answer with a cryptographic proof of its sources. Repeat the same question and a STRICT-mode cache hit replays in ~100 ms.

Setup

Get the source

git clone ssh://git@git.unturf.com:2222/engineering/unturf/aborist.git
cd aborist

HTTPS variant if SSH isn't set up:

git clone https://git.unturf.com/engineering/unturf/aborist.git

Install prerequisites

Aborist needs Python 3.10+, GNU make, curl, and bzip2. SQLite 3.35+ ships with CPython.

macOS (Homebrew)

xcode-select --install        # if you don't already have CLT
brew install python@3.12 git make

Apple's make is GNU make, no extra step needed. bzip2 and curl are bundled.

Ubuntu / Debian

sudo apt update
sudo apt install -y git python3 python3-venv python3-dev build-essential curl bzip2

22.04 ships Python 3.10; 24.04 ships 3.12 — both work.

Windows

The Makefile uses bash idioms, so the supported path is WSL2 running Ubuntu. From an admin PowerShell:

wsl --install -d Ubuntu-24.04

Then inside the WSL Ubuntu shell, follow the Ubuntu instructions above.

(Native cmd / PowerShell + Git Bash mostly works for the Python parts but several make targets call for i in $(seq…) and bash -c — easier to just use WSL2.)

OpenBSD

pkg_add git python-3.12 gmake curl

OpenBSD's default make is BSD make. Aborist's Makefile uses GNU-make features (?=, conditional functions). Substitute gmake for make in every command, e.g. gmake bootstrap, gmake query Q='…'.

Bootstrap

make bootstrap

Creates .venv/, installs the package in editable mode with the [dev,html] extras, and exposes aborist at .venv/bin/aborist. No system-wide install. Re-running make bootstrap is a no-op if the venv is up to date.

After bootstrap, every workflow lives behind a make target. Run make help to list them.

Data: Wikipedia 2003-05-16 (Phase III SQL dump)

The 2003 dataset lives at https://dumps.wikimedia.org/archive/2003/2003-05-16/en/ as three files. This is a MySQL extended-INSERT format dump; for XML-format dumps from 2006 onward see the Phase IV section below.

file	size	what
20030516_cur_tablesql.bz2	82 MB	one row per article: snapshot of every Wikipedia page on 2003-05-16
old_tablesqlbz2.1	640 MiB	first half of the revision-history table (split bzip2 stream)
old_tablesqlbz2.2	252 MiB	second half — cat them together to decompress

make fetch-cur                # snapshot only (~82 MB on disk)
make fetch-old                # full history (~1.4 GB on disk after concat)
make fetch                    # both

Files land in data/. Re-running is idempotent (curl skips if already present).

Sharded ingest (the canonical path)

Per-shard SQLite files, no write-lock contention. Each shard process owns its own DB; cross-shard reads attach all shards as UNION ALL views.

make ingest-cur-attached SHARDS=4    # cur snapshot, 4 parallel shards (~3 min)
make ingest-old-attached SHARDS=4    # full history, ~30–40 min

Shards land in ~/.aborist/shards/. Override with SHARDS_DIR=/path/to/somewhere.

Resumable

Add --resume (or just re-run the make target — --resume is the default for attached ingests). Each shard tracks its own high-water mark in meta; an interrupted ingest picks up where it left off without re-hashing already-cached docs.

Single-DB ingest (simpler, smaller)

For experiments under a few thousand docs, a single SQLite file is fine:

make ingest-cur INGEST_LIMIT=1000       # one DB at $(DB), default ~/.aborist/aborist.db
make ingest-cur-parallel SHARDS=4       # 4 processes, one shared DB (WAL serialized)

Distillation (cores feed retrieval)

Two distillers ship: first-sentence-v1 (one sentence per chunk) and tfidf-keywords-v1 (top-K distinctive terms per doc). Cores are Merkle-signed back to their source docs and serve as enriched titles for retrieval.

make distill-shards-parallel              # first-sentence cores, per-shard
make distill-shards-tfidf-parallel        # TF-IDF cores, per-shard

Run both — they generate independent cores per source. TF-IDF cores let neologisms (like a personal term that never appears in any title) match retrievals via keyword overlap.

Data: Wikipedia 2010-11 (and other Phase IV snapshots)

In 2006 MediaWiki swapped its dumps from MySQL INSERT INTO cur syntax to XML. Aborist reads both — the SQL path above for 2003-2005 cur dumps, and a streaming XML path for any dated snapshot in https://dumps.wikimedia.org/archive/. The largest single snapshot in that archive is enwiki 2010-11-08:

file	size	what
enwiki-20101011-pages-articles.xml.bz2	6.2 GB	latest revision of every main-namespace article on 2010-11-08 (~3.4M pages, ~1.9M after redirects)
enwiki-20101011-abstract.xml	2.9 GB	first-paragraph abstracts only — pre-distilled summaries at ~1/100th the chunk volume

Other useful dated snapshots in the archive: 2006-07 (1.8 GB), 2006-12 (1.9 GB), 2010-03 (varies by language). All work via the same source class.

# defaults target enwiki 20101011 (2010-11)
make fetch-xml                # ~6.2 GB compressed download
make ingest-xml-attached SHARDS=4    # ~2 hours sharded; ~95 GB on disk after

# pick any other snapshot by overriding the date variables
make fetch-xml WP_XML_YEAR=2006 WP_XML_MONTH=2006-07 WP_XML_DATE=20061104

# abstract feed (one-paragraph summaries, full coverage at ~5-10 GB total)
make fetch-abstract
make ingest-abstract

The XML source streams .xml.bz2 directly via iterparse with bounded memory (each <page> is processed and cleared). Same shard / resume / Merkle contract as the SQL source. Title-prefix namespace filtering kicks in for older export schemas that omit per-page <ns>.

To ingest historical revisions instead of just the current snapshot, point WP_XML at a pages-meta-history.xml.bz2 file and use make ingest-xml-history — the source emits one Document per revision and aborist's prior-doc detection chains them with supersedes edges.

Data: personal Grok export

If you have an xAI data-export bundle, point GROK_EXPORT at its root directory (the one containing ttl/30d/export_data/<user-id>/):

export GROK_EXPORT=$HOME/Downloads/<your-user-uuid>
make ingest-grok-attached            # conversations  -> $(SHARDS_DIR)/grok.db
make ingest-grok-media-attached      # media prompts  -> $(SHARDS_DIR)/grok.db

Both walk the export tree, find prod-grok-backend.json, and yield one Document per conversation (or media-generation post). Conversation titles, full message text, and turn ordering are preserved. Each becomes a normal queryable doc in the cluster — your prior chats become memory the corpus can consult.

--resume is the default for these targets. Re-run any time to pick up new exports.

Data: git and Mercurial repos (self-play)

Aborist can consult itself. Point a source at any local clone and every text file at HEAD becomes a queryable Document; re-ingesting after new commits chains old → new via supersedes edges, so the audit trail grows alongside the repo:

make ingest-self                         # this aborist tree, into ~/.aborist/shards/aborist-self.db
make ingest-git GIT_REPO=/path/to/repo   # any other git clone
make ingest-hg  HG_REPO=/path/to/repo    # mercurial flavor

URI shape: git://<repo-name>/file/<relative-path> (no commit hash — that's what enables the supersedes chain on re-ingest). Binary files are skipped (NUL-byte heuristic + UTF-8 decode probe). extra carries the current commit hash, timestamp, and subject for informational purposes; the cryptographic identity is the content-derived document_root as for every other Document.

Data: OpenAI / ChatGPT export (planned)

Not implemented yet. The shape will be one new Source subclass at aborist/sources/openai.py plus a Makefile target. The OpenAI ChatGPT data export is a .zip containing conversations.json with the mapping/messages tree shape. Adding it follows the same pattern as aborist/sources/grok.py — see that file as the template.

# (placeholder)
make ingest-openai-attached         # OPENAI_EXPORT=$HOME/Downloads/<chatgpt-export>

When this lands, conversations from both Grok and OpenAI will sit in the same shard cluster; queries fan out across all of them.

Asking the corpus

make query Q="What is the philosophy of stoicism?"
make query Q="tell me about permacomputer ?"
make query QUERY_TOP_K=12 Q="…"        # widen the source set
make query-dry Q="…"                    # assemble context but skip the LLM call

The query path:

1. Search — FTS5 (body) + SQL LIKE (title) + JOIN over derivations (TF-IDF core keywords) across every shard. Three accept paths to the relevance filter.
2. Concept overlay — synonym groups (Athlon ↔ AMD) widen retrieval; rivalry pairs (AMD ↔ Intel) narrow it unless the query has comparative phrasing ("compare X vs Y").
3. Context assembly — top-K sources concatenated up to a 60 KB budget, fed to Hermes-3 with strict attribution rules in the system prompt.
4. Cache — the v9.8 8-dim cache_key (source_root | question_hash | model_profile | conversation | governance_policy | schema | canonicalization | chunking) keys the answer in qa.db. Cache hits replay STRICT-mode in ~100 ms. Per-phase timings are returned in every result.

LLM endpoint defaults to https://hermes.ai.unturf.com/v1 (Hermes-3 Llama-3.1-8B, 82K context, no auth). Override:

export ABORIST_LLM_ENDPOINT="https://your-vllm.example/v1"
export ABORIST_LLM_MODEL="meta-llama/Llama-3.1-70B-Instruct"
export ABORIST_LLM_API_KEY="..."

Mark a wrong answer

If the LLM produced something incorrect, mark its cache record stale so the next ask re-runs inference fresh:

make falsify KEY=<cache_key> REASON="why it was wrong"

The original record stays in the database (history matters). A falsifications row + falsify audit event record the act. Future lookups skip records whose falsification_state != 'live'.

Inspecting

make stats-shards                    # totals across shards
make analyze-shards                  # compression spectrum, depth histogram, audit chain integrity
make verify-shards                   # round-trip Merkle proofs on a random sample
make activity                        # recent Q&A + ingests + derives + falsifications (agent timeline)
make activity ACTIVITY_LIMIT=20

activity is JSON; pipe through jq to drill in.

Architecture (one screenful)

aborist/
├── merkle.py           Python port of proxy.unturf.com/pkg/verified/merkle.go
│                       conventions: non-commutative HashCombine (0x03 prefix),
│                       explicit IsLeft per sibling, self-duplicate odd elements.
├── store.py            SQLite v9.8 schema (8-dim cache key, falsification state,
│                       audit chain, surface/core kinds, hot/warm/cold tier).
├── ingest.py           normalize → chunk → merkle → upsert. Bulk-batched writer.
├── document.py         Document, Edge, Chunker (tok-512-v1 default).
├── source.py           Source ABC: iter_documents() -> Iterator[Document].
├── sources/
│   ├── wikipedia.py    cur + old MediaWiki SQL dumps (bz2-streamed).
│   ├── html_page.py    URL list + selectolax + httpx (robots-aware).
│   └── grok.py         xAI data export (conversations + media prompts).
├── distill/            Distiller ABC + first_sentence + tfidf + recursion runner.
├── search/             FTS5 backend + SearchBackend ABC + AuditMode.
├── qa/                 ask (single-doc) + query (multi-source RAG) + concepts overlay.
├── evict.py            hot ↔ cold tier transitions; rehydrate via source pipeline.
└── cli.py              argparse entrypoint (the make targets call into here).

Source papers (read first if confused):

• ~/Downloads/merkle-providence-reverse-rag-whitepaper.pdf — public spec, AGPL-3.0
• ~/Downloads/merkle-agi-dag_v7.txt — formal substrate (TLV/canonical encoding, theorems T1–T5)

Tests

make test            # 79 tests, all stdlib + pytest
make bench           # ETL throughput across configs (serial / shared-WAL / attached)

License

License: AGPL-3.0-only · This algorithm, its implementation, & all associated code carry the GNU Affero General Public License v3.0 (only). You may use, modify, & distribute under those terms. No proprietary relicensing exists.

(Verbatim from the Merkle Providence Reverse RAG whitepaper, April 2026.)