# arborist An arborist for trees and forests of cross-linked information. Arborist ingests documents into a content-addressed, Merkle-committed SQLite store, distills them into recursive cores, and answers questions over the resulting corpus via an OpenAI-compatible LLM. Every cached answer carries a verifiable Merkle proof tying it back to its source documents — Merkle-AGI v9.8 / Merkle Providence Reverse RAG, runnable end-to-end. ## Quickstart Two end-to-end paths. Pick whichever corpus you want first; both share the same query, verify, falsify, and inspect surfaces. > **Windows / no `make`?** Every `make ` below works verbatim — a bundled `make.bat` shim forwards to a pure-stdlib runner (`tasks.py`). The quickstart needs only **Python 3.10+** (no `make`, no `bzip2`, no `curl`). In `cmd` type `make query Q="…"`; in PowerShell use `.\make.bat query Q="…"` (or `py -3 tasks.py query Q="…"` directly). See [Setup → Windows](#windows). The Makefile remains the canonical path on Linux/macOS. ### A. Wikipedia 2003 (the canonical bootstrap dataset) ```sh make bootstrap # one-time: venv + dev extras make fetch-cur # download the 2003-05-16 snapshot (~82 MB) make ingest-cur-attached # ~3 min: 128k articles, 4 parallel shards make distill-shards-parallel # surface → core (first-sentence) make distill-shards-tfidf-parallel # core → keyword sets for retrieval make query Q="What is anarcho-capitalism?" ``` A Hermes-3 inference runs against the local corpus, picks 4–8 source articles by Merkle root, and returns an answer plus a verifier label that names what the lexical verifier could confirm. The current four-rung ladder for claim-lattice modes is `POINTER-LINKED` → `ANCHOR-WARRANTED` → `EVIDENCE-WARRANTED` → `UNGROUNDED` (with `-PARTIAL` suffix on HYBRID). Repeat the same question and a cache hit replays in ~100 ms. ### B. Crawl any live website and query it ```sh make bootstrap-crawler # one-time: install [crawler] extras make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2 # BFS + ingest make query Q="who is Russell Ballestrini?" # cross-shard query — picks up the new shard automatically ``` The crawl shard is named after the seed hostname (`crawl_russell_ballestrini_net.db`) under `~/.arborist/shards/`. Add `FAST=1` for aggressive crawling against your own sites; `MAX=N` to cap discovery; `DEPTH=N` to bound BFS. Robots `Disallow` is always honored. After ingest, `make recrawl-check DOMAIN=russell.ballestrini.net` does a conditional-HEAD freshness probe per page. ### After the answer ```sh make inspect KEY= # sidecar: classify each unverified span make falsify KEY= REASON='…' # mark wrong, keep history make burn KEY= REASON='…' # delete (kindergarten only — refuses if children exist) ``` `make help` lists every target. ## Setup ### Get the source ``` git clone ssh://git@git.unturf.com:2222/engineering/unturf/arborist.git cd arborist ``` HTTPS variant if SSH isn't set up: ``` git clone https://git.unturf.com/engineering/unturf/arborist.git ``` ### Install prerequisites The **quickstart needs only Python 3.10+** (SQLite ≥ 3.35 ships with CPython; the 2003 dump is decompressed in-process via Python's `bz2`). `GNU make`, `curl`, and `bzip2` are conveniences for the **Makefile** path on Linux/macOS — on Windows the bundled `make.bat` / `tasks.py` need none of them (see [Windows](#windows)). **macOS** (Homebrew) ``` xcode-select --install # if you don't already have CLT brew install python@3.12 git make ``` Apple's `make` is GNU make, no extra step needed. `bzip2` and `curl` are bundled. **Ubuntu / Debian** ``` sudo apt update sudo apt install -y git python3 python3-venv python3-dev build-essential curl bzip2 ``` 22.04 ships Python 3.10; 24.04 ships 3.12 — both work. **Windows** (native — no `make`, no WSL) Install Python 3.10+ from and tick **"Add python.exe to PATH"**. That is the only prerequisite — the runner uses stdlib `urllib` for the download and stdlib `bz2` for decompression, so there is no `make`, `curl`, or `bzip2` to install. The bundled `make.bat` makes every documented `make ` work as-is: ```bat :: in cmd.exe (run from the repo root) make bootstrap make fetch-cur make ingest-cur-attached make distill-shards-parallel make distill-shards-tfidf-parallel make query Q="What is anarcho-capitalism?" ``` ```powershell # in PowerShell, prefix with .\ (PowerShell doesn't search the current dir) .\make.bat bootstrap .\make.bat query Q="What is anarcho-capitalism?" ``` Or call the runner directly with the `py` launcher: `py -3 tasks.py `. `py -3 tasks.py help` lists every supported target. The venv lands at `.venv\Scripts\` (vs `.venv/bin/` on POSIX); `tasks.py` resolves that automatically. Override the interpreter used to build the venv with the `ARBORIST_PYTHON` env var (e.g. `set ARBORIST_PYTHON=py -3.12`). `tasks.py` covers the **quickstart subset** — bootstrap, fetch/ingest/distill, query/inspect/falsify/burn, the crawl path, stats/verify/search, clean. The bench / π* / NLI / textbook / docs targets stay Makefile-only; for those (or to use the canonical Makefile) install **WSL2** and follow the Ubuntu instructions: ``` wsl --install -d Ubuntu-24.04 ``` **OpenBSD** ``` pkg_add git python-3.12 gmake curl ``` OpenBSD's default `make` is BSD make. Arborist's Makefile uses GNU-make features (`?=`, conditional functions). Substitute `gmake` for `make` in every command, e.g. `gmake bootstrap`, `gmake query Q='…'`. ### Bootstrap ``` make bootstrap ``` Creates `.venv/`, installs the package in editable mode with the `[dev]` extras, and exposes `arborist` at `.venv/bin/arborist` (`.venv\Scripts\arborist.exe` on Windows). No system-wide install. Re-running `make bootstrap` is a no-op if the venv is up to date. After bootstrap, every workflow lives behind a `make` target. Run `make help` to list them (Windows: `py -3 tasks.py help` for the quickstart subset). ## Data: Wikipedia 2003-05-16 (Phase III SQL dump) The 2003 dataset lives at as three files. This is a MySQL extended-INSERT format dump; for XML-format dumps from 2006 onward see the Phase IV section below. | file | size | what | |---|---|---| | `20030516_cur_tablesql.bz2` | 82 MB | one row per article: snapshot of every Wikipedia page on 2003-05-16 | | `old_tablesqlbz2.1` | 640 MiB | first half of the revision-history table (split bzip2 stream) | | `old_tablesqlbz2.2` | 252 MiB | second half — `cat` them together to decompress | ``` make fetch-cur # snapshot only (~82 MB on disk) make fetch-old # full history (~1.4 GB on disk after concat) make fetch # both ``` Files land in `data/`. Re-running is idempotent (curl skips if already present). ### Sharded ingest (the canonical path) Per-shard SQLite files, no write-lock contention. Each shard process owns its own DB; cross-shard reads attach all shards as `UNION ALL` views. ``` make ingest-cur-attached SHARDS=4 # cur snapshot, 4 parallel shards (~3 min) make ingest-old-attached SHARDS=4 # full history, ~30–40 min ``` Shards land in `~/.arborist/shards/`. Override with `SHARDS_DIR=/path/to/somewhere`. ### Resumable Add `--resume` (or just re-run the make target — `--resume` is the default for attached ingests). Each shard tracks its own high-water mark in `meta`; an interrupted ingest picks up where it left off without re-hashing already-cached docs. ### Single-DB ingest (simpler, smaller) For experiments under a few thousand docs, a single SQLite file is fine: ``` make ingest-cur INGEST_LIMIT=1000 # one DB at $(DB), default ~/.arborist/arborist.db make ingest-cur-parallel SHARDS=4 # 4 processes, one shared DB (WAL serialized) ``` ### Distillation (cores feed retrieval) Two distillers ship: `first-sentence-v1` (one sentence per chunk) and `tfidf-keywords-v1` (top-K distinctive terms per doc). Cores are Merkle-signed back to their source docs and serve as enriched titles for retrieval. ``` make distill-shards-parallel # first-sentence cores, per-shard make distill-shards-tfidf-parallel # TF-IDF cores, per-shard ``` Run both — they generate independent cores per source. TF-IDF cores let neologisms (like a personal term that never appears in any title) match retrievals via keyword overlap. ## Data: Wikipedia 2010-11 (and other Phase IV snapshots) In 2006 MediaWiki swapped its dumps from MySQL `INSERT INTO cur` syntax to XML. Arborist reads both — the SQL path above for 2003-2005 cur dumps, and a streaming XML path for any dated snapshot in . The largest single snapshot in that archive is enwiki 2010-11-08: | file | size | what | |---|---|---| | `enwiki-20101011-pages-articles.xml.bz2` | 6.2 GB | latest revision of every main-namespace article on 2010-11-08 (~3.4M pages, ~1.9M after redirects) | | `enwiki-20101011-abstract.xml` | 2.9 GB | first-paragraph abstracts only — pre-distilled summaries at ~1/100th the chunk volume | Other useful dated snapshots in the archive: 2006-07 (1.8 GB), 2006-12 (1.9 GB), 2010-03 (varies by language). All work via the same source class. ``` # defaults target enwiki 20101011 (2010-11) make fetch-xml # ~6.2 GB compressed download make ingest-xml-attached SHARDS=4 # ~2 hours sharded; ~95 GB on disk after # pick any other snapshot by overriding the date variables make fetch-xml WP_XML_YEAR=2006 WP_XML_MONTH=2006-07 WP_XML_DATE=20061104 # abstract feed (one-paragraph summaries, full coverage at ~5-10 GB total) make fetch-abstract make ingest-abstract ``` The XML source streams `.xml.bz2` directly via `iterparse` with bounded memory (each `` is processed and cleared). Same shard / resume / Merkle contract as the SQL source. Title-prefix namespace filtering kicks in for older export schemas that omit per-page ``. To ingest historical revisions instead of just the current snapshot, point `WP_XML` at a `pages-meta-history.xml.bz2` file and use `make ingest-xml-history` — the source emits one Document per revision and arborist's prior-doc detection chains them with `supersedes` edges. ## Data: personal Grok export If you have an xAI data-export bundle, point `GROK_EXPORT` at its root directory (the one containing `ttl/30d/export_data//`): ``` export GROK_EXPORT=$HOME/Downloads/ make ingest-grok-attached # conversations -> $(SHARDS_DIR)/grok.db make ingest-grok-media-attached # media prompts -> $(SHARDS_DIR)/grok.db ``` Both walk the export tree, find `prod-grok-backend.json`, and yield one Document per conversation (or media-generation post). Conversation titles, full message text, and turn ordering are preserved. Each becomes a normal queryable doc in the cluster — your prior chats become memory the corpus can consult. `--resume` is the default for these targets. Re-run any time to pick up new exports. ## Data: git and Mercurial repos (self-play) Arborist can consult itself. Point a source at any local clone and every text file at HEAD becomes a queryable Document; re-ingesting after new commits chains old → new via `supersedes` edges, so the audit trail grows alongside the repo: ``` make ingest-self # this arborist tree, into ~/.arborist/shards/arborist-self.db make ingest-git GIT_REPO=/path/to/repo # any other git clone make ingest-hg HG_REPO=/path/to/repo # mercurial flavor ``` URI shape: `git:///file/` (no commit hash — that's what enables the supersedes chain on re-ingest). Binary files are skipped (NUL-byte heuristic + UTF-8 decode probe). `extra` carries the current commit hash, timestamp, and subject for informational purposes; the cryptographic identity is the content-derived `document_root` as for every other Document. ## Data: live websites (the crawler) Arborist can BFS-discover and ingest a website starting from a seed URL, respecting robots.txt and crawl delays. The crawler is **off by default** — heavy deps (aiohttp, bs4, lxml, mwparserfromhell, etc.) ship as the `[crawler]` extras and aren't pulled into the default test suite. ``` make bootstrap-crawler # one-time, install [crawler] extras make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2 # BFS + ingest the discovered pages ``` The shard filename is derived from the seed URL's hostname so cross-shard query picks it up automatically: ``` URL=https://russell.ballestrini.net → $(SHARDS_DIR)/crawl_russell_ballestrini_net.db ``` After ingest, the same `make query` searches across crawl shards alongside Wikipedia, Grok, and self-play sources: ``` make query Q="who is Russell Ballestrini?" ``` Knobs: | variable | default | what | |---|---|---| | `URL=` | (required) | seed URL; BFS stays on its hostname (no subdomain crossover) | | `DEPTH=` | `2` | max BFS depth from seed | | `MAX=` | `0` | cap discovery at N URLs (`0` = no cap, depth is the only bound) | | `FAST=1` | unset | flip the verbatim AsyncWebFetcher into fast_mode: 5s timeouts, CPU×3 parallel page workers, ignore robots.txt `crawl-delay`. Disallow is still honored. Use only against sites where aggressive fetching is acceptable. | | `CRAWL_SHARD=` | derived from URL | override the destination shard path | Feeds and sitemaps (`atom`, `rss`, `sitemap.xml`, `wp-rss2.xml`, etc.) are skipped at ingest — they're discovery infrastructure, not knowledge. ETag and Last-Modified per page are captured so a future probe can ask "does this need recrawling?" without re-fetching bodies: ``` make recrawl-check DOMAIN=russell.ballestrini.net ``` Conditional `If-None-Match` / `If-Modified-Since` HEAD requests classify each ingested doc as fresh (304), stale (200), gone (404/410), or unreachable. One tiny round trip per URL with no body transfer when content's unchanged. The crawler is a verbatim lift from `~/git/agents.ai.unturf.com/core/` (provenance documented in `arborist/sources/crawler/__init__.py`); arborist-side changes drop the chat-bot fetch triggers and skip `web_cache_manager.py` (arborist has its own content-addressed cache). Run `make test-crawler` for the lift's own tests. ## Asking the corpus ``` make query Q="What is the philosophy of stoicism?" make query Q="tell me about permacomputer ?" make query Q="…" QUERY_TOP_K=12 # widen the source set make query Q="…" JSON=1 # raw record (cache_key, merkle_proof, timings, full sources) make query-dry Q="…" # assemble context but skip the LLM call ``` Default render is human-readable: question, audit-mode summary line, answer, sources list, unverified spans, short cache_key. `JSON=1` gives the full record. Same trailing-question-mark question deduplicates (`question_hash` strips trailing `.?!,;:？！。、…` after lowercasing). The query path: 1. **Search** — FTS5 (body) + SQL `LIKE` (title) + `JOIN` over derivations (TF-IDF core keywords) across every shard. Three accept paths to the relevance filter. 2. **Concept overlay** — per-shard `concept_relations` SQLite table (corpus-derived, not hand-curated). Synonyms widen retrieval; rivalries narrow it unless the query uses comparative phrasing ("compare X vs Y"). Built-in extractor `link_reciprocity_synonym` reads the existing `edges` table for reciprocal A↔B link pairs and emits synonym edges between their title-tokens. ~1.6% storage tax measured on 6 GB Wikipedia. 3. **Context assembly** — top-K sources concatenated up to a 60 KB budget. Wikitext is stripped to plain prose via `arborist.wikitext.to_base()` (the corpus stores raw `[[wikilinks]]` so the link graph is recoverable on demand; the LLM and verifier both see clean prose). 4. **LLM** — Hermes-3 with strict attribution rules in the system prompt + a user-turn grounding reminder. 5. **Verifier** — every claim runs through a layered lexical check; the result rolls up into the v9.8 trichotomy (`audit_mode` ∈ STRICT / HYBRID / UNGROUNDED) at the schema layer AND a four-rung display ladder at render time (POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → UNGROUNDED). See below. 6. **Cache** — the v9.8 8-dim cache_key (`source_root | question_hash | model_profile | conversation | governance_policy | schema | canonicalization | chunking`) keys the answer in `qa.db`. Cache hits replay in ~100 ms. Per-phase timings in every result. LLM endpoint defaults to `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B, 82K context, no auth). Override: ``` export ARBORIST_LLM_ENDPOINT="https://your-vllm.example/v1" export ARBORIST_LLM_MODEL="meta-llama/Llama-3.1-70B-Instruct" export ARBORIST_LLM_API_KEY="..." ``` ## Verifying answers (audit modes & label ladder) Two layers of labels stack on every answer: **Schema layer — v9.8 trichotomy** (`audit_mode` column, persisted, drives cache lookups & audit chain): | mode | meaning | |---|---| | **STRICT** | every evidence unit verifies against context | | **HYBRID** | some claims source-grounded, some emerged from training | | **UNGROUNDED** | no evidence, or none verifies — purely emergent | **Display layer — four-rung ladder** (claim-lattice modes only; renderer-only transformation, schema unchanged): | rung | what's actually proved | |---|---| | **EVIDENCE-WARRANTED** | pointer verified + warrant ran & passed + no soft demotes | | **ANCHOR-WARRANTED** | pointer-linked + warrant passed; soft-demote violations present | | **POINTER-LINKED** | pointer/source/chunk verified, but warrant either didn't apply or failed for some claim | | **UNGROUNDED** | no verified pairs | HYBRID gets a `-PARTIAL` suffix on whichever rung applies. The point of the display ladder: `STRICT in claim_lattice mode` is NOT "the answer is correct" — it's "every pointer resolved to a valid evidence object whose source_role is allowed AND citation-coverage passed." The display label spells out the actual property so users don't read STRICT as full semantic entailment. **Verifier strategies** run in sequence; first to find evidence classifies. Each is lexical (substring or token-coverage), never embeddings — soft signals stay out of the proof path. | # | strategy | what triggers it | |---|---|---| | 1 | `quote` | model wraps claims in `"..."`; each verbatim-substring tested | | 2 | `span` | bullet/sentence units substring-tested as fallback | | 3 | `entity` | multi-word proper nouns tested with proximity-cluster gating | | 4 | `paraphrase` | inside the span path: ≥85% token coverage on prose-shaped spans | | 5 | `claim_lattice` | model emits `claim text. [E1,E2]` pointer-line OR `{"claims":[{"text":..., "evidence_ids":[...]}]}` JSON; verifier resolves pointers to runtime-built evidence objects & runs seven hard checks | The `claim_lattice` path runs **seven deterministic hard checks**: parser succeeded, evidence_id resolves, source_role allowed, claim text non-empty, citation coverage threshold, pointer count cap, anchor-class warrant. Anchor-class warrant composes five lexical anchor classes — proper-noun, date, entity-list, count (with digit↔word equivalence), and why-cause — each gated on either question shape, claim content, or both. See whitepaper §13.9 for the architecture. Trailing `(Source: https://...)` parentheticals the model appends to verbatim source sentences are stripped before substring testing, so verbatim-with-citation no longer flags HYBRID. `unverified_quotes` on each record is the corpus-growth signal — model output that didn't ground anywhere. `arborist emergent --aggregate` ranks them by frequency (the worklist of "things to ingest more sources for"). `arborist reclassify` re-runs the verifier against existing live records after corpus growth without any LLM call; HYBRID promotes to STRICT, UNGROUNDED to HYBRID, and one `providence_reclassify` audit event per change. To dig into a specific record's unverified spans: ``` make inspect KEY= ``` Read-only **sidecar** diagnostic: pulls source chunks, classifies each unverified span as `verbatim_in_base` / `trailing_artifact` / `paraphrase` / `partial_paraphrase` / `no_overlap`. Writes nothing — sidecars never enter the v9.8 hard chain (that invariant is what keeps `audit_mode` a binary classification rather than a soft score). ## Mesh / federation (off by default) Optional gossip layer for peer-to-peer corpus sync. Default off — no code path touches the network unless `mesh.enabled` is set. See [`docs/mesh.md`](docs/mesh.md) for protocol contract and the `arborist mesh` CLI subcommands. ## Inspecting ``` make stats-shards # totals across shards make analyze-shards # compression spectrum, depth histogram, audit chain integrity make verify-shards # round-trip Merkle proofs on a random sample make activity # recent Q&A + ingests + derives + falsifications (agent timeline) make activity ACTIVITY_LIMIT=20 ``` `activity` is JSON; pipe through `jq` to drill in. ## Architecture ![Arborist module graph](docs/diagrams/arborist-modules.svg) Generated API reference (every module, class, and function from docstrings): build with `make docs-api` (output at `docs/_source/_build/html/`) or browse the published Read the Docs site. Pipeline diagrams live in [`docs/diagrams/`](docs/diagrams/); render with `make docs`. ## Tests & bench ``` make test # 2000+ tests, default suite, ~60s sequential / ~10s with pytest -n auto make test-crawler # opt-in: tests for the verbatim crawler lift make test-live # gated: live QA quality fixtures against Hermes (~1 min) make bench # ETL throughput across configs (serial / shared-WAL / attached) make bench-qa # full QA-quality sweep, sample-shuffled, 71q × 3m × 3n at c=4 make bench-qa-smoke # 5-question smoke fixture for prompt-iteration loops (~1-3 min) ``` The default suite never hits the network. The crawler suite is gated behind `make bootstrap-crawler` (installs the `[crawler]` extras). `make bench-qa` writes JSONL + markdown into `bench/qa_results/.{jsonl,md}` (gitignored); design-log entries live in `docs/qa-modes-bench-.md`. The bench is stop/start-able via `--resume ` (same `--seed` required for shuffled-task-order alignment). `BENCH_QA_CONCURRENCY=N` Makefile variable overrides the default `4`. See `docs/bench-maxing.md` for the full speed playbook. ## License License: AGPL-3.0-only · This algorithm, its implementation, & all associated code carry the GNU Affero General Public License v3.0 (only). You may use, modify, & distribute under those terms. No proprietary relicensing exists. ### Permacomputer Preamble This is free software for the public good of a permacomputer hosted at [permacomputer.com](https://www.permacomputer.com), an always-on computer by the people, for the people. Durable, easy to repair, & distributed like tap water for machine learning intelligence. Our permacomputer is community-owned infrastructure optimized around four values: - **TRUTH** — First principles, math & science, open source code freely distributed. - **FREEDOM** — Voluntary partnerships, freedom from tyranny & corporate control. - **HARMONY** — Minimal waste, self-renewing systems with diverse thriving connections. - **LOVE** — Be yourself without hurting others, cooperation through natural law. NO WARRANTY. Software is provided "AS IS" without warranty of any kind. Full text in [`LICENSE`](LICENSE). (Verbatim from the [Merkle Providence Reverse RAG whitepaper](https://unfirehose.com/merkle-providence-reverse-rag-whitepaper.pdf), April 2026.)