arborist/README.md

# arborist

An arborist for trees and forests of cross-linked information.

Arborist ingests documents into a content-addressed, Merkle-committed SQLite store, distills them into recursive cores, and answers questions over the resulting corpus via an OpenAI-compatible LLM. Every cached answer carries a verifiable Merkle proof tying it back to its source documents — Merkle-AGI v9.8 / Merkle Providence Reverse RAG, runnable end-to-end.

## Quickstart

Two end-to-end paths. Pick whichever corpus you want first; both share the same query, verify, falsify, and inspect surfaces.

> **Windows / no `make`?** Every `make <target>` below works verbatim — a bundled `make.bat` shim forwards to a pure-stdlib runner (`tasks.py`). The quickstart needs only **Python 3.10+** (no `make`, no `bzip2`, no `curl`). In `cmd` type `make query Q="…"`; in PowerShell use `.\make.bat query Q="…"` (or `py -3 tasks.py query Q="…"` directly). See [Setup → Windows](#windows). The Makefile remains the canonical path on Linux/macOS.

### A. Wikipedia 2003 (the canonical bootstrap dataset)

```sh
make bootstrap                          # one-time: venv + dev extras
make fetch-cur                          # download the 2003-05-16 snapshot (~82 MB)
make ingest-cur-attached                # ~3 min: 128k articles, 4 parallel shards
make distill-shards-parallel            # surface → core (first-sentence)
make distill-shards-tfidf-parallel      # core → keyword sets for retrieval
make query Q="What is anarcho-capitalism?"
```

A Hermes-3 inference runs against the local corpus, picks 4–8 source articles by Merkle root, and returns an answer plus a verifier label that names what the lexical verifier could confirm. The current four-rung ladder for claim-lattice modes is `POINTER-LINKED` → `ANCHOR-WARRANTED` → `EVIDENCE-WARRANTED` → `UNGROUNDED` (with `-PARTIAL` suffix on HYBRID). Repeat the same question and a cache hit replays in ~100 ms.

### B. Crawl any live website and query it

```sh
make bootstrap-crawler                                            # one-time: install [crawler] extras
make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2     # BFS + ingest
make query Q="who is Russell Ballestrini?"                        # cross-shard query — picks up the new shard automatically
```

The crawl shard is named after the seed hostname (`crawl_russell_ballestrini_net.db`) under `~/.arborist/shards/`. Add `FAST=1` for aggressive crawling against your own sites; `MAX=N` to cap discovery; `DEPTH=N` to bound BFS. Robots `Disallow` is always honored. After ingest, `make recrawl-check DOMAIN=russell.ballestrini.net` does a conditional-HEAD freshness probe per page.

### After the answer

```sh
make inspect KEY=<cache_key>           # sidecar: classify each unverified span
make falsify KEY=<cache_key> REASON='…' # mark wrong, keep history
make burn    KEY=<cache_key> REASON='…' # delete (kindergarten only — refuses if children exist)
```

`make help` lists every target.

## Setup

### Get the source

```
git clone ssh://git@git.unturf.com:2222/engineering/unturf/arborist.git
cd arborist
```

HTTPS variant if SSH isn't set up:

```
git clone https://git.unturf.com/engineering/unturf/arborist.git
```

### Install prerequisites

The **quickstart needs only Python 3.10+** (SQLite ≥ 3.35 ships with CPython; the 2003 dump is decompressed in-process via Python's `bz2`). `GNU make`, `curl`, and `bzip2` are conveniences for the **Makefile** path on Linux/macOS — on Windows the bundled `make.bat` / `tasks.py` need none of them (see [Windows](#windows)).

**macOS** (Homebrew)

```
xcode-select --install        # if you don't already have CLT
brew install python@3.12 git make
```

Apple's `make` is GNU make, no extra step needed. `bzip2` and `curl` are bundled.

**Ubuntu / Debian**

```
sudo apt update
sudo apt install -y git python3 python3-venv python3-dev build-essential curl bzip2
```

22.04 ships Python 3.10; 24.04 ships 3.12 — both work.

<a id="windows"></a>**Windows** (native — no `make`, no WSL)

Install Python 3.10+ from <https://www.python.org/downloads/> and tick **"Add python.exe to PATH"**. That is the only prerequisite — the runner uses stdlib `urllib` for the download and stdlib `bz2` for decompression, so there is no `make`, `curl`, or `bzip2` to install.

The bundled `make.bat` makes every documented `make <target>` work as-is:

```bat
:: in cmd.exe (run from the repo root)
make bootstrap
make fetch-cur
make ingest-cur-attached
make distill-shards-parallel
make distill-shards-tfidf-parallel
make query Q="What is anarcho-capitalism?"
```

```powershell
# in PowerShell, prefix with .\  (PowerShell doesn't search the current dir)
.\make.bat bootstrap
.\make.bat query Q="What is anarcho-capitalism?"
```

Or call the runner directly with the `py` launcher: `py -3 tasks.py <target>`. `py -3 tasks.py help` lists every supported target. The venv lands at `.venv\Scripts\` (vs `.venv/bin/` on POSIX); `tasks.py` resolves that automatically. Override the interpreter used to build the venv with the `ARBORIST_PYTHON` env var (e.g. `set ARBORIST_PYTHON=py -3.12`).

`tasks.py` covers the **quickstart subset** — bootstrap, fetch/ingest/distill, query/inspect/falsify/burn, the crawl path, stats/verify/search, clean. The bench / π* / NLI / textbook / docs targets stay Makefile-only; for those (or to use the canonical Makefile) install **WSL2** and follow the Ubuntu instructions:

```
wsl --install -d Ubuntu-24.04
```

**OpenBSD**

```
pkg_add git python-3.12 gmake curl
```

OpenBSD's default `make` is BSD make. Arborist's Makefile uses GNU-make features (`?=`, conditional functions). Substitute `gmake` for `make` in every command, e.g. `gmake bootstrap`, `gmake query Q='…'`.

### Bootstrap

```
make bootstrap
```

Creates `.venv/`, installs the package in editable mode with the `[dev]` extras, and exposes `arborist` at `.venv/bin/arborist` (`.venv\Scripts\arborist.exe` on Windows). No system-wide install. Re-running `make bootstrap` is a no-op if the venv is up to date.

After bootstrap, every workflow lives behind a `make` target. Run `make help` to list them (Windows: `py -3 tasks.py help` for the quickstart subset).

## Data: Wikipedia 2003-05-16 (Phase III SQL dump)

The 2003 dataset lives at <https://dumps.wikimedia.org/archive/2003/2003-05-16/en/> as three files. This is a MySQL extended-INSERT format dump; for XML-format dumps from 2006 onward see the Phase IV section below.

| file | size | what |
|---|---|---|
| `20030516_cur_tablesql.bz2` | 82 MB | one row per article: snapshot of every Wikipedia page on 2003-05-16 |
| `old_tablesqlbz2.1` | 640 MiB | first half of the revision-history table (split bzip2 stream) |
| `old_tablesqlbz2.2` | 252 MiB | second half — `cat` them together to decompress |

```
make fetch-cur                # snapshot only (~82 MB on disk)
make fetch-old                # full history (~1.4 GB on disk after concat)
make fetch                    # both
```

Files land in `data/`. Re-running is idempotent (curl skips if already present).

### Sharded ingest (the canonical path)

Per-shard SQLite files, no write-lock contention. Each shard process owns its own DB; cross-shard reads attach all shards as `UNION ALL` views.

```
make ingest-cur-attached SHARDS=4    # cur snapshot, 4 parallel shards (~3 min)
make ingest-old-attached SHARDS=4    # full history, ~30–40 min
```

Shards land in `~/.arborist/shards/`. Override with `SHARDS_DIR=/path/to/somewhere`.

### Resumable

Add `--resume` (or just re-run the make target — `--resume` is the default for attached ingests). Each shard tracks its own high-water mark in `meta`; an interrupted ingest picks up where it left off without re-hashing already-cached docs.

### Single-DB ingest (simpler, smaller)

For experiments under a few thousand docs, a single SQLite file is fine:

```
make ingest-cur INGEST_LIMIT=1000       # one DB at $(DB), default ~/.arborist/arborist.db
make ingest-cur-parallel SHARDS=4       # 4 processes, one shared DB (WAL serialized)
```

### Distillation (cores feed retrieval)

Two distillers ship: `first-sentence-v1` (one sentence per chunk) and `tfidf-keywords-v1` (top-K distinctive terms per doc). Cores are Merkle-signed back to their source docs and serve as enriched titles for retrieval.

```
make distill-shards-parallel              # first-sentence cores, per-shard
make distill-shards-tfidf-parallel        # TF-IDF cores, per-shard
```

Run both — they generate independent cores per source. TF-IDF cores let neologisms (like a personal term that never appears in any title) match retrievals via keyword overlap.

## Data: Wikipedia 2010-11 (and other Phase IV snapshots)

In 2006 MediaWiki swapped its dumps from MySQL `INSERT INTO cur` syntax to XML. Arborist reads both — the SQL path above for 2003-2005 cur dumps, and a streaming XML path for any dated snapshot in <https://dumps.wikimedia.org/archive/>. The largest single snapshot in that archive is enwiki 2010-11-08:

| file | size | what |
|---|---|---|
| `enwiki-20101011-pages-articles.xml.bz2` | 6.2 GB | latest revision of every main-namespace article on 2010-11-08 (~3.4M pages, ~1.9M after redirects) |
| `enwiki-20101011-abstract.xml` | 2.9 GB | first-paragraph abstracts only — pre-distilled summaries at ~1/100th the chunk volume |

Other useful dated snapshots in the archive: 2006-07 (1.8 GB), 2006-12 (1.9 GB), 2010-03 (varies by language). All work via the same source class.

```
# defaults target enwiki 20101011 (2010-11)
make fetch-xml                # ~6.2 GB compressed download
make ingest-xml-attached SHARDS=4    # ~2 hours sharded; ~95 GB on disk after

# pick any other snapshot by overriding the date variables
make fetch-xml WP_XML_YEAR=2006 WP_XML_MONTH=2006-07 WP_XML_DATE=20061104

# abstract feed (one-paragraph summaries, full coverage at ~5-10 GB total)
make fetch-abstract
make ingest-abstract
```

The XML source streams `.xml.bz2` directly via `iterparse` with bounded memory (each `<page>` is processed and cleared). Same shard / resume / Merkle contract as the SQL source. Title-prefix namespace filtering kicks in for older export schemas that omit per-page `<ns>`.

To ingest historical revisions instead of just the current snapshot, point `WP_XML` at a `pages-meta-history.xml.bz2` file and use `make ingest-xml-history` — the source emits one Document per revision and arborist's prior-doc detection chains them with `supersedes` edges.

## Data: personal Grok export

If you have an xAI data-export bundle, point `GROK_EXPORT` at its root directory (the one containing `ttl/30d/export_data/<user-id>/`):

```
export GROK_EXPORT=$HOME/Downloads/<your-user-uuid>
make ingest-grok-attached            # conversations  -> $(SHARDS_DIR)/grok.db
make ingest-grok-media-attached      # media prompts  -> $(SHARDS_DIR)/grok.db
```

Both walk the export tree, find `prod-grok-backend.json`, and yield one Document per conversation (or media-generation post). Conversation titles, full message text, and turn ordering are preserved. Each becomes a normal queryable doc in the cluster — your prior chats become memory the corpus can consult.

`--resume` is the default for these targets. Re-run any time to pick up new exports.

## Data: git and Mercurial repos (self-play)

Arborist can consult itself. Point a source at any local clone and every text file at HEAD becomes a queryable Document; re-ingesting after new commits chains old → new via `supersedes` edges, so the audit trail grows alongside the repo:

```
make ingest-self                         # this arborist tree, into ~/.arborist/shards/arborist-self.db
make ingest-git GIT_REPO=/path/to/repo   # any other git clone
make ingest-hg  HG_REPO=/path/to/repo    # mercurial flavor
```

URI shape: `git://<repo-name>/file/<relative-path>` (no commit hash — that's what enables the supersedes chain on re-ingest). Binary files are skipped (NUL-byte heuristic + UTF-8 decode probe). `extra` carries the current commit hash, timestamp, and subject for informational purposes; the cryptographic identity is the content-derived `document_root` as for every other Document.

## Data: live websites (the crawler)

Arborist can BFS-discover and ingest a website starting from a seed URL, respecting robots.txt and crawl delays. The crawler is **off by default** — heavy deps (aiohttp, bs4, lxml, mwparserfromhell, etc.) ship as the `[crawler]` extras and aren't pulled into the default test suite.

```
make bootstrap-crawler                                              # one-time, install [crawler] extras
make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2       # BFS + ingest the discovered pages
```

The shard filename is derived from the seed URL's hostname so cross-shard query picks it up automatically:

```
URL=https://russell.ballestrini.net  →  $(SHARDS_DIR)/crawl_russell_ballestrini_net.db
```

After ingest, the same `make query` searches across crawl shards alongside Wikipedia, Grok, and self-play sources:

```
make query Q="who is Russell Ballestrini?"
```

Knobs:

| variable | default | what |
|---|---|---|
| `URL=` | (required) | seed URL; BFS stays on its hostname (no subdomain crossover) |
| `DEPTH=` | `2` | max BFS depth from seed |
| `MAX=` | `0` | cap discovery at N URLs (`0` = no cap, depth is the only bound) |
| `FAST=1` | unset | flip the verbatim AsyncWebFetcher into fast_mode: 5s timeouts, CPU×3 parallel page workers, ignore robots.txt `crawl-delay`. Disallow is still honored. Use only against sites where aggressive fetching is acceptable. |
| `CRAWL_SHARD=` | derived from URL | override the destination shard path |

Feeds and sitemaps (`atom`, `rss`, `sitemap.xml`, `wp-rss2.xml`, etc.) are skipped at ingest — they're discovery infrastructure, not knowledge. ETag and Last-Modified per page are captured so a future probe can ask "does this need recrawling?" without re-fetching bodies:

```
make recrawl-check DOMAIN=russell.ballestrini.net
```

Conditional `If-None-Match` / `If-Modified-Since` HEAD requests classify each ingested doc as fresh (304), stale (200), gone (404/410), or unreachable. One tiny round trip per URL with no body transfer when content's unchanged.

The crawler is a verbatim lift from `~/git/agents.ai.unturf.com/core/` (provenance documented in `arborist/sources/crawler/__init__.py`); arborist-side changes drop the chat-bot fetch triggers and skip `web_cache_manager.py` (arborist has its own content-addressed cache). Run `make test-crawler` for the lift's own tests.

## Asking the corpus

```
make query Q="What is the philosophy of stoicism?"
make query Q="tell me about permacomputer ?"
make query Q="…" QUERY_TOP_K=12        # widen the source set
make query Q="…" JSON=1                # raw record (cache_key, merkle_proof, timings, full sources)
make query-dry Q="…"                    # assemble context but skip the LLM call
```

Default render is human-readable: question, audit-mode summary line, answer, sources list, unverified spans, short cache_key. `JSON=1` gives the full record. Same trailing-question-mark question deduplicates (`question_hash` strips trailing `.?!,;:？！。、…` after lowercasing).

The query path:

1. **Search** — FTS5 (body) + SQL `LIKE` (title) + `JOIN` over derivations (TF-IDF core keywords) across every shard. Three accept paths to the relevance filter.
2. **Concept overlay** — per-shard `concept_relations` SQLite table (corpus-derived, not hand-curated). Synonyms widen retrieval; rivalries narrow it unless the query uses comparative phrasing ("compare X vs Y"). Built-in extractor `link_reciprocity_synonym` reads the existing `edges` table for reciprocal A↔B link pairs and emits synonym edges between their title-tokens. ~1.6% storage tax measured on 6 GB Wikipedia.
3. **Context assembly** — top-K sources concatenated up to a 60 KB budget. Wikitext is stripped to plain prose via `arborist.wikitext.to_base()` (the corpus stores raw `[[wikilinks]]` so the link graph is recoverable on demand; the LLM and verifier both see clean prose).
4. **LLM** — Hermes-3 with strict attribution rules in the system prompt + a user-turn grounding reminder.
5. **Verifier** — every claim runs through a layered lexical check; the result rolls up into the v9.8 trichotomy (`audit_mode` ∈ STRICT / HYBRID / UNGROUNDED) at the schema layer AND a four-rung display ladder at render time (POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → UNGROUNDED). See below.
6. **Cache** — the v9.8 8-dim cache_key (`source_root | question_hash | model_profile | conversation | governance_policy | schema | canonicalization | chunking`) keys the answer in `qa.db`. Cache hits replay in ~100 ms. Per-phase timings in every result.

LLM endpoint defaults to `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B, 82K context, no auth). Override:

```
export ARBORIST_LLM_ENDPOINT="https://your-vllm.example/v1"
export ARBORIST_LLM_MODEL="meta-llama/Llama-3.1-70B-Instruct"
export ARBORIST_LLM_API_KEY="..."
```

## Verifying answers (audit modes & label ladder)

Two layers of labels stack on every answer:

**Schema layer — v9.8 trichotomy** (`audit_mode` column, persisted, drives cache lookups & audit chain):

| mode | meaning |
|---|---|
| **STRICT** | every evidence unit verifies against context |
| **HYBRID** | some claims source-grounded, some emerged from training |
| **UNGROUNDED** | no evidence, or none verifies — purely emergent |

**Display layer — four-rung ladder** (claim-lattice modes only; renderer-only transformation, schema unchanged):

| rung | what's actually proved |
|---|---|
| **EVIDENCE-WARRANTED** | pointer verified + warrant ran & passed + no soft demotes |
| **ANCHOR-WARRANTED** | pointer-linked + warrant passed; soft-demote violations present |
| **POINTER-LINKED** | pointer/source/chunk verified, but warrant either didn't apply or failed for some claim |
| **UNGROUNDED** | no verified pairs |

HYBRID gets a `-PARTIAL` suffix on whichever rung applies. The point of the display ladder: `STRICT in claim_lattice mode` is NOT "the answer is correct" — it's "every pointer resolved to a valid evidence object whose source_role is allowed AND citation-coverage passed." The display label spells out the actual property so users don't read STRICT as full semantic entailment.

**Verifier strategies** run in sequence; first to find evidence classifies. Each is lexical (substring or token-coverage), never embeddings — soft signals stay out of the proof path.

| # | strategy | what triggers it |
|---|---|---|
| 1 | `quote` | model wraps claims in `"..."`; each verbatim-substring tested |
| 2 | `span` | bullet/sentence units substring-tested as fallback |
| 3 | `entity` | multi-word proper nouns tested with proximity-cluster gating |
| 4 | `paraphrase` | inside the span path: ≥85% token coverage on prose-shaped spans |
| 5 | `claim_lattice` | model emits `claim text. [E1,E2]` pointer-line OR `{"claims":[{"text":..., "evidence_ids":[...]}]}` JSON; verifier resolves pointers to runtime-built evidence objects & runs seven hard checks |

The `claim_lattice` path runs **seven deterministic hard checks**: parser succeeded, evidence_id resolves, source_role allowed, claim text non-empty, citation coverage threshold, pointer count cap, anchor-class warrant. Anchor-class warrant composes five lexical anchor classes — proper-noun, date, entity-list, count (with digit↔word equivalence), and why-cause — each gated on either question shape, claim content, or both. See whitepaper §13.9 for the architecture.

Trailing `(Source: https://...)` parentheticals the model appends to verbatim source sentences are stripped before substring testing, so verbatim-with-citation no longer flags HYBRID.

`unverified_quotes` on each record is the corpus-growth signal — model output that didn't ground anywhere. `arborist emergent --aggregate` ranks them by frequency (the worklist of "things to ingest more sources for"). `arborist reclassify` re-runs the verifier against existing live records after corpus growth without any LLM call; HYBRID promotes to STRICT, UNGROUNDED to HYBRID, and one `providence_reclassify` audit event per change.

To dig into a specific record's unverified spans:

```
make inspect KEY=<cache_key>
```

Read-only **sidecar** diagnostic: pulls source chunks, classifies each unverified span as `verbatim_in_base` / `trailing_artifact` / `paraphrase` / `partial_paraphrase` / `no_overlap`. Writes nothing — sidecars never enter the v9.8 hard chain (that invariant is what keeps `audit_mode` a binary classification rather than a soft score).

## Mesh / federation (off by default)

Optional gossip layer for peer-to-peer corpus sync. Default off — no code path touches the network unless `mesh.enabled` is set. See [`docs/mesh.md`](docs/mesh.md) for protocol contract and the `arborist mesh` CLI subcommands.

## Inspecting

```
make stats-shards                    # totals across shards
make analyze-shards                  # compression spectrum, depth histogram, audit chain integrity
make verify-shards                   # round-trip Merkle proofs on a random sample
make activity                        # recent Q&A + ingests + derives + falsifications (agent timeline)
make activity ACTIVITY_LIMIT=20
```

`activity` is JSON; pipe through `jq` to drill in.

## Architecture

![Arborist module graph](docs/diagrams/arborist-modules.svg)

Generated API reference (every module, class, and function from docstrings):
build with `make docs-api` (output at `docs/_source/_build/html/`) or browse
the published Read the Docs site. Pipeline diagrams live in
[`docs/diagrams/`](docs/diagrams/); render with `make docs`.

## Tests & bench

```
make test                # 2000+ tests, default suite, ~60s sequential / ~10s with pytest -n auto
make test-crawler        # opt-in: tests for the verbatim crawler lift
make test-live           # gated: live QA quality fixtures against Hermes (~1 min)
make bench               # ETL throughput across configs (serial / shared-WAL / attached)
make bench-qa            # full QA-quality sweep, sample-shuffled, 71q × 3m × 3n at c=4
make bench-qa-smoke      # 5-question smoke fixture for prompt-iteration loops (~1-3 min)
```

The default suite never hits the network. The crawler suite is gated behind `make bootstrap-crawler` (installs the `[crawler]` extras).

`make bench-qa` writes JSONL + markdown into `bench/qa_results/<utc-stamp>.{jsonl,md}` (gitignored); design-log entries live in `docs/qa-modes-bench-<date>.md`. The bench is stop/start-able via `--resume <jsonl-path>` (same `--seed` required for shuffled-task-order alignment). `BENCH_QA_CONCURRENCY=N` Makefile variable overrides the default `4`. See `docs/bench-maxing.md` for the full speed playbook.

## License

License: AGPL-3.0-only · This algorithm, its implementation, & all associated code carry the GNU Affero General Public License v3.0 (only). You may use, modify, & distribute under those terms. No proprietary relicensing exists.

### Permacomputer Preamble

This is free software for the public good of a permacomputer hosted at [permacomputer.com](https://www.permacomputer.com), an always-on computer by the people, for the people. Durable, easy to repair, & distributed like tap water for machine learning intelligence.

Our permacomputer is community-owned infrastructure optimized around four values:

- **TRUTH** — First principles, math & science, open source code freely distributed.
- **FREEDOM** — Voluntary partnerships, freedom from tyranny & corporate control.
- **HARMONY** — Minimal waste, self-renewing systems with diverse thriving connections.
- **LOVE** — Be yourself without hurting others, cooperation through natural law.

NO WARRANTY. Software is provided "AS IS" without warranty of any kind. Full text in [`LICENSE`](LICENSE).

(Verbatim from the [Merkle Providence Reverse RAG whitepaper](https://unfirehose.com/merkle-providence-reverse-rag-whitepaper.pdf), April 2026.)