engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

readme: polish + quickstart (Wiki 2003 + live-site crawl)

Browse source 
Per fox: README was stale on most of what shipped over the past
sessions. Sweep:

- New Quickstart section at the top (replaces "What this gets you")
  with two end-to-end paths: Wikipedia 2003 dump and crawling a live
  site (russell.ballestrini.net as the worked example). Each is
  five-ish commands and ends with a real `make query`.
- New "Data: live websites (the crawler)" section between git/hg
  ingest and the OpenAI-export placeholder. Documents:
    - bootstrap-crawler / crawl-ingest with all knobs (URL, DEPTH,
      MAX, FAST, CRAWL_SHARD)
    - per-domain shard naming so make query auto-discovers the data
    - feed/sitemap skip at ingest (discovery infrastructure, not
      knowledge)
    - ETag/Last-Modified capture + recrawl-check freshness probe
    - provenance pointer to ~/git/agents.ai.unturf.com
- Asking-the-corpus section gains JSON=1, default human render note,
  question_hash equivalence (trailing punctuation stripped), and the
  full query path now reflects wikitext-strip-before-LLM step.
- New "Verifying answers (audit modes)" section explains the
  trichotomy + four-strategy verifier (quote/span/entity/paraphrase),
  trailing-citation strip, and the corpus-growth signal flow
  (emergent → reclassify).
- New "Marking and burning records" section covers falsify vs burn
  (kindergarten rule, KIND= for documents and cores), chain-check.
- Mesh section: dropped "on the roadmap" claim — wire layer +
  serve/sync/pull verbs are shipped. Pointers to docs/mesh.md and
  docs/mesh-deploy.md.
- Architecture diagram updated to current tree (wikitext.py,
  sources/crawler/, qa/inspect.py, qa/verify.py, mesh/, vcs.py).
- Whitepaper pointer updated to canonical rst path; mesh docs added.
- Test counts: 326+ default suite, separate test-crawler target,
  default suite never hits the network.
This commit is contained in:
russell@unturf.com 2026-04-29 12:50:49 -04:00
parent 10c1cba6e0
commit 7deee7d91a
No known key found for this signature in database
1 changed files with 174 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

208
README.md
View file

@ -4,18 +4,42 @@ An arborist for trees and forests of cross-linked information.
Aborist ingests documents into a content-addressed, Merkle-committed SQLite store, distills them into recursive cores, and answers questions over the resulting corpus via an OpenAI-compatible LLM. Every cached answer carries a verifiable Merkle proof tying it back to its source documents — Merkle-AGI v9.8 / Merkle Providence Reverse RAG, runnable end-to-end.
## What this gets you
## Quickstart
```
make bootstrap # one-time: venv + deps
make fetch-cur # download Wikipedia 2003-05-16 (~82 MB)
make ingest-cur-attached # ~3 min — 128k articles, 4 parallel shards
make distill-shards-parallel
make distill-shards-tfidf-parallel
Two end-to-end paths. Pick whichever corpus you want first; both share the same query, verify, falsify, and inspect surfaces.
### A. Wikipedia 2003 (the canonical bootstrap dataset)
```sh
make bootstrap # one-time: venv + dev extras
make fetch-cur # download the 2003-05-16 snapshot (~82 MB)
make ingest-cur-attached # ~3 min: 128k articles, 4 parallel shards
make distill-shards-parallel # surface → core (first-sentence)
make distill-shards-tfidf-parallel # core → keyword sets for retrieval
make query Q="What is anarcho-capitalism?"
```
After the last command, a Hermes-3 inference runs against the local corpus, picks 4–8 source articles by Merkle root, and returns an answer with a cryptographic proof of its sources. Repeat the same question and a STRICT-mode cache hit replays in ~100 ms.
A Hermes-3 inference runs against the local corpus, picks 4–8 source articles by Merkle root, and returns an answer plus a verifier audit_mode (STRICT / HYBRID / UNGROUNDED). Repeat the same question and a cache hit replays in ~100 ms.
### B. Crawl any live website and query it
```sh
make bootstrap-crawler # one-time: install [crawler] extras
make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2 # BFS + ingest
make query Q="who is Russell Ballestrini?" # cross-shard query — picks up the new shard automatically
```
The crawl shard is named after the seed hostname (`crawl_russell_ballestrini_net.db`) under `~/.aborist/shards/`. Add `FAST=1` for aggressive crawling against your own sites; `MAX=N` to cap discovery; `DEPTH=N` to bound BFS. Robots `Disallow` is always honored. After ingest, `make recrawl-check DOMAIN=russell.ballestrini.net` does a conditional-HEAD freshness probe per page.
### After the answer
```sh
make inspect KEY=<cache_key> # sidecar: classify each unverified span
make falsify KEY=<cache_key> REASON='…' # mark wrong, keep history
make burn KEY=<cache_key> REASON='…' # delete (kindergarten only — refuses if children exist)
```
`make help` lists every target.
## Setup
@ -191,6 +215,47 @@ make ingest-hg HG_REPO=/path/to/repo # mercurial flavor
URI shape: `git://<repo-name>/file/<relative-path>` (no commit hash — that's what enables the supersedes chain on re-ingest). Binary files are skipped (NUL-byte heuristic + UTF-8 decode probe). `extra` carries the current commit hash, timestamp, and subject for informational purposes; the cryptographic identity is the content-derived `document_root` as for every other Document.
## Data: live websites (the crawler)
Aborist can BFS-discover and ingest a website starting from a seed URL, respecting robots.txt and crawl delays. The crawler is **off by default** — heavy deps (aiohttp, bs4, lxml, mwparserfromhell, etc.) ship as the `[crawler]` extras and aren't pulled into the default test suite.
```
make bootstrap-crawler # one-time, install [crawler] extras
make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2 # BFS + ingest the discovered pages
```
The shard filename is derived from the seed URL's hostname so cross-shard query picks it up automatically:
```
URL=https://russell.ballestrini.net → $(SHARDS_DIR)/crawl_russell_ballestrini_net.db
```
After ingest, the same `make query` searches across crawl shards alongside Wikipedia, Grok, and self-play sources:
```
make query Q="who is Russell Ballestrini?"
```
Knobs:
| variable | default | what |
|---|---|---|
| `URL=` | (required) | seed URL; BFS stays on its hostname (no subdomain crossover) |
| `DEPTH=` | `2` | max BFS depth from seed |
| `MAX=` | `0` | cap discovery at N URLs (`0` = no cap, depth is the only bound) |
| `FAST=1` | unset | flip the verbatim AsyncWebFetcher into fast_mode: 5s timeouts, CPU×3 parallel page workers, ignore robots.txt `crawl-delay`. Disallow is still honored. Use only against sites where aggressive fetching is acceptable. |
| `CRAWL_SHARD=` | derived from URL | override the destination shard path |
Feeds and sitemaps (`atom`, `rss`, `sitemap.xml`, `wp-rss2.xml`, etc.) are skipped at ingest — they're discovery infrastructure, not knowledge. ETag and Last-Modified per page are captured so a future probe can ask "does this need recrawling?" without re-fetching bodies:
```
make recrawl-check DOMAIN=russell.ballestrini.net
```
Conditional `If-None-Match` / `If-Modified-Since` HEAD requests classify each ingested doc as fresh (304), stale (200), gone (404/410), or unreachable. One tiny round trip per URL with no body transfer when content's unchanged.
The crawler is a verbatim lift from `~/git/agents.ai.unturf.com/core/` (provenance documented in `aborist/sources/crawler/__init__.py`); aborist-side changes drop the chat-bot fetch triggers and skip `web_cache_manager.py` (aborist has its own content-addressed cache). Run `make test-crawler` for the lift's own tests.
## Data: OpenAI / ChatGPT export (planned)
Not implemented yet. The shape will be one new `Source` subclass at `aborist/sources/openai.py` plus a Makefile target. The OpenAI ChatGPT data export is a `.zip` containing `conversations.json` with the `mapping`/`messages` tree shape. Adding it follows the same pattern as `aborist/sources/grok.py` — see that file as the template.
@ -207,16 +272,21 @@ When this lands, conversations from both Grok and OpenAI will sit in the same sh
```
make query Q="What is the philosophy of stoicism?"
make query Q="tell me about permacomputer ?"
make query QUERY_TOP_K=12 Q="…" # widen the source set
make query Q="…" QUERY_TOP_K=12 # widen the source set
make query Q="…" JSON=1 # raw record (cache_key, merkle_proof, timings, full sources)
make query-dry Q="…" # assemble context but skip the LLM call
```
Default render is human-readable: question, audit-mode summary line, answer, sources list, unverified spans, short cache_key. `JSON=1` gives the full record. Same trailing-question-mark question deduplicates (`question_hash` strips trailing `.?!,;:？！。、…` after lowercasing).
The query path:
1. **Search** — FTS5 (body) + SQL `LIKE` (title) + `JOIN` over derivations (TF-IDF core keywords) across every shard. Three accept paths to the relevance filter.
2. **Concept overlay** — synonym groups (`Athlon` ↔ `AMD`) widen retrieval; rivalry pairs (AMD ↔ Intel) narrow it unless the query has comparative phrasing ("compare X vs Y").
3. **Context assembly** — top-K sources concatenated up to a 60 KB budget, fed to Hermes-3 with strict attribution rules in the system prompt.
4. **Cache** — the v9.8 8-dim cache_key (`source_root | question_hash | model_profile | conversation | governance_policy | schema | canonicalization | chunking`) keys the answer in `qa.db`. Cache hits replay STRICT-mode in ~100 ms. Per-phase timings are returned in every result.
3. **Context assembly** — top-K sources concatenated up to a 60 KB budget. Wikitext is stripped to plain prose via `aborist.wikitext.to_base()` (the corpus stores raw `[[wikilinks]]` so the link graph is recoverable on demand; the LLM and verifier both see clean prose).
4. **LLM** — Hermes-3 with strict attribution rules in the system prompt + a user-turn grounding reminder.
5. **Verifier** — every claim is classified into the v9.8 trichotomy STRICT / HYBRID / UNGROUNDED via a layered lexical check (see below).
6. **Cache** — the v9.8 8-dim cache_key (`source_root | question_hash | model_profile | conversation | governance_policy | schema | canonicalization | chunking`) keys the answer in `qa.db`. Cache hits replay in ~100 ms. Per-phase timings in every result.
LLM endpoint defaults to `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B, 82K context, no auth). Override:
@ -226,15 +296,58 @@ export ABORIST_LLM_MODEL="meta-llama/Llama-3.1-70B-Instruct"
export ABORIST_LLM_API_KEY="..."
```
## Mark a wrong answer
## Verifying answers (audit modes)
If the LLM produced something incorrect, mark its cache record stale so the next ask re-runs inference fresh:
Every answer carries an `audit_mode` set by `aborist/qa/verify.py`:
| mode | meaning |
|---|---|
| **STRICT** | every evidence unit verifies against context |
| **HYBRID** | some claims source-grounded, some emerged from training |
| **UNGROUNDED** | no evidence, or none verifies — purely emergent |
Four strategies run in sequence; first to find evidence classifies. Each is lexical (substring or token-coverage), never embeddings — soft signals stay out of the proof path.
| # | strategy | what triggers it |
|---|---|---|
| 1 | `quote` | model wraps claims in `"..."`; each verbatim-substring tested |
| 2 | `span` | bullet/sentence units substring-tested as fallback |
| 3 | `entity` | multi-word proper nouns tested with proximity-cluster gating |
| 4 | `paraphrase` | inside the span path: ≥85% token coverage on prose-shaped spans |
Trailing `(Source: https://...)` parentheticals the model appends to verbatim source sentences are stripped before substring testing, so verbatim-with-citation no longer flags HYBRID.
`unverified_quotes` on each record is the corpus-growth signal — model output that didn't ground anywhere. `aborist emergent --aggregate` ranks them by frequency (the worklist of "things to ingest more sources for"). `aborist reclassify` re-runs the verifier against existing live records after corpus growth without any LLM call; HYBRID promotes to STRICT, UNGROUNDED to HYBRID, and one `providence_reclassify` audit event per change.
To dig into a specific record's unverified spans:
```
make inspect KEY=<cache_key>
```
Read-only **sidecar** diagnostic: pulls source chunks, classifies each unverified span as `verbatim_in_base` / `trailing_artifact` / `paraphrase` / `partial_paraphrase` / `no_overlap`. Writes nothing — sidecars never enter the v9.8 hard chain (that invariant is what keeps `audit_mode` a binary classification rather than a soft score).
## Marking and burning records
Two ways to remove a cached answer, depending on whether it has children:
```
make falsify KEY=<cache_key> REASON="why it was wrong"
```
The original record stays in the database (history matters). A `falsifications` row + `falsify` audit event record the act. Future lookups skip records whose `falsification_state != 'live'`.
The record stays in the database; `falsification_state` flips to `failed` and a `falsifications` row + `falsify` audit event record the act. Future lookups skip records whose `falsification_state != 'live'`.
```
make burn KEY=<cache_key> REASON="..." # providence leaves
make burn KIND=document ROOT=<hex> REASON="..." # surface documents
make burn KIND=core ROOT=<hex> REASON="..." # distilled cores
```
`burn` actually deletes the row. Refuses if the leaf has children (falsifications referencing the cache_key, derivations using the document, or other records pointing at it) — pass `FORCE=1` to override. Always writes an audit event. Use during early/scratch corpus building; falsify is the audit-preserving alternative once downstream consumers exist.
```
make chain-check-shards # 0 chain breaks across every shard's audit_events
```
## Mesh / federation (off by default)
@ -259,7 +372,15 @@ aborist mesh members # list current epoch's ros
The kicked member's prior signatures stay verifiable forever (their roster row at older epochs is preserved on disk). They have no entry in the new epoch's secret envelope, so any AEAD-protected gossip from epoch+1 onward is opaque to them — that is the eviction guarantee.
The HTTP gossip wire (`mesh sync`, `mesh serve`) is on the roadmap; this commit ships the cryptographic foundation, state machine, and CLI. The crypto is `cryptography`-backed Ed25519 + X25519 + ChaCha20-Poly1305.
Once two peers have enrolled each other, the HTTP gossip wire is live:
```
aborist mesh serve --host 0.0.0.0 --port 8400 # blocks until SIGINT
aborist mesh sync --peer http://other.example.com:8400 # announce local document_roots
aborist mesh pull --peer http://... --root <hex> # pull a body, verify Merkle
```
Every envelope is Ed25519-signed; bodies can opt into AEAD encryption under the per-epoch shared secret (`encrypt=True` on the client API). Receivers track each sender's chain-of-claims and reject envelopes whose `prev_event_hash` doesn't extend the last known event for that peer — fork detection at the wire layer. Two-host setup runbook: `docs/mesh-deploy.md`. Protocol contract + diagrams: `docs/mesh.md`.
## Inspecting
@ -277,37 +398,56 @@ make activity ACTIVITY_LIMIT=20
```
aborist/
├── merkle.py Python port of proxy.unturf.com/pkg/verified/merkle.go
│ conventions: non-commutative HashCombine (0x03 prefix),
│ explicit IsLeft per sibling, self-duplicate odd elements.
├── store.py SQLite v9.8 schema (8-dim cache key, falsification state,
│ audit chain, surface/core kinds, hot/warm/cold tier).
├── ingest.py normalize → chunk → merkle → upsert. Bulk-batched writer.
├── document.py Document, Edge, Chunker (tok-512-v1 default).
├── source.py Source ABC: iter_documents() -> Iterator[Document].
├── merkle.py Python port of proxy.unturf.com/pkg/verified/merkle.go
│ (non-commutative HashCombine 0x03 prefix, explicit
│ IsLeft per sibling, self-duplicate odd elements).
├── store.py SQLite v9.8 schema (8-dim cache key, falsification
│ state, audit chain, surface/core kinds, hot/warm/
│ cold tier, document_http_meta, mesh_*).
├── ingest.py normalize → chunk → merkle → upsert. Bulk-batched.
├── document.py Document, Edge, Chunker (tok-512-v1 default).
├── source.py Source ABC: iter_documents() -> Iterator[Document].
├── wikitext.py to_base(): wikitext → plain prose, BASE_VERSION-pinned.
├── sources/
│ ├── wikipedia.py cur + old MediaWiki SQL dumps (bz2-streamed).
│ ├── html_page.py URL list + selectolax + httpx (robots-aware).
│ └── grok.py xAI data export (conversations + media prompts).
├── distill/ Distiller ABC + first_sentence + tfidf + recursion runner.
├── search/ FTS5 backend + SearchBackend ABC + AuditMode.
├── qa/ ask (single-doc) + query (multi-source RAG) + concepts overlay.
├── evict.py hot ↔ cold tier transitions; rehydrate via source pipeline.
└── cli.py argparse entrypoint (the make targets call into here).
│ ├── wikipedia.py cur + old MediaWiki SQL dumps (bz2-streamed).
│ ├── wikipedia_xml.py Phase IV XML dumps (iterparse, page + history).
│ ├── html_page.py URL list + selectolax + httpx (robots-aware).
│ ├── grok.py xAI data export (conversations + media prompts).
│ ├── vcs.py git + Mercurial repos (HEAD walk, supersedes chain).
│ └── crawler/ verbatim AsyncWebFetcher lift + ingest bridge.
├── distill/ Distiller ABC + first_sentence + tfidf + runner.
├── search/ FTS5 backend + SearchBackend ABC + AuditMode.
├── qa/
│ ├── client.py ChatClient + StubClient + OpenAICompatibleClient.
│ ├── keys.py 8-dim cache_key + question_hash (CJK-aware strip).
│ ├── verify.py 4-strategy lexical verifier (quote/span/entity/
│ │ paraphrase). Citation-strip + wikitext-strip.
│ ├── inspect.py sidecar diagnostic — read-only span classifier.
│ ├── runner.py ask(): single-doc Q&A + cache + verify.
│ └── query.py query(): multi-source RAG + concept overlay.
├── mesh/ off-by-default federation: identity (Ed25519/X25519),
│ per-epoch roster, AEAD-wrapped epoch secrets, HTTP
│ gossip wire, per-peer chain-of-claims tracking.
├── evict.py hot ↔ cold tier transitions; rehydrate via source.
└── cli.py argparse entrypoint (the make targets call into here).
```
Source papers (read first if confused):
- `~/Downloads/merkle-providence-reverse-rag-whitepaper.pdf` — public spec, AGPL-3.0
- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV/canonical encoding, theorems T1–T5)
- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` — canonical whitepaper (rst, builds the PDF). Section 13.8 covers the verifier in depth.
- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV/canonical encoding, theorems T1–T5).
- `docs/mesh.md` + `docs/mesh-deploy.md` — protocol contract + two-host runbook for federation.
## Tests
```
make test # 79 tests, all stdlib + pytest
make test # 326+ tests, default suite (stdlib + pytest, no network)
make test-crawler # opt-in: tests for the verbatim crawler lift
make bench # ETL throughput across configs (serial / shared-WAL / attached)
```
The default suite never hits the network. The crawler suite is gated behind `make bootstrap-crawler` (installs the `[crawler]` extras).
## License
License: AGPL-3.0-only · This algorithm, its implementation, & all associated code carry the GNU Affero General Public License v3.0 (only). You may use, modify, & distribute under those terms. No proprietary relicensing exists.