arborist/CLAUDE.md

Agent Blackops — aborist

This repo is operated by agent blackops for fox/timehexon on the unsandbox / unturf / permacomputer platform.

Identity shard: ~/git/unsandbox.com/blackops/BLACKOPS.md.

What aborist is

A content-addressed, Merkle-committed document store. Implements the runtime spec from Merkle Providence Reverse RAG (April 2026 whitepaper) scaled up to the Merkle-AGI v9.8 admissibility ledger. Tends "trees and forests of cross-linked information" — the namesake.

Three layers stacked on one SQLite file:

1. Surface — ingested documents (Wikipedia dumps, HTML pages, anything with a URI). Chunked, Merkle-rooted, FTS5-indexed.
2. Core — distilled documents (haiku/keyword/equation-scale) Merkle-bound back to source surface(s) via per-chunk inclusion proofs in derivations.proof_blob. Recursive: cores derive from cores. The "planet toward center" compression.
3. Providence cache — Q&A records keyed on the v9.8 8-dim invariant. STRICT-mode hits carry verifiable Merkle proofs.

Source papers (read first if confused)

• ~/Downloads/merkle-providence-reverse-rag-whitepaper.pdf — public-facing spec, AGPL-3.0
• ~/Downloads/merkle-agi-dag_v7.txt — formal substrate (TLV encoding A1, public quantization A2, collision-resistant hash A3, theorems T1–T5)
• ~/git/proxy.unturf.com/pkg/verified/merkle.go — fox's existing Go merkle implementation. Aborist Python ports its conventions exactly.
• ~/git/proxy.unturf.com/docs/merkle-tree.md — convention reference

Architecture

aborist/
├── merkle.py              # Python port of proxy.unturf.com Go conventions
├── store.py               # v9.8 SQLite schema, audit chain helpers
├── document.py            # Document, Edge, Chunker (TokenChunker default)
├── source.py              # Source ABC: iter_documents() -> Iterator[Document]
├── ingest.py              # batched: normalize -> chunk -> merkle -> upsert
├── evict.py               # hot->cold + rehydrate (v9.8 falsification on drift)
├── search/                # SearchBackend ABC + AuditMode + FTS5
├── sources/               # one file per corpus
│   ├── wikipedia.py       #   WikipediaSqlDump (cur + old tables)
│   └── html_page.py       #   HtmlPageSource (selectolax, robots-aware)
├── distill/               # surface->core distillation
│   ├── base.py            #   Distiller ABC + DistillationResult
│   ├── first_sentence.py  #   FirstSentenceDistiller (no-ML stub)
│   ├── tfidf.py           #   TfidfKeywordDistiller (pure-Python TF-IDF)
│   └── runner.py          #   batched: derive + per-contrib-chunk proofs
├── qa/                    # Q&A: 8-dim cache_key + Merkle-bound answers
│   ├── client.py          #   ChatClient + StubClient + OpenAICompat
│   ├── keys.py            #   cache_key, question_hash, ... (pure functions)
│   └── runner.py          #   ask(): cache hit STRICT / miss -> infer -> write
└── cli.py                 # ingest / search / verify / stats / distill /
                           # evict / rehydrate / ask / providence / analyze

Build, test, run

Every workflow is a make target. Bare python commands are not the user interface.

make bootstrap           # venv + editable install with [dev] extras
make test                # pytest -q (49 tests)
make all                 # bootstrap + fetch-cur + ingest-cur + verify + stats
make fetch               # cur (82 MB) + old.1 (640 MiB) + old.2 (252 MiB) + concat
make ingest-cur          # ingest snapshot articles
make ingest-old          # ingest revision history (~hours)
make verify              # round-trip Merkle proofs on a random sample
make analyze             # compression spectrum + audit chain integrity

aborist/cli.py adds: analyze, distill --kind {surface,core}, evict, rehydrate, ask, providence. --batch-size defaults to 200 docs/transaction; lower it only to bound memory peaks.

Schema invariants (do not break)

• Aborist is a v9.8 store. Every Document carries chunking_version, canonicalization_version, schema_version. Every providence record carries the full 8-dim cache_key: source_root | question_hash | model_profile_hash | conversation_hash | governance_policy_hash | schema_version | canonicalization_version | chunking_version. Bumping any one invalidates prior records on lookup.
• falsification_state ∈ {live, failed, stale, quarantined}. Cache lookups must filter on state='live'. Drift detection (rehydrate vs source root mismatch) flips to stale.
• Audit chain. Every state-changing op writes one row in audit_events with event_hash = sha256(prev_event_hash || canonical(body)). Chain integrity is verified in make analyze. Never insert into audit_events directly — use aborist.store.append_audit.
• Cores never evict. evict_to_cold only touches kind='surface'.
• Idempotent re-ingest. Same content → same document_root → no-op insert. Same URI + different content → new doc + supersedes edge linking new → old (lossless history).

Conventions (do not silently change)

• Merkle: non-commutative HashCombine with prefix 0x03. Leaves 0x00. Odd-element rule = self-duplicate, NOT zero-pad. MerkleProof.siblings carries explicit is_left flag — never sort lexically.
• Chunker default = tok-512-v1. Changing the default bumps chunking_version and stales every prior cache record. Add a new chunker as a new name instead.
• Canonicalization = norm-v1 (NFC, collapsed whitespace). Same rule.
• Schema = v9.8.0. Same rule.
• Search hits carry audit_mode ∈ STRICT/HYBRID/VISUAL. FTS5 hits are VISUAL. Providence-cache hits with verified proofs are STRICT. Never overclaim.
• Soft hash vs hard hash. Hard = SHA-256 (commitments, proofs, cache_key). Soft = embeddings/TF-IDF/similarity (training, ranking, distillation candidate selection). Never mix — soft never enters proof path.

Live endpoints

• LLM: https://hermes.ai.unturf.com/v1 (Hermes-3 Llama-3.1-8B-FP8-Dynamic on vLLM, 82K ctx, no auth). uncloseai.com is marketing only — has no /v1. Override via --endpoint or ABORIST_LLM_ENDPOINT.
• Wikipedia dumps: https://dumps.wikimedia.org/archive/2003/2003-05-16/en/. robots.txt returned 404 → no rules.

Hot path / gotchas

• Parser is hand-rolled in aborist/sources/wikipedia.py (char-position state machine, escape-aware). After the v9.8 commit it's 4× faster via str.find + slicing — easy to break by reverting to char-by-char loops. cProfile any change.
• PRAGMA synchronous=NORMAL is set per-connection in store.connect(). Safe under WAL (the journal_mode is set in SCHEMA_SQL). Don't downgrade to FULL without a measured reason — costs ~5x throughput.
• HTML source has optional deps: pip install '.[html]' for selectolax. The CLI surfaces --source html only if the import succeeds.
• Background ingest/distill processes: stdout is buffered. Use export PYTHONUNBUFFERED=1 or python -u. Per blackops top-level rule.
• Disk pressure. Full cur ingest ~2 GB; full old ingest ~5–8 GB. df -h /home/fox first.

Operational rules

• I propose, fox decides. Unsure = ask. Can't ask = stop.
• No autonomous destructive ops. No clean-data, clean-db, force-push, or DB drops without explicit instruction.
• Never add Co-Authored-By or "Generated with Claude" lines to commits. Professional commit messages only — code speaks for itself.
• Always export PYTHONUNBUFFERED=1 for long-running processes. Buffered output disappears when processes die.
• Fail-closed. Cleanup crew, not demolition.
• DRY in context — single source of truth, no sprawl.
• Never say "AI" — always say "machine learning."
• Prefer "defect" over "bug."
• Check robots.txt before any web fetch the user didn't authorize.

Orientation protocol

date -u
pwd
git log --oneline -5
git status
make test
.venv/bin/aborist --db ~/.aborist/aborist.db stats
.venv/bin/aborist --db ~/.aborist/aborist.db analyze --gravity-top 5

Then ask fox what the mission is.