arborist/CLAUDE.md

Agent Blackops — aborist

This repo is operated by agent blackops for fox/timehexon on the unsandbox / unturf / permacomputer platform.

Identity shard: ~/git/unsandbox.com/blackops/BLACKOPS.md.

What aborist is

A content-addressed, Merkle-committed document store. Implements the runtime spec from Merkle Providence Reverse RAG (April 2026 whitepaper) scaled up to the Merkle-AGI v9.8 admissibility ledger. Tends "trees and forests of cross-linked information" — the namesake.

Three layers stacked on one SQLite file:

1. Surface — ingested documents (Wikipedia dumps, HTML pages, anything with a URI). Chunked, Merkle-rooted, FTS5-indexed.
2. Core — distilled documents (haiku/keyword/equation-scale) Merkle-bound back to source surface(s) via per-chunk inclusion proofs in derivations.proof_blob. Recursive: cores derive from cores. The "planet toward center" compression.
3. Providence cache — Q&A records keyed on the v9.8 8-dim invariant. Each record carries an audit_mode set by the post-LLM faithfulness verifier (aborist/qa/verify.py): STRICT (every claim verbatim-grounded or token-coverage paraphrase-grounded), HYBRID (mixed source/emergent), UNGROUNDED (no recoverable grounding — purely emergent from training, or genuine model refusal). verifier_method records which of four strategies — quote / span / entity / paraphrase — fired.

Source papers (read first if confused)

• ~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst — canonical whitepaper source (rst, builds the PDF). Edit here, not the PDF.
• ~/Downloads/merkle-providence-reverse-rag-whitepaper.pdf — built artifact; mirrors the rst above
• ~/Downloads/merkle-agi-dag_v7.txt — formal substrate (TLV encoding A1, public quantization A2, collision-resistant hash A3, theorems T1–T5)
• ~/git/proxy.unturf.com/pkg/verified/merkle.go — fox's existing Go merkle implementation. Aborist Python ports its conventions exactly.
• ~/git/proxy.unturf.com/docs/merkle-tree.md — convention reference

Architecture

aborist/
├── merkle.py              # Python port of proxy.unturf.com Go conventions
├── store.py               # v9.8 SQLite schema, audit chain helpers
├── document.py            # Document, Edge, Chunker (TokenChunker default)
├── source.py              # Source ABC: iter_documents() -> Iterator[Document]
├── ingest.py              # batched: normalize -> chunk -> merkle -> upsert
├── evict.py               # hot->cold + rehydrate (v9.8 falsification on drift)
├── search/                # SearchBackend ABC + AuditMode + FTS5
├── sources/               # one file per corpus
│   ├── wikipedia.py       #   WikipediaSqlDump (cur + old tables)
│   └── html_page.py       #   HtmlPageSource (selectolax, robots-aware)
├── distill/               # surface->core distillation
│   ├── base.py            #   Distiller ABC + DistillationResult
│   ├── first_sentence.py  #   FirstSentenceDistiller (no-ML stub)
│   ├── tfidf.py           #   TfidfKeywordDistiller (pure-Python TF-IDF)
│   └── runner.py          #   batched: derive + per-contrib-chunk proofs
├── qa/                    # Q&A: 8-dim cache_key + Merkle-bound answers
│   ├── client.py          #   ChatClient + StubClient + OpenAICompat
│   ├── keys.py            #   cache_key, question_hash, ... (pure functions)
│   ├── verify.py          #   layered verifier: quote → span → entity → paraphrase
│   ├── inspect.py         #   read-only sidecar: diagnose unverified spans
│   └── runner.py          #   ask(): cache -> infer -> verify -> classify -> write
├── wikitext.py            # to_base(): wikitext → plain prose (BASE_VERSION-pinned)
└── cli.py                 # ingest / search / verify / stats / distill /
                           # evict / rehydrate / ask / providence / emergent /
                           # reclassify / inspect / analyze

Build, test, run

Every workflow is a make target. Bare python commands are not the user interface.

make bootstrap           # venv + editable install with [dev] extras
make test                # pytest -q (49 tests)
make all                 # bootstrap + fetch-cur + ingest-cur + verify + stats
make fetch               # cur (82 MB) + old.1 (640 MiB) + old.2 (252 MiB) + concat
make ingest-cur          # ingest snapshot articles
make ingest-old          # ingest revision history (~hours)
make verify-shards       # round-trip Merkle proofs on a random sample (cross-shard)
make analyze-shards      # cross-shard compression spectrum + audit integrity
make chain-check-shards  # audit-chain break count per shard (0 = intact)

Hygiene after any state-changing op (table rebuild, reclassify run, governance hash bump, mass falsify): make chain-check-shards for a one-second sanity (every shard should report 0), then make analyze-shards for the full spectrum + chain audit. Chain breaks are the loudest possible signal that something corrupted the audit log — catch them at the seam, not in production.

aborist/cli.py adds: analyze, distill --kind {surface,core}, evict, rehydrate, ask, providence, emergent (list UNGROUNDED/HYBRID records or --aggregate to rank unverified quotes — corpus-growth signal), reclassify (re-run the verifier against existing live providence records under the current entity policy; no LLM calls; --compare runs all four entity policies side-by-side, --dry-run reports without writing), inspect --cache-key X (read-only sidecar that pulls source chunks for one record & classifies each unverified span; never writes to providence_cache or audit_events). The query command human-renders by default; pass --json (or JSON=1 to make query) for the raw record. --batch-size defaults to 200 docs/transaction; lower it only to bound memory peaks.

Schema invariants (do not break)

• Aborist is a v9.8 store. Every Document carries chunking_version, canonicalization_version, schema_version. Every providence record carries the full 8-dim cache_key: source_root | question_hash | model_profile_hash | conversation_hash | governance_policy_hash | schema_version | canonicalization_version | chunking_version. Bumping any one invalidates prior records on lookup.
• falsification_state ∈ {live, failed, stale, quarantined}. Cache lookups must filter on state='live'. Drift detection (rehydrate vs source root mismatch) flips to stale.
• Audit chain. Every state-changing op writes one row in audit_events with event_hash = sha256(prev_event_hash || canonical(body)). Chain integrity is verified in make analyze-shards (full audit) or make chain-check-shards (one-second per-shard break count, 0 = intact). Never insert into audit_events directly — use aborist.store.append_audit.
• Cores never evict. evict_to_cold only touches kind='surface'.
• Idempotent re-ingest. Same content → same document_root → no-op insert. Same URI + different content → new doc + supersedes edge linking new → old (lossless history).

Conventions (do not silently change)

• Merkle: non-commutative HashCombine with prefix 0x03. Leaves 0x00. Odd-element rule = self-duplicate, NOT zero-pad. MerkleProof.siblings carries explicit is_left flag — never sort lexically.
• Chunker default = tok-512-v1. Changing the default bumps chunking_version and stales every prior cache record. Add a new chunker as a new name instead.
• Canonicalization = norm-v1 (NFC, collapsed whitespace). Same rule.
• Schema = v9.8.0. Same rule.
• question_hash is dedup-mode-aware. question_hash(q, mode=...) accepts "strict" (NFC + ws-collapse only — case-sensitive, punctuation-sensitive, article-sensitive; every variant gets its own hash) or "equivalence_class" (default — additionally lowercase + trailing-punct strip + standalone-article strip; collapses "who is X", "who is X?", "Who Is X.", "who is the X", "who is an X？" to one hash). Mode lives in policy["question_dedup"] so it folds into governance_policy_hash; agents under different modes write records under different cache_keys.
• JIT lookup fidelity. Per-call fidelity parameter on query()/ask() decouples lookup tolerance from write policy. "strict" checks only the primary cache_key (audit-grade — record reuse only on exact match). "equivalence_class" (default) tries primary first, then the alternate dedup mode's cache_key as a fallback so a fast-cache agent can reuse records written under either policy. Result includes lookup_path ∈ {"strict", "equivalence_class", "strict_fallback", "equivalence_class_fallback", "miss"}. Cross-silo fallback works because the helper rewrites policy["question_dedup"] to the alternate mode when computing the fallback ckey, so governance_policy_hash matches what an agent under that mode would have written. CLI: --question-dedup, --fidelity flags on query (and the ask API params).
• audit_mode is decided by the verifier, never asserted unconditionally. Four layered strategies in aborist/qa/verify.py, tried in order; first to find evidence classifies the answer:
  1. quote — model wrapped claims in double quotes per system prompt. Sequential pairing: 1st & 2nd ", 3rd & 4th, etc. (NOT regex pairing — that captures inter-pair prose as a phantom span when the model writes "title" prose "quote").
  2. span — bullet/sentence lines from the answer appear verbatim in context. Catches models that quote inline without "..." marks.
  3. entity — multi-word proper-noun phrases appear verbatim in context. Gated by entity_policy ∈ {strict, hybrid, drop, proximity}. Default proximity: STRICT only when N=3 verified entities cluster within W=300 chars in source (cast list / infobox / roster). Otherwise HYBRID/UNGROUNDED. Distinguishes structural grounding from incidental mention. Lives in DEFAULT_QUERY_POLICY["entity_policy"] so any change bumps governance_policy_hash.
  4. paraphrase — soft fallback on prose-shaped spans when 1-3 fail. Token-coverage probe: ≥4-char content tokens minus an English stopword set (from, with, would, which, etc.) checked for presence in normalized base context. Promotes when coverage ≥ DEFAULT_PARAPHRASE_COVERAGE (0.85) with ≥ DEFAULT_PARAPHRASE_MIN_TOKENS (4) content tokens. Fires only on prose-shaped spans (≥2 lowercase content tokens via _is_prose_span) so lists of proper nouns flow to the entity strategy instead. Quote strategy deliberately gets NO paraphrase fallback — "..." asserts verbatim citation; paraphrasing inside quotes is a model error to flag, not auto-promote. Records carry verifier_method='paraphrase' so an auditor can tell soft-verified evidence from lexical-verbatim. Trichotomy across all paths: STRICT = every evidence unit (≥1) verifies. HYBRID = some verify, some don't. UNGROUNDED = no evidence or none verifies. Persisted on providence_cache.audit_mode + verifier_method; cache-hits return the stored mode. Never overclaim — STRICT is a verifiable claim, not a default.
• Trailing-citation strip. _strip_trailing_citation peels a single trailing parenthetical at end-of-span (gated on a citation-cue word — Source:, citing, see, ref, from — OR a URL) before substring testing. Keeps "...prose. (Source: https://...)" from flunking just because the model appended a citation tail. Refuses to strip genuine prose parentheticals (no cue word, no URL).
• Verifier stays binary; falsifications carry soft signal. No per-quote diagnosis fields on hard verifier output. verify_quotes returns evidence units + classification; the falsify+reclassify loop owns "why didn't this ground" for the operator, and aborist inspect --cache-key X is the read-only sidecar that classifies each unverified span (verbatim_in_base / verbatim_in_raw_only / trailing_artifact / paraphrase / partial_paraphrase / no_overlap) — sidecars never write to providence_cache or audit_events. Don't bolt confidence scores or partial-match indicators onto verify.py.
• Wikitext base prose. aborist/wikitext.py:to_base(raw) converts MediaWiki wikitext → plain prose deterministically (mwparserfromhell-backed; pinned by BASE_VERSION = "wikitext-base-v1"). Applied before the LLM call in aborist/qa/runner.py and aborist/qa/query.py (gated on policy["base_version"]), and again inside verify_quotes so the verifier compares like-against-like. Both sides — model and verifier — see prose; the model can quote source paragraphs verbatim instead of escaping [[wikilinks]], and Wikipedia chunks ship to Hermes with ~43% fewer tokens. policy["base_version"] lives in DEFAULT_POLICY/DEFAULT_QUERY_POLICY so it folds into governance_policy_hash; bumping BASE_VERSION invalidates every prior cache record's 8-dim cache_key on next lookup. Optional dep — installs without mwparserfromhell keep _wikitext_to_base = None and policy["base_version"] = None, leaving raw wikitext in both context and verifier (graceful fallback, no failure mode).
• Soft hash vs hard hash. Hard = SHA-256 (commitments, proofs, cache_key). Soft = embeddings/TF-IDF/similarity (training, ranking, distillation candidate selection). Never mix — soft never enters proof path.

Live endpoints

• LLM: https://hermes.ai.unturf.com/v1 (Hermes-3 Llama-3.1-8B-FP8-Dynamic on vLLM, 82K ctx, no auth). uncloseai.com is marketing only — has no /v1. Override via --endpoint or ABORIST_LLM_ENDPOINT.
• Wikipedia dumps: https://dumps.wikimedia.org/archive/2003/2003-05-16/en/. robots.txt returned 404 → no rules.

Retrieval pipeline (aborist/qa/query.py)

Multi-stage. Each stage exists because something earlier wasn't enough; revert at your peril.

1. Two parallel FTS5 searches per shard, merged. _search_corpus runs body-BM25 AND title-LIKE in parallel and unions hits. The title path closes a recall gap where list-pages with many URLs (e.g. List_of_HTTP_status_codes) outrank the actual HTTP article on body BM25. See d608c39 fix retrieval recall.
2. Body-coverage sqrt rerank. BM25 systematically favors short docs with rare body tokens (Tell_(poker) outranking Back_to_the_Future on a film query). Counteract by reranking on sqrt(body_token_count) to recover long-doc hits. 1983b79.
3. Title-token boost. _rerank_by_title adds boost × (overlap) to hits whose title contains query tokens — strong topical signal that BM25 alone misses.
4. _filter_by_title_relevance — three accept paths. A hit passes if any of: (a) title-token overlap meets title_breadth (≤2 tokens require ALL; 3+ require N-1); (b) it's in core_match_roots (TF-IDF core keywords contain a query token — closes the gap for neologisms like "permacomputer" that never appear in titles); (c) body density — _body_density_passes requires distinct_present >= breadth_threshold AND total_mentions >= 3. Same breadth scaling as titles. Synonym fallback fires only for 1-token queries (otherwise intel-titled doc bleeds into AMD queries via the rivalry group).
5. Rivalry exclusion (rivalry_excluded in qa/concepts.py). Intel-titled docs drop from AMD queries; reverse holds. Applies on every accept path.
6. Stem-aware token matching. _stem_token_for_match: trailing-s strip on tokens >4 chars (skip ss-enders). Possessive (superman's → supermans → superman) and plural (girlfriends → girlfriend) collapse onto bare-stem source mentions. Caught the 2026-04-29 "who is supermans girlfriend?" defect — pre-stem, the query admitted 7 unrelated Girlfriends-titled articles.
7. Per-source context cap. Each top-K hit gets at most max_context_chars / top_k chars before the global char_budget is consumed. Fox's 2026-04-29 catch: List_of_Batman_comics (80 KB+ bibliography) was monopolizing the entire 60 KB budget at hit #1, dropping every other source.
8. Wikitext base prose runs on the assembled context BEFORE the LLM call (policy["base_version"] gates it, see Wikitext base prose convention above).

Hot path / gotchas

• Parser is hand-rolled in aborist/sources/wikipedia.py (char-position state machine, escape-aware). After the v9.8 commit it's 4× faster via str.find + slicing — easy to break by reverting to char-by-char loops. cProfile any change.
• PRAGMA synchronous=NORMAL is set per-connection in store.connect(). Safe under WAL (the journal_mode is set in SCHEMA_SQL). Don't downgrade to FULL without a measured reason — costs ~5x throughput.
• HTML source has optional deps: pip install '.[html]' for selectolax. The CLI surfaces --source html only if the import succeeds.
• Background ingest/distill processes: stdout is buffered. Use export PYTHONUNBUFFERED=1 or python -u. Per blackops top-level rule.
• Disk pressure. Full cur ingest ~2 GB; full old ingest ~5–8 GB. df -h /home/fox first.

Operational rules

• I propose, fox decides. Unsure = ask. Can't ask = stop.
• No autonomous destructive ops. No clean-data, clean-db, force-push, or DB drops without explicit instruction.
• Never add Co-Authored-By or "Generated with Claude" lines to commits. Professional commit messages only — code speaks for itself.
• Always export PYTHONUNBUFFERED=1 for long-running processes. Buffered output disappears when processes die.
• Fail-closed. Cleanup crew, not demolition.
• DRY in context — single source of truth, no sprawl.
• Never say "AI" — always say "machine learning."
• Prefer "defect" over "bug."
• Check robots.txt before any web fetch the user didn't authorize.

Orientation protocol

date -u
pwd
git log --oneline -5
git status
make test
make chain-check-shards   # per-shard audit-chain integrity (0 = intact)
.venv/bin/aborist --shards-dir ~/.aborist/shards stats
.venv/bin/aborist --shards-dir ~/.aborist/shards analyze --gravity-top 5

Then ask fox what the mission is.