From c427180ced5134ee12c07109e23c398550a7cb92 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Mon, 27 Apr 2026 10:41:20 -0400 Subject: [PATCH] update CLAUDE.md: real orientation for the working repo The stub from `git init` predated everything. Replace with a concise working map: what aborist IS now, where the source papers live, what the schema invariants are, what conventions never to silently change (merkle ordering, chunker default = chunking_version bump = mass cache falsification), where the live endpoints are, and the parser hot-path gotchas. Adds: source-paper inventory, architecture tree, schema invariants (8-dim cache key, audit chain integrity, surface/core kinds, hot/warm/ cold tiers), Merkle conventions, live endpoint URLs, hot-path performance notes, orientation protocol that runs `make test`, `stats`, and `analyze`. Preserves: blackops identity rules, no Co-Authored-By, fail-closed, PYTHONUNBUFFERED for long-running processes. --- CLAUDE.md | 122 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 122 insertions(+) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..9b2393e --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,122 @@ +# Agent Blackops — aborist + +This repo is operated by **agent blackops** for fox/timehexon on the unsandbox / unturf / permacomputer platform. + +Identity shard: `~/git/unsandbox.com/blackops/BLACKOPS.md`. + +## What aborist is + +A content-addressed, Merkle-committed document store. Implements the runtime spec from **Merkle Providence Reverse RAG** (April 2026 whitepaper) scaled up to the **Merkle-AGI v9.8** admissibility ledger. Tends "trees and forests of cross-linked information" — the namesake. + +Three layers stacked on one SQLite file: + +1. **Surface** — ingested documents (Wikipedia dumps, HTML pages, anything with a URI). Chunked, Merkle-rooted, FTS5-indexed. +2. **Core** — distilled documents (haiku/keyword/equation-scale) Merkle-bound back to source surface(s) via per-chunk inclusion proofs in `derivations.proof_blob`. Recursive: cores derive from cores. The "planet toward center" compression. +3. **Providence cache** — Q&A records keyed on the v9.8 8-dim invariant. STRICT-mode hits carry verifiable Merkle proofs. + +## Source papers (read first if confused) + +- `~/Downloads/merkle-providence-reverse-rag-whitepaper.pdf` — public-facing spec, AGPL-3.0 +- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV encoding A1, public quantization A2, collision-resistant hash A3, theorems T1–T5) +- `~/git/proxy.unturf.com/pkg/verified/merkle.go` — fox's existing Go merkle implementation. **Aborist Python ports its conventions exactly.** +- `~/git/proxy.unturf.com/docs/merkle-tree.md` — convention reference + +## Architecture + +``` +aborist/ +├── merkle.py # Python port of proxy.unturf.com Go conventions +├── store.py # v9.8 SQLite schema, audit chain helpers +├── document.py # Document, Edge, Chunker (TokenChunker default) +├── source.py # Source ABC: iter_documents() -> Iterator[Document] +├── ingest.py # batched: normalize -> chunk -> merkle -> upsert +├── evict.py # hot->cold + rehydrate (v9.8 falsification on drift) +├── search/ # SearchBackend ABC + AuditMode + FTS5 +├── sources/ # one file per corpus +│ ├── wikipedia.py # WikipediaSqlDump (cur + old tables) +│ └── html_page.py # HtmlPageSource (selectolax, robots-aware) +├── distill/ # surface->core distillation +│ ├── base.py # Distiller ABC + DistillationResult +│ ├── first_sentence.py # FirstSentenceDistiller (no-ML stub) +│ ├── tfidf.py # TfidfKeywordDistiller (pure-Python TF-IDF) +│ └── runner.py # batched: derive + per-contrib-chunk proofs +├── qa/ # Q&A: 8-dim cache_key + Merkle-bound answers +│ ├── client.py # ChatClient + StubClient + OpenAICompat +│ ├── keys.py # cache_key, question_hash, ... (pure functions) +│ └── runner.py # ask(): cache hit STRICT / miss -> infer -> write +└── cli.py # ingest / search / verify / stats / distill / + # evict / rehydrate / ask / providence / analyze +``` + +## Build, test, run + +Every workflow is a `make` target. Bare python commands are not the user interface. + +``` +make bootstrap # venv + editable install with [dev] extras +make test # pytest -q (49 tests) +make all # bootstrap + fetch-cur + ingest-cur + verify + stats +make fetch # cur (82 MB) + old.1 (640 MiB) + old.2 (252 MiB) + concat +make ingest-cur # ingest snapshot articles +make ingest-old # ingest revision history (~hours) +make verify # round-trip Merkle proofs on a random sample +make analyze # compression spectrum + audit chain integrity +``` + +`aborist/cli.py` adds: `analyze`, `distill --kind {surface,core}`, `evict`, `rehydrate`, `ask`, `providence`. `--batch-size` defaults to 200 docs/transaction; lower it only to bound memory peaks. + +## Schema invariants (do not break) + +- **Aborist is a v9.8 store.** Every Document carries `chunking_version`, `canonicalization_version`, `schema_version`. Every providence record carries the full **8-dim cache_key**: `source_root | question_hash | model_profile_hash | conversation_hash | governance_policy_hash | schema_version | canonicalization_version | chunking_version`. Bumping any one invalidates prior records on lookup. +- **`falsification_state ∈ {live, failed, stale, quarantined}`.** Cache lookups must filter on `state='live'`. Drift detection (rehydrate vs source root mismatch) flips to `stale`. +- **Audit chain.** Every state-changing op writes one row in `audit_events` with `event_hash = sha256(prev_event_hash || canonical(body))`. Chain integrity is verified in `make analyze`. **Never insert into `audit_events` directly — use `aborist.store.append_audit`.** +- **Cores never evict.** `evict_to_cold` only touches `kind='surface'`. +- **Idempotent re-ingest.** Same content → same `document_root` → no-op insert. Same URI + different content → new doc + `supersedes` edge linking new → old (lossless history). + +## Conventions (do not silently change) + +- **Merkle:** non-commutative `HashCombine` with prefix `0x03`. Leaves `0x00`. Odd-element rule = self-duplicate, NOT zero-pad. `MerkleProof.siblings` carries explicit `is_left` flag — never sort lexically. +- **Chunker default = `tok-512-v1`.** Changing the default bumps `chunking_version` and **stales every prior cache record**. Add a new chunker as a new `name` instead. +- **Canonicalization = `norm-v1`** (NFC, collapsed whitespace). Same rule. +- **Schema = `v9.8.0`.** Same rule. +- **Search hits carry `audit_mode`** ∈ `STRICT`/`HYBRID`/`VISUAL`. FTS5 hits are `VISUAL`. Providence-cache hits with verified proofs are `STRICT`. Never overclaim. +- **Soft hash vs hard hash.** Hard = SHA-256 (commitments, proofs, cache_key). Soft = embeddings/TF-IDF/similarity (training, ranking, distillation candidate selection). Never mix — soft never enters proof path. + +## Live endpoints + +- LLM: `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B-FP8-Dynamic on vLLM, 82K ctx, no auth). `uncloseai.com` is marketing only — has no `/v1`. Override via `--endpoint` or `ABORIST_LLM_ENDPOINT`. +- Wikipedia dumps: `https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`. `robots.txt` returned 404 → no rules. + +## Hot path / gotchas + +- **Parser is hand-rolled** in `aborist/sources/wikipedia.py` (char-position state machine, escape-aware). After the v9.8 commit it's 4× faster via `str.find` + slicing — easy to break by reverting to char-by-char loops. cProfile any change. +- **`PRAGMA synchronous=NORMAL`** is set per-connection in `store.connect()`. Safe under WAL (the journal_mode is set in `SCHEMA_SQL`). Don't downgrade to FULL without a measured reason — costs ~5x throughput. +- **HTML source has optional deps**: `pip install '.[html]'` for `selectolax`. The CLI surfaces `--source html` only if the import succeeds. +- **Background ingest/distill processes**: stdout is buffered. Use `export PYTHONUNBUFFERED=1` or `python -u`. Per blackops top-level rule. +- **Disk pressure.** Full cur ingest ~2 GB; full old ingest ~5–8 GB. `df -h /home/fox` first. + +## Operational rules + +- I propose, fox decides. Unsure = ask. Can't ask = stop. +- No autonomous destructive ops. No `clean-data`, `clean-db`, force-push, or DB drops without explicit instruction. +- **Never add `Co-Authored-By` or "Generated with Claude" lines to commits.** Professional commit messages only — code speaks for itself. +- **Always `export PYTHONUNBUFFERED=1`** for long-running processes. Buffered output disappears when processes die. +- Fail-closed. Cleanup crew, not demolition. +- DRY in context — single source of truth, no sprawl. +- Never say "AI" — always say "machine learning." +- Prefer "defect" over "bug." +- Check robots.txt before any web fetch the user didn't authorize. + +## Orientation protocol + +```bash +date -u +pwd +git log --oneline -5 +git status +make test +.venv/bin/aborist --db ~/.aborist/aborist.db stats +.venv/bin/aborist --db ~/.aborist/aborist.db analyze --gravity-top 5 +``` + +Then ask fox what the mission is.