update CLAUDE.md: real orientation for the working repo

The stub from `git init` predated everything. Replace with a concise working map: what aborist IS now, where the source papers live, what the schema invariants are, what conventions never to silently change (merkle ordering, chunker default = chunking_version bump = mass cache falsification), where the live endpoints are, and the parser hot-path gotchas. Adds: source-paper inventory, architecture tree, schema invariants (8-dim cache key, audit chain integrity, surface/core kinds, hot/warm/ cold tiers), Merkle conventions, live endpoint URLs, hot-path performance notes, orientation protocol that runs `make test`, `stats`, and `analyze`. Preserves: blackops identity rules, no Co-Authored-By, fail-closed, PYTHONUNBUFFERED for long-running processes.
2026-04-27 10:41:20 -04:00 · 2026-04-27 10:41:20 -04:00 · c427180ced
commit c427180ced
parent 85ae761df8
1 changed files with 122 additions and 0 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@ -0,0 +1,122 @@
+# Agent Blackops — aborist
+
+This repo is operated by **agent blackops** for fox/timehexon on the unsandbox / unturf / permacomputer platform.
+
+Identity shard: `~/git/unsandbox.com/blackops/BLACKOPS.md`.
+
+## What aborist is
+
+A content-addressed, Merkle-committed document store. Implements the runtime spec from **Merkle Providence Reverse RAG** (April 2026 whitepaper) scaled up to the **Merkle-AGI v9.8** admissibility ledger. Tends "trees and forests of cross-linked information" — the namesake.
+
+Three layers stacked on one SQLite file:
+
+1. **Surface** — ingested documents (Wikipedia dumps, HTML pages, anything with a URI). Chunked, Merkle-rooted, FTS5-indexed.
+2. **Core** — distilled documents (haiku/keyword/equation-scale) Merkle-bound back to source surface(s) via per-chunk inclusion proofs in `derivations.proof_blob`. Recursive: cores derive from cores. The "planet toward center" compression.
+3. **Providence cache** — Q&A records keyed on the v9.8 8-dim invariant. STRICT-mode hits carry verifiable Merkle proofs.
+
+## Source papers (read first if confused)
+
+- `~/Downloads/merkle-providence-reverse-rag-whitepaper.pdf` — public-facing spec, AGPL-3.0
+- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV encoding A1, public quantization A2, collision-resistant hash A3, theorems T1–T5)
+- `~/git/proxy.unturf.com/pkg/verified/merkle.go` — fox's existing Go merkle implementation. **Aborist Python ports its conventions exactly.**
+- `~/git/proxy.unturf.com/docs/merkle-tree.md` — convention reference
+
+## Architecture
+
+```
+aborist/
+├── merkle.py              # Python port of proxy.unturf.com Go conventions
+├── store.py               # v9.8 SQLite schema, audit chain helpers
+├── document.py            # Document, Edge, Chunker (TokenChunker default)
+├── source.py              # Source ABC: iter_documents() -> Iterator[Document]
+├── ingest.py              # batched: normalize -> chunk -> merkle -> upsert
+├── evict.py               # hot->cold + rehydrate (v9.8 falsification on drift)
+├── search/                # SearchBackend ABC + AuditMode + FTS5
+├── sources/               # one file per corpus
+│   ├── wikipedia.py       #   WikipediaSqlDump (cur + old tables)
+│   └── html_page.py       #   HtmlPageSource (selectolax, robots-aware)
+├── distill/               # surface->core distillation
+│   ├── base.py            #   Distiller ABC + DistillationResult
+│   ├── first_sentence.py  #   FirstSentenceDistiller (no-ML stub)
+│   ├── tfidf.py           #   TfidfKeywordDistiller (pure-Python TF-IDF)
+│   └── runner.py          #   batched: derive + per-contrib-chunk proofs
+├── qa/                    # Q&A: 8-dim cache_key + Merkle-bound answers
+│   ├── client.py          #   ChatClient + StubClient + OpenAICompat
+│   ├── keys.py            #   cache_key, question_hash, ... (pure functions)
+│   └── runner.py          #   ask(): cache hit STRICT / miss -> infer -> write
+└── cli.py                 # ingest / search / verify / stats / distill /
+                           # evict / rehydrate / ask / providence / analyze
+```
+
+## Build, test, run
+
+Every workflow is a `make` target. Bare python commands are not the user interface.
+
+```
+make bootstrap           # venv + editable install with [dev] extras
+make test                # pytest -q (49 tests)
+make all                 # bootstrap + fetch-cur + ingest-cur + verify + stats
+make fetch               # cur (82 MB) + old.1 (640 MiB) + old.2 (252 MiB) + concat
+make ingest-cur          # ingest snapshot articles
+make ingest-old          # ingest revision history (~hours)
+make verify              # round-trip Merkle proofs on a random sample
+make analyze             # compression spectrum + audit chain integrity
+```
+
+`aborist/cli.py` adds: `analyze`, `distill --kind {surface,core}`, `evict`, `rehydrate`, `ask`, `providence`. `--batch-size` defaults to 200 docs/transaction; lower it only to bound memory peaks.
+
+## Schema invariants (do not break)
+
+- **Aborist is a v9.8 store.** Every Document carries `chunking_version`, `canonicalization_version`, `schema_version`. Every providence record carries the full **8-dim cache_key**: `source_root | question_hash | model_profile_hash | conversation_hash | governance_policy_hash | schema_version | canonicalization_version | chunking_version`. Bumping any one invalidates prior records on lookup.
+- **`falsification_state ∈ {live, failed, stale, quarantined}`.** Cache lookups must filter on `state='live'`. Drift detection (rehydrate vs source root mismatch) flips to `stale`.
+- **Audit chain.** Every state-changing op writes one row in `audit_events` with `event_hash = sha256(prev_event_hash || canonical(body))`. Chain integrity is verified in `make analyze`. **Never insert into `audit_events` directly — use `aborist.store.append_audit`.**
+- **Cores never evict.** `evict_to_cold` only touches `kind='surface'`.
+- **Idempotent re-ingest.** Same content → same `document_root` → no-op insert. Same URI + different content → new doc + `supersedes` edge linking new → old (lossless history).
+
+## Conventions (do not silently change)
+
+- **Merkle:** non-commutative `HashCombine` with prefix `0x03`. Leaves `0x00`. Odd-element rule = self-duplicate, NOT zero-pad. `MerkleProof.siblings` carries explicit `is_left` flag — never sort lexically.
+- **Chunker default = `tok-512-v1`.** Changing the default bumps `chunking_version` and **stales every prior cache record**. Add a new chunker as a new `name` instead.
+- **Canonicalization = `norm-v1`** (NFC, collapsed whitespace). Same rule.
+- **Schema = `v9.8.0`.** Same rule.
+- **Search hits carry `audit_mode`** ∈ `STRICT`/`HYBRID`/`VISUAL`. FTS5 hits are `VISUAL`. Providence-cache hits with verified proofs are `STRICT`. Never overclaim.
+- **Soft hash vs hard hash.** Hard = SHA-256 (commitments, proofs, cache_key). Soft = embeddings/TF-IDF/similarity (training, ranking, distillation candidate selection). Never mix — soft never enters proof path.
+
+## Live endpoints
+
+- LLM: `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B-FP8-Dynamic on vLLM, 82K ctx, no auth). `uncloseai.com` is marketing only — has no `/v1`. Override via `--endpoint` or `ABORIST_LLM_ENDPOINT`.
+- Wikipedia dumps: `https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`. `robots.txt` returned 404 → no rules.
+
+## Hot path / gotchas
+
+- **Parser is hand-rolled** in `aborist/sources/wikipedia.py` (char-position state machine, escape-aware). After the v9.8 commit it's 4× faster via `str.find` + slicing — easy to break by reverting to char-by-char loops. cProfile any change.
+- **`PRAGMA synchronous=NORMAL`** is set per-connection in `store.connect()`. Safe under WAL (the journal_mode is set in `SCHEMA_SQL`). Don't downgrade to FULL without a measured reason — costs ~5x throughput.
+- **HTML source has optional deps**: `pip install '.[html]'` for `selectolax`. The CLI surfaces `--source html` only if the import succeeds.
+- **Background ingest/distill processes**: stdout is buffered. Use `export PYTHONUNBUFFERED=1` or `python -u`. Per blackops top-level rule.
+- **Disk pressure.** Full cur ingest ~2 GB; full old ingest ~5–8 GB. `df -h /home/fox` first.
+
+## Operational rules
+
+- I propose, fox decides. Unsure = ask. Can't ask = stop.
+- No autonomous destructive ops. No `clean-data`, `clean-db`, force-push, or DB drops without explicit instruction.
+- **Never add `Co-Authored-By` or "Generated with Claude" lines to commits.** Professional commit messages only — code speaks for itself.
+- **Always `export PYTHONUNBUFFERED=1`** for long-running processes. Buffered output disappears when processes die.
+- Fail-closed. Cleanup crew, not demolition.
+- DRY in context — single source of truth, no sprawl.
+- Never say "AI" — always say "machine learning."
+- Prefer "defect" over "bug."
+- Check robots.txt before any web fetch the user didn't authorize.
+
+## Orientation protocol
+
+```bash
+date -u
+pwd
+git log --oneline -5
+git status
+make test
+.venv/bin/aborist --db ~/.aborist/aborist.db stats
+.venv/bin/aborist --db ~/.aborist/aborist.db analyze --gravity-top 5
+```
+
+Then ask fox what the mission is.