arborist/CLAUDE.md

Agent Blackops — aborist

This repo is operated by agent blackops for fox/timehexon on the unsandbox / unturf / permacomputer platform.

Identity shard: ~/git/unsandbox.com/blackops/BLACKOPS.md.

What aborist is

A content-addressed, Merkle-committed document store. Implements the runtime spec from Merkle Providence Reverse RAG (April 2026 whitepaper) scaled up to the Merkle-AGI v9.8 admissibility ledger.

Three layers stacked on one SQLite file:

1. Surface — ingested documents (Wikipedia dumps, HTML pages, anything with a URI). Chunked, Merkle-rooted, FTS5-indexed.
2. Core — distilled documents Merkle-bound back to surfaces via per-chunk inclusion proofs in derivations.proof_blob. Recursive.
3. Providence cache — Q&A records keyed on the v9.8 8-dim invariant. Each record carries audit_mode (STRICT / HYBRID / UNGROUNDED) decided by the verifier in aborist/qa/verify.py.

Source papers

• ~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst — canonical whitepaper (rst → PDF). Edit here, not the PDF.
• ~/Downloads/merkle-agi-dag_v7.txt — formal substrate (TLV encoding A1, public quantization A2, collision-resistant hash A3, theorems T1–T5).
• ~/git/proxy.unturf.com/pkg/verified/merkle.go — fox's existing Go merkle. Aborist Python ports its conventions exactly.
• ~/git/proxy.unturf.com/docs/merkle-tree.md — convention reference.

Architecture

aborist/
├── merkle.py        # Python port of proxy.unturf.com Go conventions
├── store.py         # v9.8 SQLite schema, audit-chain helpers
├── document.py      # Document, Edge, Chunker (TokenChunker default)
├── source.py        # Source ABC: iter_documents()
├── ingest.py        # batched normalize → chunk → merkle → upsert
├── evict.py         # hot↔cold + rehydrate (v9.8 falsification on drift)
├── search/          # SearchBackend ABC + AuditMode + FTS5
├── sources/         # one file per corpus (wikipedia, html_page)
├── distill/         # surface → core distillation (tfidf, first_sentence)
├── qa/              # Q&A: 8-dim cache_key + Merkle-bound answers
│   ├── client.py    #   ChatClient + StubClient + OpenAICompat
│   ├── keys.py      #   cache_key, question_hash (pure functions)
│   ├── verify.py    #   layered verifier + claim-lattice verifiers (G0)
│   ├── evidence.py  #   EvidenceObject + spotlight excerpt
│   ├── inspect.py   #   read-only sidecars (deflection, title-relevance)
│   ├── dag.py       #   per-run Merkle-DAG (7-stage quote / 9-stage CTI)
│   └── runner.py    #   ask(): cache → infer → verify → write
├── wikitext.py      # to_base(): wikitext → plain prose (BASE_VERSION-pinned)
└── cli.py           # ingest / search / verify / stats / distill /
                     # evict / rehydrate / ask / providence / emergent /
                     # reclassify / inspect / analyze

Build, test, run

Every workflow is a make target. Bare python is not the user interface. See the Makefile for the full list.

make bootstrap           # venv + editable install with [dev] extras
make test                # pytest -q
make all                 # bootstrap + fetch-cur + ingest-cur + verify + stats
make verify-shards       # round-trip Merkle proofs (cross-shard sample)
make analyze-shards      # cross-shard compression + audit integrity
make chain-check-shards  # audit-chain break count per shard (0 = intact)
make query Q="..." [JSON=1 BURN=1 K="extra retrieval keywords" ANSWER_MODE=…]
make bench-qa            # QA-quality sweep (live LLM)

Hygiene after any state-changing op (rebuild, reclassify, hash bump, mass falsify): make chain-check-shards first (every shard should report 0), then make analyze-shards for the spectrum + chain audit. Chain breaks are the loudest possible signal.

Schema invariants (do not break)

• v9.8 8-dim cache_key: source_root | question_hash | model_profile_hash | conversation_hash | governance_policy_hash | schema_version | canonicalization_version | chunking_version. Bumping any one invalidates prior records on lookup.
• falsification_state ∈ {live, failed, stale, quarantined}. Cache lookups filter on state='live'. Drift → stale.
• Audit chain: every state-changing op writes one row in audit_events with event_hash = sha256(prev || canonical(body)). Verified by make chain-check-shards. Use aborist.store.append_audit — never insert into audit_events directly.
• Cores never evict. evict_to_cold only touches kind='surface'.
• Idempotent re-ingest. Same content → same document_root → no-op insert. Same URI + different content → new doc + supersedes edge (lossless history).

Conventions (do not silently change)

Each rule below has full rationale in the named source file. Don't revert without reading why. When in doubt, walk the Five-step algorithm first.

• Merkle conventions: non-commutative HashCombine prefix 0x03, leaves 0x00, odd-element rule = self-duplicate (NOT zero-pad). MerkleProof.siblings carries is_left flag — never sort lexically. See aborist/merkle.py.
• Versioned defaults: tok-512-v1 (chunker), norm-v1 (canonicalization), v9.8.0 (schema), wikitext-base-v1 (prose). Changing any default stales every prior cache record. Add a new name instead.
• question_hash is dedup-mode-aware (strict | equivalence_class); folds into governance_policy_hash. JIT fidelity parameter on query()/ask() decouples lookup tolerance from write policy. See aborist/qa/keys.py.
• audit_mode is decided by the verifier, never asserted. Four layered strategies tried in order, first to find evidence classifies: quote (sequential pair-matching, NOT regex — prevents phantom inter-pair captures), span (verbatim line match), entity (proximity-clustered proper nouns; entity_policy ∈ {strict, hybrid, drop, proximity}), paraphrase (token-coverage, prose-shaped only; verifier_method='paraphrase'). Trichotomy: STRICT = every unit verifies, HYBRID = mixed, UNGROUNDED = none. Never overclaim. See aborist/qa/verify.py.
• Verifier stays binary; falsifications carry soft signal. No per-quote diagnosis fields on hard verifier output. Sidecars (aborist.qa.inspect.diagnose_*, aborist inspect --cache-key X) classify unverified spans, deflection, title-relevance — never write to providence_cache or audit_events.
• Trailing-citation strip: _strip_trailing_citation peels one trailing parenthetical at end-of-span (gated on a citation cue or URL) before substring testing. See aborist/qa/verify.py.
• Soft hash vs hard hash: hard = SHA-256 (commitments, proofs, cache_key); soft = embeddings/TF-IDF/similarity (training, ranking, distillation). Soft never enters proof path.
• Three answer modes: policy["answer_mode"] ∈ {"quote", "claim_lattice_pointer", "claim_lattice"}, default "quote". Bench 2026-04-30 on Hermes-3-8B: quote 0.47 strict-rate, pointer 0.36, JSON 0.49 — JSON wins. Both lattice modes share verifier_method="claim_lattice"; answer_mode on the run-DAG
  • json_fixups disambiguate. Each mode folds into governance_policy_hash. See aborist/qa/verify.py, docs/qa-modes-bench-2026-04-30.md.
• Claim-lattice-pointer mode (G0 / CTI): runtime mints pointer_id (E1, E2, … — what the model sees) and content-addressed evidence_id (what the cache & run-DAG store). Renderer interpolates literal source spans via _spotlight_excerpt. Synthetic-elision-by- construction-impossible — model never types the quote string. 9-stage run-DAG. See aborist/qa/evidence.py, docs/cti-architecture.md.
• Claim-count ceiling (TOO_MANY_CLAIMS): default 12 per answer. Catches "tell me all there is to know about X" runaway. Demotes STRICT → HYBRID without truncating. Folds into governance_policy_hash. See aborist/qa/verify.py.
• Wikitext base prose: aborist/wikitext.py:to_base() runs before the LLM call AND inside verify_quotes so model and verifier see the same prose. Optional dep — graceful fallback when mwparserfromhell is missing.
• Deflection sidecar: diagnose_deflection(question, answer) detects topic-shift via subject-anchor heuristic (LAST content token in question must appear in answer). Suppressed for date / count / cause shapes ("when", "why", "how many"). See aborist/qa/inspect.py.
• Title-relevance hard check (Rule 8): _claim_title_overlap in aborist/qa/verify.py. For each claim that resolved, at least one cited evidence's source title must share ≥1 stemmed content token with the claim text. When NO cited title overlaps, record a TITLE_MISMATCH violation & demote STRICT → HYBRID. Catches retrieval-driven hallucinations where the cited chunk's SOURCE is structurally unrelated to the claim's subject (2026-05-02 spin-glass case: claim about spin glass cited to a chunk from Quantum chromodynamics; span had incidental physics-vocab overlap, but the source title shared zero stems with the claim). Renderer surfaces a · title mismatch tail on the audit-line label alongside · warrant missing.
• Title-relevance sidecar (legacy diagnostic): diagnose_title_relevance(claim, cited_titles) in aborist.qa.inspect returns the same signal in dict form for per-cache-key inspection. Sidecar; never enters proof path. Pre-dates the Rule 8 promotion (2026-05-02).

Live endpoints

• LLM: https://hermes.ai.unturf.com/v1 (Hermes-3 Llama-3.1-8B-FP8- Dynamic on vLLM, 82K ctx, no auth). uncloseai.com is marketing only. Override via --endpoint or ABORIST_LLM_ENDPOINT.
• Wikipedia dumps: https://dumps.wikimedia.org/archive/2003/2003-05-16/en/. robots.txt returned 404 → no rules.

Retrieval pipeline (aborist/qa/query.py)

Multi-stage. Each stage exists because something earlier wasn't enough; revert at your peril. Order:

1. Four parallel FTS5 search routes per shard, merged — body BM25, title-LIKE, core-keyword (TF-IDF cores), and phrase-pattern (verbatim n=5/n=6 sequences from the question). Phrase route closes the allusion gap (Orwell case: "always been at war" verbatim matches the 1984 article whose title shares zero tokens with the query). See docs/reference-frame-failure-class.md.
2. Body-coverage sqrt rerank — counters BM25's short-doc bias.
3. Title-token boost — boost × overlap on title-token-matching hits.
4. _filter_by_title_relevance — four accept paths: title-token overlap, TF-IDF core match, body density, phrase match (accept-path 4 lets phrase-route hits with no title overlap survive).
5. Rivalry exclusion + synonym expansion (aborist/concepts/) — Intel-titled docs drop from AMD queries; reverse holds. Backed by the per-shard concept_relations SQLite table (corpus-derived, not hand-curated). 1.6% storage tax measured at backfill on 6 GB wiki — kept flat, no further compaction. See docs/concept-relations-design.md for the storage choice rationale.
6. Stem-aware token matching — possessive / plural collapse (superman's → supermans → superman).
7. Per-source context cap — max_context_chars / top_k. Prevents one huge doc from monopolizing the budget.
8. Wikitext base prose runs on assembled context BEFORE the LLM.
9. Template-phrase stopwords (_FTS5_STOPWORDS and _TITLE_STOPWORDS must stay in sync) — strips tell show describe explain summarize say give list find make please all there know everything anything something so "tell me all there is to know about X" doesn't dilute query tokens.

--retrieval-keywords (CLI: K="...") lets an operator augment retrieval-side tokens without changing what the LLM sees as its question. Provenance gap on this is tracked in Ticket #000001.

Hot path / gotchas

• Hand-rolled wikitext parser (aborist/sources/wikipedia.py): char-position state machine, escape-aware, 4× faster than char- by-char loops via str.find + slicing. cProfile any change.
• PRAGMA synchronous=NORMAL per-connection in store.connect(). Safe under WAL. Don't downgrade without measured reason (~5× cost).
• HTML source has optional deps: pip install '.[html]' for selectolax. CLI surfaces --source html only if import succeeds.
• Background ingest/distill processes: stdout is buffered. Use export PYTHONUNBUFFERED=1 or python -u.
• Disk pressure: full cur ~2 GB; full old ~5–8 GB. df -h first.

Operational rules

• I propose, fox decides. Unsure = ask. Can't ask = stop.
• No autonomous destructive ops (clean-data, clean-db, force-push, DB drops) without explicit instruction.
• Never add Co-Authored-By or "Generated with Claude" lines to commits. Code speaks for itself.
• Always export PYTHONUNBUFFERED=1 for long-running processes.
• Fail-closed. Cleanup crew, not demolition.
• DRY in context — single source of truth, no sprawl.
• Never say "AI" — always say "machine learning."
• Prefer "defect" over "bug."
• Check robots.txt before any web fetch the user didn't authorize.

Five-step algorithm

When proposing or evaluating change, walk these in sequence. Skipping a step makes the next ones expensive and the system worse.

1. Make the requirements less dumb. Every requirement gets a person's name, not a department. If you can't name who asked or which defect closed, the requirement is suspect.
2. Delete the part or the process. If you aren't putting back at least 10% of what you delete, you aren't deleting hard enough. Verifier-stays-binary and "no soft signals in hard chain" are deletion-first guardrails.
3. Simplify and optimize. Only after 1 + 2. Don't optimize a process that shouldn't exist.
4. Accelerate cycle time. Only after simplifying. "Don't dig the grave faster."
5. Automate. Last, not first. Hand-rolled before scripted, scripted before declarative, declarative before generated.

When in doubt, ask "have we tried deleting it?" before reaching for steps 3-5.

Bench-maxing — measure deltas, not opinions

Full discipline + worked examples in docs/bench-maxing.md. Headlines:

• Bench before AND after every change (n=3, signal under 5pp is noise).
• Avoid negation in prompts (Hermes-3-8B inverts under attention).
• Bench is the scoreboard; live fixtures are the gates.
• Self-heal beats retry (preserve partial output, never fabricate).
• Honest verdicts beat optimistic ones (false-positive STRICT is corruption).
• Old maps vs runtime maps — when the model and runtime disagree, the runtime wins. Pointer IDs, runtime-interpolated spans, evidence maps, policy hashes, hard verifier checks all move authority OUT of the model's prior and INTO runtime artifacts.

Docs index

North-star:

• docs/seven-point-program.md — the architectural directive distilled 2026-05-01. Every new ticket / feature / prompt edit walks past this. Bench harness reports per-mode directive coverage.

Architecture / ongoing work:

• docs/cti-architecture.md — CTI Clause Tree Intelligence.
• docs/mesh.md, docs/mesh-deploy.md — mesh wire + deploy.
• docs/qa-modes-bench-2026-04-30.md — JSON-mode hardening journey.
• docs/reference-frame-failure-class.md — Orwell case + phrase-route fix.
• docs/bench-maxing.md — bench discipline.
• docs/test-coverage-audit-2026-05-01.md — test-suite coverage audit.
• docs/verifier-semantic-gap-design.md — soft-signal NLI proposal.
• docs/self-reference-thought-chains-design.md, docs/self-reference-distillation-design.md — recursive distillation.
• docs/naming-deferral.md — naming convention notes.

Tickets: docs/TICKETS.md (index + Next ID). Open tickets:

• docs/ticket-000001-retrieval-keywords-audit-gap.md — provenance binding for --retrieval-keywords. Directive D4.
• docs/ticket-000002-reference-frame-polarity-contract.md — Module L: multi-frame answer compilation. Directive D3.
• docs/ticket-000003-anchor-class-warrant.md — anchor-class warrant generalization (entity-list / count / why-cause shapes). Directive D6.

Orientation protocol

date -u
pwd
git log --oneline -5
git status
make test
make chain-check-shards   # 0 per shard = intact
.venv/bin/aborist --shards-dir ~/.aborist/shards stats
.venv/bin/aborist --shards-dir ~/.aborist/shards analyze --gravity-top 5

Then ask fox what the mission is.