arborist/CLAUDE.md

# Agent Blackops — arborist

This repo is operated by **agent blackops** for fox/timehexon on the
unsandbox / unturf / permacomputer platform.

Identity shard: `~/git/unsandbox.com/blackops/BLACKOPS.md`.

## What arborist is

A content-addressed, Merkle-committed document store. Implements the
runtime spec from **Merkle Providence Reverse RAG** (April 2026
whitepaper) scaled up to the **Merkle-AGI v9.8** admissibility ledger.

Three layers stacked on one SQLite file:

1. **Surface** — ingested documents (Wikipedia dumps, HTML pages,
   anything with a URI). Chunked, Merkle-rooted, FTS5-indexed.
2. **Core** — distilled documents Merkle-bound back to surfaces via
   per-chunk inclusion proofs in `derivations.proof_blob`. Recursive.
3. **Providence cache** — Q&A records keyed on the v9.8 8-dim
   invariant. Each record carries `audit_mode` (STRICT / HYBRID /
   UNGROUNDED) decided by the verifier in `arborist/qa/verify.py`.
   For claim-lattice modes the renderer maps that token to a four-
   rung ladder (POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED;
   ENTAILMENT-VERIFIED reserved); UNGROUNDED below all rungs. The
   schema column stays unchanged — programmatic callers see the
   trichotomy, human-facing surfaces see the ladder. See
   `arborist/cli.py:_render_audit_label`.

## Source papers

- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst`
  — canonical whitepaper (rst → PDF). Edit here, not the PDF.
- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV
  encoding A1, public quantization A2, collision-resistant hash A3,
  theorems T1–T5).
- `~/git/proxy.unturf.com/pkg/verified/merkle.go` — fox's existing
  Go merkle. Arborist Python ports its conventions exactly.
- `~/git/proxy.unturf.com/docs/merkle-tree.md` — convention reference.

## Architecture

```
arborist/
├── merkle.py        # Python port of proxy.unturf.com Go conventions
├── store.py         # v9.8 SQLite schema, audit-chain helpers
├── document.py      # Document, Edge, Chunker (TokenChunker default)
├── source.py        # Source ABC: iter_documents()
├── ingest.py        # batched normalize → chunk → merkle → upsert
├── evict.py         # hot↔cold + rehydrate (v9.8 falsification on drift)
├── search/          # SearchBackend ABC + AuditMode + FTS5
├── sources/         # one file per corpus (wikipedia, html_page)
├── distill/         # surface → core distillation (tfidf, first_sentence)
├── qa/              # Q&A: 8-dim cache_key + Merkle-bound answers
│   ├── client.py    #   ChatClient + StubClient + OpenAICompat
│   ├── keys.py      #   cache_key, question_hash (pure functions)
│   ├── verify.py    #   layered verifier + claim-lattice verifiers (G0)
│   ├── evidence.py  #   EvidenceObject + spotlight excerpt
│   ├── inspect.py   #   read-only sidecars (deflection, title-relevance)
│   ├── dag.py       #   per-run Merkle-DAG (7/8 quote · 9/10 CTI · 3 reject)
│   ├── quantifier.py        # broad-quantifier classifier (#000008 P1)
│   ├── model_profiles.py    # per-model claim-cap profiles (#000008 P2)
│   ├── quantifier_reminder.py # broad-query reminder text (#000008 P3)
│   └── runner.py    #   ask(): cache → infer → verify → write
├── wikitext.py      # to_base(): wikitext → plain prose (BASE_VERSION-pinned)
└── cli.py           # ingest / search / verify / stats / distill /
                     # evict / rehydrate / ask / providence / emergent /
                     # reclassify / inspect / analyze
```

## Build, test, run

Every workflow is a `make` target. Bare python is not the user
interface. See the Makefile for the full list.

```
make bootstrap           # venv + editable install with [dev] extras
make test                # pytest -q
make all                 # bootstrap + fetch-cur + ingest-cur + verify + stats
make verify-shards       # round-trip Merkle proofs (cross-shard sample)
make analyze-shards      # cross-shard compression + audit integrity
make chain-check-shards  # audit-chain break count per shard (0 = intact)
make query Q="..." [JSON=1 BURN=1 K="extra retrieval keywords" ANSWER_MODE=…]
make bench-qa            # QA-quality sweep (live LLM)
```

**Hygiene after any state-changing op** (rebuild, reclassify, hash
bump, mass falsify): `make chain-check-shards` first (every shard
should report `0`), then `make analyze-shards` for the spectrum +
chain audit. Chain breaks are the loudest possible signal.

## Schema invariants (do not break)

- **v9.8 8-dim cache_key**: `source_root | question_hash |
  model_profile_hash | conversation_hash | governance_policy_hash |
  schema_version | canonicalization_version | chunking_version`.
  Bumping any one invalidates prior records on lookup.
- **`falsification_state ∈ {live, failed, stale, quarantined}`**.
  Cache lookups filter on `state='live'`. Drift → `stale`.
- **Audit chain**: every state-changing op writes one row in
  `audit_events` with `event_hash = sha256(prev || canonical(body))`.
  Verified by `make chain-check-shards`. Use `arborist.store.append_audit`
  — never insert into `audit_events` directly.
- **Cores never evict.** `evict_to_cold` only touches `kind='surface'`.
- **Idempotent re-ingest.** Same content → same `document_root` →
  no-op insert. Same URI + different content → new doc + `supersedes`
  edge (lossless history).

## Conventions (do not silently change)

Each rule below has full rationale in the named source file. Don't
revert without reading why. When in doubt, walk the
[Five-step algorithm](#five-step-algorithm) first.

- **Merkle conventions**: non-commutative `HashCombine` prefix `0x03`,
  leaves `0x00`, odd-element rule = self-duplicate (NOT zero-pad).
  `MerkleProof.siblings` carries `is_left` flag — never sort lexically.
  See `arborist/merkle.py`.
- **Versioned defaults**: `tok-512-v1` (chunker), `norm-v1`
  (canonicalization), `v9.8.0` (schema), `wikitext-base-v1` (prose).
  Changing any default stales every prior cache record. Add a new
  `name` instead.
- **`question_hash` is dedup-mode-aware** (`strict` |
  `equivalence_class`); folds into `governance_policy_hash`. JIT
  `fidelity` parameter on `query()`/`ask()` decouples lookup tolerance
  from write policy. See `arborist/qa/keys.py`.
- **`audit_mode` is decided by the verifier, never asserted.** Four
  layered strategies tried in order, first to find evidence
  classifies: **quote** (sequential pair-matching, NOT regex —
  prevents phantom inter-pair captures), **span** (verbatim line
  match), **entity** (proximity-clustered proper nouns;
  `entity_policy ∈ {strict, hybrid, drop, proximity}`), **paraphrase**
  (token-coverage, prose-shaped only; `verifier_method='paraphrase'`).
  Trichotomy: STRICT = every unit verifies, HYBRID = mixed,
  UNGROUNDED = none. Never overclaim. See `arborist/qa/verify.py`.
- **Verifier stays binary; falsifications carry soft signal.** No
  per-quote diagnosis fields on hard verifier output. Sidecars
  (`arborist.qa.inspect.diagnose_*`, `arborist inspect --cache-key X`)
  classify unverified spans, deflection, title-relevance — never
  write to `providence_cache` or `audit_events`.
- **Trailing-citation strip**: `_strip_trailing_citation` peels one
  trailing parenthetical at end-of-span (gated on a citation cue or
  URL) before substring testing. See `arborist/qa/verify.py`.
- **Soft hash vs hard hash**: hard = SHA-256 (commitments, proofs,
  cache_key); soft = embeddings/TF-IDF/similarity (training, ranking,
  distillation). Soft never enters proof path.
- **Three answer modes**: `policy["answer_mode"] ∈ {"quote",
  "claim_lattice_pointer", "claim_lattice"}`, default `"quote"`.
  Bench 2026-05-02T15:07Z on Hermes-3-8B (post-Sprint-1b/2, n=3
  × 71 questions, sample-shuffled @ c=4): quote 0.54 strict-rate,
  pointer 0.20, JSON 0.42. Quote leads on raw lexical grounding;
  JSON leads among lattice modes. Per-mode peak buckets: quote
  8-16KB (0.58), pointer 16-32KB (0.20), JSON 32-64KB (0.48) —
  these drive `max_context_chars_by_mode`. 99% directive coverage
  (D2/D3/D4/D6/D7) on lattice modes. Both lattice modes share
  `verifier_method="claim_lattice"`; `answer_mode` on the run-DAG
  + `json_fixups` disambiguate. Each mode folds into
  `governance_policy_hash`. See `arborist/qa/verify.py`,
  `docs/qa-modes-bench.md`.
- **Four-rung ladder (lattice-mode display layer)**: POINTER-LINKED
  → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → (ENTAILMENT-VERIFIED
  reserved); UNGROUNDED below. Each rung names a strictly stronger
  property the lexical verifier could confirm. WARRANT_MISSING drops
  to POINTER-LINKED; soft-demote violations (LAZY_ANCHOR_DEMOTED,
  POINTER_OVERFLOW_TRIMMED, TOO_MANY_CLAIMS, BARE_NAME_CLAIM,
  TITLE_MISMATCH) cap at ANCHOR-WARRANTED. Quote / span / entity /
  paraphrase modes keep their original audit_mode tokens (those
  verify against pinned spans, not synthesis). Pure render-layer —
  cache_key, governance_policy_hash, & all programmatic callers see
  the underlying audit_mode unchanged. See
  `arborist/cli.py:_render_audit_label`.
- **Claim-lattice-pointer mode (G0 / CTI)**: runtime mints
  `pointer_id` (E1, E2, … — what the model sees) and content-addressed
  `evidence_id` (what the cache & run-DAG store). Renderer interpolates
  literal source spans via `_spotlight_excerpt`. Synthetic-elision-by-
  construction-impossible — model never types the quote string. 9-stage
  run-DAG. See `arborist/qa/evidence.py`, `docs/cti-architecture.md`.
- **Claim-count ceiling (`TOO_MANY_CLAIMS`)**: default 12 per answer.
  Catches "tell me all there is to know about X" runaway. Demotes
  STRICT → HYBRID without truncating. Folds into
  `governance_policy_hash`. See `arborist/qa/verify.py`.
- **Broad-quantifier preflight guard (Ticket #000008)**: pure lexical
  classifier (`arborist/qa/quantifier.py`) maps a question onto a
  10-rung intensity ladder (ABSENT → SINGULAR → … → ALL →
  COMPREHENSIVE). Per-model `arborist/qa/model_profiles.py` PROFILES
  dict picks a per-call claim-cap from the (intensity, model) pair;
  `arborist/qa/quantifier_reminder.py` synthesizes a one-line user-turn
  reminder for broad questions. Six-level disable hierarchy
  (per-test, per-call CLI, per-phase policy, per-mode, per-model,
  master-via-governance-hash). 7 policy fields fold into
  `governance_policy_hash` so flipping any of them invalidates prior
  records. Defaults preserve dry-run discipline:
  `quantifier_guard_apply_caps=False`,
  `quantifier_reminder_enabled=False`,
  `quantifier_reject_broad=False`. CLI flags on `arborist query`:
  `--no-quantifier-guard`, `--allow-broad`, `--reject-broad`,
  `--apply-quantifier-caps`. Bench A/B (2026-05-03, n=3 × 9 broad
  questions × 3 modes): reminder default-on supported (FORMAT_COLLAPSED
  −100%, NO_EVIDENCE_POINTER −33%, JSON UNGROUNDED −22pp); cap default-on
  for JSON only (+14pp STRICT-rate, no gain on pointer); cap+reminder
  best on pointer mean ratio (0.684) but not strictly best on JSON
  STRICT-rate. See `docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md`
  §12 for the four-cell A/B data.
- **Wikitext base prose**: `arborist/wikitext.py:to_base()` runs
  before the LLM call AND inside `verify_quotes` so model and verifier
  see the same prose. Optional dep — graceful fallback when
  `mwparserfromhell` is missing.
- **Deflection sidecar**: `diagnose_deflection(question, answer)`
  detects topic-shift via subject-anchor heuristic (LAST content
  token in question must appear in answer). Suppressed for date /
  count / cause shapes ("when", "why", "how many"). See
  `arborist/qa/inspect.py`.
- **Title-relevance hard check (Rule 8)**: `_claim_title_overlap`
  in `arborist/qa/verify.py`. For each claim that resolved, at least
  one cited evidence's source title must share ≥1 stemmed content
  token with the claim text. When NO cited title overlaps, record
  a `TITLE_MISMATCH` violation & demote `STRICT → HYBRID`. Catches
  retrieval-driven hallucinations where the cited chunk's SOURCE
  is structurally unrelated to the claim's subject (2026-05-02
  spin-glass case: claim about spin glass cited to a chunk from
  *Quantum chromodynamics*; span had incidental physics-vocab
  overlap, but the source title shared zero stems with the claim).
  Renderer surfaces a `· title mismatch` tail on the audit-line
  label alongside `· warrant missing`.
- **Title-relevance sidecar (legacy diagnostic)**:
  `diagnose_title_relevance(claim, cited_titles)` in
  `arborist.qa.inspect` returns the same signal in dict form for
  per-cache-key inspection. Sidecar; never enters proof path.
  Pre-dates the Rule 8 promotion (2026-05-02).

## Live endpoints

- LLM: `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B-FP8-
  Dynamic on vLLM, 82K ctx, no auth). `uncloseai.com` is marketing
  only. Override via `--endpoint` or `ARBORIST_LLM_ENDPOINT`.
- Wikipedia dumps: `https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`.
  `robots.txt` returned 404 → no rules.

## Retrieval pipeline (`arborist/qa/query.py`)

Multi-stage. Each stage exists because something earlier wasn't
enough; revert at your peril. Order:

1. **Four parallel FTS5 search routes per shard, merged** — body BM25,
   title-LIKE, core-keyword (TF-IDF cores), and **phrase-pattern**
   (verbatim n=5/n=6 sequences from the question). Phrase route
   closes the allusion gap (Orwell case: "always been at war" verbatim
   matches the 1984 article whose title shares zero tokens with the
   query).
2. **Body-coverage `sqrt` rerank** — counters BM25's short-doc bias.
3. **Title-token boost** — `boost × overlap` on title-token-matching hits.
4. **`_filter_by_title_relevance` — four accept paths**: title-token
   overlap, TF-IDF core match, body density, **phrase match**
   (accept-path 4 lets phrase-route hits with no title overlap survive).
5. **Rivalry exclusion + synonym expansion** (`arborist/concepts/`) —
   Intel-titled docs drop from AMD queries; reverse holds. Backed
   by the per-shard `concept_relations` SQLite table (corpus-derived,
   not hand-curated). 1.6% storage tax measured at backfill on 6 GB
   wiki — kept flat, no further compaction. See
   `docs/concept-relations-design.md` for the storage choice rationale.
6. **Stem-aware token matching** — possessive / plural collapse
   (`superman's → supermans → superman`).
7. **Per-source context cap** — `max_context_chars / top_k`. Prevents
   one huge doc from monopolizing the budget.
8. **Wikitext base prose** runs on assembled context BEFORE the LLM.
9. **Template-phrase stopwords** (`_FTS5_STOPWORDS` and
   `_TITLE_STOPWORDS` must stay in sync) — strips `tell show describe
   explain summarize say give list find make please all there know
   everything anything something` so "tell me all there is to know
   about X" doesn't dilute query tokens.

`--retrieval-keywords` (CLI: `K="..."`) lets an operator augment
retrieval-side tokens without changing what the LLM sees as its
question. Provenance gap on this is tracked in
[Ticket #000001](docs/tickets/ticket-000001-retrieval-keywords-audit-gap.md).

## Hot path / gotchas

- **Hand-rolled wikitext parser** (`arborist/sources/wikipedia.py`):
  char-position state machine, escape-aware, 4× faster than char-
  by-char loops via `str.find` + slicing. cProfile any change.
- **`PRAGMA synchronous=NORMAL`** per-connection in `store.connect()`.
  Safe under WAL. Don't downgrade without measured reason (~5× cost).
- **HTML source has optional deps**: `pip install '.[html]'` for
  `selectolax`. CLI surfaces `--source html` only if import succeeds.
- **Background ingest/distill processes**: stdout is buffered. Use
  `export PYTHONUNBUFFERED=1` or `python -u`.
- **Disk pressure**: full cur ~2 GB; full old ~5–8 GB. `df -h` first.

## Operational rules

- I propose, fox decides. Unsure = ask. Can't ask = stop.
- No autonomous destructive ops (`clean-data`, `clean-db`, force-push,
  DB drops) without explicit instruction.
- **Never add `Co-Authored-By` or "Generated with Claude" lines to
  commits.** Code speaks for itself.
- **Python only in arborist.** No Rust, C, JS, or other languages
  inside this repo. arborist is the source-of-truth implementation;
  forks and downstream clients/servers in any language follow our
  schemas, canonical encodings, and audit protocols. Optional
  toolchains for ZK/world-model/etc. live in sibling repos
  (`arborist-zk-bench`, `arborist-world`, etc.) so a fresh checkout
  needs only `python3.12 + venv + sqlite3`.
- **Always `export PYTHONUNBUFFERED=1`** for long-running processes.
- Fail-closed. Cleanup crew, not demolition.
- DRY in context — single source of truth, no sprawl.
- Never say "AI" — always say "machine learning."
- Prefer "defect" over "bug."
- Check `robots.txt` before any web fetch the user didn't authorize.

## Five-step algorithm

When proposing or evaluating change, walk these in sequence. Skipping
a step makes the next ones expensive and the system worse.

1. **Make the requirements less dumb.** Every requirement gets a
   person's name, not a department. If you can't name who asked or
   which defect closed, the requirement is suspect.
2. **Delete the part or the process.** If you aren't putting back at
   least 10% of what you delete, you aren't deleting hard enough.
   Verifier-stays-binary and "no soft signals in hard chain" are
   deletion-first guardrails.
3. **Simplify and optimize.** Only after 1 + 2. Don't optimize a
   process that shouldn't exist.
4. **Accelerate cycle time.** Only after simplifying. "Don't dig the
   grave faster."
5. **Automate.** Last, not first. Hand-rolled before scripted,
   scripted before declarative, declarative before generated.

When in doubt, ask "have we tried deleting it?" before reaching for
steps 3-5.

## Bench-maxing — measure deltas, not opinions

Full discipline + worked examples in `docs/bench-maxing.md`. Headlines:

- Bench before AND after every change (n=3, signal under 5pp is noise).
- Avoid negation in prompts (Hermes-3-8B inverts under attention).
- Bench is the scoreboard; live fixtures are the gates.
- Self-heal beats retry (preserve partial output, never fabricate).
- Honest verdicts beat optimistic ones (false-positive STRICT is
  corruption).
- Old maps vs runtime maps — when the model and runtime disagree,
  the runtime wins. Pointer IDs, runtime-interpolated spans, evidence
  maps, policy hashes, hard verifier checks all move authority OUT
  of the model's prior and INTO runtime artifacts.

## Docs index

North-star:

- `docs/seven-point-program.md` — the architectural directive distilled
  2026-05-01. Every new ticket / feature / prompt edit walks past this.
  Bench harness reports per-mode directive coverage.

Architecture / ongoing work:

- `docs/cti-architecture.md` — CTI Clause Tree Intelligence.
- `docs/mesh.md`, `docs/mesh-deploy.md` — mesh wire + deploy.
- `docs/benchmarks.md` — orientation: harnesses, fixtures,
  signal floor, make targets, bench-row schema, addenda index.
  Read first when running a bench.
- `docs/qa-modes-bench.md` — bench journal (rolling addenda).
  Headlines + cross-references to per-ticket bench data.
- `docs/qa-modes-bench-2026-04-30.md` — JSON-mode hardening journey
  (prior bench reference).
- `docs/bench-maxing.md` — bench discipline (5pp signal floor etc).
- `docs/bench-emergent-design.md` — random-word stress-test design.
- `docs/verifier-semantic-gap-design.md` — soft-signal NLI proposal.
- `docs/self-reference-design.md`,
  `docs/self-reference-design.md` — recursive distillation.

Tickets: `docs/TICKETS.md` is the authoritative index with `Next
ID`. Closed tickets stay in place as the design log. New tickets
bump `Next ID` atomically. **Close tickets when the work lands —
flip Status to `closed · landed in commit <sha>` (or
`closed · YYYY-MM-DD`) in the ticket file AND in the index row, in
the same commit as the implementation.** An open ticket whose code
already shipped is a stale map.

## Orientation protocol

```bash
date -u
pwd
git log --oneline -5
git status
make test
make chain-check-shards   # 0 per shard = intact
.venv/bin/arborist --shards-dir ~/.arborist/shards stats
.venv/bin/arborist --shards-dir ~/.arborist/shards analyze --gravity-top 5
```

Then ask fox what the mission is.