# Agent Blackops — arborist

This repo is operated by **agent blackops** for fox/timehexon on the
unsandbox / unturf / permacomputer platform.

Identity shard: `~/git/unsandbox.com/blackops/BLACKOPS.md`.

## What arborist is

A content-addressed, Merkle-committed document store. Implements the
runtime spec from **Merkle Providence Reverse RAG** (April 2026
whitepaper) scaled up to the **Merkle-AGI v9.8** admissibility ledger.

Three layers stacked on one SQLite file:

1. **Surface** — ingested documents (Wikipedia dumps, HTML pages,
   anything with a URI). Chunked, Merkle-rooted, FTS5-indexed.
2. **Core** — distilled documents Merkle-bound back to surfaces via
   per-chunk inclusion proofs in `derivations.proof_blob`. Recursive.
3. **Providence cache** — Q&A records keyed on the v9.8 8-dim
   invariant. Each record carries `audit_mode` (STRICT / HYBRID /
   UNGROUNDED) decided by the verifier in `arborist/qa/verify.py`.
   For claim-lattice modes the renderer maps that token to a four-
   rung ladder (POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED;
   ENTAILMENT-VERIFIED reserved); UNGROUNDED below all rungs. The
   schema column stays unchanged — programmatic callers see the
   trichotomy, human-facing surfaces see the ladder. See
   `arborist/cli.py:_render_audit_label`.

## Source papers

- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst`
  — canonical whitepaper (rst → PDF). Edit here, not the PDF.
- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV
  encoding A1, public quantization A2, collision-resistant hash A3,
  theorems T1–T5).
- `~/git/proxy.unturf.com/pkg/verified/merkle.go` — fox's existing
  Go merkle. Arborist Python ports its conventions exactly.
- `~/git/proxy.unturf.com/docs/merkle-tree.md` — convention reference.

## Architecture

```
arborist/
├── merkle.py        # Python port of proxy.unturf.com Go conventions
├── store.py         # v9.8 SQLite schema, audit-chain helpers
├── document.py      # Document, Edge, Chunker (TokenChunker default)
├── source.py        # Source ABC: iter_documents()
├── ingest.py        # batched normalize → chunk → merkle → upsert
├── evict.py         # hot↔cold + rehydrate (v9.8 falsification on drift)
├── compress.py      # chunk pack/unpack (zstd dictionary trained on corpus)
├── snapshot.py      # snapshot.db creation + load
├── journal.py       # NDJSON unfirehose journal sink
├── wikitext.py      # to_base(): wikitext → plain prose (BASE_VERSION-pinned)
├── search/          # SearchBackend ABC + AuditMode + FTS5
├── sources/         # one file per corpus (wikipedia, html_page,
│                   # claim_pack, textbook_tex, grok, vcs, …)
├── distill/         # surface → core distillation (tfidf, first_sentence)
├── qa/              # Q&A: 8-dim cache_key + Merkle-bound answers
│   ├── client.py    #   ChatClient + StubClient + OpenAICompat
│   ├── keys.py      #   cache_key, question_hash (pure functions)
│   ├── verify.py    #   layered verifier + claim-lattice verifiers (G0)
│   ├── evidence.py  #   EvidenceObject + spotlight excerpt
│   ├── inspect.py   #   read-only sidecars (deflection, title-relevance)
│   ├── dag.py       #   per-run Merkle-DAG (7/8 quote · 9/10 CTI · 3 reject)
│   ├── quantifier.py        # broad-quantifier classifier (#000008 P1)
│   ├── model_profiles.py    # per-model claim-cap profiles (#000008 P2)
│   ├── quantifier_reminder.py # broad-query reminder text (#000008 P3)
│   ├── canonical_cache.py   # canonical-projection persistence (#000027)
│   ├── witness.py           # multi-witness fan-out (#000028)
│   ├── warrant_resolver.py  # claim-pack warrant chain resolution (#000031)
│   ├── crosslang.py         # cross-lang guard: signal + es stoppack (#000001 §7 P0)
│   ├── mt/                  # Operation Sandwich MT edges (#000056)
│   │                        # opus-mt es/fr/ru↔en + entity_mask
│   └── runner.py    #   ask(): cache → infer → verify → write
├── concepts/        # corpus-derived synonym + rivalry layer (#000018 sib.)
├── pi_star/         # canonical projection π* registry (#000015)
│                    # arithmetic@v1, logic-kernel@v1, algebra-symbolic@v1, …
├── memory/          # MemoryRoot lifelong-learning audit chain (#000017)
├── selfmodel/       # SelfModel snapshot + falsify (#000014)
├── capital/         # 8-form capital ledger (#000020)
├── substrate/       # Merkle-AGI substrate primitives (paper-spec'd)
│   ├── anchor_prg.py       #   φ_PRG HMAC-SHA-512 (v7 §9.10; #000035)
│   ├── fork_score.py       #   ScoredFork decision fn (v8; #000012 P1a)
│   └── weights.py          #   ForkScore weight set
├── world/           # v7-W spatial-temporal substrate reservation (#000013;
│                   # namespace stub; future kernels under world/pi_star/,
│                   # world/frontier/, world/adapters/)
├── mesh/            # mesh wire format + group-key state machine
└── cli.py           # ingest / search / verify / stats / distill /
                     # evict / rehydrate / ask / providence / emergent /
                     # reclassify / inspect / analyze / canon / sweep /
                     # warrant-resolve / alias / capital / selfmodel
```

**Dir naming convention.** Topic-named, never version-prefixed. The
substrate-paper version (v7 plastic-training, v8 selection/consensus,
v9 falsification controller, …) and the live SQLite schema version
(v9.8) are two unrelated numbering schemes that share decimals;
version-prefixed dirs (``arborist/v7/``, ``arborist/v8/``) were tried
2026-05-10 and retired the same day because readers asked "is this
schema-v7 or paper-v7?". Substrate-paper-spec'd primitives now live
under ``arborist/substrate/``; topic dirs (``capital/``, ``memory/``,
``selfmodel/``, ``concepts/``, ``pi_star/``) hold cross-version
mechanisms.

## Build, test, run

Every workflow is a `make` target. Bare python is not the user
interface. See the Makefile for the full list.

```
make bootstrap           # venv + editable install with [dev] extras
make test                # pytest -q
make all                 # bootstrap + fetch-cur + ingest-cur + verify + stats
make verify-shards       # round-trip Merkle proofs (cross-shard sample)
make analyze-shards      # cross-shard compression + audit integrity
make chain-check-shards  # audit-chain break count per shard (0 = intact)
make query Q="..." [JSON=1 BURN=1 K="extra retrieval keywords" ANSWER_MODE=…]
make bench-qa            # QA-quality sweep (live LLM)

# Textbook surface-ingest layer (#000031 — PD/open-licensed
# math/logic/CS textbooks for warrant promotion):
make textbooks-summary   # license + URL counts per manifest entry
make crawl-textbooks     # BFS-crawl every entry with crawl_url
                         #   → ~/.arborist/crawl/textbook_<id>.db
make textbooks-tex       # PG LaTeX-source ingest (Hilbert, Boole)
make textbook ID=<id>    # ingest one textbook by manifest id
                         # (idempotent at DB layer)
make textbooks-base-knowledge  # bulk: Cantor + De Morgan + Russell
                               # IMP + Judson (the four 2026-05-09
                               # base-knowledge additions)

# Per-textbook convenience targets (one per active manifest id):
make textbook-bogart textbook-keller-trotter textbook-levin
make textbook-aristotle-prior textbook-aristotle-posterior
make textbook-newton textbook-morin
make textbook-judson textbook-cantor textbook-demorgan
make textbook-russell-imp textbook-russell-pom
make textbook-laplace textbook-pm textbook-grinstead-snell
make textbook-hilbert textbook-boole textbook-peano textbook-dedekind
make textbook-plfa textbook-sf-lf

# Claim-pack warrant-chain resolver (#000031 Phase 2 + 2.5):
make sweep TARGET=warrants  # warrant-resolve --use-aliases --write
arborist warrant-resolve --use-aliases --write
                         # → 92 / 92 (100%) coverage as of 2026-05-10:
                         # 18 textbook substrates + 74 fox-decided
                         # citation-aliases + 13 term-aliases.
                         # (alias counts grow as fox adds substitutions
                         # — `arborist alias citation list | jq length`
                         # for live count). Per-pillar 13/13 · 10/10 ·
                         # 13/13 · 18/18 · 5/5 · 5/5 · 14/14 · 14/14.
arborist alias citation list  # see substitutions
arborist alias term list      # see vocabulary aliases
```

**Hygiene after any state-changing op** (rebuild, reclassify, hash
bump, mass falsify): `make chain-check-shards` first (every shard
should report `0`), then `make analyze-shards` for the spectrum +
chain audit. Chain breaks are the loudest possible signal.

## Schema invariants (do not break)

- **v9.8 8-dim cache_key** (+ optional legible 9th): `source_root |
  question_hash | model_profile_hash | conversation_hash |
  governance_policy_hash | schema_version | canonicalization_version |
  chunking_version`. Bumping any one invalidates prior records on
  lookup. `cache_key()` also accepts an **optional 9th dimension**,
  `verifier_policy_hash` (`keys.py`), for *audit legibility* — it does
  NOT add correctness coverage, because the verifier-policy fields are
  a subset of the policy dict and so already fold into
  `governance_policy_hash` (a verifier-rule change already changes the
  cache_key today; the 9th just makes "did the *verifier* rules
  change?" answerable from one hash diff). 8-dim is the default write
  form; the 9-dim form is opt-in. Mandatory-vs-legible is #000058.
- **`falsification_state ∈ {live, failed, stale, quarantined}`**.
  Cache lookups filter on `state='live'`. Drift → `stale`.
- **Audit chain**: every state-changing op writes one row in
  `audit_events` with `event_hash = sha256(prev || canonical(body))`.
  Verified by `make chain-check-shards`. Use `arborist.store.append_audit`
  — never insert into `audit_events` directly.
- **Cores never evict.** `evict_to_cold` only touches `kind='surface'`.
- **Idempotent re-ingest.** Same content → same `document_root` →
  no-op insert. Same URI + different content → new doc + `supersedes`
  edge (lossless history).

## Conventions (do not silently change)

Each rule below has full rationale in the named source file. Don't
revert without reading why. When in doubt, walk the
[Five-step algorithm](#five-step-algorithm) first.

- **Merkle conventions**: non-commutative `HashCombine` prefix `0x03`,
  leaves `0x00`, odd-element rule = self-duplicate (NOT zero-pad).
  `MerkleProof.siblings` carries `is_left` flag — never sort lexically.
  See `arborist/merkle.py`.
- **Versioned defaults**: `tok-512-v1` (chunker), `norm-v1`
  (canonicalization), `v9.8.0` (schema), `wikitext-base-v1` (prose).
  Changing any default stales every prior cache record. Add a new
  `name` instead.
- **`question_hash` is dedup-mode-aware** (`strict` |
  `equivalence_class`); folds into `governance_policy_hash`. JIT
  `fidelity` parameter on `query()`/`ask()` decouples lookup tolerance
  from write policy. See `arborist/qa/keys.py`.
- **`audit_mode` is decided by the verifier, never asserted.** Four
  layered strategies tried in order, first to find evidence
  classifies: **quote** (sequential pair-matching, NOT regex —
  prevents phantom inter-pair captures), **span** (verbatim line
  match), **entity** (proximity-clustered proper nouns;
  `entity_policy ∈ {strict, hybrid, drop, proximity}`), **paraphrase**
  (token-coverage, prose-shaped only; `verifier_method='paraphrase'`).
  Trichotomy: STRICT = every unit verifies, HYBRID = mixed,
  UNGROUNDED = none. Never overclaim. See `arborist/qa/verify.py`.
- **Verifier stays binary; falsifications carry soft signal.** No
  per-quote diagnosis fields on hard verifier output. Sidecars
  (`arborist.qa.inspect.diagnose_*`, `arborist inspect --cache-key X`)
  classify unverified spans, deflection, title-relevance — never
  write to `providence_cache` or `audit_events`.
- **Trailing-citation strip**: `_strip_trailing_citation` peels one
  trailing parenthetical at end-of-span (gated on a citation cue or
  URL) before substring testing. See `arborist/qa/verify.py`.
- **Soft hash vs hard hash**: hard = SHA-256 (commitments, proofs,
  cache_key); soft = embeddings/TF-IDF/similarity (training, ranking,
  distillation). Soft never enters proof path.
- **Cross-language = the sandwich, MT on the edges only**: translate
  query in (retrieval + LLM prompt) → English answer → the
  **byte-identical verifier** grounds English-vs-English → translate
  the verified answer out as **display-only** (`display_*`, banner-
  labelled, zero grounding — the `_render_audit_label` projection
  discipline). Translation NEVER re-enters the verifier (that's the
  #000049 model-in-proof-path cage). Invariants: `question_hash` =
  the user's original question (untranslated); `verifier_policy_hash`
  unchanged; MT engine identity binds into `RetrievalPlan.mt_*` (run-
  DAG), NOT `governance_policy_hash` (the *flag* moves it like any
  policy flag — correct cache partition — but that hash covers the
  whole policy, `keys.py:182`; don't mistake "no new governance
  field" for "governance untouched"). Engine = local `[mt]` opus-mt,
  hash-pinned, never Hermes-for-translation, never an API. Default
  OFF (`crosslang_guard_enabled` P0; `crosslang_translate_enabled`
  Sandwich; `crosslang_entity_mask` **default-OFF** — measured
  net-negative, kept only behind the flag). Measured net win over
  the real "nothing" baseline (raw es → noise/UNGROUNDED): es ≈0 % →
  71 % grounded; the −14 pp vs *English* is the cost of a new
  capability, not a regression (compare to no-cross-lang, never to
  native English). The recall lever (entity-preservation) is **still
  open** — `entity_mask` v1 failed at bench scale; corpus-title
  anchoring is the untried idea. See `arborist/qa/crosslang.py`,
  `arborist/qa/mt/`, #000001 §7, #000056 §9.
- **Three answer modes**: `policy["answer_mode"] ∈ {"quote",
  "claim_lattice_pointer", "claim_lattice"}`, default `"quote"`.
  Bench 2026-05-02T15:07Z on Hermes-3-8B (post-Sprint-1b/2, n=3
  × 71 questions, sample-shuffled @ c=4): quote 0.54 strict-rate,
  pointer 0.20, JSON 0.42. Quote leads on raw lexical grounding;
  JSON leads among lattice modes. Per-mode peak buckets: quote
  8-16KB (0.58), pointer 16-32KB (0.20), JSON 32-64KB (0.48) —
  these drive `max_context_chars_by_mode`. 99% directive coverage
  (D2/D3/D4/D6/D7) on lattice modes. Both lattice modes share
  `verifier_method="claim_lattice"`; `answer_mode` on the run-DAG
  + `json_fixups` disambiguate. Each mode folds into
  `governance_policy_hash`. See `arborist/qa/verify.py`,
  `docs/qa-modes-bench.md`.
- **Four-rung ladder (lattice-mode display layer)**: POINTER-LINKED
  → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → (ENTAILMENT-VERIFIED
  reserved); UNGROUNDED below. Each rung names a strictly stronger
  property the lexical verifier could confirm. WARRANT_MISSING drops
  to POINTER-LINKED; soft-demote violations (LAZY_ANCHOR_DEMOTED,
  POINTER_OVERFLOW_TRIMMED, TOO_MANY_CLAIMS, BARE_NAME_CLAIM,
  TITLE_MISMATCH) cap at ANCHOR-WARRANTED. Quote / span / entity /
  paraphrase modes keep their original audit_mode tokens (those
  verify against pinned spans, not synthesis). Pure render-layer —
  cache_key, governance_policy_hash, & all programmatic callers see
  the underlying audit_mode unchanged. See
  `arborist/cli.py:_render_audit_label`.
- **Claim-lattice-pointer mode (G0 / CTI)**: runtime mints
  `pointer_id` (E1, E2, … — what the model sees) and content-addressed
  `evidence_id` (what the cache & run-DAG store). Renderer interpolates
  literal source spans via `_spotlight_excerpt`. Synthetic-elision-by-
  construction-impossible — model never types the quote string. 9-stage
  run-DAG. See `arborist/qa/evidence.py`, `docs/cti-architecture.md`.
- **Claim-count ceiling (`TOO_MANY_CLAIMS`)**: default 12 per answer.
  Catches "tell me all there is to know about X" runaway. Demotes
  STRICT → HYBRID without truncating. Folds into
  `governance_policy_hash`. See `arborist/qa/verify.py`.
- **Broad-quantifier preflight guard (Ticket #000008)**: pure lexical
  classifier (`arborist/qa/quantifier.py`) maps a question onto a
  10-rung intensity ladder (ABSENT → SINGULAR → … → ALL →
  COMPREHENSIVE). Per-model `arborist/qa/model_profiles.py` PROFILES
  dict picks a per-call claim-cap from the (intensity, model) pair;
  `arborist/qa/quantifier_reminder.py` synthesizes a one-line user-turn
  reminder for broad questions. Six-level disable hierarchy
  (per-test, per-call CLI, per-phase policy, per-mode, per-model,
  master-via-governance-hash). 7 policy fields fold into
  `governance_policy_hash` so flipping any of them invalidates prior
  records. Defaults preserve dry-run discipline:
  `quantifier_guard_apply_caps=False`,
  `quantifier_reminder_enabled=False`,
  `quantifier_reject_broad=False`. CLI flags on `arborist query`:
  `--no-quantifier-guard`, `--allow-broad`, `--reject-broad`,
  `--apply-quantifier-caps`. Bench A/B (2026-05-03, n=3 × 9 broad
  questions × 3 modes): reminder default-on supported (FORMAT_COLLAPSED
  −100%, NO_EVIDENCE_POINTER −33%, JSON UNGROUNDED −22pp); cap default-on
  for JSON only (+14pp STRICT-rate, no gain on pointer); cap+reminder
  best on pointer mean ratio (0.684) but not strictly best on JSON
  STRICT-rate. See `docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md`
  §12 for the four-cell A/B data.
- **Wikitext base prose**: `arborist/wikitext.py:to_base()` runs
  before the LLM call AND inside `verify_quotes` so model and verifier
  see the same prose. Optional dep — graceful fallback when
  `mwparserfromhell` is missing.
- **Deflection sidecar**: `diagnose_deflection(question, answer)`
  detects topic-shift via subject-anchor heuristic (LAST content
  token in question must appear in answer). Suppressed for date /
  count / cause shapes ("when", "why", "how many"). See
  `arborist/qa/inspect.py`.
- **Coherence sidecar** (#000052 §3.1): `diagnose_coherence(answer)`
  in `arborist/qa/inspect.py` — per-sentence lexical shape check,
  **no model**. Emits `kind ∈ {phrase_component_reuse, circular,
  vacuous, ok, empty}`: `circular` = subject content-tokens ⊆
  predicate's and the predicate leads with a subject token ("X is
  X"); `phrase_component_reuse` = the subject quotes a phrase & the
  predicate reuses one of that phrase's own tokens as a bare
  `the/a/an <token>` referent (the 2026-05-12 "the phrase 'Zionist
  entity' is used as the entity" field case — a token collision the
  verifier + deflection + title-relevance all pass and NLI returns
  *neutral* on); `vacuous` = predicate is only placeholder hypernyms
  + filler ("X is a thing"). Surfaced in `inspect_cache_key` + the
  `arborist inspect` human view (`· incoherent: <kind>`). Advisory
  sidecar — **never** writes `providence_cache` / `audit_events` /
  `run_dag_root`; a demote-only verifier hook is possible but
  deliberately not wired (would fold into `governance_policy_hash`).
- **Title-relevance hard check (Rule 8)**: `_claim_title_overlap`
  in `arborist/qa/verify.py`. For each claim that resolved, at least
  one cited evidence's source title must share ≥1 stemmed content
  token with the claim text. When NO cited title overlaps, record
  a `TITLE_MISMATCH` violation & demote `STRICT → HYBRID`. Catches
  retrieval-driven hallucinations where the cited chunk's SOURCE
  is structurally unrelated to the claim's subject (2026-05-02
  spin-glass case: claim about spin glass cited to a chunk from
  *Quantum chromodynamics*; span had incidental physics-vocab
  overlap, but the source title shared zero stems with the claim).
  Renderer surfaces a `· title mismatch` tail on the audit-line
  label alongside `· warrant missing`.
- **Title-relevance sidecar (legacy diagnostic)**:
  `diagnose_title_relevance(claim, cited_titles)` in
  `arborist.qa.inspect` returns the same signal in dict form for
  per-cache-key inspection. Sidecar; never enters proof path.
  Pre-dates the Rule 8 promotion (2026-05-02).

## Live endpoints

- LLM: `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B-FP8-
  Dynamic on vLLM, 82K ctx, no auth). `uncloseai.com` is marketing
  only. Override via `--endpoint` or `ARBORIST_LLM_ENDPOINT`.
- Wikipedia dumps: `https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`.
  `robots.txt` returned 404 → no rules.

### Budget discipline — Hermes / Qwen first, Opus deferred

2026-05-19: the huge-N `#000057` control-arm sweep
(`f63b00d` → `9dc02e4`) burned our Opus quota. Until fox explicitly
re-adds it, sweep + `bench-qa` LLM calls route to **Hermes-3-8B**
(`hermes.ai.unturf.com/v1`) and **Qwen via uncloseai.com**; Opus drops
out of control-arm grids. This is a budget rule, not a quality claim —
bench results already on disk under `bench/results/` stay valid; just
don't expand them with new Opus tokens without a go. Translation at
scale is still NOT a Hermes/Qwen 8B job (sandwich MT uses local pinned
`opus-mt`, see crosslang rule). Grok candidacy noted from ajax synthetic-
data-distillation benchmarks.

<!-- Grok API rejoins the grid once fox approves credit-card spend.
     When unflagged, add a `grok-*` arm to `bench/control_sweep.py`
     and document model id + endpoint here. -->

### Per-call model selection (general agent work, not sweep arms)

Within a 4-hour Max-plan window, haiku/sonnet/opus draw equally from
quota — prefer the largest model that fits (fewer retries = better
quota efficiency). During **paid overflow** (beyond Max), pay-per-token
applies: start haiku, escalate sonnet then opus only on failure.
Hermes/Qwen via `uncloseai.com` serve classification & code-adjacent
work, **never human-language translation** (8B quality too low; that's
the sandwich-MT `opus-mt` edge).

## Retrieval pipeline (`arborist/qa/query.py`)

Multi-stage. Each stage exists because something earlier wasn't
enough; revert at your peril. Order:

1. **Four parallel FTS5 search routes per shard, merged** — body BM25,
   title-LIKE, core-keyword (TF-IDF cores), and **phrase-pattern**
   (verbatim n=5/n=6 sequences from the question). Phrase route
   closes the allusion gap (Orwell case: "always been at war" verbatim
   matches the 1984 article whose title shares zero tokens with the
   query).
2. **Body-coverage `sqrt` rerank** — counters BM25's short-doc bias.
3. **Title-token boost** — `boost × overlap` on title-token-matching hits.
4. **`_filter_by_title_relevance` — four accept paths**: title-token
   overlap, TF-IDF core match, body density, **phrase match**
   (accept-path 4 lets phrase-route hits with no title overlap survive).
5. **Rivalry exclusion + synonym expansion** (`arborist/concepts/`) —
   Intel-titled docs drop from AMD queries; reverse holds. Backed
   by the per-shard `concept_relations` SQLite table (corpus-derived,
   not hand-curated). 1.6% storage tax measured at backfill on 6 GB
   wiki — kept flat, no further compaction. (Storage choice rationale
   lived in `docs/concept-relations-design.md` prior to its deletion
   in `bb6a89c` — the design is now documented inline in
   `arborist/concepts/extract.py` + the data is self-describing.)
6. **Stem-aware token matching** — possessive / plural collapse
   (`superman's → supermans → superman`).
7. **Per-source context cap** — `max_context_chars / top_k`. Prevents
   one huge doc from monopolizing the budget.
8. **Wikitext base prose** runs on assembled context BEFORE the LLM.
9. **Template-phrase stopwords** (`_FTS5_STOPWORDS` and
   `_TITLE_STOPWORDS` must stay in sync) — strips `tell show describe
   explain summarize say give list find make please all there know
   everything anything something` so "tell me all there is to know
   about X" doesn't dilute query tokens.

`--retrieval-keywords` (CLI: `K="..."`) lets an operator augment
retrieval-side tokens without changing what the LLM sees as its
question. Provenance gap on this is tracked in
[Ticket #000001](docs/tickets/ticket-000001-retrieval-keywords-audit-gap.md).

## Hot path / gotchas

- **Hand-rolled wikitext parser** (`arborist/sources/wikipedia.py`):
  char-position state machine, escape-aware, 4× faster than char-
  by-char loops via `str.find` + slicing. cProfile any change.
- **`PRAGMA synchronous=NORMAL`** per-connection in `store.connect()`.
  Safe under WAL. Don't downgrade without measured reason (~5× cost).
- **HTML source has optional deps**: `pip install '.[html]'` for
  `selectolax`. CLI surfaces `--source html` only if import succeeds.
- **Background ingest/distill processes**: stdout is buffered. Use
  `export PYTHONUNBUFFERED=1` or `python -u`.
- **Disk pressure**: full cur ~2 GB; full old ~5–8 GB. `df -h` first.

## Operational rules

- I propose, fox decides. Unsure = ask. Can't ask = stop.
- **Never offer "stop here / take a break / fresh eyes tomorrow" as a
  next-action option.** Fox keeps going. When proposing options at a
  decision point, list the actual moves (high-payoff vs low-risk
  cleanup, etc.); don't pad the menu with a no-op "we're done for
  today" choice. If a task is genuinely complete, say so flat —
  don't dress it up as a third option.
- No autonomous destructive ops (`clean-data`, `clean-db`, force-push,
  DB drops) without explicit instruction.
- **Never add `Co-Authored-By` or "Generated with Claude" lines to
  commits.** Code speaks for itself.
- **Python only in arborist.** No Rust, C, JS, or other languages
  inside this repo. arborist is the source-of-truth implementation;
  forks and downstream clients/servers in any language follow our
  schemas, canonical encodings, and audit protocols. Optional
  toolchains for ZK/world-model/etc. live in sibling repos
  (`arborist-zk-bench`, `arborist-world`, etc.) so a fresh checkout
  needs only `python3.12 + venv + sqlite3`.
- **Always `export PYTHONUNBUFFERED=1`** for long-running processes.
- **AUTOCOUNT discipline on numeric claims in `docs/`.** Any
  numeric claim added to a doc (test count, fixture-row count,
  SQLite row count, filtered-row count) should be wrapped in an
  AUTOCOUNT tag at write time so future drift fires the
  regression test. Format:
  ``<!--AUTOCOUNT:metric:path-->N<!--/AUTOCOUNT-->`` where
  ``metric`` is one of ``tests`` / ``fixture-rows`` / ``db-rows``
  / ``db-where``. Tags are invisible in rendered markdown
  (GitHub strips HTML comments). Tags inside ```` ``` ```` fenced
  code blocks are auto-skipped (illustrative examples, not live
  claims). Closed-ticket "N tests pass" snapshots stay UNtagged
  (they're point-in-time historical records). Full discipline +
  4-metric reference + future-metric recipe in
  ``docs/tickets/ticket-000044-autocount-doc-drift-discipline.md``;
  harness at ``tests/test_doc_counts.py``.
- Fail-closed. Cleanup crew, not demolition.
- DRY in context — single source of truth, no sprawl.
- Never say "AI" — always say "machine learning."
- Prefer "defect" over "bug."
- Check `robots.txt` before any web fetch the user didn't authorize.

## Five-step algorithm

When proposing or evaluating change, walk these in sequence. Skipping
a step makes the next ones expensive and the system worse.

1. **Make the requirements less dumb.** Every requirement gets a
   person's name, not a department. If you can't name who asked or
   which defect closed, the requirement is suspect.
2. **Delete the part or the process.** If you aren't putting back at
   least 10% of what you delete, you aren't deleting hard enough.
   Verifier-stays-binary and "no soft signals in hard chain" are
   deletion-first guardrails.
3. **Simplify and optimize.** Only after 1 + 2. Don't optimize a
   process that shouldn't exist.
4. **Accelerate cycle time.** Only after simplifying. "Don't dig the
   grave faster."
5. **Automate.** Last, not first. Hand-rolled before scripted,
   scripted before declarative, declarative before generated.

When in doubt, ask "have we tried deleting it?" before reaching for
steps 3-5.

## Bench-maxing — measure deltas, not opinions

Full discipline + worked examples in `docs/bench-maxing.md`. Headlines:

- Bench before AND after every change (n=3, signal under 5pp is noise).
- **When a lever's failure class is sub-noise-floor, fix the
  instrument, not just the lever.** The curated n=3 audit_mode bench
  can't resolve a class that is ≤3-5 of 75 q (four 2026-05-18
  hypotheses died there — round-trip, entity-mask, disambiguation).
  Mine **ground-truth-carrying** questions from corpus titles
  (`bench/mine_questions.py`) and grade by **deterministic retrieval
  recall@k** via `query --dry-run` (`bench/recall_at_k.py`): no LLM,
  no verifier, no n=3 noise, no 5pp floor, scalable to the corpus.
  Resolves a single lever to ±1 question (worked example: numeral-
  fold recall 55→75%, `a3ac653`). Caveats: measures *retrieval
  surfacing* — necessary-not-sufficient for STRICT; mined fixtures
  are answerable-by-construction so they **complement, never
  replace**, the curated adversarial set (the verifier-honesty/trap
  gate). Conflating the two is itself a bench-maxing error.
- **Report recall@1/@3/@k, not one lenient k — a coarse k hides a
  rank-only lift.** `recall_at_k.py` returns the target's rank, so
  recall at every k is free from one retrieval. Measured 2026-05-18:
  accent-fold looked inert at recall@8 (95→98, noise) but the OFF
  baseline was recall@1 55% vs @8 95% — a too-lenient k flattered
  it to a near-ceiling and nearly got a real lever wrongly reverted.
  recall@1/@3 is the resolution that matters (primary-source
  selection keys on rank, not mere top-k presence). Prevalence ≠
  miss-rate either: the corpus survey ranked accent #1 at 8.1% of
  *titles*, but the measured miss-rate (recall@1) is what decides —
  measure headroom, never rank candidates by raw prevalence.
- **Fan out independent measurements; serial-by-caution is halting
  in disguise.** Mined recall is deterministic per query (read-only,
  no LLM, no shared state) — concurrency cannot change which sources
  rank; the only risk is the per-probe timeout, and dry-run
  retrieval (~2 s) has huge margin under the 120 s cap. Run the whole
  fold-search backlog (accent / hyphen / honorific / …) as parallel
  background sweeps; only same-fixture A/B that toggles `query.py`
  state needs git-stash serialisation.
- Avoid negation in prompts (Hermes-3-8B inverts under attention).
- Bench is the scoreboard; live fixtures are the gates.
- Self-heal beats retry (preserve partial output, never fabricate).
- Honest verdicts beat optimistic ones (false-positive STRICT is
  corruption).
- Old maps vs runtime maps — when the model and runtime disagree,
  the runtime wins. Pointer IDs, runtime-interpolated spans, evidence
  maps, policy hashes, hard verifier checks all move authority OUT
  of the model's prior and INTO runtime artifacts.
- **When researching a config/model decision: sweep wide, sweep
  deep, on real-traffic-sized samples — don't guess, and don't trust
  small denominators.** Three times running (ticket #000049 §7
  #18→#24) the answer flipped: a clean synthetic eval said one thing,
  bench-qa pipeline output said another; the *default* config said one
  thing, a `{model × hyperparams}` grid sweep said another; an n=1 FP
  sample said "this config passes", an n=3 FP sample said "that one
  collapses — a *different* config passes". So: (1) gate numbers come
  from bench-qa pipeline output, never contrived fixtures; (2) the
  sweep must be wide enough — multiple checkpoints, the full
  hyperparam grid; there is no "bigger model is better" law, the
  *specific checkpoint* and the *score-shape* (e.g. a single-threshold
  margin vs a two-threshold rule) dominate — wide enough to include
  the config that survives; (3) every gate number is provisional until
  the denominators are big enough — bump `BENCH_QA_N` and re-confirm
  before promoting anything. The GPU box (`ai`, the 4090 — `make
  bootstrap-nli-only`, then `ARBORIST_NLI_DEVICE` auto-detects cuda)
  makes this cheap: a 7-model × full-grid × ~300-record sweep is
  ~3.5 min. **That electricity is well spent — burn it; a guessed
  config that ships is far more expensive than a sweep that doesn't.**
  Heavy off-device passes (NLI sweeps, embedding backfills) run on the
  GPU producer, never in arborist's `python+sqlite3` core (cf. the
  #000051 vecpack pattern).

## Docs index

North-star:

- `docs/seven-point-program.md` — the architectural directive distilled
  2026-05-01. Every new ticket / feature / prompt edit walks past this.
  Bench harness reports per-mode directive coverage.
- `docs/warrant-substrate-cookbook.md` — per-pillar map of the 18
  open-licensed textbook substrates that back every claim-pack
  record. Ingest patterns (HTML / textbook_tex / PDF→localhost),
  alias discipline, cascade tuning, honest tier breakdown.
  Read first when extending substrate coverage.

Architecture / ongoing work:

- `docs/cti-architecture.md` — CTI Clause Tree Intelligence.
- `docs/mesh.md` — mesh wire format + group-key state machine.
- `docs/benchmarks.md` — orientation: harnesses, fixtures,
  signal floor, make targets, bench-row schema, addenda index.
  Read first when running a bench.
- `docs/qa-modes-bench.md` — bench journal (rolling addenda).
  Headlines + cross-references to per-ticket bench data.
- `docs/bench-maxing.md` — bench discipline (5pp signal floor etc).
- `docs/soft-hash-channel-analysis.md` — #000018 closure; analysis
  of soft-hash covert channel risks under M0/M1/M2 threat models.
- `docs/soft-hash-channel-t3-bound.md` — #000036 closed-form
  per-window budget bound; pairs with t3_bound_calculator.
- `docs/onnx-vendor-capture-immunity.md` — why the model-in-proof-
  path cage (#000049) makes the inference engine (ORT / torch /
  tinygrad) an interchangeable sidecar, never a trust dependency;
  public-domain positioning capital.
- `docs/calculator-test-patterns.md` — sister discipline for
  testing calculator-style code (12 patterns; see #000044 for
  numeric drift discipline).
- `docs/spec-methodology.md` — #000019 specification methodology
  for π* canonical projections.
- `docs/v7w-frontier-catalog.md` — #000013 v7-W spatial-temporal
  frontier catalog (4 ε-frontiers).
- `docs/v8-fork-score.md` — #000012 ForkScore Phase 1a reference.
- `docs/pi-star-composition.md` — π* cross-domain composition
  reference (#000015 deliverable).

Tickets: `docs/TICKETS.md` is the authoritative index with `Next
ID`. Closed tickets stay in place as the design log. New tickets
bump `Next ID` atomically. **Close tickets when the work lands —
flip Status to `closed · landed in commit <sha>` (or
`closed · YYYY-MM-DD`) in the ticket file AND in the index row, in
the same commit as the implementation.** An open ticket whose code
already shipped is a stale map.

**Default: extend an existing ticket. Don't proliferate.** When
follow-up work surfaces during implementation (a consumer-side fix
the new code needs to actually take effect; a bench-gated tuning
step; a small downstream tweak), **add it to the open / most-related
ticket**: reopen `closed` → `in progress` if the prior closure was
premature, extend the scope, list the new sub-items, keep the design
log linear. fox tracks the index by skimming a small set of threads;
spawning #000054.1 / #000055 / #000056 / … for every follow-up turns
the index into a wall of micro-tickets he can't easily keep state on,
and dilutes the design log instead of concentrating it. **Split only
when the new piece needs a *distinct audience*** — specifically, when
it's a self-contained design decision a Dav1d de-novo review (the
external code-review thread that runs against one ticket at a time)
needs to read independently. Architectural inflection points, large
scope changes, or fundamental discipline questions (NLI may touch
`audit_mode`? — #000049 split from #000048) cross that bar. "The
extractor needs a small consumer-side tweak" does not. When in doubt,
ask before opening a new ticket — a two-sentence "fold into #X or
spawn a sibling?" is cheap; an unwanted ticket is friction. Closing
on the implementation commit is good practice **only** when the work
is actually complete end-to-end; pattern-match closure on the
user-visible outcome, not on the commit.

## Orientation protocol

```bash
date -u
pwd
git log --oneline -5
git status
make test
make chain-check-shards   # 0 per shard = intact
.venv/bin/arborist --shards-dir ~/.arborist/shards stats
.venv/bin/arborist --shards-dir ~/.arborist/shards analyze --gravity-top 5
```

Then ask fox what the mission is.
