engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(claude): compress + attribute — split bench-maxing, add docs index

Browse source 
Walked the five-step algorithm on CLAUDE.md itself.

Step 1 — requirements check: every section earned its place via a
specific operator-failure context. Sections that had grown into
prose essays got compressed back to load-bearing rules + pointers.

Step 2 — delete:
  - Bench-maxing prose section (~100 lines) → moved verbatim to
    docs/bench-maxing.md. CLAUDE.md keeps a one-block headline list
    pointing at the doc.
  - Conventions section: each rule tightened to one paragraph max,
    with a `See aborist/qa/verify.py` (or similar) trailing pointer
    so the operator can jump straight to the rationale in code.
    Verbose duplications between conventions and retrieval-pipeline
    sections (e.g. wikitext base prose explained twice) collapsed.
  - Retrieval pipeline: each of the 9 stages now references the
    relevant module (`qa/concepts.py`, `query.py`) instead of
    re-narrating the failure case in prose.
  - Architecture tree: minor trim, removed redundant comments
    where the filename already names the role.

Step 3 — simplify: source papers section dropped one item that was
duplicate (PDF + RST point to same content); kept the canonical
source.

Steps 4 + 5 — n/a (this is a doc, not a process).

New section: Docs index. Lists every architectural / design doc
in docs/ plus a ticket sub-list (open tickets with their files).
Closes the discoverability gap where TICKETS.md existed but agents
didn't know to look at it.

Net: CLAUDE.md goes from 308 → 311 lines BUT the avg information
density is up — bench prose doesn't bloat the entry-point doc, and
each convention now points to its source. The `docs/bench-maxing.md`
extraction is the real win: separation of "rules I must not break"
(CLAUDE.md) from "discipline I should internalize" (bench-maxing.md).
This commit is contained in:
russell@unturf.com 2026-05-01 14:12:37 -04:00
parent ac17e6e4ab
commit 24198ab05e
No known key found for this signature in database
2 changed files with 348 additions and 233 deletions
Download patch file Download diff file Expand all files Collapse all files

469
CLAUDE.md
View file

@ -1,296 +1,299 @@
# Agent Blackops — aborist
This repo is operated by **agent blackops** for fox/timehexon on the unsandbox / unturf / permacomputer platform.
This repo is operated by **agent blackops** for fox/timehexon on the
unsandbox / unturf / permacomputer platform.
Identity shard: `~/git/unsandbox.com/blackops/BLACKOPS.md`.
## What aborist is
A content-addressed, Merkle-committed document store. Implements the runtime spec from **Merkle Providence Reverse RAG** (April 2026 whitepaper) scaled up to the **Merkle-AGI v9.8** admissibility ledger. Tends "trees and forests of cross-linked information" — the namesake.
A content-addressed, Merkle-committed document store. Implements the
runtime spec from **Merkle Providence Reverse RAG** (April 2026
whitepaper) scaled up to the **Merkle-AGI v9.8** admissibility ledger.
Three layers stacked on one SQLite file:
1. **Surface** — ingested documents (Wikipedia dumps, HTML pages, anything with a URI). Chunked, Merkle-rooted, FTS5-indexed.
2. **Core** — distilled documents (haiku/keyword/equation-scale) Merkle-bound back to source surface(s) via per-chunk inclusion proofs in `derivations.proof_blob`. Recursive: cores derive from cores. The "planet toward center" compression.
3. **Providence cache** — Q&A records keyed on the v9.8 8-dim invariant. Each record carries an `audit_mode` set by the post-LLM faithfulness verifier (`aborist/qa/verify.py`): STRICT (every claim verbatim-grounded or token-coverage paraphrase-grounded), HYBRID (mixed source/emergent), UNGROUNDED (no recoverable grounding — purely emergent from training, or genuine model refusal). `verifier_method` records which of four strategies — quote / span / entity / paraphrase — fired.
1. **Surface** — ingested documents (Wikipedia dumps, HTML pages,
anything with a URI). Chunked, Merkle-rooted, FTS5-indexed.
2. **Core** — distilled documents Merkle-bound back to surfaces via
per-chunk inclusion proofs in `derivations.proof_blob`. Recursive.
3. **Providence cache** — Q&A records keyed on the v9.8 8-dim
invariant. Each record carries `audit_mode` (STRICT / HYBRID /
UNGROUNDED) decided by the verifier in `aborist/qa/verify.py`.
## Source papers (read first if confused)
## Source papers
- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` — **canonical whitepaper source** (rst, builds the PDF). Edit here, not the PDF.
- `~/Downloads/merkle-providence-reverse-rag-whitepaper.pdf` — built artifact; mirrors the rst above
- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV encoding A1, public quantization A2, collision-resistant hash A3, theorems T1–T5)
- `~/git/proxy.unturf.com/pkg/verified/merkle.go` — fox's existing Go merkle implementation. **Aborist Python ports its conventions exactly.**
- `~/git/proxy.unturf.com/docs/merkle-tree.md` — convention reference
- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst`
— canonical whitepaper (rst → PDF). Edit here, not the PDF.
- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV
encoding A1, public quantization A2, collision-resistant hash A3,
theorems T1–T5).
- `~/git/proxy.unturf.com/pkg/verified/merkle.go` — fox's existing
Go merkle. Aborist Python ports its conventions exactly.
- `~/git/proxy.unturf.com/docs/merkle-tree.md` — convention reference.
## Architecture
```
aborist/
├── merkle.py # Python port of proxy.unturf.com Go conventions
├── store.py # v9.8 SQLite schema, audit chain helpers
├── document.py # Document, Edge, Chunker (TokenChunker default)
├── source.py # Source ABC: iter_documents() -> Iterator[Document]
├── ingest.py # batched: normalize -> chunk -> merkle -> upsert
├── evict.py # hot->cold + rehydrate (v9.8 falsification on drift)
├── search/ # SearchBackend ABC + AuditMode + FTS5
├── sources/ # one file per corpus
│ ├── wikipedia.py # WikipediaSqlDump (cur + old tables)
│ └── html_page.py # HtmlPageSource (selectolax, robots-aware)
├── distill/ # surface->core distillation
│ ├── base.py # Distiller ABC + DistillationResult
│ ├── first_sentence.py # FirstSentenceDistiller (no-ML stub)
│ ├── tfidf.py # TfidfKeywordDistiller (pure-Python TF-IDF)
│ └── runner.py # batched: derive + per-contrib-chunk proofs
├── qa/ # Q&A: 8-dim cache_key + Merkle-bound answers
│ ├── client.py # ChatClient + StubClient + OpenAICompat
│ ├── keys.py # cache_key, question_hash, ... (pure functions)
│ ├── verify.py # layered verifier: quote → span → entity → paraphrase;
│ │ # plus claim-lattice-pointer verifier (G0 / CTI)
│ ├── evidence.py # EvidenceObject + spotlight excerpt (G0)
│ ├── parse_claims.py # pointer-line parser (G0)
│ ├── inspect.py # read-only sidecar: diagnose unverified spans
│ ├── dag.py # per-run Merkle-DAG (7-stage quote / 9-stage CTI)
│ └── runner.py # ask(): cache -> infer -> verify -> classify -> write
├── wikitext.py # to_base(): wikitext → plain prose (BASE_VERSION-pinned)
└── cli.py # ingest / search / verify / stats / distill /
# evict / rehydrate / ask / providence / emergent /
# reclassify / inspect / analyze
├── merkle.py # Python port of proxy.unturf.com Go conventions
├── store.py # v9.8 SQLite schema, audit-chain helpers
├── document.py # Document, Edge, Chunker (TokenChunker default)
├── source.py # Source ABC: iter_documents()
├── ingest.py # batched normalize → chunk → merkle → upsert
├── evict.py # hot↔cold + rehydrate (v9.8 falsification on drift)
├── search/ # SearchBackend ABC + AuditMode + FTS5
├── sources/ # one file per corpus (wikipedia, html_page)
├── distill/ # surface → core distillation (tfidf, first_sentence)
├── qa/ # Q&A: 8-dim cache_key + Merkle-bound answers
│ ├── client.py # ChatClient + StubClient + OpenAICompat
│ ├── keys.py # cache_key, question_hash (pure functions)
│ ├── verify.py # layered verifier + claim-lattice verifiers (G0)
│ ├── evidence.py # EvidenceObject + spotlight excerpt
│ ├── inspect.py # read-only sidecars (deflection, title-relevance)
│ ├── dag.py # per-run Merkle-DAG (7-stage quote / 9-stage CTI)
│ └── runner.py # ask(): cache → infer → verify → write
├── wikitext.py # to_base(): wikitext → plain prose (BASE_VERSION-pinned)
└── cli.py # ingest / search / verify / stats / distill /
# evict / rehydrate / ask / providence / emergent /
# reclassify / inspect / analyze
```
## Build, test, run
Every workflow is a `make` target. Bare python commands are not the user interface.
Every workflow is a `make` target. Bare python is not the user
interface. See the Makefile for the full list.
```
make bootstrap # venv + editable install with [dev] extras
make test # pytest -q (49 tests)
make test # pytest -q
make all # bootstrap + fetch-cur + ingest-cur + verify + stats
make fetch # cur (82 MB) + old.1 (640 MiB) + old.2 (252 MiB) + concat
make ingest-cur # ingest snapshot articles
make ingest-old # ingest revision history (~hours)
make verify-shards # round-trip Merkle proofs on a random sample (cross-shard)
make analyze-shards # cross-shard compression spectrum + audit integrity
make verify-shards # round-trip Merkle proofs (cross-shard sample)
make analyze-shards # cross-shard compression + audit integrity
make chain-check-shards # audit-chain break count per shard (0 = intact)
make query Q="..." [JSON=1 BURN=1 K="extra retrieval keywords" ANSWER_MODE=…]
make bench-qa # QA-quality sweep (live LLM)
```
**Hygiene after any state-changing op** (table rebuild, reclassify run, governance hash bump, mass falsify): `make chain-check-shards` for a one-second sanity (every shard should report `0`), then `make analyze-shards` for the full spectrum + chain audit. Chain breaks are the loudest possible signal that something corrupted the audit log — catch them at the seam, not in production.
`aborist/cli.py` adds: `analyze`, `distill --kind {surface,core}`, `evict`, `rehydrate`, `ask`, `providence`, `emergent` (list UNGROUNDED/HYBRID records or `--aggregate` to rank unverified quotes — corpus-growth signal), `reclassify` (re-run the verifier against existing live providence records under the current entity policy; no LLM calls; `--compare` runs all four entity policies side-by-side, `--dry-run` reports without writing), `inspect --cache-key X` (read-only sidecar that pulls source chunks for one record & classifies each unverified span; never writes to providence_cache or audit_events). The `query` command human-renders by default; pass `--json` (or `JSON=1` to `make query`) for the raw record. `--batch-size` defaults to 200 docs/transaction; lower it only to bound memory peaks.
**Hygiene after any state-changing op** (rebuild, reclassify, hash
bump, mass falsify): `make chain-check-shards` first (every shard
should report `0`), then `make analyze-shards` for the spectrum +
chain audit. Chain breaks are the loudest possible signal.
## Schema invariants (do not break)
- **Aborist is a v9.8 store.** Every Document carries `chunking_version`, `canonicalization_version`, `schema_version`. Every providence record carries the full **8-dim cache_key**: `source_root | question_hash | model_profile_hash | conversation_hash | governance_policy_hash | schema_version | canonicalization_version | chunking_version`. Bumping any one invalidates prior records on lookup.
- **`falsification_state ∈ {live, failed, stale, quarantined}`.** Cache lookups must filter on `state='live'`. Drift detection (rehydrate vs source root mismatch) flips to `stale`.
- **Audit chain.** Every state-changing op writes one row in `audit_events` with `event_hash = sha256(prev_event_hash || canonical(body))`. Chain integrity is verified in `make analyze-shards` (full audit) or `make chain-check-shards` (one-second per-shard break count, `0` = intact). **Never insert into `audit_events` directly — use `aborist.store.append_audit`.**
- **v9.8 8-dim cache_key**: `source_root | question_hash |
model_profile_hash | conversation_hash | governance_policy_hash |
schema_version | canonicalization_version | chunking_version`.
Bumping any one invalidates prior records on lookup.
- **`falsification_state ∈ {live, failed, stale, quarantined}`**.
Cache lookups filter on `state='live'`. Drift → `stale`.
- **Audit chain**: every state-changing op writes one row in
`audit_events` with `event_hash = sha256(prev || canonical(body))`.
Verified by `make chain-check-shards`. Use `aborist.store.append_audit`
— never insert into `audit_events` directly.
- **Cores never evict.** `evict_to_cold` only touches `kind='surface'`.
- **Idempotent re-ingest.** Same content → same `document_root` → no-op insert. Same URI + different content → new doc + `supersedes` edge linking new → old (lossless history).
- **Idempotent re-ingest.** Same content → same `document_root` →
no-op insert. Same URI + different content → new doc + `supersedes`
edge (lossless history).
## Conventions (do not silently change)
- **Merkle:** non-commutative `HashCombine` with prefix `0x03`. Leaves `0x00`. Odd-element rule = self-duplicate, NOT zero-pad. `MerkleProof.siblings` carries explicit `is_left` flag — never sort lexically.
- **Chunker default = `tok-512-v1`.** Changing the default bumps `chunking_version` and **stales every prior cache record**. Add a new chunker as a new `name` instead.
- **Canonicalization = `norm-v1`** (NFC, collapsed whitespace). Same rule.
- **Schema = `v9.8.0`.** Same rule.
- **`question_hash` is dedup-mode-aware.** `question_hash(q, mode=...)` accepts `"strict"` (NFC + ws-collapse only — case-sensitive, punctuation-sensitive, article-sensitive; every variant gets its own hash) or `"equivalence_class"` (default — additionally lowercase + trailing-punct strip + standalone-article strip; collapses `"who is X"`, `"who is X?"`, `"Who Is X."`, `"who is the X"`, `"who is an X？"` to one hash). Mode lives in `policy["question_dedup"]` so it folds into `governance_policy_hash`; agents under different modes write records under different cache_keys.
- **JIT lookup fidelity.** Per-call `fidelity` parameter on `query()`/`ask()` decouples lookup tolerance from write policy. `"strict"` checks only the primary cache_key (audit-grade — record reuse only on exact match). `"equivalence_class"` (default) tries primary first, then the alternate dedup mode's cache_key as a fallback so a fast-cache agent can reuse records written under either policy. Result includes `lookup_path` ∈ `{"strict", "equivalence_class", "strict_fallback", "equivalence_class_fallback", "miss"}`. Cross-silo fallback works because the helper rewrites `policy["question_dedup"]` to the alternate mode when computing the fallback ckey, so `governance_policy_hash` matches what an agent under that mode would have written. CLI: `--question-dedup`, `--fidelity` flags on `query` (and the `ask` API params).
- **`audit_mode` is decided by the verifier, never asserted unconditionally.** Four layered strategies in `aborist/qa/verify.py`, tried in order; first to find evidence classifies the answer:
1. **quote** — model wrapped claims in double quotes per system prompt. Sequential pairing: 1st & 2nd `"`, 3rd & 4th, etc. (NOT regex pairing — that captures inter-pair prose as a phantom span when the model writes `"title" prose "quote"`).
2. **span** — bullet/sentence lines from the answer appear verbatim in context. Catches models that quote inline without `"..."` marks.
3. **entity** — multi-word proper-noun phrases appear verbatim in context. Gated by `entity_policy ∈ {strict, hybrid, drop, proximity}`. Default `proximity`: STRICT only when N=3 verified entities cluster within W=300 chars in source (cast list / infobox / roster). Otherwise HYBRID/UNGROUNDED. Distinguishes structural grounding from incidental mention. Lives in `DEFAULT_QUERY_POLICY["entity_policy"]` so any change bumps `governance_policy_hash`.
4. **paraphrase** — soft fallback on prose-shaped spans when 1-3 fail. Token-coverage probe: ≥4-char content tokens minus an English stopword set (`from`, `with`, `would`, `which`, etc.) checked for presence in normalized base context. Promotes when coverage ≥ `DEFAULT_PARAPHRASE_COVERAGE` (0.85) with ≥ `DEFAULT_PARAPHRASE_MIN_TOKENS` (4) content tokens. Fires only on prose-shaped spans (≥2 lowercase content tokens via `_is_prose_span`) so lists of proper nouns flow to the entity strategy instead. Quote strategy deliberately gets NO paraphrase fallback — `"..."` asserts verbatim citation; paraphrasing inside quotes is a model error to flag, not auto-promote. Records carry `verifier_method='paraphrase'` so an auditor can tell soft-verified evidence from lexical-verbatim.
Trichotomy across all paths: STRICT = every evidence unit (≥1) verifies. HYBRID = some verify, some don't. UNGROUNDED = no evidence or none verifies. Persisted on `providence_cache.audit_mode` + `verifier_method`; cache-hits return the stored mode. Never overclaim — STRICT is a verifiable claim, not a default.
- **Trailing-citation strip.** `_strip_trailing_citation` peels a single trailing parenthetical at end-of-span (gated on a citation-cue word — `Source:`, `citing`, `see`, `ref`, `from` — OR a URL) before substring testing. Keeps `"...prose. (Source: https://...)"` from flunking just because the model appended a citation tail. Refuses to strip genuine prose parentheticals (no cue word, no URL).
- **Verifier stays binary; falsifications carry soft signal.** No per-quote diagnosis fields on hard verifier output. `verify_quotes` returns evidence units + classification; the falsify+reclassify loop owns "why didn't this ground" for the operator, and `aborist inspect --cache-key X` is the read-only sidecar that classifies each unverified span (`verbatim_in_base` / `verbatim_in_raw_only` / `trailing_artifact` / `paraphrase` / `partial_paraphrase` / `no_overlap`) — sidecars never write to `providence_cache` or `audit_events`. Don't bolt confidence scores or partial-match indicators onto `verify.py`.
- **Deflection sidecar.** `aborist.qa.inspect.diagnose_deflection(question, answer)` detects topic-shift on adversarial-premise questions (Mars-BDFL pattern: 2026-04-30 'who is a benevolent dictator for life for mars?' returned STRICT with answer about Guido/Python — verifier did its job, but the user's question went unanswered). **Subject-anchor heuristic**: the LAST content token in the question (after stopword strip) is treated as the question's primary subject. If the subject anchor is missing from the answer, classify as `deflection` regardless of generic-vocabulary overlap. Returns `kind ∈ {deflection, partial_overlap, on_topic, no_question_tokens}`. Wired into `inspect_cache_key()` and per-row in `bench/qa_sweep.py` (summary table grows a `deflections` column tracking STRICT/HYBRID rows where subject anchor is missing). Sidecar-only — never feeds back into providence; deflection on a STRICT record means "substrate did the right thing structurally, but agent's question wasn't answered."
- **Claim-count ceiling (TOO_MANY_CLAIMS).** `policy["claim_lattice_max_claims_per_answer"]` caps the number of claims a single answer can emit, default `12`. Bench finding (york-england 2026-04-30): the prompt shape "tell me all there is to know about X" prompted Hermes to spam 26-59 claim-pointer pairs of which only 2-4 verified. Atomic-claim prompt rule (commit b5925c8) reduced the typical case to ~10 claims; the cap is defense in depth. Cap of 12 admits typical entity-list questions (5-7 dinosaurs, simpsons + pets) while flagging the runaway. Cap doesn't truncate — every claim still verifies so the operator sees the full evidence; the violation demotes STRICT → HYBRID via the existing violation-check path. Plumbed through both pointer (`verify_claim_lattice`) and JSON (`verify_claim_lattice_json`) verifiers. Folds into `governance_policy_hash` on change.
- **Wikitext base prose.** `aborist/wikitext.py:to_base(raw)` converts MediaWiki wikitext → plain prose deterministically (mwparserfromhell-backed; pinned by `BASE_VERSION = "wikitext-base-v1"`). Applied **before the LLM call** in `aborist/qa/runner.py` and `aborist/qa/query.py` (gated on `policy["base_version"]`), and again inside `verify_quotes` so the verifier compares like-against-like. Both sides — model and verifier — see prose; the model can quote source paragraphs verbatim instead of escaping `[[wikilinks]]`, and Wikipedia chunks ship to Hermes with ~43% fewer tokens. `policy["base_version"]` lives in `DEFAULT_POLICY`/`DEFAULT_QUERY_POLICY` so it folds into `governance_policy_hash`; bumping `BASE_VERSION` invalidates every prior cache record's 8-dim cache_key on next lookup. Optional dep — installs without `mwparserfromhell` keep `_wikitext_to_base = None` and `policy["base_version"] = None`, leaving raw wikitext in both context and verifier (graceful fallback, no failure mode).
- **Soft hash vs hard hash.** Hard = SHA-256 (commitments, proofs, cache_key). Soft = embeddings/TF-IDF/similarity (training, ranking, distillation candidate selection). Never mix — soft never enters proof path.
- **Three answer modes.** `policy["answer_mode"] ∈ {"quote", "claim_lattice_pointer", "claim_lattice"}`, default `"quote"`. The substrate exposes all three; agents pick by inference profile. `quote` for prose-shape models that emit verbatim citations inline. `claim_lattice_pointer` (G0 / CTI) for prose-distribution models like Hermes-3-8B that handle pointer-line bracketed tags well. `claim_lattice` (JSON) for grammar-constrained inference (vLLM `guided_json`, Claude/GPT-4 native JSON, Qwen 3.6 reasoner) — pairs with the `_lenient_json_parse` pre-parser that handles markdown fences / preamble / curly quotes / trailing commas / unbalanced brackets so non-strict JSON inference paths stay survivable. Bench evidence (2026-04-30, n=3, 24 questions on Hermes-3-8B): quote 0.47 strict-rate, pointer 0.36, JSON 0.49 — JSON wins on Hermes. Both lattice modes share `verifier_method="claim_lattice"` so the providence_cache CHECK constraint accepts both; downstream disambiguation lives in the `answer_mode` field on the run-DAG and (for JSON) in the `json_fixups` list on the verdict. Each mode folds into `governance_policy_hash` so different modes write under different cache_keys and never alias.
- **Claim-lattice-pointer mode (G0 / CTI Clause Lattice Intelligence).** In pointer mode the runtime builds an evidence map from retrieved chunks; each `EvidenceObject` carries TWO ids — `pointer_id` (`E1`, `E2`, …, sequential, what the model sees) and `evidence_id` (sha256-derived `E########`, content-addressed, what the cache & run-DAG use). Hermes emits pointer-line prose (`Claim text. [E12]` per line); `aborist.qa.parse_claims.parse_pointer_claims` parses `(claim_text, pointer_ids[])`; `verify_claim_lattice` resolves pointers to `EvidenceObject`s and runs deterministic checks ONLY: parser succeeded, evidence_id resolves, source_role allowed, no manual quotes (any `"` char violates — strict), claim text non-empty. Soft signals (entailment, completeness, predicate compatibility, scope) stay sidecar; never enter the proof path. Renderer interpolates literal source spans at display time via spotlight excerpt (`_spotlight_excerpt` finds the first claim-content-token match in the cited span and centers a window on it; falls back to leading window when no token matches). Synthetic-elision-by-construction-impossible — the model never types the quote string. No iterative repair in pointer mode (one-shot benchmark discipline). Run-DAG grows from 7 stages to 9: `question / retrieval / evidence_map / prompt / raw_answer / parsed_claim_lattice / verify / render / final_label`. `policy["answer_mode"]` folds into `governance_policy_hash` so two modes write under different cache_keys and never alias. `make query Q="..."` defaults to `ANSWER_MODE=claim_lattice_pointer` so the testing harness exercises G0 by default; library `DEFAULT_POLICY` / `DEFAULT_QUERY_POLICY` stay `"quote"` so Python callers and unit tests aren't surprised. Within each retrieved source, chunks are ranked by query-token overlap before pointer-id assignment so the chunk most likely to textually support the question gets `E1` — counters Hermes-3-8B's lazy-anchor habit on doc-order-first chunks.
Each rule below has full rationale in the named source file. Don't
revert without reading why. When in doubt, walk the
[Five-step algorithm](#five-step-algorithm) first.
- **Merkle conventions**: non-commutative `HashCombine` prefix `0x03`,
leaves `0x00`, odd-element rule = self-duplicate (NOT zero-pad).
`MerkleProof.siblings` carries `is_left` flag — never sort lexically.
See `aborist/merkle.py`.
- **Versioned defaults**: `tok-512-v1` (chunker), `norm-v1`
(canonicalization), `v9.8.0` (schema), `wikitext-base-v1` (prose).
Changing any default stales every prior cache record. Add a new
`name` instead.
- **`question_hash` is dedup-mode-aware** (`strict` |
`equivalence_class`); folds into `governance_policy_hash`. JIT
`fidelity` parameter on `query()`/`ask()` decouples lookup tolerance
from write policy. See `aborist/qa/keys.py`.
- **`audit_mode` is decided by the verifier, never asserted.** Four
layered strategies tried in order, first to find evidence
classifies: **quote** (sequential pair-matching, NOT regex —
prevents phantom inter-pair captures), **span** (verbatim line
match), **entity** (proximity-clustered proper nouns;
`entity_policy ∈ {strict, hybrid, drop, proximity}`), **paraphrase**
(token-coverage, prose-shaped only; `verifier_method='paraphrase'`).
Trichotomy: STRICT = every unit verifies, HYBRID = mixed,
UNGROUNDED = none. Never overclaim. See `aborist/qa/verify.py`.
- **Verifier stays binary; falsifications carry soft signal.** No
per-quote diagnosis fields on hard verifier output. Sidecars
(`aborist.qa.inspect.diagnose_*`, `aborist inspect --cache-key X`)
classify unverified spans, deflection, title-relevance — never
write to `providence_cache` or `audit_events`.
- **Trailing-citation strip**: `_strip_trailing_citation` peels one
trailing parenthetical at end-of-span (gated on a citation cue or
URL) before substring testing. See `aborist/qa/verify.py`.
- **Soft hash vs hard hash**: hard = SHA-256 (commitments, proofs,
cache_key); soft = embeddings/TF-IDF/similarity (training, ranking,
distillation). Soft never enters proof path.
- **Three answer modes**: `policy["answer_mode"] ∈ {"quote",
"claim_lattice_pointer", "claim_lattice"}`, default `"quote"`.
Bench 2026-04-30 on Hermes-3-8B: quote 0.47 strict-rate, pointer
0.36, JSON 0.49 — JSON wins. Both lattice modes share
`verifier_method="claim_lattice"`; `answer_mode` on the run-DAG
+ `json_fixups` disambiguate. Each mode folds into
`governance_policy_hash`. See `aborist/qa/verify.py`,
`docs/qa-modes-bench-2026-04-30.md`.
- **Claim-lattice-pointer mode (G0 / CTI)**: runtime mints
`pointer_id` (E1, E2, … — what the model sees) and content-addressed
`evidence_id` (what the cache & run-DAG store). Renderer interpolates
literal source spans via `_spotlight_excerpt`. Synthetic-elision-by-
construction-impossible — model never types the quote string. 9-stage
run-DAG. See `aborist/qa/evidence.py`, `docs/cti-architecture.md`.
- **Claim-count ceiling (`TOO_MANY_CLAIMS`)**: default 12 per answer.
Catches "tell me all there is to know about X" runaway. Demotes
STRICT → HYBRID without truncating. Folds into
`governance_policy_hash`. See `aborist/qa/verify.py`.
- **Wikitext base prose**: `aborist/wikitext.py:to_base()` runs
before the LLM call AND inside `verify_quotes` so model and verifier
see the same prose. Optional dep — graceful fallback when
`mwparserfromhell` is missing.
- **Deflection sidecar**: `diagnose_deflection(question, answer)`
detects topic-shift via subject-anchor heuristic (LAST content
token in question must appear in answer). Suppressed for date /
count / cause shapes ("when", "why", "how many"). See
`aborist/qa/inspect.py`.
- **Title-relevance sidecar**: `diagnose_title_relevance(claim,
cited_titles)` flags retrieval-driven hallucinations where the
cited chunk's source title shares zero stems with the claim's
content tokens. See `aborist/qa/inspect.py`.
## Live endpoints
- LLM: `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B-FP8-Dynamic on vLLM, 82K ctx, no auth). `uncloseai.com` is marketing only — has no `/v1`. Override via `--endpoint` or `ABORIST_LLM_ENDPOINT`.
- Wikipedia dumps: `https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`. `robots.txt` returned 404 → no rules.
- LLM: `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B-FP8-
Dynamic on vLLM, 82K ctx, no auth). `uncloseai.com` is marketing
only. Override via `--endpoint` or `ABORIST_LLM_ENDPOINT`.
- Wikipedia dumps: `https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`.
`robots.txt` returned 404 → no rules.
## Retrieval pipeline (`aborist/qa/query.py`)
Multi-stage. Each stage exists because something earlier wasn't enough; revert at your peril.
Multi-stage. Each stage exists because something earlier wasn't
enough; revert at your peril. Order:
1. **Two parallel FTS5 searches per shard, merged.** `_search_corpus` runs body-BM25 AND title-LIKE in parallel and unions hits. The title path closes a recall gap where list-pages with many URLs (e.g. `List_of_HTTP_status_codes`) outrank the actual `HTTP` article on body BM25. See `d608c39 fix retrieval recall`.
2. **Body-coverage `sqrt` rerank.** BM25 systematically favors short docs with rare body tokens (`Tell_(poker)` outranking `Back_to_the_Future` on a film query). Counteract by reranking on `sqrt(body_token_count)` to recover long-doc hits. `1983b79`.
3. **Title-token boost.** `_rerank_by_title` adds `boost × (overlap)` to hits whose title contains query tokens — strong topical signal that BM25 alone misses.
4. **`_filter_by_title_relevance` — three accept paths.** A hit passes if any of: (a) **title-token overlap** meets `title_breadth` (≤2 tokens require ALL; 3+ require N-1); (b) it's in `core_match_roots` (TF-IDF core keywords contain a query token — closes the gap for neologisms like "permacomputer" that never appear in titles); (c) **body density** — `_body_density_passes` requires `distinct_present >= breadth_threshold AND total_mentions >= 3`. Same breadth scaling as titles. Synonym fallback fires only for 1-token queries (otherwise `intel`-titled doc bleeds into AMD queries via the rivalry group).
5. **Rivalry exclusion** (`rivalry_excluded` in `qa/concepts.py`). Intel-titled docs drop from AMD queries; reverse holds. Applies on every accept path.
6. **Stem-aware token matching.** `_stem_token_for_match`: trailing-`s` strip on tokens >4 chars (skip `ss`-enders). Possessive (`superman's` → `supermans` → `superman`) and plural (`girlfriends` → `girlfriend`) collapse onto bare-stem source mentions. Caught the 2026-04-29 *"who is supermans girlfriend?"* defect — pre-stem, the query admitted 7 unrelated `Girlfriends`-titled articles.
7. **Per-source context cap.** Each top-K hit gets at most `max_context_chars / top_k` chars before the global `char_budget` is consumed. Fox's 2026-04-29 catch: `List_of_Batman_comics` (80 KB+ bibliography) was monopolizing the entire 60 KB budget at hit #1, dropping every other source.
8. **Wikitext base prose** runs on the assembled context BEFORE the LLM call (`policy["base_version"]` gates it, see Wikitext base prose convention above).
9. **Template-phrase stopwords on the FTS5 MATCH.** `_FTS5_STOPWORDS` (search/fts5.py) and `_TITLE_STOPWORDS` (qa/query.py) must stay in sync — both filter `tell show describe explain summarize say give list find make please all there know everything anything something` plus standard English stopwords. The `all there know` additions (2026-04-30) close the york-england miss: pre-fix, "tell me all there is to know about york england?" expanded to `"all" AND "there" AND "know" AND "york" AND "england"` in AND-mode FTS5, which favored "X of England" articles that incidentally mention all five tokens over the actual `York` article. Retrieval-time stopwords only — they don't enter `cache_key` or `governance_policy_hash`.
1. **Four parallel FTS5 search routes per shard, merged** — body BM25,
title-LIKE, core-keyword (TF-IDF cores), and **phrase-pattern**
(verbatim n=5/n=6 sequences from the question). Phrase route
closes the allusion gap (Orwell case: "always been at war" verbatim
matches the 1984 article whose title shares zero tokens with the
query). See `docs/reference-frame-failure-class.md`.
2. **Body-coverage `sqrt` rerank** — counters BM25's short-doc bias.
3. **Title-token boost** — `boost × overlap` on title-token-matching hits.
4. **`_filter_by_title_relevance` — four accept paths**: title-token
overlap, TF-IDF core match, body density, **phrase match**
(accept-path 4 lets phrase-route hits with no title overlap survive).
5. **Rivalry exclusion** (`qa/concepts.py`) — Intel-titled docs drop
from AMD queries; reverse holds.
6. **Stem-aware token matching** — possessive / plural collapse
(`superman's → supermans → superman`).
7. **Per-source context cap** — `max_context_chars / top_k`. Prevents
one huge doc from monopolizing the budget.
8. **Wikitext base prose** runs on assembled context BEFORE the LLM.
9. **Template-phrase stopwords** (`_FTS5_STOPWORDS` and
`_TITLE_STOPWORDS` must stay in sync) — strips `tell show describe
explain summarize say give list find make please all there know
everything anything something` so "tell me all there is to know
about X" doesn't dilute query tokens.
`--retrieval-keywords` (CLI: `K="..."`) lets an operator augment
retrieval-side tokens without changing what the LLM sees as its
question. Provenance gap on this is tracked in
[Ticket #000001](docs/ticket-000001-retrieval-keywords-audit-gap.md).
## Hot path / gotchas
- **Parser is hand-rolled** in `aborist/sources/wikipedia.py` (char-position state machine, escape-aware). After the v9.8 commit it's 4× faster via `str.find` + slicing — easy to break by reverting to char-by-char loops. cProfile any change.
- **`PRAGMA synchronous=NORMAL`** is set per-connection in `store.connect()`. Safe under WAL (the journal_mode is set in `SCHEMA_SQL`). Don't downgrade to FULL without a measured reason — costs ~5x throughput.
- **HTML source has optional deps**: `pip install '.[html]'` for `selectolax`. The CLI surfaces `--source html` only if the import succeeds.
- **Background ingest/distill processes**: stdout is buffered. Use `export PYTHONUNBUFFERED=1` or `python -u`. Per blackops top-level rule.
- **Disk pressure.** Full cur ingest ~2 GB; full old ingest ~5–8 GB. `df -h /home/fox` first.
## Bench-maxing — measure deltas, not opinions
The QA-quality bench (`make bench-qa`) and live functional fixtures
(`make test-live`) are not decoration. They are the discipline that
keeps prompt + verifier work honest. Patterns established 2026-04-30
during the JSON-mode hardening journey:
- **Bench before AND after every change.** Single-sample bench has
±20pp noise; n=3 narrows the band but signal under 5pp is still
noise. The only way to know a change moved the needle is to run
the same harness on identical inputs immediately before and
immediately after. Don't skip this — opinions about whether a
prompt change "should help" are routinely wrong.
- **Slight prompt adjustments are fine — local-minima are not.**
Prompts can be nudged with a worked example or a single-sentence
rule. They should NOT be padded with many rules that incentivize
empty / over-cautious output ("if uncertain, omit" can collapse
the model into refusing valid answers). When in doubt, change
one thing per bench cycle.
- **Avoid negation in every prompt.** Small instruction-tuned models
(Hermes-3-8B observed) struggle with "do NOT X" / "never Y" /
"without Z" — the negation often pattern-matches away or inverts
under attention. Rewrite every rule to its positive form. Some
swaps that work:
- "Do not invent IDs" → "Use only the IDs that appear in the EVIDENCE blocks above"
- "Never begin a line with E1:" → "Each line begins with the claim text, then a bracket tag"
- "No commentary, no preamble" → "Your output begins with `{` and ends with `}`"
- "If no evidence supports a claim, omit the claim" → "Emit only claims that an EVIDENCE block directly supports"
- "No double-quote characters in text" → "Write text as plain prose; punctuation appears in the source span the runtime renders for you"
Audit all the `*_system_prompt` and `*_grounding_reminder` policy
fields for residual negation when a model behaves erratically —
it is the cheapest fix in the prompt-iteration toolkit.
- **Bench is the scoreboard; live fixtures are the gates.** Bench
measures aggregate (STRICT/HYBRID/UNGROUNDED counts across N×Q
samples). Live fixtures (`tests/test_qa_quality_live.py`) gate
on specific known-good answers. When a change improves things
the bench number climbs AND every fixture passes; when something
regresses the bench falls AND named fixtures fail by name,
pointing at where the regression landed.
- **Self-heal beats retry.** When the model emits malformed output
(truncated JSON, trailing comma, partial key), repair the
artifact rather than re-running the LLM call. Self-healing
preserves whatever partial content the model produced and lands
in the unverified bucket honestly. Retrying spends another
inference round and may produce identical garbage. See
`_repair_truncated_json` in `aborist/qa/verify.py` for the
pattern: walk once tracking string state + bracket stack, close
/ drop / balance at end-of-input. Conservative repairs only —
never insert content, never fabricate keys.
- **Name the failure → fix in code → re-bench → confirm.** Each
bench-driven commit should reference the specific failure mode
it addresses (e.g. "Apollo runaway: 3/3 → 2/3" or "JSON
hallucinated near-miss content-IDs on cross-doc relationships").
When the next bench shows the named failure didn't budge, you
fixed the wrong thing. Name it again, try again.
- **Honest verdicts beat optimistic ones.** A change that drops
STRICT count by 5 but moves those 5 to HYBRID with real
grounding is a WIN — false-positive STRICTs are corruption.
The bench grade improves when the verifier reports closer to
ground truth, not closer to 100%. Trust HYBRID with smell-
sidecar warnings over STRICT-with-bogus-citation every time.
- **Per-question fixtures > marker-string assertions.** When
designing a live fixture, prefer entity-presence checks
("Burns in the answer", "October or 1991") over byte-identical
matching. Hermes is non-deterministic; the right entity in the
answer is the gate, not the exact phrasing. When a fixture
starts failing intermittently, treat it as quality drift on
that question shape and tighten retrieval / verifier upstream
rather than loosening the test.
- **Old maps vs runtime maps — move authority toward runtime artifacts.**
Every base model carries old maps: trained patterns from the time
the weights froze. It "knows" how API X used to work, what JSON
shape Y used to take, which ritual citation phrase was popular.
Those priors collide with the present-day runtime — the actual
retrieved chunks, the current evidence map, the live policy hash,
the ID space the runtime built this minute. When the model and
the runtime disagree, the runtime wins. Engineering moves
authority OUT of the model's old map and INTO runtime artifacts:
- Pointer IDs (E1, E2, …) the runtime mints, so the model can't
fabricate a content-addressed hash that "looks right."
- Source spans the runtime interpolates by
`(chunk_root, offset_start, offset_end)`, so the model never
owns the quote text.
- Evidence maps the retrieval pipeline assembles per-query, so
the model can't self-supply context from training memory.
- Policy hashes that fold prompt + verifier + retrieval choices
into the cache_key, so old governance can't silently reuse
new-rule records.
- Hard checks run by the verifier, never the model self-grading.
When a defect smells like "model brought a stale map," the fix is
to make the runtime more authoritative, not to negotiate with the
model's prior. Hermes' content-addressed-evidence-id hallucination
(commit bb8450d, 2026-04-30) is the canonical case: the fix was
to swap the prompt-facing surface to runtime-minted pointer IDs
the model can't fabricate, not to add an "are you sure?" step.
- **Doc captures the journey, not just the end state.**
`docs/qa-modes-bench-2026-04-30.md` shows the day's progression
(baseline → retry → trim-and-verify → pointer-IDs →
stop-sequence → self-heal). Each row is a named failure +
fix + bench delta. Future-you (or a new agent) can read the
doc and see WHY each commit landed, not just WHAT.
- **Hand-rolled wikitext parser** (`aborist/sources/wikipedia.py`):
char-position state machine, escape-aware, 4× faster than char-
by-char loops via `str.find` + slicing. cProfile any change.
- **`PRAGMA synchronous=NORMAL`** per-connection in `store.connect()`.
Safe under WAL. Don't downgrade without measured reason (~5× cost).
- **HTML source has optional deps**: `pip install '.[html]'` for
`selectolax`. CLI surfaces `--source html` only if import succeeds.
- **Background ingest/distill processes**: stdout is buffered. Use
`export PYTHONUNBUFFERED=1` or `python -u`.
- **Disk pressure**: full cur ~2 GB; full old ~5–8 GB. `df -h` first.
## Operational rules
- I propose, fox decides. Unsure = ask. Can't ask = stop.
- No autonomous destructive ops. No `clean-data`, `clean-db`, force-push, or DB drops without explicit instruction.
- **Never add `Co-Authored-By` or "Generated with Claude" lines to commits.** Professional commit messages only — code speaks for itself.
- **Always `export PYTHONUNBUFFERED=1`** for long-running processes. Buffered output disappears when processes die.
- No autonomous destructive ops (`clean-data`, `clean-db`, force-push,
DB drops) without explicit instruction.
- **Never add `Co-Authored-By` or "Generated with Claude" lines to
commits.** Code speaks for itself.
- **Always `export PYTHONUNBUFFERED=1`** for long-running processes.
- Fail-closed. Cleanup crew, not demolition.
- DRY in context — single source of truth, no sprawl.
- Never say "AI" — always say "machine learning."
- Prefer "defect" over "bug."
- Check robots.txt before any web fetch the user didn't authorize.
- Check `robots.txt` before any web fetch the user didn't authorize.
## Five-step algorithm (apply in order)
## Five-step algorithm
When proposing or evaluating change, walk these in sequence. Skipping a
step makes the next ones expensive and the system worse.
When proposing or evaluating change, walk these in sequence. Skipping
a step makes the next ones expensive and the system worse.
1. **Make the requirements less dumb.** Challenge every requirement and
assumption — *especially* the ones from "smart" sources. Every
requirement should come with a person's name, not a department or
a generic appeal to "best practices." For aborist that means: every
policy knob, schema invariant, prompt rule, and verifier check
should trace to a specific failure or owner. If you can't name who
asked for it or which defect it closed, the requirement is suspect.
1. **Make the requirements less dumb.** Every requirement gets a
person's name, not a department. If you can't name who asked or
which defect closed, the requirement is suspect.
2. **Delete the part or the process.** If you aren't putting back at
least 10% of what you delete, you aren't deleting hard enough.
Verifier-stays-binary and "no soft signals in hard chain" are
deletion-first guardrails.
3. **Simplify and optimize.** Only after 1 + 2. Don't optimize a
process that shouldn't exist.
4. **Accelerate cycle time.** Only after simplifying. "Don't dig the
grave faster."
5. **Automate.** Last, not first. Hand-rolled before scripted,
scripted before declarative, declarative before generated.
2. **Delete the part or the process.** Actively remove unnecessary
steps, fields, components, or rules. If you aren't putting back at
least 10% of what you delete, you aren't deleting hard enough. For
aborist that means: prefer pruning prompt rules / policy fields /
rerank stages / cache columns over piling new ones on. The
verifier-stays-binary discipline and the "no soft signals in the
hard chain" rules are deletion-first guardrails.
When in doubt, ask "have we tried deleting it?" before reaching for
steps 3-5.
3. **Simplify and optimize.** Refine what remains — *only after*
steps 1 and 2. A common mistake is optimizing a process that
shouldn't exist. Don't add a faster index, a better tokenizer, or
a tighter loop until the surrounding stage has survived steps 1+2.
## Bench-maxing — measure deltas, not opinions
4. **Accelerate cycle time.** Speed up the process — *only after*
simplifying. "If you're digging your grave, don't dig it faster."
For aborist: latency wins on a poorly-shaped retrieval pipeline
are graveyard-digging.
Full discipline + worked examples in `docs/bench-maxing.md`. Headlines:
5. **Automate.** Automate last, not first. Automating too early
produces inefficient, high-cost systems that nobody understands.
For aborist that means: hand-rolled before scripted, scripted
before declarative, declarative before generated. The bench
harness, distill runner, and ingest CLI all earned their
automation only after the underlying ops had been simplified
into single-purpose stages.
- Bench before AND after every change (n=3, signal under 5pp is noise).
- Avoid negation in prompts (Hermes-3-8B inverts under attention).
- Bench is the scoreboard; live fixtures are the gates.
- Self-heal beats retry (preserve partial output, never fabricate).
- Honest verdicts beat optimistic ones (false-positive STRICT is
corruption).
- Old maps vs runtime maps — when the model and runtime disagree,
the runtime wins. Pointer IDs, runtime-interpolated spans, evidence
maps, policy hashes, hard verifier checks all move authority OUT
of the model's prior and INTO runtime artifacts.
Apply to tickets, prompt edits, retrieval-pipeline additions, schema
changes, and our own behavior. When in doubt, ask "have we tried
deleting it?" before reaching for steps 3-5.
## Docs index
Architecture / ongoing work:
- `docs/cti-architecture.md` — CTI Clause Tree Intelligence.
- `docs/mesh.md`, `docs/mesh-deploy.md` — mesh wire + deploy.
- `docs/qa-modes-bench-2026-04-30.md` — JSON-mode hardening journey.
- `docs/reference-frame-failure-class.md` — Orwell case + phrase-route fix.
- `docs/bench-maxing.md` — bench discipline.
- `docs/test-coverage-audit-2026-05-01.md` — test-suite coverage audit.
- `docs/verifier-semantic-gap-design.md` — soft-signal NLI proposal.
- `docs/self-reference-thought-chains-design.md`,
`docs/self-reference-distillation-design.md` — recursive distillation.
- `docs/naming-deferral.md` — naming convention notes.
Tickets: `docs/TICKETS.md` (index + Next ID). Open tickets:
- `docs/ticket-000001-retrieval-keywords-audit-gap.md` — provenance
binding for `--retrieval-keywords`.
- `docs/ticket-000002-reference-frame-polarity-contract.md` —
Module L: multi-frame answer compilation.
## Orientation protocol
@ -300,7 +303,7 @@ pwd
git log --oneline -5
git status
make test
make chain-check-shards # per-shard audit-chain integrity (0 = intact)
make chain-check-shards # 0 per shard = intact
.venv/bin/aborist --shards-dir ~/.aborist/shards stats
.venv/bin/aborist --shards-dir ~/.aborist/shards analyze --gravity-top 5
```

112
docs/bench-maxing.md Normal file
View file

@ -0,0 +1,112 @@
# Bench-maxing — measure deltas, not opinions
The QA-quality bench (`make bench-qa`) and live functional fixtures
(`make test-live`) are not decoration. They are the discipline that
keeps prompt + verifier work honest. Patterns established 2026-04-30
during the JSON-mode hardening journey:
- **Bench before AND after every change.** Single-sample bench has
±20pp noise; n=3 narrows the band but signal under 5pp is still
noise. The only way to know a change moved the needle is to run
the same harness on identical inputs immediately before and
immediately after. Don't skip this — opinions about whether a
prompt change "should help" are routinely wrong.
- **Slight prompt adjustments are fine — local-minima are not.**
Prompts can be nudged with a worked example or a single-sentence
rule. They should NOT be padded with many rules that incentivize
empty / over-cautious output ("if uncertain, omit" can collapse
the model into refusing valid answers). When in doubt, change
one thing per bench cycle.
- **Avoid negation in every prompt.** Small instruction-tuned models
(Hermes-3-8B observed) struggle with "do NOT X" / "never Y" /
"without Z" — the negation often pattern-matches away or inverts
under attention. Rewrite every rule to its positive form. Some
swaps that work:
- "Do not invent IDs" → "Use only the IDs that appear in the EVIDENCE blocks above"
- "Never begin a line with E1:" → "Each line begins with the claim text, then a bracket tag"
- "No commentary, no preamble" → "Your output begins with `{` and ends with `}`"
- "If no evidence supports a claim, omit the claim" → "Emit only claims that an EVIDENCE block directly supports"
- "No double-quote characters in text" → "Write text as plain prose; punctuation appears in the source span the runtime renders for you"
Audit all the `*_system_prompt` and `*_grounding_reminder` policy
fields for residual negation when a model behaves erratically — it
is the cheapest fix in the prompt-iteration toolkit.
- **Bench is the scoreboard; live fixtures are the gates.** Bench
measures aggregate (STRICT/HYBRID/UNGROUNDED counts across N×Q
samples). Live fixtures (`tests/test_qa_quality_live.py`) gate on
specific known-good answers. When a change improves things the
bench number climbs AND every fixture passes; when something
regresses the bench falls AND named fixtures fail by name,
pointing at where the regression landed.
- **Self-heal beats retry.** When the model emits malformed output
(truncated JSON, trailing comma, partial key), repair the artifact
rather than re-running the LLM call. Self-healing preserves
whatever partial content the model produced and lands in the
unverified bucket honestly. Retrying spends another inference
round and may produce identical garbage. See
`_repair_truncated_json` in `aborist/qa/verify.py` for the
pattern: walk once tracking string state + bracket stack, close /
drop / balance at end-of-input. Conservative repairs only — never
insert content, never fabricate keys.
- **Name the failure → fix in code → re-bench → confirm.** Each
bench-driven commit should reference the specific failure mode it
addresses (e.g. "Apollo runaway: 3/3 → 2/3" or "JSON hallucinated
near-miss content-IDs on cross-doc relationships"). When the next
bench shows the named failure didn't budge, you fixed the wrong
thing. Name it again, try again.
- **Honest verdicts beat optimistic ones.** A change that drops
STRICT count by 5 but moves those 5 to HYBRID with real grounding
is a WIN — false-positive STRICTs are corruption. The bench grade
improves when the verifier reports closer to ground truth, not
closer to 100%. Trust HYBRID with smell-sidecar warnings over
STRICT-with-bogus-citation every time.
- **Per-question fixtures > marker-string assertions.** When
designing a live fixture, prefer entity-presence checks ("Burns
in the answer", "October or 1991") over byte-identical matching.
Hermes is non-deterministic; the right entity in the answer is
the gate, not the exact phrasing. When a fixture starts failing
intermittently, treat it as quality drift on that question shape
and tighten retrieval / verifier upstream rather than loosening
the test.
- **Old maps vs runtime maps — move authority toward runtime artifacts.**
Every base model carries old maps: trained patterns from the time
the weights froze. It "knows" how API X used to work, what JSON
shape Y used to take, which ritual citation phrase was popular.
Those priors collide with the present-day runtime — the actual
retrieved chunks, the current evidence map, the live policy hash,
the ID space the runtime built this minute. When the model and
the runtime disagree, the runtime wins. Engineering moves
authority OUT of the model's old map and INTO runtime artifacts:
- Pointer IDs (E1, E2, …) the runtime mints, so the model can't
fabricate a content-addressed hash that "looks right."
- Source spans the runtime interpolates by
`(chunk_root, offset_start, offset_end)`, so the model never
owns the quote text.
- Evidence maps the retrieval pipeline assembles per-query, so
the model can't self-supply context from training memory.
- Policy hashes that fold prompt + verifier + retrieval choices
into the cache_key, so old governance can't silently reuse
new-rule records.
- Hard checks run by the verifier, never the model self-grading.
When a defect smells like "model brought a stale map," the fix is
to make the runtime more authoritative, not to negotiate with the
model's prior. Hermes' content-addressed-evidence-id hallucination
(commit `bb8450d`, 2026-04-30) is the canonical case: the fix was
to swap the prompt-facing surface to runtime-minted pointer IDs
the model can't fabricate, not to add an "are you sure?" step.
- **Doc captures the journey, not just the end state.**
`docs/qa-modes-bench-2026-04-30.md` shows the day's progression
(baseline → retry → trim-and-verify → pointer-IDs → stop-sequence
→ self-heal). Each row is a named failure + fix + bench delta.
Future-you (or a new agent) can read the doc and see WHY each
commit landed, not just WHAT.