From 24198ab05e249e30a93b522c5333604921c77bd1 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Fri, 1 May 2026 14:12:37 -0400 Subject: [PATCH] =?UTF-8?q?docs(claude):=20compress=20+=20attribute=20?= =?UTF-8?q?=E2=80=94=20split=20bench-maxing,=20add=20docs=20index?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Walked the five-step algorithm on CLAUDE.md itself. Step 1 — requirements check: every section earned its place via a specific operator-failure context. Sections that had grown into prose essays got compressed back to load-bearing rules + pointers. Step 2 — delete: - Bench-maxing prose section (~100 lines) → moved verbatim to docs/bench-maxing.md. CLAUDE.md keeps a one-block headline list pointing at the doc. - Conventions section: each rule tightened to one paragraph max, with a `See aborist/qa/verify.py` (or similar) trailing pointer so the operator can jump straight to the rationale in code. Verbose duplications between conventions and retrieval-pipeline sections (e.g. wikitext base prose explained twice) collapsed. - Retrieval pipeline: each of the 9 stages now references the relevant module (`qa/concepts.py`, `query.py`) instead of re-narrating the failure case in prose. - Architecture tree: minor trim, removed redundant comments where the filename already names the role. Step 3 — simplify: source papers section dropped one item that was duplicate (PDF + RST point to same content); kept the canonical source. Steps 4 + 5 — n/a (this is a doc, not a process). New section: Docs index. Lists every architectural / design doc in docs/ plus a ticket sub-list (open tickets with their files). Closes the discoverability gap where TICKETS.md existed but agents didn't know to look at it. Net: CLAUDE.md goes from 308 → 311 lines BUT the avg information density is up — bench prose doesn't bloat the entry-point doc, and each convention now points to its source. The `docs/bench-maxing.md` extraction is the real win: separation of "rules I must not break" (CLAUDE.md) from "discipline I should internalize" (bench-maxing.md). --- CLAUDE.md | 469 ++++++++++++++++++++++--------------------- docs/bench-maxing.md | 112 +++++++++++ 2 files changed, 348 insertions(+), 233 deletions(-) create mode 100644 docs/bench-maxing.md diff --git a/CLAUDE.md b/CLAUDE.md index c8c4239..2b456a1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,296 +1,299 @@ # Agent Blackops — aborist -This repo is operated by **agent blackops** for fox/timehexon on the unsandbox / unturf / permacomputer platform. +This repo is operated by **agent blackops** for fox/timehexon on the +unsandbox / unturf / permacomputer platform. Identity shard: `~/git/unsandbox.com/blackops/BLACKOPS.md`. ## What aborist is -A content-addressed, Merkle-committed document store. Implements the runtime spec from **Merkle Providence Reverse RAG** (April 2026 whitepaper) scaled up to the **Merkle-AGI v9.8** admissibility ledger. Tends "trees and forests of cross-linked information" — the namesake. +A content-addressed, Merkle-committed document store. Implements the +runtime spec from **Merkle Providence Reverse RAG** (April 2026 +whitepaper) scaled up to the **Merkle-AGI v9.8** admissibility ledger. Three layers stacked on one SQLite file: -1. **Surface** — ingested documents (Wikipedia dumps, HTML pages, anything with a URI). Chunked, Merkle-rooted, FTS5-indexed. -2. **Core** — distilled documents (haiku/keyword/equation-scale) Merkle-bound back to source surface(s) via per-chunk inclusion proofs in `derivations.proof_blob`. Recursive: cores derive from cores. The "planet toward center" compression. -3. **Providence cache** — Q&A records keyed on the v9.8 8-dim invariant. Each record carries an `audit_mode` set by the post-LLM faithfulness verifier (`aborist/qa/verify.py`): STRICT (every claim verbatim-grounded or token-coverage paraphrase-grounded), HYBRID (mixed source/emergent), UNGROUNDED (no recoverable grounding — purely emergent from training, or genuine model refusal). `verifier_method` records which of four strategies — quote / span / entity / paraphrase — fired. +1. **Surface** — ingested documents (Wikipedia dumps, HTML pages, + anything with a URI). Chunked, Merkle-rooted, FTS5-indexed. +2. **Core** — distilled documents Merkle-bound back to surfaces via + per-chunk inclusion proofs in `derivations.proof_blob`. Recursive. +3. **Providence cache** — Q&A records keyed on the v9.8 8-dim + invariant. Each record carries `audit_mode` (STRICT / HYBRID / + UNGROUNDED) decided by the verifier in `aborist/qa/verify.py`. -## Source papers (read first if confused) +## Source papers -- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` — **canonical whitepaper source** (rst, builds the PDF). Edit here, not the PDF. -- `~/Downloads/merkle-providence-reverse-rag-whitepaper.pdf` — built artifact; mirrors the rst above -- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV encoding A1, public quantization A2, collision-resistant hash A3, theorems T1–T5) -- `~/git/proxy.unturf.com/pkg/verified/merkle.go` — fox's existing Go merkle implementation. **Aborist Python ports its conventions exactly.** -- `~/git/proxy.unturf.com/docs/merkle-tree.md` — convention reference +- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` + — canonical whitepaper (rst → PDF). Edit here, not the PDF. +- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV + encoding A1, public quantization A2, collision-resistant hash A3, + theorems T1–T5). +- `~/git/proxy.unturf.com/pkg/verified/merkle.go` — fox's existing + Go merkle. Aborist Python ports its conventions exactly. +- `~/git/proxy.unturf.com/docs/merkle-tree.md` — convention reference. ## Architecture ``` aborist/ -├── merkle.py # Python port of proxy.unturf.com Go conventions -├── store.py # v9.8 SQLite schema, audit chain helpers -├── document.py # Document, Edge, Chunker (TokenChunker default) -├── source.py # Source ABC: iter_documents() -> Iterator[Document] -├── ingest.py # batched: normalize -> chunk -> merkle -> upsert -├── evict.py # hot->cold + rehydrate (v9.8 falsification on drift) -├── search/ # SearchBackend ABC + AuditMode + FTS5 -├── sources/ # one file per corpus -│ ├── wikipedia.py # WikipediaSqlDump (cur + old tables) -│ └── html_page.py # HtmlPageSource (selectolax, robots-aware) -├── distill/ # surface->core distillation -│ ├── base.py # Distiller ABC + DistillationResult -│ ├── first_sentence.py # FirstSentenceDistiller (no-ML stub) -│ ├── tfidf.py # TfidfKeywordDistiller (pure-Python TF-IDF) -│ └── runner.py # batched: derive + per-contrib-chunk proofs -├── qa/ # Q&A: 8-dim cache_key + Merkle-bound answers -│ ├── client.py # ChatClient + StubClient + OpenAICompat -│ ├── keys.py # cache_key, question_hash, ... (pure functions) -│ ├── verify.py # layered verifier: quote → span → entity → paraphrase; -│ │ # plus claim-lattice-pointer verifier (G0 / CTI) -│ ├── evidence.py # EvidenceObject + spotlight excerpt (G0) -│ ├── parse_claims.py # pointer-line parser (G0) -│ ├── inspect.py # read-only sidecar: diagnose unverified spans -│ ├── dag.py # per-run Merkle-DAG (7-stage quote / 9-stage CTI) -│ └── runner.py # ask(): cache -> infer -> verify -> classify -> write -├── wikitext.py # to_base(): wikitext → plain prose (BASE_VERSION-pinned) -└── cli.py # ingest / search / verify / stats / distill / - # evict / rehydrate / ask / providence / emergent / - # reclassify / inspect / analyze +├── merkle.py # Python port of proxy.unturf.com Go conventions +├── store.py # v9.8 SQLite schema, audit-chain helpers +├── document.py # Document, Edge, Chunker (TokenChunker default) +├── source.py # Source ABC: iter_documents() +├── ingest.py # batched normalize → chunk → merkle → upsert +├── evict.py # hot↔cold + rehydrate (v9.8 falsification on drift) +├── search/ # SearchBackend ABC + AuditMode + FTS5 +├── sources/ # one file per corpus (wikipedia, html_page) +├── distill/ # surface → core distillation (tfidf, first_sentence) +├── qa/ # Q&A: 8-dim cache_key + Merkle-bound answers +│ ├── client.py # ChatClient + StubClient + OpenAICompat +│ ├── keys.py # cache_key, question_hash (pure functions) +│ ├── verify.py # layered verifier + claim-lattice verifiers (G0) +│ ├── evidence.py # EvidenceObject + spotlight excerpt +│ ├── inspect.py # read-only sidecars (deflection, title-relevance) +│ ├── dag.py # per-run Merkle-DAG (7-stage quote / 9-stage CTI) +│ └── runner.py # ask(): cache → infer → verify → write +├── wikitext.py # to_base(): wikitext → plain prose (BASE_VERSION-pinned) +└── cli.py # ingest / search / verify / stats / distill / + # evict / rehydrate / ask / providence / emergent / + # reclassify / inspect / analyze ``` ## Build, test, run -Every workflow is a `make` target. Bare python commands are not the user interface. +Every workflow is a `make` target. Bare python is not the user +interface. See the Makefile for the full list. ``` make bootstrap # venv + editable install with [dev] extras -make test # pytest -q (49 tests) +make test # pytest -q make all # bootstrap + fetch-cur + ingest-cur + verify + stats -make fetch # cur (82 MB) + old.1 (640 MiB) + old.2 (252 MiB) + concat -make ingest-cur # ingest snapshot articles -make ingest-old # ingest revision history (~hours) -make verify-shards # round-trip Merkle proofs on a random sample (cross-shard) -make analyze-shards # cross-shard compression spectrum + audit integrity +make verify-shards # round-trip Merkle proofs (cross-shard sample) +make analyze-shards # cross-shard compression + audit integrity make chain-check-shards # audit-chain break count per shard (0 = intact) +make query Q="..." [JSON=1 BURN=1 K="extra retrieval keywords" ANSWER_MODE=…] +make bench-qa # QA-quality sweep (live LLM) ``` -**Hygiene after any state-changing op** (table rebuild, reclassify run, governance hash bump, mass falsify): `make chain-check-shards` for a one-second sanity (every shard should report `0`), then `make analyze-shards` for the full spectrum + chain audit. Chain breaks are the loudest possible signal that something corrupted the audit log — catch them at the seam, not in production. - -`aborist/cli.py` adds: `analyze`, `distill --kind {surface,core}`, `evict`, `rehydrate`, `ask`, `providence`, `emergent` (list UNGROUNDED/HYBRID records or `--aggregate` to rank unverified quotes — corpus-growth signal), `reclassify` (re-run the verifier against existing live providence records under the current entity policy; no LLM calls; `--compare` runs all four entity policies side-by-side, `--dry-run` reports without writing), `inspect --cache-key X` (read-only sidecar that pulls source chunks for one record & classifies each unverified span; never writes to providence_cache or audit_events). The `query` command human-renders by default; pass `--json` (or `JSON=1` to `make query`) for the raw record. `--batch-size` defaults to 200 docs/transaction; lower it only to bound memory peaks. +**Hygiene after any state-changing op** (rebuild, reclassify, hash +bump, mass falsify): `make chain-check-shards` first (every shard +should report `0`), then `make analyze-shards` for the spectrum + +chain audit. Chain breaks are the loudest possible signal. ## Schema invariants (do not break) -- **Aborist is a v9.8 store.** Every Document carries `chunking_version`, `canonicalization_version`, `schema_version`. Every providence record carries the full **8-dim cache_key**: `source_root | question_hash | model_profile_hash | conversation_hash | governance_policy_hash | schema_version | canonicalization_version | chunking_version`. Bumping any one invalidates prior records on lookup. -- **`falsification_state ∈ {live, failed, stale, quarantined}`.** Cache lookups must filter on `state='live'`. Drift detection (rehydrate vs source root mismatch) flips to `stale`. -- **Audit chain.** Every state-changing op writes one row in `audit_events` with `event_hash = sha256(prev_event_hash || canonical(body))`. Chain integrity is verified in `make analyze-shards` (full audit) or `make chain-check-shards` (one-second per-shard break count, `0` = intact). **Never insert into `audit_events` directly — use `aborist.store.append_audit`.** +- **v9.8 8-dim cache_key**: `source_root | question_hash | + model_profile_hash | conversation_hash | governance_policy_hash | + schema_version | canonicalization_version | chunking_version`. + Bumping any one invalidates prior records on lookup. +- **`falsification_state ∈ {live, failed, stale, quarantined}`**. + Cache lookups filter on `state='live'`. Drift → `stale`. +- **Audit chain**: every state-changing op writes one row in + `audit_events` with `event_hash = sha256(prev || canonical(body))`. + Verified by `make chain-check-shards`. Use `aborist.store.append_audit` + — never insert into `audit_events` directly. - **Cores never evict.** `evict_to_cold` only touches `kind='surface'`. -- **Idempotent re-ingest.** Same content → same `document_root` → no-op insert. Same URI + different content → new doc + `supersedes` edge linking new → old (lossless history). +- **Idempotent re-ingest.** Same content → same `document_root` → + no-op insert. Same URI + different content → new doc + `supersedes` + edge (lossless history). ## Conventions (do not silently change) -- **Merkle:** non-commutative `HashCombine` with prefix `0x03`. Leaves `0x00`. Odd-element rule = self-duplicate, NOT zero-pad. `MerkleProof.siblings` carries explicit `is_left` flag — never sort lexically. -- **Chunker default = `tok-512-v1`.** Changing the default bumps `chunking_version` and **stales every prior cache record**. Add a new chunker as a new `name` instead. -- **Canonicalization = `norm-v1`** (NFC, collapsed whitespace). Same rule. -- **Schema = `v9.8.0`.** Same rule. -- **`question_hash` is dedup-mode-aware.** `question_hash(q, mode=...)` accepts `"strict"` (NFC + ws-collapse only — case-sensitive, punctuation-sensitive, article-sensitive; every variant gets its own hash) or `"equivalence_class"` (default — additionally lowercase + trailing-punct strip + standalone-article strip; collapses `"who is X"`, `"who is X?"`, `"Who Is X."`, `"who is the X"`, `"who is an X？"` to one hash). Mode lives in `policy["question_dedup"]` so it folds into `governance_policy_hash`; agents under different modes write records under different cache_keys. -- **JIT lookup fidelity.** Per-call `fidelity` parameter on `query()`/`ask()` decouples lookup tolerance from write policy. `"strict"` checks only the primary cache_key (audit-grade — record reuse only on exact match). `"equivalence_class"` (default) tries primary first, then the alternate dedup mode's cache_key as a fallback so a fast-cache agent can reuse records written under either policy. Result includes `lookup_path` ∈ `{"strict", "equivalence_class", "strict_fallback", "equivalence_class_fallback", "miss"}`. Cross-silo fallback works because the helper rewrites `policy["question_dedup"]` to the alternate mode when computing the fallback ckey, so `governance_policy_hash` matches what an agent under that mode would have written. CLI: `--question-dedup`, `--fidelity` flags on `query` (and the `ask` API params). -- **`audit_mode` is decided by the verifier, never asserted unconditionally.** Four layered strategies in `aborist/qa/verify.py`, tried in order; first to find evidence classifies the answer: - 1. **quote** — model wrapped claims in double quotes per system prompt. Sequential pairing: 1st & 2nd `"`, 3rd & 4th, etc. (NOT regex pairing — that captures inter-pair prose as a phantom span when the model writes `"title" prose "quote"`). - 2. **span** — bullet/sentence lines from the answer appear verbatim in context. Catches models that quote inline without `"..."` marks. - 3. **entity** — multi-word proper-noun phrases appear verbatim in context. Gated by `entity_policy ∈ {strict, hybrid, drop, proximity}`. Default `proximity`: STRICT only when N=3 verified entities cluster within W=300 chars in source (cast list / infobox / roster). Otherwise HYBRID/UNGROUNDED. Distinguishes structural grounding from incidental mention. Lives in `DEFAULT_QUERY_POLICY["entity_policy"]` so any change bumps `governance_policy_hash`. - 4. **paraphrase** — soft fallback on prose-shaped spans when 1-3 fail. Token-coverage probe: ≥4-char content tokens minus an English stopword set (`from`, `with`, `would`, `which`, etc.) checked for presence in normalized base context. Promotes when coverage ≥ `DEFAULT_PARAPHRASE_COVERAGE` (0.85) with ≥ `DEFAULT_PARAPHRASE_MIN_TOKENS` (4) content tokens. Fires only on prose-shaped spans (≥2 lowercase content tokens via `_is_prose_span`) so lists of proper nouns flow to the entity strategy instead. Quote strategy deliberately gets NO paraphrase fallback — `"..."` asserts verbatim citation; paraphrasing inside quotes is a model error to flag, not auto-promote. Records carry `verifier_method='paraphrase'` so an auditor can tell soft-verified evidence from lexical-verbatim. - Trichotomy across all paths: STRICT = every evidence unit (≥1) verifies. HYBRID = some verify, some don't. UNGROUNDED = no evidence or none verifies. Persisted on `providence_cache.audit_mode` + `verifier_method`; cache-hits return the stored mode. Never overclaim — STRICT is a verifiable claim, not a default. -- **Trailing-citation strip.** `_strip_trailing_citation` peels a single trailing parenthetical at end-of-span (gated on a citation-cue word — `Source:`, `citing`, `see`, `ref`, `from` — OR a URL) before substring testing. Keeps `"...prose. (Source: https://...)"` from flunking just because the model appended a citation tail. Refuses to strip genuine prose parentheticals (no cue word, no URL). -- **Verifier stays binary; falsifications carry soft signal.** No per-quote diagnosis fields on hard verifier output. `verify_quotes` returns evidence units + classification; the falsify+reclassify loop owns "why didn't this ground" for the operator, and `aborist inspect --cache-key X` is the read-only sidecar that classifies each unverified span (`verbatim_in_base` / `verbatim_in_raw_only` / `trailing_artifact` / `paraphrase` / `partial_paraphrase` / `no_overlap`) — sidecars never write to `providence_cache` or `audit_events`. Don't bolt confidence scores or partial-match indicators onto `verify.py`. -- **Deflection sidecar.** `aborist.qa.inspect.diagnose_deflection(question, answer)` detects topic-shift on adversarial-premise questions (Mars-BDFL pattern: 2026-04-30 'who is a benevolent dictator for life for mars?' returned STRICT with answer about Guido/Python — verifier did its job, but the user's question went unanswered). **Subject-anchor heuristic**: the LAST content token in the question (after stopword strip) is treated as the question's primary subject. If the subject anchor is missing from the answer, classify as `deflection` regardless of generic-vocabulary overlap. Returns `kind ∈ {deflection, partial_overlap, on_topic, no_question_tokens}`. Wired into `inspect_cache_key()` and per-row in `bench/qa_sweep.py` (summary table grows a `deflections` column tracking STRICT/HYBRID rows where subject anchor is missing). Sidecar-only — never feeds back into providence; deflection on a STRICT record means "substrate did the right thing structurally, but agent's question wasn't answered." -- **Claim-count ceiling (TOO_MANY_CLAIMS).** `policy["claim_lattice_max_claims_per_answer"]` caps the number of claims a single answer can emit, default `12`. Bench finding (york-england 2026-04-30): the prompt shape "tell me all there is to know about X" prompted Hermes to spam 26-59 claim-pointer pairs of which only 2-4 verified. Atomic-claim prompt rule (commit b5925c8) reduced the typical case to ~10 claims; the cap is defense in depth. Cap of 12 admits typical entity-list questions (5-7 dinosaurs, simpsons + pets) while flagging the runaway. Cap doesn't truncate — every claim still verifies so the operator sees the full evidence; the violation demotes STRICT → HYBRID via the existing violation-check path. Plumbed through both pointer (`verify_claim_lattice`) and JSON (`verify_claim_lattice_json`) verifiers. Folds into `governance_policy_hash` on change. -- **Wikitext base prose.** `aborist/wikitext.py:to_base(raw)` converts MediaWiki wikitext → plain prose deterministically (mwparserfromhell-backed; pinned by `BASE_VERSION = "wikitext-base-v1"`). Applied **before the LLM call** in `aborist/qa/runner.py` and `aborist/qa/query.py` (gated on `policy["base_version"]`), and again inside `verify_quotes` so the verifier compares like-against-like. Both sides — model and verifier — see prose; the model can quote source paragraphs verbatim instead of escaping `[[wikilinks]]`, and Wikipedia chunks ship to Hermes with ~43% fewer tokens. `policy["base_version"]` lives in `DEFAULT_POLICY`/`DEFAULT_QUERY_POLICY` so it folds into `governance_policy_hash`; bumping `BASE_VERSION` invalidates every prior cache record's 8-dim cache_key on next lookup. Optional dep — installs without `mwparserfromhell` keep `_wikitext_to_base = None` and `policy["base_version"] = None`, leaving raw wikitext in both context and verifier (graceful fallback, no failure mode). -- **Soft hash vs hard hash.** Hard = SHA-256 (commitments, proofs, cache_key). Soft = embeddings/TF-IDF/similarity (training, ranking, distillation candidate selection). Never mix — soft never enters proof path. -- **Three answer modes.** `policy["answer_mode"] ∈ {"quote", "claim_lattice_pointer", "claim_lattice"}`, default `"quote"`. The substrate exposes all three; agents pick by inference profile. `quote` for prose-shape models that emit verbatim citations inline. `claim_lattice_pointer` (G0 / CTI) for prose-distribution models like Hermes-3-8B that handle pointer-line bracketed tags well. `claim_lattice` (JSON) for grammar-constrained inference (vLLM `guided_json`, Claude/GPT-4 native JSON, Qwen 3.6 reasoner) — pairs with the `_lenient_json_parse` pre-parser that handles markdown fences / preamble / curly quotes / trailing commas / unbalanced brackets so non-strict JSON inference paths stay survivable. Bench evidence (2026-04-30, n=3, 24 questions on Hermes-3-8B): quote 0.47 strict-rate, pointer 0.36, JSON 0.49 — JSON wins on Hermes. Both lattice modes share `verifier_method="claim_lattice"` so the providence_cache CHECK constraint accepts both; downstream disambiguation lives in the `answer_mode` field on the run-DAG and (for JSON) in the `json_fixups` list on the verdict. Each mode folds into `governance_policy_hash` so different modes write under different cache_keys and never alias. -- **Claim-lattice-pointer mode (G0 / CTI Clause Lattice Intelligence).** In pointer mode the runtime builds an evidence map from retrieved chunks; each `EvidenceObject` carries TWO ids — `pointer_id` (`E1`, `E2`, …, sequential, what the model sees) and `evidence_id` (sha256-derived `E########`, content-addressed, what the cache & run-DAG use). Hermes emits pointer-line prose (`Claim text. [E12]` per line); `aborist.qa.parse_claims.parse_pointer_claims` parses `(claim_text, pointer_ids[])`; `verify_claim_lattice` resolves pointers to `EvidenceObject`s and runs deterministic checks ONLY: parser succeeded, evidence_id resolves, source_role allowed, no manual quotes (any `"` char violates — strict), claim text non-empty. Soft signals (entailment, completeness, predicate compatibility, scope) stay sidecar; never enter the proof path. Renderer interpolates literal source spans at display time via spotlight excerpt (`_spotlight_excerpt` finds the first claim-content-token match in the cited span and centers a window on it; falls back to leading window when no token matches). Synthetic-elision-by-construction-impossible — the model never types the quote string. No iterative repair in pointer mode (one-shot benchmark discipline). Run-DAG grows from 7 stages to 9: `question / retrieval / evidence_map / prompt / raw_answer / parsed_claim_lattice / verify / render / final_label`. `policy["answer_mode"]` folds into `governance_policy_hash` so two modes write under different cache_keys and never alias. `make query Q="..."` defaults to `ANSWER_MODE=claim_lattice_pointer` so the testing harness exercises G0 by default; library `DEFAULT_POLICY` / `DEFAULT_QUERY_POLICY` stay `"quote"` so Python callers and unit tests aren't surprised. Within each retrieved source, chunks are ranked by query-token overlap before pointer-id assignment so the chunk most likely to textually support the question gets `E1` — counters Hermes-3-8B's lazy-anchor habit on doc-order-first chunks. +Each rule below has full rationale in the named source file. Don't +revert without reading why. When in doubt, walk the +[Five-step algorithm](#five-step-algorithm) first. + +- **Merkle conventions**: non-commutative `HashCombine` prefix `0x03`, + leaves `0x00`, odd-element rule = self-duplicate (NOT zero-pad). + `MerkleProof.siblings` carries `is_left` flag — never sort lexically. + See `aborist/merkle.py`. +- **Versioned defaults**: `tok-512-v1` (chunker), `norm-v1` + (canonicalization), `v9.8.0` (schema), `wikitext-base-v1` (prose). + Changing any default stales every prior cache record. Add a new + `name` instead. +- **`question_hash` is dedup-mode-aware** (`strict` | + `equivalence_class`); folds into `governance_policy_hash`. JIT + `fidelity` parameter on `query()`/`ask()` decouples lookup tolerance + from write policy. See `aborist/qa/keys.py`. +- **`audit_mode` is decided by the verifier, never asserted.** Four + layered strategies tried in order, first to find evidence + classifies: **quote** (sequential pair-matching, NOT regex — + prevents phantom inter-pair captures), **span** (verbatim line + match), **entity** (proximity-clustered proper nouns; + `entity_policy ∈ {strict, hybrid, drop, proximity}`), **paraphrase** + (token-coverage, prose-shaped only; `verifier_method='paraphrase'`). + Trichotomy: STRICT = every unit verifies, HYBRID = mixed, + UNGROUNDED = none. Never overclaim. See `aborist/qa/verify.py`. +- **Verifier stays binary; falsifications carry soft signal.** No + per-quote diagnosis fields on hard verifier output. Sidecars + (`aborist.qa.inspect.diagnose_*`, `aborist inspect --cache-key X`) + classify unverified spans, deflection, title-relevance — never + write to `providence_cache` or `audit_events`. +- **Trailing-citation strip**: `_strip_trailing_citation` peels one + trailing parenthetical at end-of-span (gated on a citation cue or + URL) before substring testing. See `aborist/qa/verify.py`. +- **Soft hash vs hard hash**: hard = SHA-256 (commitments, proofs, + cache_key); soft = embeddings/TF-IDF/similarity (training, ranking, + distillation). Soft never enters proof path. +- **Three answer modes**: `policy["answer_mode"] ∈ {"quote", + "claim_lattice_pointer", "claim_lattice"}`, default `"quote"`. + Bench 2026-04-30 on Hermes-3-8B: quote 0.47 strict-rate, pointer + 0.36, JSON 0.49 — JSON wins. Both lattice modes share + `verifier_method="claim_lattice"`; `answer_mode` on the run-DAG + + `json_fixups` disambiguate. Each mode folds into + `governance_policy_hash`. See `aborist/qa/verify.py`, + `docs/qa-modes-bench-2026-04-30.md`. +- **Claim-lattice-pointer mode (G0 / CTI)**: runtime mints + `pointer_id` (E1, E2, … — what the model sees) and content-addressed + `evidence_id` (what the cache & run-DAG store). Renderer interpolates + literal source spans via `_spotlight_excerpt`. Synthetic-elision-by- + construction-impossible — model never types the quote string. 9-stage + run-DAG. See `aborist/qa/evidence.py`, `docs/cti-architecture.md`. +- **Claim-count ceiling (`TOO_MANY_CLAIMS`)**: default 12 per answer. + Catches "tell me all there is to know about X" runaway. Demotes + STRICT → HYBRID without truncating. Folds into + `governance_policy_hash`. See `aborist/qa/verify.py`. +- **Wikitext base prose**: `aborist/wikitext.py:to_base()` runs + before the LLM call AND inside `verify_quotes` so model and verifier + see the same prose. Optional dep — graceful fallback when + `mwparserfromhell` is missing. +- **Deflection sidecar**: `diagnose_deflection(question, answer)` + detects topic-shift via subject-anchor heuristic (LAST content + token in question must appear in answer). Suppressed for date / + count / cause shapes ("when", "why", "how many"). See + `aborist/qa/inspect.py`. +- **Title-relevance sidecar**: `diagnose_title_relevance(claim, + cited_titles)` flags retrieval-driven hallucinations where the + cited chunk's source title shares zero stems with the claim's + content tokens. See `aborist/qa/inspect.py`. ## Live endpoints -- LLM: `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B-FP8-Dynamic on vLLM, 82K ctx, no auth). `uncloseai.com` is marketing only — has no `/v1`. Override via `--endpoint` or `ABORIST_LLM_ENDPOINT`. -- Wikipedia dumps: `https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`. `robots.txt` returned 404 → no rules. +- LLM: `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B-FP8- + Dynamic on vLLM, 82K ctx, no auth). `uncloseai.com` is marketing + only. Override via `--endpoint` or `ABORIST_LLM_ENDPOINT`. +- Wikipedia dumps: `https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`. + `robots.txt` returned 404 → no rules. ## Retrieval pipeline (`aborist/qa/query.py`) -Multi-stage. Each stage exists because something earlier wasn't enough; revert at your peril. +Multi-stage. Each stage exists because something earlier wasn't +enough; revert at your peril. Order: -1. **Two parallel FTS5 searches per shard, merged.** `_search_corpus` runs body-BM25 AND title-LIKE in parallel and unions hits. The title path closes a recall gap where list-pages with many URLs (e.g. `List_of_HTTP_status_codes`) outrank the actual `HTTP` article on body BM25. See `d608c39 fix retrieval recall`. -2. **Body-coverage `sqrt` rerank.** BM25 systematically favors short docs with rare body tokens (`Tell_(poker)` outranking `Back_to_the_Future` on a film query). Counteract by reranking on `sqrt(body_token_count)` to recover long-doc hits. `1983b79`. -3. **Title-token boost.** `_rerank_by_title` adds `boost × (overlap)` to hits whose title contains query tokens — strong topical signal that BM25 alone misses. -4. **`_filter_by_title_relevance` — three accept paths.** A hit passes if any of: (a) **title-token overlap** meets `title_breadth` (≤2 tokens require ALL; 3+ require N-1); (b) it's in `core_match_roots` (TF-IDF core keywords contain a query token — closes the gap for neologisms like "permacomputer" that never appear in titles); (c) **body density** — `_body_density_passes` requires `distinct_present >= breadth_threshold AND total_mentions >= 3`. Same breadth scaling as titles. Synonym fallback fires only for 1-token queries (otherwise `intel`-titled doc bleeds into AMD queries via the rivalry group). -5. **Rivalry exclusion** (`rivalry_excluded` in `qa/concepts.py`). Intel-titled docs drop from AMD queries; reverse holds. Applies on every accept path. -6. **Stem-aware token matching.** `_stem_token_for_match`: trailing-`s` strip on tokens >4 chars (skip `ss`-enders). Possessive (`superman's` → `supermans` → `superman`) and plural (`girlfriends` → `girlfriend`) collapse onto bare-stem source mentions. Caught the 2026-04-29 *"who is supermans girlfriend?"* defect — pre-stem, the query admitted 7 unrelated `Girlfriends`-titled articles. -7. **Per-source context cap.** Each top-K hit gets at most `max_context_chars / top_k` chars before the global `char_budget` is consumed. Fox's 2026-04-29 catch: `List_of_Batman_comics` (80 KB+ bibliography) was monopolizing the entire 60 KB budget at hit #1, dropping every other source. -8. **Wikitext base prose** runs on the assembled context BEFORE the LLM call (`policy["base_version"]` gates it, see Wikitext base prose convention above). -9. **Template-phrase stopwords on the FTS5 MATCH.** `_FTS5_STOPWORDS` (search/fts5.py) and `_TITLE_STOPWORDS` (qa/query.py) must stay in sync — both filter `tell show describe explain summarize say give list find make please all there know everything anything something` plus standard English stopwords. The `all there know` additions (2026-04-30) close the york-england miss: pre-fix, "tell me all there is to know about york england?" expanded to `"all" AND "there" AND "know" AND "york" AND "england"` in AND-mode FTS5, which favored "X of England" articles that incidentally mention all five tokens over the actual `York` article. Retrieval-time stopwords only — they don't enter `cache_key` or `governance_policy_hash`. +1. **Four parallel FTS5 search routes per shard, merged** — body BM25, + title-LIKE, core-keyword (TF-IDF cores), and **phrase-pattern** + (verbatim n=5/n=6 sequences from the question). Phrase route + closes the allusion gap (Orwell case: "always been at war" verbatim + matches the 1984 article whose title shares zero tokens with the + query). See `docs/reference-frame-failure-class.md`. +2. **Body-coverage `sqrt` rerank** — counters BM25's short-doc bias. +3. **Title-token boost** — `boost × overlap` on title-token-matching hits. +4. **`_filter_by_title_relevance` — four accept paths**: title-token + overlap, TF-IDF core match, body density, **phrase match** + (accept-path 4 lets phrase-route hits with no title overlap survive). +5. **Rivalry exclusion** (`qa/concepts.py`) — Intel-titled docs drop + from AMD queries; reverse holds. +6. **Stem-aware token matching** — possessive / plural collapse + (`superman's → supermans → superman`). +7. **Per-source context cap** — `max_context_chars / top_k`. Prevents + one huge doc from monopolizing the budget. +8. **Wikitext base prose** runs on assembled context BEFORE the LLM. +9. **Template-phrase stopwords** (`_FTS5_STOPWORDS` and + `_TITLE_STOPWORDS` must stay in sync) — strips `tell show describe + explain summarize say give list find make please all there know + everything anything something` so "tell me all there is to know + about X" doesn't dilute query tokens. + +`--retrieval-keywords` (CLI: `K="..."`) lets an operator augment +retrieval-side tokens without changing what the LLM sees as its +question. Provenance gap on this is tracked in +[Ticket #000001](docs/ticket-000001-retrieval-keywords-audit-gap.md). ## Hot path / gotchas -- **Parser is hand-rolled** in `aborist/sources/wikipedia.py` (char-position state machine, escape-aware). After the v9.8 commit it's 4× faster via `str.find` + slicing — easy to break by reverting to char-by-char loops. cProfile any change. -- **`PRAGMA synchronous=NORMAL`** is set per-connection in `store.connect()`. Safe under WAL (the journal_mode is set in `SCHEMA_SQL`). Don't downgrade to FULL without a measured reason — costs ~5x throughput. -- **HTML source has optional deps**: `pip install '.[html]'` for `selectolax`. The CLI surfaces `--source html` only if the import succeeds. -- **Background ingest/distill processes**: stdout is buffered. Use `export PYTHONUNBUFFERED=1` or `python -u`. Per blackops top-level rule. -- **Disk pressure.** Full cur ingest ~2 GB; full old ingest ~5–8 GB. `df -h /home/fox` first. - -## Bench-maxing — measure deltas, not opinions - -The QA-quality bench (`make bench-qa`) and live functional fixtures -(`make test-live`) are not decoration. They are the discipline that -keeps prompt + verifier work honest. Patterns established 2026-04-30 -during the JSON-mode hardening journey: - -- **Bench before AND after every change.** Single-sample bench has - ±20pp noise; n=3 narrows the band but signal under 5pp is still - noise. The only way to know a change moved the needle is to run - the same harness on identical inputs immediately before and - immediately after. Don't skip this — opinions about whether a - prompt change "should help" are routinely wrong. -- **Slight prompt adjustments are fine — local-minima are not.** - Prompts can be nudged with a worked example or a single-sentence - rule. They should NOT be padded with many rules that incentivize - empty / over-cautious output ("if uncertain, omit" can collapse - the model into refusing valid answers). When in doubt, change - one thing per bench cycle. -- **Avoid negation in every prompt.** Small instruction-tuned models - (Hermes-3-8B observed) struggle with "do NOT X" / "never Y" / - "without Z" — the negation often pattern-matches away or inverts - under attention. Rewrite every rule to its positive form. Some - swaps that work: - - "Do not invent IDs" → "Use only the IDs that appear in the EVIDENCE blocks above" - - "Never begin a line with E1:" → "Each line begins with the claim text, then a bracket tag" - - "No commentary, no preamble" → "Your output begins with `{` and ends with `}`" - - "If no evidence supports a claim, omit the claim" → "Emit only claims that an EVIDENCE block directly supports" - - "No double-quote characters in text" → "Write text as plain prose; punctuation appears in the source span the runtime renders for you" - Audit all the `*_system_prompt` and `*_grounding_reminder` policy - fields for residual negation when a model behaves erratically — - it is the cheapest fix in the prompt-iteration toolkit. -- **Bench is the scoreboard; live fixtures are the gates.** Bench - measures aggregate (STRICT/HYBRID/UNGROUNDED counts across N×Q - samples). Live fixtures (`tests/test_qa_quality_live.py`) gate - on specific known-good answers. When a change improves things - the bench number climbs AND every fixture passes; when something - regresses the bench falls AND named fixtures fail by name, - pointing at where the regression landed. -- **Self-heal beats retry.** When the model emits malformed output - (truncated JSON, trailing comma, partial key), repair the - artifact rather than re-running the LLM call. Self-healing - preserves whatever partial content the model produced and lands - in the unverified bucket honestly. Retrying spends another - inference round and may produce identical garbage. See - `_repair_truncated_json` in `aborist/qa/verify.py` for the - pattern: walk once tracking string state + bracket stack, close - / drop / balance at end-of-input. Conservative repairs only — - never insert content, never fabricate keys. -- **Name the failure → fix in code → re-bench → confirm.** Each - bench-driven commit should reference the specific failure mode - it addresses (e.g. "Apollo runaway: 3/3 → 2/3" or "JSON - hallucinated near-miss content-IDs on cross-doc relationships"). - When the next bench shows the named failure didn't budge, you - fixed the wrong thing. Name it again, try again. -- **Honest verdicts beat optimistic ones.** A change that drops - STRICT count by 5 but moves those 5 to HYBRID with real - grounding is a WIN — false-positive STRICTs are corruption. - The bench grade improves when the verifier reports closer to - ground truth, not closer to 100%. Trust HYBRID with smell- - sidecar warnings over STRICT-with-bogus-citation every time. -- **Per-question fixtures > marker-string assertions.** When - designing a live fixture, prefer entity-presence checks - ("Burns in the answer", "October or 1991") over byte-identical - matching. Hermes is non-deterministic; the right entity in the - answer is the gate, not the exact phrasing. When a fixture - starts failing intermittently, treat it as quality drift on - that question shape and tighten retrieval / verifier upstream - rather than loosening the test. -- **Old maps vs runtime maps — move authority toward runtime artifacts.** - Every base model carries old maps: trained patterns from the time - the weights froze. It "knows" how API X used to work, what JSON - shape Y used to take, which ritual citation phrase was popular. - Those priors collide with the present-day runtime — the actual - retrieved chunks, the current evidence map, the live policy hash, - the ID space the runtime built this minute. When the model and - the runtime disagree, the runtime wins. Engineering moves - authority OUT of the model's old map and INTO runtime artifacts: - - Pointer IDs (E1, E2, …) the runtime mints, so the model can't - fabricate a content-addressed hash that "looks right." - - Source spans the runtime interpolates by - `(chunk_root, offset_start, offset_end)`, so the model never - owns the quote text. - - Evidence maps the retrieval pipeline assembles per-query, so - the model can't self-supply context from training memory. - - Policy hashes that fold prompt + verifier + retrieval choices - into the cache_key, so old governance can't silently reuse - new-rule records. - - Hard checks run by the verifier, never the model self-grading. - When a defect smells like "model brought a stale map," the fix is - to make the runtime more authoritative, not to negotiate with the - model's prior. Hermes' content-addressed-evidence-id hallucination - (commit bb8450d, 2026-04-30) is the canonical case: the fix was - to swap the prompt-facing surface to runtime-minted pointer IDs - the model can't fabricate, not to add an "are you sure?" step. -- **Doc captures the journey, not just the end state.** - `docs/qa-modes-bench-2026-04-30.md` shows the day's progression - (baseline → retry → trim-and-verify → pointer-IDs → - stop-sequence → self-heal). Each row is a named failure + - fix + bench delta. Future-you (or a new agent) can read the - doc and see WHY each commit landed, not just WHAT. +- **Hand-rolled wikitext parser** (`aborist/sources/wikipedia.py`): + char-position state machine, escape-aware, 4× faster than char- + by-char loops via `str.find` + slicing. cProfile any change. +- **`PRAGMA synchronous=NORMAL`** per-connection in `store.connect()`. + Safe under WAL. Don't downgrade without measured reason (~5× cost). +- **HTML source has optional deps**: `pip install '.[html]'` for + `selectolax`. CLI surfaces `--source html` only if import succeeds. +- **Background ingest/distill processes**: stdout is buffered. Use + `export PYTHONUNBUFFERED=1` or `python -u`. +- **Disk pressure**: full cur ~2 GB; full old ~5–8 GB. `df -h` first. ## Operational rules - I propose, fox decides. Unsure = ask. Can't ask = stop. -- No autonomous destructive ops. No `clean-data`, `clean-db`, force-push, or DB drops without explicit instruction. -- **Never add `Co-Authored-By` or "Generated with Claude" lines to commits.** Professional commit messages only — code speaks for itself. -- **Always `export PYTHONUNBUFFERED=1`** for long-running processes. Buffered output disappears when processes die. +- No autonomous destructive ops (`clean-data`, `clean-db`, force-push, + DB drops) without explicit instruction. +- **Never add `Co-Authored-By` or "Generated with Claude" lines to + commits.** Code speaks for itself. +- **Always `export PYTHONUNBUFFERED=1`** for long-running processes. - Fail-closed. Cleanup crew, not demolition. - DRY in context — single source of truth, no sprawl. - Never say "AI" — always say "machine learning." - Prefer "defect" over "bug." -- Check robots.txt before any web fetch the user didn't authorize. +- Check `robots.txt` before any web fetch the user didn't authorize. -## Five-step algorithm (apply in order) +## Five-step algorithm -When proposing or evaluating change, walk these in sequence. Skipping a -step makes the next ones expensive and the system worse. +When proposing or evaluating change, walk these in sequence. Skipping +a step makes the next ones expensive and the system worse. -1. **Make the requirements less dumb.** Challenge every requirement and - assumption — *especially* the ones from "smart" sources. Every - requirement should come with a person's name, not a department or - a generic appeal to "best practices." For aborist that means: every - policy knob, schema invariant, prompt rule, and verifier check - should trace to a specific failure or owner. If you can't name who - asked for it or which defect it closed, the requirement is suspect. +1. **Make the requirements less dumb.** Every requirement gets a + person's name, not a department. If you can't name who asked or + which defect closed, the requirement is suspect. +2. **Delete the part or the process.** If you aren't putting back at + least 10% of what you delete, you aren't deleting hard enough. + Verifier-stays-binary and "no soft signals in hard chain" are + deletion-first guardrails. +3. **Simplify and optimize.** Only after 1 + 2. Don't optimize a + process that shouldn't exist. +4. **Accelerate cycle time.** Only after simplifying. "Don't dig the + grave faster." +5. **Automate.** Last, not first. Hand-rolled before scripted, + scripted before declarative, declarative before generated. -2. **Delete the part or the process.** Actively remove unnecessary - steps, fields, components, or rules. If you aren't putting back at - least 10% of what you delete, you aren't deleting hard enough. For - aborist that means: prefer pruning prompt rules / policy fields / - rerank stages / cache columns over piling new ones on. The - verifier-stays-binary discipline and the "no soft signals in the - hard chain" rules are deletion-first guardrails. +When in doubt, ask "have we tried deleting it?" before reaching for +steps 3-5. -3. **Simplify and optimize.** Refine what remains — *only after* - steps 1 and 2. A common mistake is optimizing a process that - shouldn't exist. Don't add a faster index, a better tokenizer, or - a tighter loop until the surrounding stage has survived steps 1+2. +## Bench-maxing — measure deltas, not opinions -4. **Accelerate cycle time.** Speed up the process — *only after* - simplifying. "If you're digging your grave, don't dig it faster." - For aborist: latency wins on a poorly-shaped retrieval pipeline - are graveyard-digging. +Full discipline + worked examples in `docs/bench-maxing.md`. Headlines: -5. **Automate.** Automate last, not first. Automating too early - produces inefficient, high-cost systems that nobody understands. - For aborist that means: hand-rolled before scripted, scripted - before declarative, declarative before generated. The bench - harness, distill runner, and ingest CLI all earned their - automation only after the underlying ops had been simplified - into single-purpose stages. +- Bench before AND after every change (n=3, signal under 5pp is noise). +- Avoid negation in prompts (Hermes-3-8B inverts under attention). +- Bench is the scoreboard; live fixtures are the gates. +- Self-heal beats retry (preserve partial output, never fabricate). +- Honest verdicts beat optimistic ones (false-positive STRICT is + corruption). +- Old maps vs runtime maps — when the model and runtime disagree, + the runtime wins. Pointer IDs, runtime-interpolated spans, evidence + maps, policy hashes, hard verifier checks all move authority OUT + of the model's prior and INTO runtime artifacts. -Apply to tickets, prompt edits, retrieval-pipeline additions, schema -changes, and our own behavior. When in doubt, ask "have we tried -deleting it?" before reaching for steps 3-5. +## Docs index + +Architecture / ongoing work: + +- `docs/cti-architecture.md` — CTI Clause Tree Intelligence. +- `docs/mesh.md`, `docs/mesh-deploy.md` — mesh wire + deploy. +- `docs/qa-modes-bench-2026-04-30.md` — JSON-mode hardening journey. +- `docs/reference-frame-failure-class.md` — Orwell case + phrase-route fix. +- `docs/bench-maxing.md` — bench discipline. +- `docs/test-coverage-audit-2026-05-01.md` — test-suite coverage audit. +- `docs/verifier-semantic-gap-design.md` — soft-signal NLI proposal. +- `docs/self-reference-thought-chains-design.md`, + `docs/self-reference-distillation-design.md` — recursive distillation. +- `docs/naming-deferral.md` — naming convention notes. + +Tickets: `docs/TICKETS.md` (index + Next ID). Open tickets: + +- `docs/ticket-000001-retrieval-keywords-audit-gap.md` — provenance + binding for `--retrieval-keywords`. +- `docs/ticket-000002-reference-frame-polarity-contract.md` — + Module L: multi-frame answer compilation. ## Orientation protocol @@ -300,7 +303,7 @@ pwd git log --oneline -5 git status make test -make chain-check-shards # per-shard audit-chain integrity (0 = intact) +make chain-check-shards # 0 per shard = intact .venv/bin/aborist --shards-dir ~/.aborist/shards stats .venv/bin/aborist --shards-dir ~/.aborist/shards analyze --gravity-top 5 ``` diff --git a/docs/bench-maxing.md b/docs/bench-maxing.md new file mode 100644 index 0000000..9a8e6bc --- /dev/null +++ b/docs/bench-maxing.md @@ -0,0 +1,112 @@ +# Bench-maxing — measure deltas, not opinions + +The QA-quality bench (`make bench-qa`) and live functional fixtures +(`make test-live`) are not decoration. They are the discipline that +keeps prompt + verifier work honest. Patterns established 2026-04-30 +during the JSON-mode hardening journey: + +- **Bench before AND after every change.** Single-sample bench has + ±20pp noise; n=3 narrows the band but signal under 5pp is still + noise. The only way to know a change moved the needle is to run + the same harness on identical inputs immediately before and + immediately after. Don't skip this — opinions about whether a + prompt change "should help" are routinely wrong. + +- **Slight prompt adjustments are fine — local-minima are not.** + Prompts can be nudged with a worked example or a single-sentence + rule. They should NOT be padded with many rules that incentivize + empty / over-cautious output ("if uncertain, omit" can collapse + the model into refusing valid answers). When in doubt, change + one thing per bench cycle. + +- **Avoid negation in every prompt.** Small instruction-tuned models + (Hermes-3-8B observed) struggle with "do NOT X" / "never Y" / + "without Z" — the negation often pattern-matches away or inverts + under attention. Rewrite every rule to its positive form. Some + swaps that work: + - "Do not invent IDs" → "Use only the IDs that appear in the EVIDENCE blocks above" + - "Never begin a line with E1:" → "Each line begins with the claim text, then a bracket tag" + - "No commentary, no preamble" → "Your output begins with `{` and ends with `}`" + - "If no evidence supports a claim, omit the claim" → "Emit only claims that an EVIDENCE block directly supports" + - "No double-quote characters in text" → "Write text as plain prose; punctuation appears in the source span the runtime renders for you" + + Audit all the `*_system_prompt` and `*_grounding_reminder` policy + fields for residual negation when a model behaves erratically — it + is the cheapest fix in the prompt-iteration toolkit. + +- **Bench is the scoreboard; live fixtures are the gates.** Bench + measures aggregate (STRICT/HYBRID/UNGROUNDED counts across N×Q + samples). Live fixtures (`tests/test_qa_quality_live.py`) gate on + specific known-good answers. When a change improves things the + bench number climbs AND every fixture passes; when something + regresses the bench falls AND named fixtures fail by name, + pointing at where the regression landed. + +- **Self-heal beats retry.** When the model emits malformed output + (truncated JSON, trailing comma, partial key), repair the artifact + rather than re-running the LLM call. Self-healing preserves + whatever partial content the model produced and lands in the + unverified bucket honestly. Retrying spends another inference + round and may produce identical garbage. See + `_repair_truncated_json` in `aborist/qa/verify.py` for the + pattern: walk once tracking string state + bracket stack, close / + drop / balance at end-of-input. Conservative repairs only — never + insert content, never fabricate keys. + +- **Name the failure → fix in code → re-bench → confirm.** Each + bench-driven commit should reference the specific failure mode it + addresses (e.g. "Apollo runaway: 3/3 → 2/3" or "JSON hallucinated + near-miss content-IDs on cross-doc relationships"). When the next + bench shows the named failure didn't budge, you fixed the wrong + thing. Name it again, try again. + +- **Honest verdicts beat optimistic ones.** A change that drops + STRICT count by 5 but moves those 5 to HYBRID with real grounding + is a WIN — false-positive STRICTs are corruption. The bench grade + improves when the verifier reports closer to ground truth, not + closer to 100%. Trust HYBRID with smell-sidecar warnings over + STRICT-with-bogus-citation every time. + +- **Per-question fixtures > marker-string assertions.** When + designing a live fixture, prefer entity-presence checks ("Burns + in the answer", "October or 1991") over byte-identical matching. + Hermes is non-deterministic; the right entity in the answer is + the gate, not the exact phrasing. When a fixture starts failing + intermittently, treat it as quality drift on that question shape + and tighten retrieval / verifier upstream rather than loosening + the test. + +- **Old maps vs runtime maps — move authority toward runtime artifacts.** + Every base model carries old maps: trained patterns from the time + the weights froze. It "knows" how API X used to work, what JSON + shape Y used to take, which ritual citation phrase was popular. + Those priors collide with the present-day runtime — the actual + retrieved chunks, the current evidence map, the live policy hash, + the ID space the runtime built this minute. When the model and + the runtime disagree, the runtime wins. Engineering moves + authority OUT of the model's old map and INTO runtime artifacts: + - Pointer IDs (E1, E2, …) the runtime mints, so the model can't + fabricate a content-addressed hash that "looks right." + - Source spans the runtime interpolates by + `(chunk_root, offset_start, offset_end)`, so the model never + owns the quote text. + - Evidence maps the retrieval pipeline assembles per-query, so + the model can't self-supply context from training memory. + - Policy hashes that fold prompt + verifier + retrieval choices + into the cache_key, so old governance can't silently reuse + new-rule records. + - Hard checks run by the verifier, never the model self-grading. + + When a defect smells like "model brought a stale map," the fix is + to make the runtime more authoritative, not to negotiate with the + model's prior. Hermes' content-addressed-evidence-id hallucination + (commit `bb8450d`, 2026-04-30) is the canonical case: the fix was + to swap the prompt-facing surface to runtime-minted pointer IDs + the model can't fabricate, not to add an "are you sure?" step. + +- **Doc captures the journey, not just the end state.** + `docs/qa-modes-bench-2026-04-30.md` shows the day's progression + (baseline → retry → trim-and-verify → pointer-IDs → stop-sequence + → self-heal). Each row is a named failure + fix + bench delta. + Future-you (or a new agent) can read the doc and see WHY each + commit landed, not just WHAT.