From a08b9f618fca93c6f7e9f595955cc266a1ff2ffd Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Tue, 28 Apr 2026 17:30:14 -0400 Subject: [PATCH] docs: catch CLAUDE.md up to wikitext-before-LLM, hygiene targets, UNGROUNDED MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Wikitext base prose section now reflects reality: to_base() runs in two places (before LLM call in query.py/runner.py AND inside verify_quotes) gated by policy["base_version"]. Both sides see prose; Hermes can quote source paragraphs verbatim instead of escaping [[wikilinks]], and Wikipedia chunks ship with ~43% fewer tokens. - Build/test block now points at the real targets: verify-shards, analyze-shards, chain-check-shards. The bare 'make analyze' target doesn't exist — only the cross-shard variants do. - New "Hygiene after any state-changing op" callout: chain-check-shards for one-second per-shard break count, analyze-shards for full audit. - Audit-chain invariant block points at the same targets. - Orientation protocol updated: --shards-dir replaces --db (the latter refers to a single-DB setup that's no longer the canonical path), and chain-check-shards is added to the per-session ritual. --- CLAUDE.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 6192038..da3adbf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -63,17 +63,20 @@ make all # bootstrap + fetch-cur + ingest-cur + verify + stats make fetch # cur (82 MB) + old.1 (640 MiB) + old.2 (252 MiB) + concat make ingest-cur # ingest snapshot articles make ingest-old # ingest revision history (~hours) -make verify # round-trip Merkle proofs on a random sample -make analyze # compression spectrum + audit chain integrity +make verify-shards # round-trip Merkle proofs on a random sample (cross-shard) +make analyze-shards # cross-shard compression spectrum + audit integrity +make chain-check-shards # audit-chain break count per shard (0 = intact) ``` +**Hygiene after any state-changing op** (table rebuild, reclassify run, governance hash bump, mass falsify): `make chain-check-shards` for a one-second sanity (every shard should report `0`), then `make analyze-shards` for the full spectrum + chain audit. Chain breaks are the loudest possible signal that something corrupted the audit log — catch them at the seam, not in production. + `aborist/cli.py` adds: `analyze`, `distill --kind {surface,core}`, `evict`, `rehydrate`, `ask`, `providence`, `emergent` (list UNGROUNDED/HYBRID records or `--aggregate` to rank unverified quotes — corpus-growth signal), `reclassify` (re-run the verifier against existing live providence records under the current entity policy; no LLM calls; `--compare` runs all four policies side-by-side, `--dry-run` reports without writing). `--batch-size` defaults to 200 docs/transaction; lower it only to bound memory peaks. ## Schema invariants (do not break) - **Aborist is a v9.8 store.** Every Document carries `chunking_version`, `canonicalization_version`, `schema_version`. Every providence record carries the full **8-dim cache_key**: `source_root | question_hash | model_profile_hash | conversation_hash | governance_policy_hash | schema_version | canonicalization_version | chunking_version`. Bumping any one invalidates prior records on lookup. - **`falsification_state ∈ {live, failed, stale, quarantined}`.** Cache lookups must filter on `state='live'`. Drift detection (rehydrate vs source root mismatch) flips to `stale`. -- **Audit chain.** Every state-changing op writes one row in `audit_events` with `event_hash = sha256(prev_event_hash || canonical(body))`. Chain integrity is verified in `make analyze`. **Never insert into `audit_events` directly — use `aborist.store.append_audit`.** +- **Audit chain.** Every state-changing op writes one row in `audit_events` with `event_hash = sha256(prev_event_hash || canonical(body))`. Chain integrity is verified in `make analyze-shards` (full audit) or `make chain-check-shards` (one-second per-shard break count, `0` = intact). **Never insert into `audit_events` directly — use `aborist.store.append_audit`.** - **Cores never evict.** `evict_to_cold` only touches `kind='surface'`. - **Idempotent re-ingest.** Same content → same `document_root` → no-op insert. Same URI + different content → new doc + `supersedes` edge linking new → old (lossless history). @@ -89,7 +92,7 @@ make analyze # compression spectrum + audit chain integrity 3. **entity** — multi-word proper-noun phrases appear verbatim in context. Gated by `entity_policy ∈ {strict, hybrid, drop, proximity}`. Default `proximity`: STRICT only when N=3 verified entities cluster within W=300 chars in source (cast list / infobox / roster). Otherwise HYBRID/UNGROUNDED. Distinguishes structural grounding from incidental mention. Lives in `DEFAULT_QUERY_POLICY["entity_policy"]` so any change bumps `governance_policy_hash`. Trichotomy across all paths: STRICT = every evidence unit (≥1) verifies. HYBRID = some verify, some don't. UNGROUNDED = no evidence or none verifies. Persisted on `providence_cache.audit_mode` + `verifier_method`; cache-hits return the stored mode. Never overclaim — STRICT is a verifiable claim, not a default. - **Verifier stays binary; falsifications carry soft signal.** No per-quote diagnosis fields on hard verifier output. `verify_quotes` returns evidence units + classification; the falsify+reclassify loop owns "why didn't this ground" for the operator. Don't bolt confidence scores or partial-match indicators onto `verify.py`. -- **Wikitext context strip.** `aborist/wikitext.py:to_base(raw)` converts MediaWiki wikitext → plain prose deterministically (mwparserfromhell-backed; pinned by `BASE_VERSION`). Applied to context inside `verify_quotes` before substring tests so a model's clean prose ("Stephen King") matches against `[[Stephen King]]`-style source markup. Optional dep — falls back to identity if mwparserfromhell isn't installed. Bumping `BASE_VERSION` should fold into `governance_policy_hash` via `policy["base_version"]` to invalidate cache. +- **Wikitext base prose.** `aborist/wikitext.py:to_base(raw)` converts MediaWiki wikitext → plain prose deterministically (mwparserfromhell-backed; pinned by `BASE_VERSION = "wikitext-base-v1"`). Applied **before the LLM call** in `aborist/qa/runner.py` and `aborist/qa/query.py` (gated on `policy["base_version"]`), and again inside `verify_quotes` so the verifier compares like-against-like. Both sides — model and verifier — see prose; the model can quote source paragraphs verbatim instead of escaping `[[wikilinks]]`, and Wikipedia chunks ship to Hermes with ~43% fewer tokens. `policy["base_version"]` lives in `DEFAULT_POLICY`/`DEFAULT_QUERY_POLICY` so it folds into `governance_policy_hash`; bumping `BASE_VERSION` invalidates every prior cache record's 8-dim cache_key on next lookup. Optional dep — installs without `mwparserfromhell` keep `_wikitext_to_base = None` and `policy["base_version"] = None`, leaving raw wikitext in both context and verifier (graceful fallback, no failure mode). - **Soft hash vs hard hash.** Hard = SHA-256 (commitments, proofs, cache_key). Soft = embeddings/TF-IDF/similarity (training, ranking, distillation candidate selection). Never mix — soft never enters proof path. ## Live endpoints @@ -125,8 +128,9 @@ pwd git log --oneline -5 git status make test -.venv/bin/aborist --db ~/.aborist/aborist.db stats -.venv/bin/aborist --db ~/.aborist/aborist.db analyze --gravity-top 5 +make chain-check-shards # per-shard audit-chain integrity (0 = intact) +.venv/bin/aborist --shards-dir ~/.aborist/shards stats +.venv/bin/aborist --shards-dir ~/.aborist/shards analyze --gravity-top 5 ``` Then ask fox what the mission is.