diff --git a/CLAUDE.md b/CLAUDE.md index da3adbf..cc2fce2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ Three layers stacked on one SQLite file: 1. **Surface** — ingested documents (Wikipedia dumps, HTML pages, anything with a URI). Chunked, Merkle-rooted, FTS5-indexed. 2. **Core** — distilled documents (haiku/keyword/equation-scale) Merkle-bound back to source surface(s) via per-chunk inclusion proofs in `derivations.proof_blob`. Recursive: cores derive from cores. The "planet toward center" compression. -3. **Providence cache** — Q&A records keyed on the v9.8 8-dim invariant. Each record carries an `audit_mode` set by the post-LLM faithfulness verifier (`aborist/qa/verify.py`): STRICT (every quoted claim verbatim-grounded), HYBRID (mixed source/emergent), UNGROUNDED (no verbatim grounding — purely emergent from training). +3. **Providence cache** — Q&A records keyed on the v9.8 8-dim invariant. Each record carries an `audit_mode` set by the post-LLM faithfulness verifier (`aborist/qa/verify.py`): STRICT (every claim verbatim-grounded or token-coverage paraphrase-grounded), HYBRID (mixed source/emergent), UNGROUNDED (no recoverable grounding — purely emergent from training, or genuine model refusal). `verifier_method` records which of four strategies — quote / span / entity / paraphrase — fired. ## Source papers (read first if confused) @@ -44,12 +44,13 @@ aborist/ ├── qa/ # Q&A: 8-dim cache_key + Merkle-bound answers │ ├── client.py # ChatClient + StubClient + OpenAICompat │ ├── keys.py # cache_key, question_hash, ... (pure functions) -│ ├── verify.py # layered verifier: quote → span → entity +│ ├── verify.py # layered verifier: quote → span → entity → paraphrase +│ ├── inspect.py # read-only sidecar: diagnose unverified spans │ └── runner.py # ask(): cache -> infer -> verify -> classify -> write ├── wikitext.py # to_base(): wikitext → plain prose (BASE_VERSION-pinned) └── cli.py # ingest / search / verify / stats / distill / # evict / rehydrate / ask / providence / emergent / - # reclassify / analyze + # reclassify / inspect / analyze ``` ## Build, test, run @@ -70,7 +71,7 @@ make chain-check-shards # audit-chain break count per shard (0 = intact) **Hygiene after any state-changing op** (table rebuild, reclassify run, governance hash bump, mass falsify): `make chain-check-shards` for a one-second sanity (every shard should report `0`), then `make analyze-shards` for the full spectrum + chain audit. Chain breaks are the loudest possible signal that something corrupted the audit log — catch them at the seam, not in production. -`aborist/cli.py` adds: `analyze`, `distill --kind {surface,core}`, `evict`, `rehydrate`, `ask`, `providence`, `emergent` (list UNGROUNDED/HYBRID records or `--aggregate` to rank unverified quotes — corpus-growth signal), `reclassify` (re-run the verifier against existing live providence records under the current entity policy; no LLM calls; `--compare` runs all four policies side-by-side, `--dry-run` reports without writing). `--batch-size` defaults to 200 docs/transaction; lower it only to bound memory peaks. +`aborist/cli.py` adds: `analyze`, `distill --kind {surface,core}`, `evict`, `rehydrate`, `ask`, `providence`, `emergent` (list UNGROUNDED/HYBRID records or `--aggregate` to rank unverified quotes — corpus-growth signal), `reclassify` (re-run the verifier against existing live providence records under the current entity policy; no LLM calls; `--compare` runs all four entity policies side-by-side, `--dry-run` reports without writing), `inspect --cache-key X` (read-only sidecar that pulls source chunks for one record & classifies each unverified span; never writes to providence_cache or audit_events). The `query` command human-renders by default; pass `--json` (or `JSON=1` to `make query`) for the raw record. `--batch-size` defaults to 200 docs/transaction; lower it only to bound memory peaks. ## Schema invariants (do not break) @@ -86,12 +87,15 @@ make chain-check-shards # audit-chain break count per shard (0 = intact) - **Chunker default = `tok-512-v1`.** Changing the default bumps `chunking_version` and **stales every prior cache record**. Add a new chunker as a new `name` instead. - **Canonicalization = `norm-v1`** (NFC, collapsed whitespace). Same rule. - **Schema = `v9.8.0`.** Same rule. -- **`audit_mode` is decided by the verifier, never asserted unconditionally.** Three layered strategies in `aborist/qa/verify.py`, tried in order; first to find evidence classifies the answer: +- **`question_hash` canonicalization** drops trailing punctuation (ASCII `.?!,;:`, CJK `?!。、`, ellipsis `…`) AND standalone English articles (`the`, `a`, `an`) after lowercase + NFC + whitespace-collapse. So `"who is X"`, `"who is X?"`, `"Who Is X."`, `"who is the X"`, `"who is an X?"` all hash to the same value. Tokens are matched as exact lowercase strings, so substrings like `"thesis"` (contains `the`) stay untouched. Conservative on purpose — only ASCII English articles. Bumping `_QUESTION_TRAILING_STRIP` or `_QUESTION_ARTICLE_STRIP` orphans prior records whose canonical question contained newly-stripped tokens; they live as history but won't be re-hit on lookup. +- **`audit_mode` is decided by the verifier, never asserted unconditionally.** Four layered strategies in `aborist/qa/verify.py`, tried in order; first to find evidence classifies the answer: 1. **quote** — model wrapped claims in double quotes per system prompt. Sequential pairing: 1st & 2nd `"`, 3rd & 4th, etc. (NOT regex pairing — that captures inter-pair prose as a phantom span when the model writes `"title" prose "quote"`). 2. **span** — bullet/sentence lines from the answer appear verbatim in context. Catches models that quote inline without `"..."` marks. 3. **entity** — multi-word proper-noun phrases appear verbatim in context. Gated by `entity_policy ∈ {strict, hybrid, drop, proximity}`. Default `proximity`: STRICT only when N=3 verified entities cluster within W=300 chars in source (cast list / infobox / roster). Otherwise HYBRID/UNGROUNDED. Distinguishes structural grounding from incidental mention. Lives in `DEFAULT_QUERY_POLICY["entity_policy"]` so any change bumps `governance_policy_hash`. + 4. **paraphrase** — soft fallback on prose-shaped spans when 1-3 fail. Token-coverage probe: ≥4-char content tokens minus an English stopword set (`from`, `with`, `would`, `which`, etc.) checked for presence in normalized base context. Promotes when coverage ≥ `DEFAULT_PARAPHRASE_COVERAGE` (0.85) with ≥ `DEFAULT_PARAPHRASE_MIN_TOKENS` (4) content tokens. Fires only on prose-shaped spans (≥2 lowercase content tokens via `_is_prose_span`) so lists of proper nouns flow to the entity strategy instead. Quote strategy deliberately gets NO paraphrase fallback — `"..."` asserts verbatim citation; paraphrasing inside quotes is a model error to flag, not auto-promote. Records carry `verifier_method='paraphrase'` so an auditor can tell soft-verified evidence from lexical-verbatim. Trichotomy across all paths: STRICT = every evidence unit (≥1) verifies. HYBRID = some verify, some don't. UNGROUNDED = no evidence or none verifies. Persisted on `providence_cache.audit_mode` + `verifier_method`; cache-hits return the stored mode. Never overclaim — STRICT is a verifiable claim, not a default. -- **Verifier stays binary; falsifications carry soft signal.** No per-quote diagnosis fields on hard verifier output. `verify_quotes` returns evidence units + classification; the falsify+reclassify loop owns "why didn't this ground" for the operator. Don't bolt confidence scores or partial-match indicators onto `verify.py`. +- **Trailing-citation strip.** `_strip_trailing_citation` peels a single trailing parenthetical at end-of-span (gated on a citation-cue word — `Source:`, `citing`, `see`, `ref`, `from` — OR a URL) before substring testing. Keeps `"...prose. (Source: https://...)"` from flunking just because the model appended a citation tail. Refuses to strip genuine prose parentheticals (no cue word, no URL). +- **Verifier stays binary; falsifications carry soft signal.** No per-quote diagnosis fields on hard verifier output. `verify_quotes` returns evidence units + classification; the falsify+reclassify loop owns "why didn't this ground" for the operator, and `aborist inspect --cache-key X` is the read-only sidecar that classifies each unverified span (`verbatim_in_base` / `verbatim_in_raw_only` / `trailing_artifact` / `paraphrase` / `partial_paraphrase` / `no_overlap`) — sidecars never write to `providence_cache` or `audit_events`. Don't bolt confidence scores or partial-match indicators onto `verify.py`. - **Wikitext base prose.** `aborist/wikitext.py:to_base(raw)` converts MediaWiki wikitext → plain prose deterministically (mwparserfromhell-backed; pinned by `BASE_VERSION = "wikitext-base-v1"`). Applied **before the LLM call** in `aborist/qa/runner.py` and `aborist/qa/query.py` (gated on `policy["base_version"]`), and again inside `verify_quotes` so the verifier compares like-against-like. Both sides — model and verifier — see prose; the model can quote source paragraphs verbatim instead of escaping `[[wikilinks]]`, and Wikipedia chunks ship to Hermes with ~43% fewer tokens. `policy["base_version"]` lives in `DEFAULT_POLICY`/`DEFAULT_QUERY_POLICY` so it folds into `governance_policy_hash`; bumping `BASE_VERSION` invalidates every prior cache record's 8-dim cache_key on next lookup. Optional dep — installs without `mwparserfromhell` keep `_wikitext_to_base = None` and `policy["base_version"] = None`, leaving raw wikitext in both context and verifier (graceful fallback, no failure mode). - **Soft hash vs hard hash.** Hard = SHA-256 (commitments, proofs, cache_key). Soft = embeddings/TF-IDF/similarity (training, ranking, distillation candidate selection). Never mix — soft never enters proof path.