docs: catch CLAUDE.md up to 4-strategy verifier, inspect sidecar, JSON knob
- Top-level providence summary now mentions 4 strategies (quote, span, entity, paraphrase) and that verifier_method records which fired. - audit_mode bullet expanded with strategy 4 (paraphrase): token-coverage ≥0.85 with ≥4 content tokens after stopword filter, fires only on prose-shaped spans (entity strategy still owns proper-noun lists). Note that quote strategy gets NO paraphrase fallback — verbatim citation is the contract. - Trailing-citation strip documented: peels (Source: ...)/URL tails before substring testing, refuses genuine prose parentheticals. - Sidecar invariant now points at aborist inspect --cache-key X: read-only, diagnoses each unverified span (verbatim_in_base / trailing_artifact / paraphrase / partial_paraphrase / no_overlap), never writes to providence_cache or audit_events. - File tree shows aborist/qa/inspect.py; CLI line lists `inspect`. Notes that `query` human-renders by default (--json or JSON=1). - New question_hash convention: trailing-punctuation strip + standalone English article strip so 'who is X', 'who is X?', 'who is the X' all hash to the same equivalence class. Whitepaper section 13.8 already current — fox shipped 13.8.5 (citation strip) and 13.8.6 (sidecar) plus the 4-strategy 13.8.1 rewrite.
This commit is contained in:
parent
3a9aa729c5
commit
8acdd6c884
1 changed files with 10 additions and 6 deletions
16
CLAUDE.md
16
CLAUDE.md
|
|
@ -12,7 +12,7 @@ Three layers stacked on one SQLite file:
|
|||
|
||||
1. **Surface** — ingested documents (Wikipedia dumps, HTML pages, anything with a URI). Chunked, Merkle-rooted, FTS5-indexed.
|
||||
2. **Core** — distilled documents (haiku/keyword/equation-scale) Merkle-bound back to source surface(s) via per-chunk inclusion proofs in `derivations.proof_blob`. Recursive: cores derive from cores. The "planet toward center" compression.
|
||||
3. **Providence cache** — Q&A records keyed on the v9.8 8-dim invariant. Each record carries an `audit_mode` set by the post-LLM faithfulness verifier (`aborist/qa/verify.py`): STRICT (every quoted claim verbatim-grounded), HYBRID (mixed source/emergent), UNGROUNDED (no verbatim grounding — purely emergent from training).
|
||||
3. **Providence cache** — Q&A records keyed on the v9.8 8-dim invariant. Each record carries an `audit_mode` set by the post-LLM faithfulness verifier (`aborist/qa/verify.py`): STRICT (every claim verbatim-grounded or token-coverage paraphrase-grounded), HYBRID (mixed source/emergent), UNGROUNDED (no recoverable grounding — purely emergent from training, or genuine model refusal). `verifier_method` records which of four strategies — quote / span / entity / paraphrase — fired.
|
||||
|
||||
## Source papers (read first if confused)
|
||||
|
||||
|
|
@ -44,12 +44,13 @@ aborist/
|
|||
├── qa/ # Q&A: 8-dim cache_key + Merkle-bound answers
|
||||
│ ├── client.py # ChatClient + StubClient + OpenAICompat
|
||||
│ ├── keys.py # cache_key, question_hash, ... (pure functions)
|
||||
│ ├── verify.py # layered verifier: quote → span → entity
|
||||
│ ├── verify.py # layered verifier: quote → span → entity → paraphrase
|
||||
│ ├── inspect.py # read-only sidecar: diagnose unverified spans
|
||||
│ └── runner.py # ask(): cache -> infer -> verify -> classify -> write
|
||||
├── wikitext.py # to_base(): wikitext → plain prose (BASE_VERSION-pinned)
|
||||
└── cli.py # ingest / search / verify / stats / distill /
|
||||
# evict / rehydrate / ask / providence / emergent /
|
||||
# reclassify / analyze
|
||||
# reclassify / inspect / analyze
|
||||
```
|
||||
|
||||
## Build, test, run
|
||||
|
|
@ -70,7 +71,7 @@ make chain-check-shards # audit-chain break count per shard (0 = intact)
|
|||
|
||||
**Hygiene after any state-changing op** (table rebuild, reclassify run, governance hash bump, mass falsify): `make chain-check-shards` for a one-second sanity (every shard should report `0`), then `make analyze-shards` for the full spectrum + chain audit. Chain breaks are the loudest possible signal that something corrupted the audit log — catch them at the seam, not in production.
|
||||
|
||||
`aborist/cli.py` adds: `analyze`, `distill --kind {surface,core}`, `evict`, `rehydrate`, `ask`, `providence`, `emergent` (list UNGROUNDED/HYBRID records or `--aggregate` to rank unverified quotes — corpus-growth signal), `reclassify` (re-run the verifier against existing live providence records under the current entity policy; no LLM calls; `--compare` runs all four policies side-by-side, `--dry-run` reports without writing). `--batch-size` defaults to 200 docs/transaction; lower it only to bound memory peaks.
|
||||
`aborist/cli.py` adds: `analyze`, `distill --kind {surface,core}`, `evict`, `rehydrate`, `ask`, `providence`, `emergent` (list UNGROUNDED/HYBRID records or `--aggregate` to rank unverified quotes — corpus-growth signal), `reclassify` (re-run the verifier against existing live providence records under the current entity policy; no LLM calls; `--compare` runs all four entity policies side-by-side, `--dry-run` reports without writing), `inspect --cache-key X` (read-only sidecar that pulls source chunks for one record & classifies each unverified span; never writes to providence_cache or audit_events). The `query` command human-renders by default; pass `--json` (or `JSON=1` to `make query`) for the raw record. `--batch-size` defaults to 200 docs/transaction; lower it only to bound memory peaks.
|
||||
|
||||
## Schema invariants (do not break)
|
||||
|
||||
|
|
@ -86,12 +87,15 @@ make chain-check-shards # audit-chain break count per shard (0 = intact)
|
|||
- **Chunker default = `tok-512-v1`.** Changing the default bumps `chunking_version` and **stales every prior cache record**. Add a new chunker as a new `name` instead.
|
||||
- **Canonicalization = `norm-v1`** (NFC, collapsed whitespace). Same rule.
|
||||
- **Schema = `v9.8.0`.** Same rule.
|
||||
- **`audit_mode` is decided by the verifier, never asserted unconditionally.** Three layered strategies in `aborist/qa/verify.py`, tried in order; first to find evidence classifies the answer:
|
||||
- **`question_hash` canonicalization** drops trailing punctuation (ASCII `.?!,;:`, CJK `?!。、`, ellipsis `…`) AND standalone English articles (`the`, `a`, `an`) after lowercase + NFC + whitespace-collapse. So `"who is X"`, `"who is X?"`, `"Who Is X."`, `"who is the X"`, `"who is an X?"` all hash to the same value. Tokens are matched as exact lowercase strings, so substrings like `"thesis"` (contains `the`) stay untouched. Conservative on purpose — only ASCII English articles. Bumping `_QUESTION_TRAILING_STRIP` or `_QUESTION_ARTICLE_STRIP` orphans prior records whose canonical question contained newly-stripped tokens; they live as history but won't be re-hit on lookup.
|
||||
- **`audit_mode` is decided by the verifier, never asserted unconditionally.** Four layered strategies in `aborist/qa/verify.py`, tried in order; first to find evidence classifies the answer:
|
||||
1. **quote** — model wrapped claims in double quotes per system prompt. Sequential pairing: 1st & 2nd `"`, 3rd & 4th, etc. (NOT regex pairing — that captures inter-pair prose as a phantom span when the model writes `"title" prose "quote"`).
|
||||
2. **span** — bullet/sentence lines from the answer appear verbatim in context. Catches models that quote inline without `"..."` marks.
|
||||
3. **entity** — multi-word proper-noun phrases appear verbatim in context. Gated by `entity_policy ∈ {strict, hybrid, drop, proximity}`. Default `proximity`: STRICT only when N=3 verified entities cluster within W=300 chars in source (cast list / infobox / roster). Otherwise HYBRID/UNGROUNDED. Distinguishes structural grounding from incidental mention. Lives in `DEFAULT_QUERY_POLICY["entity_policy"]` so any change bumps `governance_policy_hash`.
|
||||
4. **paraphrase** — soft fallback on prose-shaped spans when 1-3 fail. Token-coverage probe: ≥4-char content tokens minus an English stopword set (`from`, `with`, `would`, `which`, etc.) checked for presence in normalized base context. Promotes when coverage ≥ `DEFAULT_PARAPHRASE_COVERAGE` (0.85) with ≥ `DEFAULT_PARAPHRASE_MIN_TOKENS` (4) content tokens. Fires only on prose-shaped spans (≥2 lowercase content tokens via `_is_prose_span`) so lists of proper nouns flow to the entity strategy instead. Quote strategy deliberately gets NO paraphrase fallback — `"..."` asserts verbatim citation; paraphrasing inside quotes is a model error to flag, not auto-promote. Records carry `verifier_method='paraphrase'` so an auditor can tell soft-verified evidence from lexical-verbatim.
|
||||
Trichotomy across all paths: STRICT = every evidence unit (≥1) verifies. HYBRID = some verify, some don't. UNGROUNDED = no evidence or none verifies. Persisted on `providence_cache.audit_mode` + `verifier_method`; cache-hits return the stored mode. Never overclaim — STRICT is a verifiable claim, not a default.
|
||||
- **Verifier stays binary; falsifications carry soft signal.** No per-quote diagnosis fields on hard verifier output. `verify_quotes` returns evidence units + classification; the falsify+reclassify loop owns "why didn't this ground" for the operator. Don't bolt confidence scores or partial-match indicators onto `verify.py`.
|
||||
- **Trailing-citation strip.** `_strip_trailing_citation` peels a single trailing parenthetical at end-of-span (gated on a citation-cue word — `Source:`, `citing`, `see`, `ref`, `from` — OR a URL) before substring testing. Keeps `"...prose. (Source: https://...)"` from flunking just because the model appended a citation tail. Refuses to strip genuine prose parentheticals (no cue word, no URL).
|
||||
- **Verifier stays binary; falsifications carry soft signal.** No per-quote diagnosis fields on hard verifier output. `verify_quotes` returns evidence units + classification; the falsify+reclassify loop owns "why didn't this ground" for the operator, and `aborist inspect --cache-key X` is the read-only sidecar that classifies each unverified span (`verbatim_in_base` / `verbatim_in_raw_only` / `trailing_artifact` / `paraphrase` / `partial_paraphrase` / `no_overlap`) — sidecars never write to `providence_cache` or `audit_events`. Don't bolt confidence scores or partial-match indicators onto `verify.py`.
|
||||
- **Wikitext base prose.** `aborist/wikitext.py:to_base(raw)` converts MediaWiki wikitext → plain prose deterministically (mwparserfromhell-backed; pinned by `BASE_VERSION = "wikitext-base-v1"`). Applied **before the LLM call** in `aborist/qa/runner.py` and `aborist/qa/query.py` (gated on `policy["base_version"]`), and again inside `verify_quotes` so the verifier compares like-against-like. Both sides — model and verifier — see prose; the model can quote source paragraphs verbatim instead of escaping `[[wikilinks]]`, and Wikipedia chunks ship to Hermes with ~43% fewer tokens. `policy["base_version"]` lives in `DEFAULT_POLICY`/`DEFAULT_QUERY_POLICY` so it folds into `governance_policy_hash`; bumping `BASE_VERSION` invalidates every prior cache record's 8-dim cache_key on next lookup. Optional dep — installs without `mwparserfromhell` keep `_wikitext_to_base = None` and `policy["base_version"] = None`, leaving raw wikitext in both context and verifier (graceful fallback, no failure mode).
|
||||
- **Soft hash vs hard hash.** Hard = SHA-256 (commitments, proofs, cache_key). Soft = embeddings/TF-IDF/similarity (training, ranking, distillation candidate selection). Never mix — soft never enters proof path.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue