diff --git a/CLAUDE.md b/CLAUDE.md index 20c1514..7972366 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -101,9 +101,12 @@ make chain-check-shards # audit-chain break count per shard (0 = intact) Trichotomy across all paths: STRICT = every evidence unit (≥1) verifies. HYBRID = some verify, some don't. UNGROUNDED = no evidence or none verifies. Persisted on `providence_cache.audit_mode` + `verifier_method`; cache-hits return the stored mode. Never overclaim — STRICT is a verifiable claim, not a default. - **Trailing-citation strip.** `_strip_trailing_citation` peels a single trailing parenthetical at end-of-span (gated on a citation-cue word — `Source:`, `citing`, `see`, `ref`, `from` — OR a URL) before substring testing. Keeps `"...prose. (Source: https://...)"` from flunking just because the model appended a citation tail. Refuses to strip genuine prose parentheticals (no cue word, no URL). - **Verifier stays binary; falsifications carry soft signal.** No per-quote diagnosis fields on hard verifier output. `verify_quotes` returns evidence units + classification; the falsify+reclassify loop owns "why didn't this ground" for the operator, and `aborist inspect --cache-key X` is the read-only sidecar that classifies each unverified span (`verbatim_in_base` / `verbatim_in_raw_only` / `trailing_artifact` / `paraphrase` / `partial_paraphrase` / `no_overlap`) — sidecars never write to `providence_cache` or `audit_events`. Don't bolt confidence scores or partial-match indicators onto `verify.py`. +- **Deflection sidecar.** `aborist.qa.inspect.diagnose_deflection(question, answer)` detects topic-shift on adversarial-premise questions (Mars-BDFL pattern: 2026-04-30 'who is a benevolent dictator for life for mars?' returned STRICT with answer about Guido/Python — verifier did its job, but the user's question went unanswered). **Subject-anchor heuristic**: the LAST content token in the question (after stopword strip) is treated as the question's primary subject. If the subject anchor is missing from the answer, classify as `deflection` regardless of generic-vocabulary overlap. Returns `kind ∈ {deflection, partial_overlap, on_topic, no_question_tokens}`. Wired into `inspect_cache_key()` and per-row in `bench/qa_sweep.py` (summary table grows a `deflections` column tracking STRICT/HYBRID rows where subject anchor is missing). Sidecar-only — never feeds back into providence; deflection on a STRICT record means "substrate did the right thing structurally, but agent's question wasn't answered." +- **Claim-count ceiling (TOO_MANY_CLAIMS).** `policy["claim_lattice_max_claims_per_answer"]` caps the number of claims a single answer can emit, default `12`. Bench finding (york-england 2026-04-30): the prompt shape "tell me all there is to know about X" prompted Hermes to spam 26-59 claim-pointer pairs of which only 2-4 verified. Atomic-claim prompt rule (commit b5925c8) reduced the typical case to ~10 claims; the cap is defense in depth. Cap of 12 admits typical entity-list questions (5-7 dinosaurs, simpsons + pets) while flagging the runaway. Cap doesn't truncate — every claim still verifies so the operator sees the full evidence; the violation demotes STRICT → HYBRID via the existing violation-check path. Plumbed through both pointer (`verify_claim_lattice`) and JSON (`verify_claim_lattice_json`) verifiers. Folds into `governance_policy_hash` on change. - **Wikitext base prose.** `aborist/wikitext.py:to_base(raw)` converts MediaWiki wikitext → plain prose deterministically (mwparserfromhell-backed; pinned by `BASE_VERSION = "wikitext-base-v1"`). Applied **before the LLM call** in `aborist/qa/runner.py` and `aborist/qa/query.py` (gated on `policy["base_version"]`), and again inside `verify_quotes` so the verifier compares like-against-like. Both sides — model and verifier — see prose; the model can quote source paragraphs verbatim instead of escaping `[[wikilinks]]`, and Wikipedia chunks ship to Hermes with ~43% fewer tokens. `policy["base_version"]` lives in `DEFAULT_POLICY`/`DEFAULT_QUERY_POLICY` so it folds into `governance_policy_hash`; bumping `BASE_VERSION` invalidates every prior cache record's 8-dim cache_key on next lookup. Optional dep — installs without `mwparserfromhell` keep `_wikitext_to_base = None` and `policy["base_version"] = None`, leaving raw wikitext in both context and verifier (graceful fallback, no failure mode). - **Soft hash vs hard hash.** Hard = SHA-256 (commitments, proofs, cache_key). Soft = embeddings/TF-IDF/similarity (training, ranking, distillation candidate selection). Never mix — soft never enters proof path. -- **Claim-lattice-pointer mode (G0 / CTI Clause Lattice Intelligence).** `policy["answer_mode"] ∈ {"quote", "claim_lattice_pointer"}`, default `"quote"`. In pointer mode the runtime builds an evidence map from retrieved chunks; each `EvidenceObject` carries TWO ids — `pointer_id` (`E1`, `E2`, …, sequential, what the model sees) and `evidence_id` (sha256-derived `E########`, content-addressed, what the cache & run-DAG use). Hermes emits pointer-line prose (`Claim text. [E12]` per line); `aborist.qa.parse_claims.parse_pointer_claims` parses `(claim_text, pointer_ids[])`; `verify_claim_lattice` resolves pointers to `EvidenceObject`s and runs deterministic checks ONLY: parser succeeded, evidence_id resolves, source_role allowed, no manual quotes (any `"` char violates — strict), claim text non-empty. Soft signals (entailment, completeness, predicate compatibility, scope) stay sidecar; never enter the proof path. Renderer interpolates literal source spans at display time via spotlight excerpt (`_spotlight_excerpt` finds the first claim-content-token match in the cited span and centers a window on it; falls back to leading window when no token matches). Synthetic-elision-by-construction-impossible — the model never types the quote string. No iterative repair in pointer mode (one-shot benchmark discipline). Run-DAG grows from 7 stages to 9: `question / retrieval / evidence_map / prompt / raw_answer / parsed_claim_lattice / verify / render / final_label`. `policy["answer_mode"]` folds into `governance_policy_hash` so two modes write under different cache_keys and never alias. `make query Q="..."` defaults to `ANSWER_MODE=claim_lattice_pointer` so the testing harness exercises G0 by default; library `DEFAULT_POLICY` / `DEFAULT_QUERY_POLICY` stay `"quote"` so Python callers and unit tests aren't surprised. Within each retrieved source, chunks are ranked by query-token overlap before pointer-id assignment so the chunk most likely to textually support the question gets `E1` — counters Hermes-3-8B's lazy-anchor habit on doc-order-first chunks. +- **Three answer modes.** `policy["answer_mode"] ∈ {"quote", "claim_lattice_pointer", "claim_lattice"}`, default `"quote"`. The substrate exposes all three; agents pick by inference profile. `quote` for prose-shape models that emit verbatim citations inline. `claim_lattice_pointer` (G0 / CTI) for prose-distribution models like Hermes-3-8B that handle pointer-line bracketed tags well. `claim_lattice` (JSON) for grammar-constrained inference (vLLM `guided_json`, Claude/GPT-4 native JSON, Qwen 3.6 reasoner) — pairs with the `_lenient_json_parse` pre-parser that handles markdown fences / preamble / curly quotes / trailing commas / unbalanced brackets so non-strict JSON inference paths stay survivable. Bench evidence (2026-04-30, n=3, 24 questions on Hermes-3-8B): quote 0.47 strict-rate, pointer 0.36, JSON 0.49 — JSON wins on Hermes. Both lattice modes share `verifier_method="claim_lattice"` so the providence_cache CHECK constraint accepts both; downstream disambiguation lives in the `answer_mode` field on the run-DAG and (for JSON) in the `json_fixups` list on the verdict. Each mode folds into `governance_policy_hash` so different modes write under different cache_keys and never alias. +- **Claim-lattice-pointer mode (G0 / CTI Clause Lattice Intelligence).** In pointer mode the runtime builds an evidence map from retrieved chunks; each `EvidenceObject` carries TWO ids — `pointer_id` (`E1`, `E2`, …, sequential, what the model sees) and `evidence_id` (sha256-derived `E########`, content-addressed, what the cache & run-DAG use). Hermes emits pointer-line prose (`Claim text. [E12]` per line); `aborist.qa.parse_claims.parse_pointer_claims` parses `(claim_text, pointer_ids[])`; `verify_claim_lattice` resolves pointers to `EvidenceObject`s and runs deterministic checks ONLY: parser succeeded, evidence_id resolves, source_role allowed, no manual quotes (any `"` char violates — strict), claim text non-empty. Soft signals (entailment, completeness, predicate compatibility, scope) stay sidecar; never enter the proof path. Renderer interpolates literal source spans at display time via spotlight excerpt (`_spotlight_excerpt` finds the first claim-content-token match in the cited span and centers a window on it; falls back to leading window when no token matches). Synthetic-elision-by-construction-impossible — the model never types the quote string. No iterative repair in pointer mode (one-shot benchmark discipline). Run-DAG grows from 7 stages to 9: `question / retrieval / evidence_map / prompt / raw_answer / parsed_claim_lattice / verify / render / final_label`. `policy["answer_mode"]` folds into `governance_policy_hash` so two modes write under different cache_keys and never alias. `make query Q="..."` defaults to `ANSWER_MODE=claim_lattice_pointer` so the testing harness exercises G0 by default; library `DEFAULT_POLICY` / `DEFAULT_QUERY_POLICY` stay `"quote"` so Python callers and unit tests aren't surprised. Within each retrieved source, chunks are ranked by query-token overlap before pointer-id assignment so the chunk most likely to textually support the question gets `E1` — counters Hermes-3-8B's lazy-anchor habit on doc-order-first chunks. ## Live endpoints @@ -122,6 +125,7 @@ Multi-stage. Each stage exists because something earlier wasn't enough; revert a 6. **Stem-aware token matching.** `_stem_token_for_match`: trailing-`s` strip on tokens >4 chars (skip `ss`-enders). Possessive (`superman's` → `supermans` → `superman`) and plural (`girlfriends` → `girlfriend`) collapse onto bare-stem source mentions. Caught the 2026-04-29 *"who is supermans girlfriend?"* defect — pre-stem, the query admitted 7 unrelated `Girlfriends`-titled articles. 7. **Per-source context cap.** Each top-K hit gets at most `max_context_chars / top_k` chars before the global `char_budget` is consumed. Fox's 2026-04-29 catch: `List_of_Batman_comics` (80 KB+ bibliography) was monopolizing the entire 60 KB budget at hit #1, dropping every other source. 8. **Wikitext base prose** runs on the assembled context BEFORE the LLM call (`policy["base_version"]` gates it, see Wikitext base prose convention above). +9. **Template-phrase stopwords on the FTS5 MATCH.** `_FTS5_STOPWORDS` (search/fts5.py) and `_TITLE_STOPWORDS` (qa/query.py) must stay in sync — both filter `tell show describe explain summarize say give list find make please all there know everything anything something` plus standard English stopwords. The `all there know` additions (2026-04-30) close the york-england miss: pre-fix, "tell me all there is to know about york england?" expanded to `"all" AND "there" AND "know" AND "york" AND "england"` in AND-mode FTS5, which favored "X of England" articles that incidentally mention all five tokens over the actual `York` article. Retrieval-time stopwords only — they don't enter `cache_key` or `governance_policy_hash`. ## Hot path / gotchas