From b5f7c93fca466d7657b989c556e918c24dd227d3 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Fri, 1 May 2026 16:20:07 -0400 Subject: [PATCH] docs(ticket-000003): anchor-class warrant generalization (Directive D6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Today's warrant-lite (aborist/qa/warrant.py) covers two anchor classes: proper-noun (relation-shape questions) and date (any claim with a 4-digit year). Three failure shapes pass the existing seven hard checks unchecked: - entity-list: 'name the simpsons family + pets' — claim enumerates entities, but the warrant doesn't demand all named entities appear in some cited span. - count: 'how many wives did henry the eighth have?' — claim says 'six', but the warrant doesn't demand the count token appear in some cited span. - why-cause: 'why did the titanic sink?' — claim names a cause noun (iceberg, asteroid, propaganda), but the warrant doesn't demand the cause token appear in some cited span. Ticket proposes per-shape anchor extractors dispatched by question- shape detector (existing _question_is_relation_shape pattern). Conservative scope on cause anchors: gate by why-shape only since lowercase common-noun extraction has higher false-positive risk. Folds into governance_policy_hash via per-class policy dict so an operator can disable individual classes per-corpus when they over-fire. Out of scope per docs/naming-deferral.md: typed-contract framework (per-question-type rule libraries). General anchor-class primitive catches the failures listed without per-type rule books; typed contracts earn slots only when bench evidence shows the general primitive misses cases. Forecast cost ~3-4 hours; risk medium (false-positive risk on why-shape cause-noun extraction). Closes the lazy-anchor gap on the question shapes the bench question set already exercises. --- docs/ticket-000003-anchor-class-warrant.md | 261 +++++++++++++++++++++ 1 file changed, 261 insertions(+) create mode 100644 docs/ticket-000003-anchor-class-warrant.md diff --git a/docs/ticket-000003-anchor-class-warrant.md b/docs/ticket-000003-anchor-class-warrant.md new file mode 100644 index 0000000..8954d0a --- /dev/null +++ b/docs/ticket-000003-anchor-class-warrant.md @@ -0,0 +1,261 @@ +# Ticket #000003 — Anchor-class warrant generalization (Module H+) + +**Status:** open · awaiting go/no-go +**Opened:** 2026-05-01 +**Directive:** [D6](seven-point-program.md) — General anchor-class +warrant before semantic NLI. +**Scope:** Generalize the warrant-lite hard check (Rule 7 in +`verify_claim_lattice`) beyond relation-shape and date-shape anchors +to cover entity-list, why-cause, count, and how-much shapes. Pin the +broader class of lazy-anchor failures structurally before any NLI +substrate exists in the proof path. +**Audience:** fox + future blackops shifts. +**Hard constraint:** the warrant check stays **lexical and binary**. +No NLI, no embeddings, no model self-grading. Soft signals continue +to live in the sidecar layer (`docs/verifier-semantic-gap-design.md` +covers that path separately). + +--- + +## 1. Problem statement + +Today's warrant-lite (`aborist/qa/warrant.py:warrant_check`) covers +two anchor classes: + +- **Proper-noun anchors** (relation-shape questions): for + `"who is X's Y?"` / `"who founded Z?"` / etc., at least one + Title-Case proper-noun phrase from the claim must appear in some + cited span. Catches the Homer-Simpson lazy-anchor where the cited + span talks about Castellaneta's voice work but never names + "Mr. Burns". + +- **Date anchors** (any question shape): for any claim that names a + specific year, that year must appear in some cited span. Catches + the Back-to-the-Future lazy-anchor where the cited span lacks the + asserted "1985". + +These two classes leave four other shapes uncovered, where the same +lazy-anchor pattern can pass the existing seven hard checks: + +### 1.1 Concrete failure shapes pending coverage + +**Entity-list shape** (`"name the simpsons family members including pets?"`): + +- Claim: `"The Simpsons family consists of Homer, Marge, Bart, Lisa, Maggie."` +- Cited span: a Simpsons-family page paragraph that mentions Homer + and Marge but not the children. The bare-name guard demands ≥2 + content tokens per claim, but doesn't demand all named entities + appear in some cited span. The verifier passes on the partial + evidence. +- Warrant gap: at least one of the named entities (the *most + specific* one — `"Maggie"` over the more-common `"Homer"`) must + appear in a cited span, OR the claim must be split. + +**Why-cause shape** (`"why did the titanic sink?"`): + +- Claim: `"The Titanic sank after striking an iceberg in 1912."` +- Cited span: a passenger-list page that mentions the Titanic + but not the iceberg. +- Warrant gap: a cause word (or the claim's primary noun for the + cause) must appear in a cited span. Without this, a model can + cite the topical-but-causally-irrelevant chunk. + +**Count shape** (`"how many wives did henry the eighth have?"`): + +- Claim: `"Henry VIII had six wives."` +- Cited span: a wife's individual bio page that doesn't say "six". +- Warrant gap: the asserted count number must appear (in numeric or + word form) in some cited span. Same shape as the date anchor but + on count-tokens. + +**How-much shape** (`"what is the boltzmann constant?"`): + +- Claim: `"The Boltzmann constant is approximately 1.38 × 10⁻²³ J/K."` +- Cited span: a Boltzmann biography paragraph. +- Warrant gap: the asserted numeric value must appear in some cited + span. Same shape as date/count. + +## 2. Generalization + +The pattern across these shapes: **the claim asserts a structurally +load-bearing token (an entity, a year, a count, a numeric value, a +cause noun); the warrant requires that token to appear in some cited +span**. + +Three new anchor classes proposed: + +### 2.1 Entity-list anchor + +When the question shape is `name X` / `list X` / `who are the +members of X` / `name X family + pets`, the claim is expected to +enumerate proper nouns. Warrant: every Title-Case proper noun in +the claim text (after a small stopword set) must appear in some +cited span. + +Conservative: a single un-anchored proper noun in an +otherwise-anchored claim does NOT fail the warrant — the bench +evidence on entity-list questions shows models occasionally name an +extra entity from training. Mark such claims `EVIDENCE_LINKED_PARTIAL` +rather than fail outright (per the demote-don't-reject pattern). + +### 2.2 Count anchor + +Claim contains a number (digit form `6`, word form `six`, ordinal +form `sixth`). For each such number, at least one cited span must +contain the same number in any of the equivalent forms. Bidirectional +match: word→digit and digit→word both count. + +The numeric-value variant (Boltzmann constant `1.38 × 10⁻²³`) is a +special case of count-anchor where the number includes scientific +notation; treat by tokenizing on `[-+×^×*0-9.eE]+` and string-matching. + +### 2.3 Cause anchor (why-shape questions) + +Question starts with `"why"`. Claim is expected to name the cause +noun (e.g. `"iceberg"`, `"asteroid impact"`, `"propaganda"`). The +existing proper-noun extractor mostly catches this when the cause is +Title-Cased, but lowercase common nouns slip through. Extension: for +`why`-shape questions, also extract bare common-noun candidates from +the claim (≥5-char tokens not in a generic stopword list) and check +that at least one appears in some cited span. + +Conservative scope: only fire on `why`-shape questions. False-positive +risk on lowercase common-noun extraction is real; gate by question +shape to constrain it. + +## 3. Implementation sketch + +### 3.1 Question-shape detector extension + +`aborist/qa/warrant.py` already has `_question_is_relation_shape`. +Add: + +```python +_QUESTION_IS_ENTITY_LIST_RE = re.compile( + r"^\s*(?:name|list|who\s+are|name\s+the|list\s+the|" + r"members?\s+of|family\s+of)\b", + re.IGNORECASE, +) + +_QUESTION_IS_COUNT_RE = re.compile( + r"^\s*how\s+(?:many|much)\b", re.IGNORECASE, +) + +_QUESTION_IS_WHY_RE = re.compile( + r"^\s*why\b", re.IGNORECASE, +) +``` + +### 3.2 Per-shape anchor extractors + +```python +def _extract_entity_anchors(claim_text: str) -> list[str]: + """Title-Case proper-noun phrases (existing extractor reused).""" + +def _extract_count_anchors(claim_text: str) -> list[str]: + """Tokens matching count-shape regex; expand digit↔word + equivalents (six↔6, twelve↔12) before substring testing.""" + +def _extract_cause_anchors(claim_text: str) -> list[str]: + """≥5-char common nouns + Title-Case proper nouns in the claim. + Gated to fire only on why-shape questions.""" +``` + +### 3.3 Dispatch in `warrant_check` + +Today's implementation runs proper-noun anchor + date anchor in +sequence and returns `(ok, missing)`. Extend to dispatch by question +shape: + +```python +def warrant_check(claim_text, cited_spans, *, question): + anchors_required = [] + if _question_is_relation_shape(question): + anchors_required.append(_extract_entity_anchors(claim_text)) + if _question_is_entity_list_shape(question): + anchors_required.append(_extract_entity_anchors(claim_text)) + if _question_is_count_shape(question): + anchors_required.append(_extract_count_anchors(claim_text)) + if _question_is_why_shape(question): + anchors_required.append(_extract_cause_anchors(claim_text)) + # Date anchor stays unconditional (any claim with a 4-digit year). + anchors_required.append(_extract_date_anchors(claim_text)) + # ... existing pass/fail logic ... +``` + +Each anchor class returns its own `missing` list; failure on any +class fires `WARRANT_MISSING` with `missing_anchors` carrying the +union. + +### 3.4 Policy gate + +`policy["claim_lattice_warrant_check_enabled"]` (already exists) +extended to a per-class dict: + +```python +"claim_lattice_warrant_classes": { + "relation": True, + "date": True, + "entity_list": True, # new + "count": True, # new + "why_cause": True, # new +}, +``` + +Folds into `governance_policy_hash` so disabling a class invalidates +prior records on lookup. Default-on for safety; operators can disable +individual classes per-query if they over-fire on a specific corpus. + +## 4. Tests required + +Already covered (relation shape + date anchor — landed earlier): +- `[x]` Relation-shape claim with proper-noun anchor missing → fails. +- `[x]` Date-anchor claim with year missing → fails. +- `[x]` Vacuous-pass when claim has no anchor. + +To add when this ticket lands: +- `[ ]` Entity-list claim with all named entities present in some + cited span → passes. +- `[ ]` Entity-list claim with one Title-Case entity missing → + demotes to EVIDENCE_LINKED_PARTIAL (not failure). +- `[ ]` Count-shape claim with `"six"` and span containing `"6"` → + passes (digit↔word equivalence). +- `[ ]` Count-shape claim with `"twelve"` and span containing + `"12"` → passes. +- `[ ]` Count-shape claim with `"six"` and no count token in any + span → fails. +- `[ ]` Why-shape claim with cause noun missing from spans → fails. +- `[ ]` Why-shape claim where claim's Title-Case anchor IS present + (existing extractor) → passes via proper-noun path. +- `[ ]` Bench: entity-list / count / why-cause coverage measured + on the existing question set; baseline strict-rate before vs + after. + +## 5. Out of scope + +- **Per-question-type rule libraries.** The general anchor-class + primitive catches the failures listed in §1.1 deterministically + without a per-type rule book. Typed contracts (e.g. + `release_date_lookup with required predicate cues`) earn their + slot only when bench evidence shows the general primitive misses + cases. See `docs/naming-deferral.md`. +- **NLI-grade entailment.** Soft signal, separate ticket if/when + the verifier-semantic-gap design (`docs/verifier-semantic-gap-design.md`) + lands. +- **Cross-claim warrant.** When claim A asserts X and claim B + asserts Y, the warrant runs per-claim independently. Cross-claim + consistency (does claim B contradict claim A's evidence?) is + out of scope. + +## 6. Status + +**Proposal.** No code yet. Forecast cost: ~3-4 hours (per-shape +detector + extractor + dispatch + ~8 tests). Risk: medium — +false-positive risk on `why-shape` cause-noun extraction; the +conservative gate-by-question-shape limits exposure. + +Forecast value: pins three failure classes structurally before any +NLI substrate exists. Brings warrant coverage from "relation + +date" (today) to "relation + date + entity-list + count + +why-cause" — closing the lazy-anchor gap on the question shapes +the bench question set already exercises.