docs(ticket-000003): anchor-class warrant generalization (Directive D6)
Today's warrant-lite (aborist/qa/warrant.py) covers two anchor
classes: proper-noun (relation-shape questions) and date (any
claim with a 4-digit year). Three failure shapes pass the existing
seven hard checks unchecked:
- entity-list: 'name the simpsons family + pets' — claim
enumerates entities, but the warrant doesn't demand all named
entities appear in some cited span.
- count: 'how many wives did henry the eighth have?' — claim
says 'six', but the warrant doesn't demand the count token
appear in some cited span.
- why-cause: 'why did the titanic sink?' — claim names a cause
noun (iceberg, asteroid, propaganda), but the warrant doesn't
demand the cause token appear in some cited span.
Ticket proposes per-shape anchor extractors dispatched by question-
shape detector (existing _question_is_relation_shape pattern).
Conservative scope on cause anchors: gate by why-shape only since
lowercase common-noun extraction has higher false-positive risk.
Folds into governance_policy_hash via per-class policy dict so an
operator can disable individual classes per-corpus when they
over-fire.
Out of scope per docs/naming-deferral.md: typed-contract framework
(per-question-type rule libraries). General anchor-class primitive
catches the failures listed without per-type rule books; typed
contracts earn slots only when bench evidence shows the general
primitive misses cases.
Forecast cost ~3-4 hours; risk medium (false-positive risk on
why-shape cause-noun extraction). Closes the lazy-anchor gap on
the question shapes the bench question set already exercises.
This commit is contained in:
parent
c1dcd3142e
commit
b5f7c93fca
1 changed files with 261 additions and 0 deletions
261
docs/ticket-000003-anchor-class-warrant.md
Normal file
261
docs/ticket-000003-anchor-class-warrant.md
Normal file
|
|
@ -0,0 +1,261 @@
|
|||
# Ticket #000003 — Anchor-class warrant generalization (Module H+)
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Opened:** 2026-05-01
|
||||
**Directive:** [D6](seven-point-program.md) — General anchor-class
|
||||
warrant before semantic NLI.
|
||||
**Scope:** Generalize the warrant-lite hard check (Rule 7 in
|
||||
`verify_claim_lattice`) beyond relation-shape and date-shape anchors
|
||||
to cover entity-list, why-cause, count, and how-much shapes. Pin the
|
||||
broader class of lazy-anchor failures structurally before any NLI
|
||||
substrate exists in the proof path.
|
||||
**Audience:** fox + future blackops shifts.
|
||||
**Hard constraint:** the warrant check stays **lexical and binary**.
|
||||
No NLI, no embeddings, no model self-grading. Soft signals continue
|
||||
to live in the sidecar layer (`docs/verifier-semantic-gap-design.md`
|
||||
covers that path separately).
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
|
||||
Today's warrant-lite (`aborist/qa/warrant.py:warrant_check`) covers
|
||||
two anchor classes:
|
||||
|
||||
- **Proper-noun anchors** (relation-shape questions): for
|
||||
`"who is X's Y?"` / `"who founded Z?"` / etc., at least one
|
||||
Title-Case proper-noun phrase from the claim must appear in some
|
||||
cited span. Catches the Homer-Simpson lazy-anchor where the cited
|
||||
span talks about Castellaneta's voice work but never names
|
||||
"Mr. Burns".
|
||||
|
||||
- **Date anchors** (any question shape): for any claim that names a
|
||||
specific year, that year must appear in some cited span. Catches
|
||||
the Back-to-the-Future lazy-anchor where the cited span lacks the
|
||||
asserted "1985".
|
||||
|
||||
These two classes leave four other shapes uncovered, where the same
|
||||
lazy-anchor pattern can pass the existing seven hard checks:
|
||||
|
||||
### 1.1 Concrete failure shapes pending coverage
|
||||
|
||||
**Entity-list shape** (`"name the simpsons family members including pets?"`):
|
||||
|
||||
- Claim: `"The Simpsons family consists of Homer, Marge, Bart, Lisa, Maggie."`
|
||||
- Cited span: a Simpsons-family page paragraph that mentions Homer
|
||||
and Marge but not the children. The bare-name guard demands ≥2
|
||||
content tokens per claim, but doesn't demand all named entities
|
||||
appear in some cited span. The verifier passes on the partial
|
||||
evidence.
|
||||
- Warrant gap: at least one of the named entities (the *most
|
||||
specific* one — `"Maggie"` over the more-common `"Homer"`) must
|
||||
appear in a cited span, OR the claim must be split.
|
||||
|
||||
**Why-cause shape** (`"why did the titanic sink?"`):
|
||||
|
||||
- Claim: `"The Titanic sank after striking an iceberg in 1912."`
|
||||
- Cited span: a passenger-list page that mentions the Titanic
|
||||
but not the iceberg.
|
||||
- Warrant gap: a cause word (or the claim's primary noun for the
|
||||
cause) must appear in a cited span. Without this, a model can
|
||||
cite the topical-but-causally-irrelevant chunk.
|
||||
|
||||
**Count shape** (`"how many wives did henry the eighth have?"`):
|
||||
|
||||
- Claim: `"Henry VIII had six wives."`
|
||||
- Cited span: a wife's individual bio page that doesn't say "six".
|
||||
- Warrant gap: the asserted count number must appear (in numeric or
|
||||
word form) in some cited span. Same shape as the date anchor but
|
||||
on count-tokens.
|
||||
|
||||
**How-much shape** (`"what is the boltzmann constant?"`):
|
||||
|
||||
- Claim: `"The Boltzmann constant is approximately 1.38 × 10⁻²³ J/K."`
|
||||
- Cited span: a Boltzmann biography paragraph.
|
||||
- Warrant gap: the asserted numeric value must appear in some cited
|
||||
span. Same shape as date/count.
|
||||
|
||||
## 2. Generalization
|
||||
|
||||
The pattern across these shapes: **the claim asserts a structurally
|
||||
load-bearing token (an entity, a year, a count, a numeric value, a
|
||||
cause noun); the warrant requires that token to appear in some cited
|
||||
span**.
|
||||
|
||||
Three new anchor classes proposed:
|
||||
|
||||
### 2.1 Entity-list anchor
|
||||
|
||||
When the question shape is `name X` / `list X` / `who are the
|
||||
members of X` / `name X family + pets`, the claim is expected to
|
||||
enumerate proper nouns. Warrant: every Title-Case proper noun in
|
||||
the claim text (after a small stopword set) must appear in some
|
||||
cited span.
|
||||
|
||||
Conservative: a single un-anchored proper noun in an
|
||||
otherwise-anchored claim does NOT fail the warrant — the bench
|
||||
evidence on entity-list questions shows models occasionally name an
|
||||
extra entity from training. Mark such claims `EVIDENCE_LINKED_PARTIAL`
|
||||
rather than fail outright (per the demote-don't-reject pattern).
|
||||
|
||||
### 2.2 Count anchor
|
||||
|
||||
Claim contains a number (digit form `6`, word form `six`, ordinal
|
||||
form `sixth`). For each such number, at least one cited span must
|
||||
contain the same number in any of the equivalent forms. Bidirectional
|
||||
match: word→digit and digit→word both count.
|
||||
|
||||
The numeric-value variant (Boltzmann constant `1.38 × 10⁻²³`) is a
|
||||
special case of count-anchor where the number includes scientific
|
||||
notation; treat by tokenizing on `[-+×^×*0-9.eE]+` and string-matching.
|
||||
|
||||
### 2.3 Cause anchor (why-shape questions)
|
||||
|
||||
Question starts with `"why"`. Claim is expected to name the cause
|
||||
noun (e.g. `"iceberg"`, `"asteroid impact"`, `"propaganda"`). The
|
||||
existing proper-noun extractor mostly catches this when the cause is
|
||||
Title-Cased, but lowercase common nouns slip through. Extension: for
|
||||
`why`-shape questions, also extract bare common-noun candidates from
|
||||
the claim (≥5-char tokens not in a generic stopword list) and check
|
||||
that at least one appears in some cited span.
|
||||
|
||||
Conservative scope: only fire on `why`-shape questions. False-positive
|
||||
risk on lowercase common-noun extraction is real; gate by question
|
||||
shape to constrain it.
|
||||
|
||||
## 3. Implementation sketch
|
||||
|
||||
### 3.1 Question-shape detector extension
|
||||
|
||||
`aborist/qa/warrant.py` already has `_question_is_relation_shape`.
|
||||
Add:
|
||||
|
||||
```python
|
||||
_QUESTION_IS_ENTITY_LIST_RE = re.compile(
|
||||
r"^\s*(?:name|list|who\s+are|name\s+the|list\s+the|"
|
||||
r"members?\s+of|family\s+of)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
_QUESTION_IS_COUNT_RE = re.compile(
|
||||
r"^\s*how\s+(?:many|much)\b", re.IGNORECASE,
|
||||
)
|
||||
|
||||
_QUESTION_IS_WHY_RE = re.compile(
|
||||
r"^\s*why\b", re.IGNORECASE,
|
||||
)
|
||||
```
|
||||
|
||||
### 3.2 Per-shape anchor extractors
|
||||
|
||||
```python
|
||||
def _extract_entity_anchors(claim_text: str) -> list[str]:
|
||||
"""Title-Case proper-noun phrases (existing extractor reused)."""
|
||||
|
||||
def _extract_count_anchors(claim_text: str) -> list[str]:
|
||||
"""Tokens matching count-shape regex; expand digit↔word
|
||||
equivalents (six↔6, twelve↔12) before substring testing."""
|
||||
|
||||
def _extract_cause_anchors(claim_text: str) -> list[str]:
|
||||
"""≥5-char common nouns + Title-Case proper nouns in the claim.
|
||||
Gated to fire only on why-shape questions."""
|
||||
```
|
||||
|
||||
### 3.3 Dispatch in `warrant_check`
|
||||
|
||||
Today's implementation runs proper-noun anchor + date anchor in
|
||||
sequence and returns `(ok, missing)`. Extend to dispatch by question
|
||||
shape:
|
||||
|
||||
```python
|
||||
def warrant_check(claim_text, cited_spans, *, question):
|
||||
anchors_required = []
|
||||
if _question_is_relation_shape(question):
|
||||
anchors_required.append(_extract_entity_anchors(claim_text))
|
||||
if _question_is_entity_list_shape(question):
|
||||
anchors_required.append(_extract_entity_anchors(claim_text))
|
||||
if _question_is_count_shape(question):
|
||||
anchors_required.append(_extract_count_anchors(claim_text))
|
||||
if _question_is_why_shape(question):
|
||||
anchors_required.append(_extract_cause_anchors(claim_text))
|
||||
# Date anchor stays unconditional (any claim with a 4-digit year).
|
||||
anchors_required.append(_extract_date_anchors(claim_text))
|
||||
# ... existing pass/fail logic ...
|
||||
```
|
||||
|
||||
Each anchor class returns its own `missing` list; failure on any
|
||||
class fires `WARRANT_MISSING` with `missing_anchors` carrying the
|
||||
union.
|
||||
|
||||
### 3.4 Policy gate
|
||||
|
||||
`policy["claim_lattice_warrant_check_enabled"]` (already exists)
|
||||
extended to a per-class dict:
|
||||
|
||||
```python
|
||||
"claim_lattice_warrant_classes": {
|
||||
"relation": True,
|
||||
"date": True,
|
||||
"entity_list": True, # new
|
||||
"count": True, # new
|
||||
"why_cause": True, # new
|
||||
},
|
||||
```
|
||||
|
||||
Folds into `governance_policy_hash` so disabling a class invalidates
|
||||
prior records on lookup. Default-on for safety; operators can disable
|
||||
individual classes per-query if they over-fire on a specific corpus.
|
||||
|
||||
## 4. Tests required
|
||||
|
||||
Already covered (relation shape + date anchor — landed earlier):
|
||||
- `[x]` Relation-shape claim with proper-noun anchor missing → fails.
|
||||
- `[x]` Date-anchor claim with year missing → fails.
|
||||
- `[x]` Vacuous-pass when claim has no anchor.
|
||||
|
||||
To add when this ticket lands:
|
||||
- `[ ]` Entity-list claim with all named entities present in some
|
||||
cited span → passes.
|
||||
- `[ ]` Entity-list claim with one Title-Case entity missing →
|
||||
demotes to EVIDENCE_LINKED_PARTIAL (not failure).
|
||||
- `[ ]` Count-shape claim with `"six"` and span containing `"6"` →
|
||||
passes (digit↔word equivalence).
|
||||
- `[ ]` Count-shape claim with `"twelve"` and span containing
|
||||
`"12"` → passes.
|
||||
- `[ ]` Count-shape claim with `"six"` and no count token in any
|
||||
span → fails.
|
||||
- `[ ]` Why-shape claim with cause noun missing from spans → fails.
|
||||
- `[ ]` Why-shape claim where claim's Title-Case anchor IS present
|
||||
(existing extractor) → passes via proper-noun path.
|
||||
- `[ ]` Bench: entity-list / count / why-cause coverage measured
|
||||
on the existing question set; baseline strict-rate before vs
|
||||
after.
|
||||
|
||||
## 5. Out of scope
|
||||
|
||||
- **Per-question-type rule libraries.** The general anchor-class
|
||||
primitive catches the failures listed in §1.1 deterministically
|
||||
without a per-type rule book. Typed contracts (e.g.
|
||||
`release_date_lookup with required predicate cues`) earn their
|
||||
slot only when bench evidence shows the general primitive misses
|
||||
cases. See `docs/naming-deferral.md`.
|
||||
- **NLI-grade entailment.** Soft signal, separate ticket if/when
|
||||
the verifier-semantic-gap design (`docs/verifier-semantic-gap-design.md`)
|
||||
lands.
|
||||
- **Cross-claim warrant.** When claim A asserts X and claim B
|
||||
asserts Y, the warrant runs per-claim independently. Cross-claim
|
||||
consistency (does claim B contradict claim A's evidence?) is
|
||||
out of scope.
|
||||
|
||||
## 6. Status
|
||||
|
||||
**Proposal.** No code yet. Forecast cost: ~3-4 hours (per-shape
|
||||
detector + extractor + dispatch + ~8 tests). Risk: medium —
|
||||
false-positive risk on `why-shape` cause-noun extraction; the
|
||||
conservative gate-by-question-shape limits exposure.
|
||||
|
||||
Forecast value: pins three failure classes structurally before any
|
||||
NLI substrate exists. Brings warrant coverage from "relation +
|
||||
date" (today) to "relation + date + entity-list + count +
|
||||
why-cause" — closing the lazy-anchor gap on the question shapes
|
||||
the bench question set already exercises.
|
||||
Loading…
Add table
Add a link
Reference in a new issue