qa(verify): Rule 9 — SUBJECT_TOKENS_ABSENT premise-parroting demote

Closes the first confirmed EVIDENCE-WARRANTED false-positive
surfaced by the 200-cycle bench-emergent run on
`steer/reply/correcter` (Ticket #000006 amend 2026-05-02b). The
model parroted three question-distinctive tokens (correcter,
steer, reply) into its claim while citing a glossary article whose
33.5K-char content contains ZERO occurrences of any of them.
Generic linguistic vocabulary (language, communication, terms,
relationships) carried Rule 5's citation-coverage check on its
own; the actual subject tokens rode along unverified.

New per-claim check `_parroted_subject_tokens_absent`: for each
resolving claim, compute the question∩claim content-token set,
then check substring presence in the union of cited evidence
spans (lower-cased, mirroring Rule 5). When ≥ threshold parroted
tokens are absent, emit `SUBJECT_TOKENS_ABSENT` and demote STRICT
→ HYBRID. Default threshold = 3 — single-token absence is often
stem-variant noise; three+ is the parrot fingerprint.

Plumbing:
- New default `DEFAULT_SUBJECT_TOKENS_ABSENT_THRESHOLD = 3`
- Both `verify_claim_lattice` and `verify_claim_lattice_json`
  gain `subject_tokens_absent_threshold` kwarg + per-claim check
  block (mirrors TITLE_MISMATCH plumbing, sits right after it
  in the rule order)
- `claim_lattice_subject_tokens_absent_threshold` policy field
  added to `DEFAULT_QUERY_POLICY` and `DEFAULT_POLICY`; folds
  into `governance_policy_hash` and (via _VERIFIER_POLICY_FIELDS)
  `verifier_policy_hash`
- All four runner/query call sites pass the policy-derived value

Live verification (cache-split cleanly via policy-hash bump):
  pre-fix cache_key 08dbd2c1… : STRICT (false positive)
  post-fix cache_key 6a519636… : UNGROUNDED

Three new unit tests in `tests/test_verify_json.py`:
- threshold-meeting parrot demotes STRICT → HYBRID
- no-op when question is None
- below-threshold absence stays STRICT

Full suite: 776 passed, 34 skipped.
This commit is contained in:
russell@unturf.com 2026-05-02 15:51:58 -04:00
parent d3ad520529
commit 39bebe3fdb
No known key found for this signature in database
6 changed files with 311 additions and 13 deletions

View file

@ -345,19 +345,29 @@ that the lexical Rule-5 check cannot catch alone.
### Action items emerging from 200-cycle delta
1. **Open new ticket #000008 — PREMISE_PARROTING detector.**
Detector signal: question-distinctive tokens (rare in cited
chunk) absent from cited evidence span while generic vocabulary
carries the coverage check. Specifically:
- Compute `question_distinctive = question_content_tokens \\
stop_corpus` (or TF-IDF top-K from question over corpus).
- For each cited evidence span: count `question_distinctive ∩
cited_chunk_tokens`.
- If 0 question-distinctive tokens appear in cited chunk while
citation-coverage Rule 5 still passes → emit
`SUBJECT_TOKENS_ABSENT` violation, demote STRICT → HYBRID.
This stays lexical (no NLI). Folds into `verifier_policy_hash`
and `governance_policy_hash`.
1. **Rule 9 — SUBJECT_TOKENS_ABSENT detector — landed inline (no
separate ticket).** New per-claim check in
`aborist/qa/verify.py`. For each resolving claim, compute the
set of content tokens shared by question AND claim
(`_parroted_subject_tokens_absent`). For each, check substring
presence in the union of cited evidence spans (lower-cased,
same as Rule 5). When ≥ `subject_tokens_absent_threshold` (default
3) parroted tokens are absent → emit `SUBJECT_TOKENS_ABSENT`
violation, demote STRICT → HYBRID. Stays lexical, stays binary,
folds into `verifier_policy_hash` + `governance_policy_hash`.
**Live verification on the original case** (cache-split via
policy-hash bump):
| cache_key | policy | audit_mode |
|---|---|---|
| `08dbd2c1…` | pre-fix | STRICT (false positive) |
| `6a519636…` | post-fix | UNGROUNDED |
Ladder now correctly demotes the parroted answer. Three new
unit tests pin the signature: demote on threshold-meeting
parrot, no-op when question is None, no-demote below threshold.
2. **`metaphor_deflection` sidecar** still rare (low single-digit
fires across 200 cycles). Calibration deferred until 300+.