qa(verify): Rule 9 — SUBJECT_TOKENS_ABSENT premise-parroting demote
Closes the first confirmed EVIDENCE-WARRANTED false-positive surfaced by the 200-cycle bench-emergent run on `steer/reply/correcter` (Ticket #000006 amend 2026-05-02b). The model parroted three question-distinctive tokens (correcter, steer, reply) into its claim while citing a glossary article whose 33.5K-char content contains ZERO occurrences of any of them. Generic linguistic vocabulary (language, communication, terms, relationships) carried Rule 5's citation-coverage check on its own; the actual subject tokens rode along unverified. New per-claim check `_parroted_subject_tokens_absent`: for each resolving claim, compute the question∩claim content-token set, then check substring presence in the union of cited evidence spans (lower-cased, mirroring Rule 5). When ≥ threshold parroted tokens are absent, emit `SUBJECT_TOKENS_ABSENT` and demote STRICT → HYBRID. Default threshold = 3 — single-token absence is often stem-variant noise; three+ is the parrot fingerprint. Plumbing: - New default `DEFAULT_SUBJECT_TOKENS_ABSENT_THRESHOLD = 3` - Both `verify_claim_lattice` and `verify_claim_lattice_json` gain `subject_tokens_absent_threshold` kwarg + per-claim check block (mirrors TITLE_MISMATCH plumbing, sits right after it in the rule order) - `claim_lattice_subject_tokens_absent_threshold` policy field added to `DEFAULT_QUERY_POLICY` and `DEFAULT_POLICY`; folds into `governance_policy_hash` and (via _VERIFIER_POLICY_FIELDS) `verifier_policy_hash` - All four runner/query call sites pass the policy-derived value Live verification (cache-split cleanly via policy-hash bump): pre-fix cache_key 08dbd2c1… : STRICT (false positive) post-fix cache_key 6a519636… : UNGROUNDED Three new unit tests in `tests/test_verify_json.py`: - threshold-meeting parrot demotes STRICT → HYBRID - no-op when question is None - below-threshold absence stays STRICT Full suite: 776 passed, 34 skipped.
This commit is contained in:
parent
d3ad520529
commit
39bebe3fdb
6 changed files with 311 additions and 13 deletions
|
|
@ -345,19 +345,29 @@ that the lexical Rule-5 check cannot catch alone.
|
|||
|
||||
### Action items emerging from 200-cycle delta
|
||||
|
||||
1. **Open new ticket #000008 — PREMISE_PARROTING detector.**
|
||||
Detector signal: question-distinctive tokens (rare in cited
|
||||
chunk) absent from cited evidence span while generic vocabulary
|
||||
carries the coverage check. Specifically:
|
||||
- Compute `question_distinctive = question_content_tokens \\
|
||||
stop_corpus` (or TF-IDF top-K from question over corpus).
|
||||
- For each cited evidence span: count `question_distinctive ∩
|
||||
cited_chunk_tokens`.
|
||||
- If 0 question-distinctive tokens appear in cited chunk while
|
||||
citation-coverage Rule 5 still passes → emit
|
||||
`SUBJECT_TOKENS_ABSENT` violation, demote STRICT → HYBRID.
|
||||
This stays lexical (no NLI). Folds into `verifier_policy_hash`
|
||||
and `governance_policy_hash`.
|
||||
1. **Rule 9 — SUBJECT_TOKENS_ABSENT detector — landed inline (no
|
||||
separate ticket).** New per-claim check in
|
||||
`aborist/qa/verify.py`. For each resolving claim, compute the
|
||||
set of content tokens shared by question AND claim
|
||||
(`_parroted_subject_tokens_absent`). For each, check substring
|
||||
presence in the union of cited evidence spans (lower-cased,
|
||||
same as Rule 5). When ≥ `subject_tokens_absent_threshold` (default
|
||||
3) parroted tokens are absent → emit `SUBJECT_TOKENS_ABSENT`
|
||||
violation, demote STRICT → HYBRID. Stays lexical, stays binary,
|
||||
folds into `verifier_policy_hash` + `governance_policy_hash`.
|
||||
|
||||
**Live verification on the original case** (cache-split via
|
||||
policy-hash bump):
|
||||
|
||||
| cache_key | policy | audit_mode |
|
||||
|---|---|---|
|
||||
| `08dbd2c1…` | pre-fix | STRICT (false positive) |
|
||||
| `6a519636…` | post-fix | UNGROUNDED |
|
||||
|
||||
Ladder now correctly demotes the parroted answer. Three new
|
||||
unit tests pin the signature: demote on threshold-meeting
|
||||
parrot, no-op when question is None, no-demote below threshold.
|
||||
|
||||
2. **`metaphor_deflection` sidecar** still rare (low single-digit
|
||||
fires across 200 cycles). Calibration deferred until 300+.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue