arborist/aborist/qa
russell@unturf.com 39bebe3fdb
qa(verify): Rule 9 — SUBJECT_TOKENS_ABSENT premise-parroting demote
Closes the first confirmed EVIDENCE-WARRANTED false-positive
surfaced by the 200-cycle bench-emergent run on
`steer/reply/correcter` (Ticket #000006 amend 2026-05-02b). The
model parroted three question-distinctive tokens (correcter,
steer, reply) into its claim while citing a glossary article whose
33.5K-char content contains ZERO occurrences of any of them.
Generic linguistic vocabulary (language, communication, terms,
relationships) carried Rule 5's citation-coverage check on its
own; the actual subject tokens rode along unverified.

New per-claim check `_parroted_subject_tokens_absent`: for each
resolving claim, compute the question∩claim content-token set,
then check substring presence in the union of cited evidence
spans (lower-cased, mirroring Rule 5). When ≥ threshold parroted
tokens are absent, emit `SUBJECT_TOKENS_ABSENT` and demote STRICT
→ HYBRID. Default threshold = 3 — single-token absence is often
stem-variant noise; three+ is the parrot fingerprint.

Plumbing:
- New default `DEFAULT_SUBJECT_TOKENS_ABSENT_THRESHOLD = 3`
- Both `verify_claim_lattice` and `verify_claim_lattice_json`
  gain `subject_tokens_absent_threshold` kwarg + per-claim check
  block (mirrors TITLE_MISMATCH plumbing, sits right after it
  in the rule order)
- `claim_lattice_subject_tokens_absent_threshold` policy field
  added to `DEFAULT_QUERY_POLICY` and `DEFAULT_POLICY`; folds
  into `governance_policy_hash` and (via _VERIFIER_POLICY_FIELDS)
  `verifier_policy_hash`
- All four runner/query call sites pass the policy-derived value

Live verification (cache-split cleanly via policy-hash bump):
  pre-fix cache_key 08dbd2c1… : STRICT (false positive)
  post-fix cache_key 6a519636… : UNGROUNDED

Three new unit tests in `tests/test_verify_json.py`:
- threshold-meeting parrot demotes STRICT → HYBRID
- no-op when question is None
- below-threshold absence stays STRICT

Full suite: 776 passed, 34 skipped.
2026-05-02 15:51:58 -04:00
..
__init__.py multi-source corpus query: pose a question, the tree pulls related cached docs 2026-04-27 11:43:57 -04:00
client.py qa(client): scrub lone UTF-16 surrogates before httpx encode 2026-05-02 11:58:25 -04:00
concepts.py qa(concepts): corpus-derived concept_relations table replaces frozensets 2026-05-01 21:14:53 -04:00
dag.py speed: pytest-xdist, bench smoke, concurrency default; UTF surrogate fix 2026-05-02 09:29:40 -04:00
evidence.py speed: pytest-xdist, bench smoke, concurrency default; UTF surrogate fix 2026-05-02 09:29:40 -04:00
frame.py docs: tickets/ subdir + delete naming-deferral, ref-frame, coverage-audit 2026-05-02 12:51:09 -04:00
inspect.py qa(inspect): wordlist union (US+UK) + configurable supplemental dictionaries 2026-05-02 14:27:24 -04:00
keys.py qa(verify): Rule 9 — SUBJECT_TOKENS_ABSENT premise-parroting demote 2026-05-02 15:51:58 -04:00
parse_claims.py qa: G0 claim-lattice-pointer answer mode — CTI quote-by-pointer 2026-04-29 21:19:19 -04:00
prompts.py docs(bench): rename qa-modes-bench-2026-05-02.md → qa-modes-bench.md 2026-05-02 12:27:38 -04:00
query.py qa(verify): Rule 9 — SUBJECT_TOKENS_ABSENT premise-parroting demote 2026-05-02 15:51:58 -04:00
repair.py qa: chain-segment failure localization + re-prompt repair tier 2026-04-29 19:27:33 -04:00
retrieval_plan.py docs: tickets/ subdir + delete naming-deferral, ref-frame, coverage-audit 2026-05-02 12:51:09 -04:00
runner.py qa(verify): Rule 9 — SUBJECT_TOKENS_ABSENT premise-parroting demote 2026-05-02 15:51:58 -04:00
verify.py qa(verify): Rule 9 — SUBJECT_TOKENS_ABSENT premise-parroting demote 2026-05-02 15:51:58 -04:00
warrant.py docs: tickets/ subdir + delete naming-deferral, ref-frame, coverage-audit 2026-05-02 12:51:09 -04:00