Two related changes that tighten the dedup + grounding signals
without lowering quality bars.
1. question_hash drops standalone English articles (the/a/an).
Fox 2026-04-29: `who is the batman?` and `who is batman?`
produced different cache records; same question, different hash.
Articles are filler at the question-equivalence layer. Add a 4th
canonicalization step to question_hash: after lowercase + trailing
punctuation strip, split on whitespace & drop tokens equal to
"the" / "a" / "an", then rejoin.
Equivalence class now includes:
"who is X" ┐
"who is the X" │
"who is a X" │ -> same question_hash
"who is an X" │
"Who Is X？" ┘ (CJK question mark)
"thesis" stays untouched — exact-match standalone tokens only,
not substring. Conservative on i18n: ASCII English articles only;
"el / la / le / der / die / das" etc. await demand.
2. _token_coverage filters stopwords and per-token punctuation.
Fox asked: should we lower the 0.85 paraphrase threshold? Honest
answer: no — that would promote the Q1 Batman fabrication
("wealthy/businessman/resides" missing from corpus) to STRICT.
Tighten the signal instead so 0.85 means more.
- Per-token punctuation strip (.,;:!?\"()[]{}) so `wayne,` lines
up with bare `wayne` in context. Apostrophes deliberately stay
so `batman's` is distinct from `batman`.
- English stopword filter on length-≥4 fillers (from / with /
have / been / would / which / where / their / etc., curated set
in _ENGLISH_STOPWORDS). These match almost any English text &
inflate coverage scores when topical content is missing.
Net effect on Q1 Batman case: missing tokens are
`wealthy/businessman/resides` — all topical. Coverage stays well
below 0.85, span stays UNGROUNDED. Net effect on a stylistic
paraphrase (model wrote `from` instead of `with`): coverage
computed only over topical tokens, both copies match → 1.0 →
correctly promoted.
Tests: 8 new in tests/test_keys.py (article-strip equivalence
classes, substring preservation, distinct-topic non-collapse) +
2 in tests/test_verify.py (stopword filter doesn't inflate; 0.85
threshold still rejects fabrication). 335 passed, 1 skipped.
Note for fox: the underlying retrieval issue surfaced in Q1/Q2 is
independent — Batman main article IS in shard 002.db but FTS5 +
title rerank ranked List_of_Batman_comics higher. Different fix,
separate commit.
russell@unturf.com
439319f960