russell@unturf.com
439319f960
Two related changes that tighten the dedup + grounding signals
without lowering quality bars.
1. question_hash drops standalone English articles (the/a/an).
Fox 2026-04-29: `who is the batman?` and `who is batman?`
produced different cache records; same question, different hash.
Articles are filler at the question-equivalence layer. Add a 4th
canonicalization step to question_hash: after lowercase + trailing
punctuation strip, split on whitespace & drop tokens equal to
"the" / "a" / "an", then rejoin.
Equivalence class now includes:
"who is X" ┐
"who is the X" │
"who is a X" │ -> same question_hash
"who is an X" │
"Who Is X？" ┘ (CJK question mark)
"thesis" stays untouched — exact-match standalone tokens only,
not substring. Conservative on i18n: ASCII English articles only;
"el / la / le / der / die / das" etc. await demand.
2. _token_coverage filters stopwords and per-token punctuation.
Fox asked: should we lower the 0.85 paraphrase threshold? Honest
answer: no — that would promote the Q1 Batman fabrication
("wealthy/businessman/resides" missing from corpus) to STRICT.
Tighten the signal instead so 0.85 means more.
- Per-token punctuation strip (.,;:!?\"()[]{}) so `wayne,` lines
up with bare `wayne` in context. Apostrophes deliberately stay
so `batman's` is distinct from `batman`.
- English stopword filter on length-≥4 fillers (from / with /
have / been / would / which / where / their / etc., curated set
in _ENGLISH_STOPWORDS). These match almost any English text &
inflate coverage scores when topical content is missing.
Net effect on Q1 Batman case: missing tokens are
`wealthy/businessman/resides` — all topical. Coverage stays well
below 0.85, span stays UNGROUNDED. Net effect on a stylistic
paraphrase (model wrote `from` instead of `with`): coverage
computed only over topical tokens, both copies match → 1.0 →
correctly promoted.
Tests: 8 new in tests/test_keys.py (article-strip equivalence
classes, substring preservation, distinct-topic non-collapse) +
2 in tests/test_verify.py (stopword filter doesn't inflate; 0.85
threshold still rejects fabrication). 335 passed, 1 skipped.
Note for fox: the underlying retrieval issue surfaced in Q1/Q2 is
independent — Batman main article IS in shard 002.db but FTS5 +
title rerank ranked List_of_Batman_comics higher. Different fix,
separate commit.
239 lines
9 KiB
Python
239 lines
9 KiB
Python
"""Cache-key dimension hashes — pure functions, deterministic.
|
||
|
||
The 8-dim cache_key invariants are spread across `aborist/qa/keys.py`.
|
||
Most behavior is covered indirectly by `test_qa.py` (cache hits, etc.);
|
||
this file pins the hash-input canonicalization rules so a future tweak
|
||
to question normalization can't silently change the equivalence class
|
||
of "what counts as the same question."
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
from aborist.qa.keys import (
|
||
cache_key,
|
||
conversation_hash,
|
||
governance_policy_hash,
|
||
model_profile_hash,
|
||
question_hash,
|
||
)
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# question_hash equivalence classes
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def test_question_hash_strips_trailing_question_mark():
|
||
"""Fox's catch on 2026-04-29: `who is X?` and `who is X` should hit
|
||
the same providence_cache record. Trailing punctuation carries no
|
||
semantic difference for retrieval."""
|
||
assert question_hash("who is X") == question_hash("who is X?")
|
||
|
||
|
||
def test_question_hash_strips_trailing_period():
|
||
assert question_hash("explain X") == question_hash("explain X.")
|
||
|
||
|
||
def test_question_hash_strips_trailing_exclamation():
|
||
assert question_hash("explain X") == question_hash("explain X!")
|
||
|
||
|
||
def test_question_hash_strips_multiple_trailing_punct():
|
||
"""Multiple trailing chars all stripped — `?!` is a single class."""
|
||
assert question_hash("explain X") == question_hash("explain X?!")
|
||
assert question_hash("explain X") == question_hash("explain X.?")
|
||
|
||
|
||
def test_question_hash_lowercases():
|
||
assert question_hash("Who Is X") == question_hash("who is x")
|
||
|
||
|
||
def test_question_hash_collapses_internal_spaces():
|
||
"""canonicalize() collapses runs of spaces to a single space. Tabs/
|
||
newlines map to newline (a separate equivalence class — see
|
||
aborist.document.canonicalize), so tab vs space is intentionally
|
||
NOT in the same bucket."""
|
||
assert question_hash("who is X") == question_hash("who is X")
|
||
|
||
|
||
def test_question_hash_does_not_strip_internal_punctuation():
|
||
"""Internal `?` is a different question — ``what's up?`` mid-sentence
|
||
can carry meaning. Only trailing punctuation gets stripped."""
|
||
assert question_hash("X, then Y") != question_hash("X then Y")
|
||
|
||
|
||
# CJK + ellipsis equivalence — non-ASCII trailing punctuation should
|
||
# behave identically to ASCII forms. Pin so a future "let's go ASCII-only"
|
||
# refactor can't silently regress i18n.
|
||
|
||
def test_question_hash_strips_full_width_question_mark():
|
||
"""U+FF1F (CJK full-width question mark)."""
|
||
assert question_hash("X") == question_hash("X？")
|
||
|
||
|
||
def test_question_hash_strips_full_width_exclamation():
|
||
"""U+FF01."""
|
||
assert question_hash("X") == question_hash("X！")
|
||
|
||
|
||
def test_question_hash_strips_ideographic_full_stop():
|
||
"""U+3002 (Chinese / Japanese sentence-ending period)."""
|
||
assert question_hash("X") == question_hash("X。")
|
||
|
||
|
||
def test_question_hash_strips_ideographic_comma():
|
||
"""U+3001."""
|
||
assert question_hash("X") == question_hash("X、")
|
||
|
||
|
||
def test_question_hash_strips_ellipsis():
|
||
"""U+2026 — single-codepoint ellipsis, plus the three-dot ASCII form
|
||
that already worked via repeated `.` strip."""
|
||
assert question_hash("X") == question_hash("X…")
|
||
assert question_hash("X") == question_hash("X...")
|
||
|
||
|
||
def test_question_hash_strips_mixed_ascii_and_cjk_trailing():
|
||
"""A question ending with both kinds of trailing punctuation strips
|
||
cleanly. rstrip walks char-by-char from the right."""
|
||
assert question_hash("X") == question_hash("X？！.")
|
||
|
||
|
||
# Quotes and brackets PAIR — naive trailing-strip would break balance.
|
||
# Pin that they stay so a future "expand the strip set" refactor knows
|
||
# the design rule.
|
||
|
||
def test_question_hash_preserves_trailing_double_quote():
|
||
"""`who said "X"?` (after stripping `?`) ends in `"`. Stripping the
|
||
quote would unbalance the surface; the equivalence class would
|
||
no longer match a balanced version."""
|
||
assert question_hash('who said "X"') != question_hash('who said "X')
|
||
|
||
|
||
def test_question_hash_preserves_trailing_single_quote():
|
||
"""Apostrophes look like quotes. `X's` (possessive) must stay distinct
|
||
from `X` (the bare entity)."""
|
||
assert question_hash("who is X's") != question_hash("who is X")
|
||
|
||
|
||
def test_question_hash_preserves_trailing_close_paren():
|
||
"""`(foo)` — a paren pair carries grouping meaning."""
|
||
assert question_hash("who is X (Y)") != question_hash("who is X (Y")
|
||
|
||
|
||
def test_question_hash_preserves_trailing_close_bracket():
|
||
assert question_hash("who is X [a]") != question_hash("who is X [a")
|
||
|
||
|
||
def test_question_hash_preserves_trailing_close_brace():
|
||
assert question_hash("who is X {a}") != question_hash("who is X {a")
|
||
|
||
|
||
# Article stripping — fox's 2026-04-29 catch: "the batman" and "batman"
|
||
# produced different cache records. Articles are filler at the question-
|
||
# equivalence layer; strip them so cache lookups dedupe.
|
||
|
||
def test_question_hash_strips_leading_article_the():
|
||
"""`who is the batman` and `who is batman` ask the same question."""
|
||
assert question_hash("who is the batman") == question_hash("who is batman")
|
||
|
||
|
||
def test_question_hash_strips_leading_article_a():
|
||
assert question_hash("what is a hot dog") == question_hash("what is hot dog")
|
||
|
||
|
||
def test_question_hash_strips_leading_article_an():
|
||
assert question_hash("what is an apple") == question_hash("what is apple")
|
||
|
||
|
||
def test_question_hash_strips_articles_anywhere_in_question():
|
||
"""Articles in the middle of the sentence also strip."""
|
||
assert (
|
||
question_hash("does the cat sit on a mat")
|
||
== question_hash("does cat sit on mat")
|
||
)
|
||
|
||
|
||
def test_question_hash_does_not_strip_article_substrings():
|
||
"""`thesis` contains `the` but is one token; substring matching
|
||
would corrupt it. Tokens are matched as exact lowercase strings."""
|
||
assert (
|
||
question_hash("explain my thesis")
|
||
!= question_hash("explain my sis")
|
||
)
|
||
# And `thesis` survives the strip:
|
||
assert "explain my thesis" in question_hash.__doc__ or True # docstring sanity
|
||
# The actual canonical form has `thesis`:
|
||
h_thesis = question_hash("thesis")
|
||
h_other = question_hash("the sis")
|
||
# "the sis" → "sis" after article strip; "thesis" stays "thesis"
|
||
assert h_thesis != h_other
|
||
|
||
|
||
def test_question_hash_handles_question_with_only_articles_and_topic():
|
||
"""Edge: very short question. `the X` and `X` and `a X` collapse."""
|
||
assert question_hash("the batman") == question_hash("batman")
|
||
assert question_hash("a batman") == question_hash("batman")
|
||
assert question_hash("an batman") == question_hash("batman")
|
||
|
||
|
||
def test_question_hash_distinct_topics_still_distinct():
|
||
"""Article strip doesn't accidentally collapse different topics."""
|
||
assert question_hash("the batman") != question_hash("the superman")
|
||
assert question_hash("a batman") != question_hash("a superman")
|
||
|
||
|
||
def test_question_hash_distinct_for_different_text():
|
||
"""Sanity: two genuinely different questions hash to different values."""
|
||
assert question_hash("who is X") != question_hash("who is Y")
|
||
|
||
|
||
def test_question_hash_is_deterministic():
|
||
"""Same input → same output, every time. No randomness."""
|
||
a = question_hash("who is russell ballestrini?")
|
||
b = question_hash("who is russell ballestrini?")
|
||
assert a == b
|
||
assert len(a) == 64 # sha256 hex
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Other dim hashes — light coverage so future refactors don't drift
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def test_model_profile_hash_includes_revision_and_quantization():
|
||
a = model_profile_hash("m", revision="r1", quantization="q1")
|
||
b = model_profile_hash("m", revision="r2", quantization="q1")
|
||
c = model_profile_hash("m", revision="r1", quantization="q2")
|
||
assert a != b
|
||
assert a != c
|
||
assert b != c
|
||
|
||
|
||
def test_conversation_hash_canonicalizes_message_order():
|
||
"""Message ordering matters — a different order is a different
|
||
conversation. Pin so anyone tempted to sort messages knows it would
|
||
invalidate cache."""
|
||
msgs1 = [{"role": "system", "content": "a"}, {"role": "user", "content": "b"}]
|
||
msgs2 = [{"role": "user", "content": "b"}, {"role": "system", "content": "a"}]
|
||
assert conversation_hash(msgs1) != conversation_hash(msgs2)
|
||
|
||
|
||
def test_governance_policy_hash_sensitive_to_keys():
|
||
a = governance_policy_hash({"k": 1})
|
||
b = governance_policy_hash({"k": 2})
|
||
c = governance_policy_hash({"j": 1})
|
||
assert a != b
|
||
assert a != c
|
||
|
||
|
||
def test_cache_key_combines_all_eight_dims():
|
||
"""If any of the 8 dims changes, cache_key changes. v9.8 invariant."""
|
||
base = ("src", "qh", "mh", "ch", "gh", "v9.8.0", "norm-v1", "tok-512-v1")
|
||
base_key = cache_key(*base)
|
||
for i in range(8):
|
||
mutated = list(base)
|
||
mutated[i] = mutated[i] + "_modified"
|
||
assert cache_key(*mutated) != base_key, (
|
||
f"dim {i} change did not bump cache_key — 8-dim invariant broken"
|
||
)
|