russell@unturf.com
c00639ed1d
The review's central, correct finding: _title_query_tokens is hot-path (every query AND title) and its fold set (hyphen #000007 + numeral/accent/honorific/brit) changes which documents retrieve, but that normalization's version was bound nowhere → a replay cannot identify which token-normalization produced an old providence record's sources. Same provenance class as the #000001 keyword gap. Severity is honest: replay-provenance gap, NOT cache corruption — different folds → different sources → different context_root → different cache_key, so no false answer-cache aliasing or false STRICT. Verifier/proof path unchanged. Fix follows the repo's OWN #000001 §5/§6 decision (bind retrieval transforms into the run-DAG RetrievalPlan/retrieval_plan_hash, NOT governance_policy_hash). The review suggested governance "Option A"; repo precedent is run-DAG binding (same status as retrieval_keywords and #000056 MT-engine identity) — the discrepancy is surfaced for fox as an explicit call, not silently overridden. - RetrievalPlan.title_token_policy (empty default → omitted from canonical() → every prior retrieval_plan_hash byte-identical; the §5 zero-churn discipline, same as the #000056 MT fields). - _TITLE_TOKEN_POLICY single source of truth in query.py, bound at the plan construction site; bump on any fold change. - Plus the review's edge cases: Roman-substring-in-word not folded, out-of-range not folded, Unicode Roman explicitly unsupported, hyphen∘numeral composition. Full suite 2498, 0 regressions. Declined (not engineering, per don't-proliferate): the review's SelfModel/MemoryRoot/5S-5T-5F/capital-ledger ceremony — the ticket design log is the single source of truth; scope recorded there.
69 lines
3.1 KiB
Python
69 lines
3.1 KiB
Python
"""Title-token fold provenance + edge cases (Dav1d review 2026-05-19).
|
|
|
|
The review's central finding: `_title_query_tokens` is hot-path and
|
|
its fold set changes which docs retrieve, so a replay must know
|
|
which fold set was active. Fix is #000001-family: bind
|
|
`title_token_policy` into the run-DAG `RetrievalPlan`/`retrieval_plan
|
|
_hash` (NOT governance — repo §5/§6 precedent). Plus the review's
|
|
edge cases for the deterministic guarantee.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
from arborist.qa.query import (
|
|
_TITLE_TOKEN_POLICY,
|
|
_numeral_fold_variants,
|
|
_title_query_tokens,
|
|
)
|
|
from arborist.qa.retrieval_plan import RetrievalPlan, retrieval_plan_hash
|
|
|
|
|
|
# --- provenance binding (the review's #7/#13, the one that matters) ---
|
|
|
|
def test_title_token_policy_binds_into_retrieval_plan_hash_zero_churn():
|
|
base = RetrievalPlan(retrieval_keywords="", top_k=8, over_fetch=32,
|
|
max_context_chars=60000)
|
|
# empty (pre-existing records) → omitted → byte-identical hash
|
|
assert "title_token_policy" not in base.canonical()
|
|
bound = RetrievalPlan(retrieval_keywords="", top_k=8, over_fetch=32,
|
|
max_context_chars=60000,
|
|
title_token_policy=_TITLE_TOKEN_POLICY)
|
|
assert bound.canonical()["title_token_policy"] == _TITLE_TOKEN_POLICY
|
|
assert retrieval_plan_hash(bound) != retrieval_plan_hash(base)
|
|
# a different fold set → different replayable hash
|
|
other = RetrievalPlan(retrieval_keywords="", top_k=8, over_fetch=32,
|
|
max_context_chars=60000,
|
|
title_token_policy="tt-v1:hyphen")
|
|
assert retrieval_plan_hash(other) != retrieval_plan_hash(bound)
|
|
|
|
|
|
# --- review edge cases for the deterministic fold guarantee ---
|
|
|
|
def test_roman_substring_inside_word_not_folded():
|
|
# only whole-token Romans; "civil"/"mixture"/"divide" must not fold
|
|
assert _numeral_fold_variants("the civil mixture divide") == set()
|
|
|
|
|
|
def test_out_of_range_roman_not_folded():
|
|
# strict 2..40; "forty-first"/"hundredth" out of the ordinal map
|
|
assert _numeral_fold_variants("the forty-first hundredth element") == set()
|
|
|
|
|
|
def test_unicode_roman_numerals_explicitly_unsupported():
|
|
# ASCII-only by design (the helper keys on ASCII tokens). Document
|
|
# the boundary rather than let it silently half-work.
|
|
assert _numeral_fold_variants("Alexander Ⅱ") == set() # Ⅱ (U+2161)
|
|
|
|
|
|
def test_hyphen_and_numeral_folds_compose_in_title_tokens():
|
|
# _title_query_tokens now layers hyphen + numeral (+accent/honor/
|
|
# brit) folds; the composition surface must be exercised.
|
|
q = _title_query_tokens("anarcho-capitalism Alexander the second")
|
|
assert {"anarchocapitalism", "ii"} <= q # hyphen-fold AND numeral-fold
|
|
assert {"anarcho", "capitalism", "alexander", "second"} <= q # additive
|
|
|
|
|
|
def test_policy_constant_is_stable_and_descriptive():
|
|
# the single source of truth; bumping it is the replay signal
|
|
assert _TITLE_TOKEN_POLICY.startswith("tt-")
|
|
for f in ("hyphen", "numeral", "accent", "honorific", "brit"):
|
|
assert f in _TITLE_TOKEN_POLICY
|