russell@unturf.com
5990834ce8
Six items from fox's de-novo synthesis (2026-05-01) landing as one atomic commit. Each item references its motivation and respects the v9.8 honesty boundary (no claims of semantic truth, no proof-path contamination by soft signals). NEW DOCS -------- docs/cti-architecture.md Maps today's modules onto the architectural layer cake fox named in his synthesis: PROMETHEUS-Σ (controller) / CTI (claim-lattice reasoning IR) / Merkle-AGI-DAG (commitment) / Reverse-RAG (evidence direction) / v9.8 Providence (admissibility ledger) / Hermes (weak proposer). Per-layer responsibility table + module map showing which existing files implement which layer. The architecture is real; the labels above name what's already there. docs/naming-deferral.md Argues why we keep internal terms (claim_lattice, claim_lattice_pointer, verify_claim_lattice) instead of renaming to architectural labels (CTI, PROMETHEUS-Σ). The rename costs cache invalidation, ~150 test fixture references, schema CHECK migration, blame-history disconnect, mesh peer coordination. The bridge is the cti-architecture mapping doc — read it once, then read code in code's vocabulary and prose in prose's vocabulary. Lists four triggers that would invert the decision. docs/self-reference-distillation-design.md Successor to docs/self-reference-thought-chains-design.md (the flat-source MVP). Maps STRICT claims onto the existing Distiller / Core / derivations infrastructure: each STRICT record becomes a Fact-Core via a new ProvidenceDistiller, with per-cited-chunk Merkle inclusion proofs back to Wikipedia source_roots. The fact-graph grows; new claims attach by inheriting the proof chain. CompositionDistiller (combining facts into new claims) is explicitly deferred — that's reasoning machinery, not infrastructure. docs/test-coverage-audit-2026-05-01.md Maps fox's §11 test list (16 items) against the codebase. 16/16 covered. Documents item #9's manual-quote-rule lifecycle: the rule was retired from pointer mode in commit224bfd6after the bench journey identified it was rejecting factually correct claims; retained in JSON variant where the punctuation-rationale argument doesn't apply. The audit doc itself is the requirements- drift defense. CODE ---- aborist/qa/keys.py:verifier_policy_hash New pure function. Hashes the verifier-relevant subset of policy (answer_mode + claim_lattice_* verifier knobs + entity_policy fields + base_version). Folded into cache_key as an optional 9th dimension via a new keyword arg with backward-compat default None — legacy 8-dim callers continue to work and produce the pre-2026-05-01 hash result. aborist/qa/{runner,query}.py Compute verifier_policy_hash from the policy_variant and pass to cache_key as the 9th dim. Records written from this commit forward bind to the verifier-policy identity. Pre-existing 8-dim records become cache-misses on next lookup — same migration cost as any governance_policy_hash bump. The 9-dim form makes the question "did the verifier rules change?" answerable from cache_key diff alone, separate from "did the prompt change?" CLAUDE.md Adds "old maps vs runtime maps" engineering discipline to the bench-maxing section. Codifies the principle: every base model carries old maps from training time; the runtime carries the fresh map; when they disagree, the runtime wins. Move authority OUT of the model's prior and INTO runtime artifacts (pointer IDs the runtime mints, source spans the runtime interpolates by offset, evidence maps assembled per query, policy hashes that fold prompt/verifier/retrieval into cache_key, hard checks run by the verifier). Hermes' content-addressed-evidence-id hallucination (commitbb8450d) is the canonical case study. TESTS ----- tests/test_keys.py Six new tests for verifier_policy_hash: - only hashes verifier subset (non-verifier fields don't change it) - changes when verifier-relevant field changes - empty-policy → stable - 9-dim cache_key distinct from 8-dim - 9-dim distinct under different verifier hashes - 8-dim form preserved for legacy callers (None == omit) DEFERRED (per fox + naming-deferral.md) --------------------------------------- - PROMETHEUS-Σ as an extracted controller module — the dispatch logic in runner.py + query.py already IS PROMETHEUS-Σ; an explicit prometheus.py is a refactor with no behavior change. Defer until a §5-rename-trigger fires. 507 tests pass (was 501 before, +6 from verifier_policy_hash coverage).
328 lines
12 KiB
Python
328 lines
12 KiB
Python
"""Cache-key dimension hashes — pure functions, deterministic.
|
||
|
||
The 8-dim cache_key invariants are spread across `aborist/qa/keys.py`.
|
||
Most behavior is covered indirectly by `test_qa.py` (cache hits, etc.);
|
||
this file pins the hash-input canonicalization rules so a future tweak
|
||
to question normalization can't silently change the equivalence class
|
||
of "what counts as the same question."
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
from aborist.qa.keys import (
|
||
cache_key,
|
||
conversation_hash,
|
||
governance_policy_hash,
|
||
model_profile_hash,
|
||
question_hash,
|
||
)
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# question_hash equivalence classes
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def test_question_hash_strips_trailing_question_mark():
|
||
"""Fox's catch on 2026-04-29: `who is X?` and `who is X` should hit
|
||
the same providence_cache record. Trailing punctuation carries no
|
||
semantic difference for retrieval."""
|
||
assert question_hash("who is X") == question_hash("who is X?")
|
||
|
||
|
||
def test_question_hash_strips_trailing_period():
|
||
assert question_hash("explain X") == question_hash("explain X.")
|
||
|
||
|
||
def test_question_hash_strips_trailing_exclamation():
|
||
assert question_hash("explain X") == question_hash("explain X!")
|
||
|
||
|
||
def test_question_hash_strips_multiple_trailing_punct():
|
||
"""Multiple trailing chars all stripped — `?!` is a single class."""
|
||
assert question_hash("explain X") == question_hash("explain X?!")
|
||
assert question_hash("explain X") == question_hash("explain X.?")
|
||
|
||
|
||
def test_question_hash_lowercases():
|
||
assert question_hash("Who Is X") == question_hash("who is x")
|
||
|
||
|
||
def test_question_hash_collapses_internal_spaces():
|
||
"""canonicalize() collapses runs of spaces to a single space. Tabs/
|
||
newlines map to newline (a separate equivalence class — see
|
||
aborist.document.canonicalize), so tab vs space is intentionally
|
||
NOT in the same bucket."""
|
||
assert question_hash("who is X") == question_hash("who is X")
|
||
|
||
|
||
def test_question_hash_does_not_strip_internal_punctuation():
|
||
"""Internal `?` is a different question — ``what's up?`` mid-sentence
|
||
can carry meaning. Only trailing punctuation gets stripped."""
|
||
assert question_hash("X, then Y") != question_hash("X then Y")
|
||
|
||
|
||
# CJK + ellipsis equivalence — non-ASCII trailing punctuation should
|
||
# behave identically to ASCII forms. Pin so a future "let's go ASCII-only"
|
||
# refactor can't silently regress i18n.
|
||
|
||
def test_question_hash_strips_full_width_question_mark():
|
||
"""U+FF1F (CJK full-width question mark)."""
|
||
assert question_hash("X") == question_hash("X？")
|
||
|
||
|
||
def test_question_hash_strips_full_width_exclamation():
|
||
"""U+FF01."""
|
||
assert question_hash("X") == question_hash("X！")
|
||
|
||
|
||
def test_question_hash_strips_ideographic_full_stop():
|
||
"""U+3002 (Chinese / Japanese sentence-ending period)."""
|
||
assert question_hash("X") == question_hash("X。")
|
||
|
||
|
||
def test_question_hash_strips_ideographic_comma():
|
||
"""U+3001."""
|
||
assert question_hash("X") == question_hash("X、")
|
||
|
||
|
||
def test_question_hash_strips_ellipsis():
|
||
"""U+2026 — single-codepoint ellipsis, plus the three-dot ASCII form
|
||
that already worked via repeated `.` strip."""
|
||
assert question_hash("X") == question_hash("X…")
|
||
assert question_hash("X") == question_hash("X...")
|
||
|
||
|
||
def test_question_hash_strips_mixed_ascii_and_cjk_trailing():
|
||
"""A question ending with both kinds of trailing punctuation strips
|
||
cleanly. rstrip walks char-by-char from the right."""
|
||
assert question_hash("X") == question_hash("X？！.")
|
||
|
||
|
||
# Quotes and brackets PAIR — naive trailing-strip would break balance.
|
||
# Pin that they stay so a future "expand the strip set" refactor knows
|
||
# the design rule.
|
||
|
||
def test_question_hash_preserves_trailing_double_quote():
|
||
"""`who said "X"?` (after stripping `?`) ends in `"`. Stripping the
|
||
quote would unbalance the surface; the equivalence class would
|
||
no longer match a balanced version."""
|
||
assert question_hash('who said "X"') != question_hash('who said "X')
|
||
|
||
|
||
def test_question_hash_preserves_trailing_single_quote():
|
||
"""Apostrophes look like quotes. `X's` (possessive) must stay distinct
|
||
from `X` (the bare entity)."""
|
||
assert question_hash("who is X's") != question_hash("who is X")
|
||
|
||
|
||
def test_question_hash_preserves_trailing_close_paren():
|
||
"""`(foo)` — a paren pair carries grouping meaning."""
|
||
assert question_hash("who is X (Y)") != question_hash("who is X (Y")
|
||
|
||
|
||
def test_question_hash_preserves_trailing_close_bracket():
|
||
assert question_hash("who is X [a]") != question_hash("who is X [a")
|
||
|
||
|
||
def test_question_hash_preserves_trailing_close_brace():
|
||
assert question_hash("who is X {a}") != question_hash("who is X {a")
|
||
|
||
|
||
# Article stripping — fox's 2026-04-29 catch: "the batman" and "batman"
|
||
# produced different cache records. Articles are filler at the question-
|
||
# equivalence layer; strip them so cache lookups dedupe.
|
||
|
||
def test_question_hash_strips_leading_article_the():
|
||
"""`who is the batman` and `who is batman` ask the same question."""
|
||
assert question_hash("who is the batman") == question_hash("who is batman")
|
||
|
||
|
||
def test_question_hash_strips_leading_article_a():
|
||
assert question_hash("what is a hot dog") == question_hash("what is hot dog")
|
||
|
||
|
||
def test_question_hash_strips_leading_article_an():
|
||
assert question_hash("what is an apple") == question_hash("what is apple")
|
||
|
||
|
||
def test_question_hash_strips_articles_anywhere_in_question():
|
||
"""Articles in the middle of the sentence also strip."""
|
||
assert (
|
||
question_hash("does the cat sit on a mat")
|
||
== question_hash("does cat sit on mat")
|
||
)
|
||
|
||
|
||
def test_question_hash_does_not_strip_article_substrings():
|
||
"""`thesis` contains `the` but is one token; substring matching
|
||
would corrupt it. Tokens are matched as exact lowercase strings."""
|
||
assert (
|
||
question_hash("explain my thesis")
|
||
!= question_hash("explain my sis")
|
||
)
|
||
# And `thesis` survives the strip:
|
||
assert "explain my thesis" in question_hash.__doc__ or True # docstring sanity
|
||
# The actual canonical form has `thesis`:
|
||
h_thesis = question_hash("thesis")
|
||
h_other = question_hash("the sis")
|
||
# "the sis" → "sis" after article strip; "thesis" stays "thesis"
|
||
assert h_thesis != h_other
|
||
|
||
|
||
def test_question_hash_handles_question_with_only_articles_and_topic():
|
||
"""Edge: very short question. `the X` and `X` and `a X` collapse."""
|
||
assert question_hash("the batman") == question_hash("batman")
|
||
assert question_hash("a batman") == question_hash("batman")
|
||
assert question_hash("an batman") == question_hash("batman")
|
||
|
||
|
||
def test_question_hash_distinct_topics_still_distinct():
|
||
"""Article strip doesn't accidentally collapse different topics."""
|
||
assert question_hash("the batman") != question_hash("the superman")
|
||
assert question_hash("a batman") != question_hash("a superman")
|
||
|
||
|
||
def test_question_hash_distinct_for_different_text():
|
||
"""Sanity: two genuinely different questions hash to different values."""
|
||
assert question_hash("who is X") != question_hash("who is Y")
|
||
|
||
|
||
def test_question_hash_is_deterministic():
|
||
"""Same input → same output, every time. No randomness."""
|
||
a = question_hash("who is russell ballestrini?")
|
||
b = question_hash("who is russell ballestrini?")
|
||
assert a == b
|
||
assert len(a) == 64 # sha256 hex
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Other dim hashes — light coverage so future refactors don't drift
|
||
# ---------------------------------------------------------------------------
|
||
|
||
|
||
def test_model_profile_hash_includes_revision_and_quantization():
|
||
a = model_profile_hash("m", revision="r1", quantization="q1")
|
||
b = model_profile_hash("m", revision="r2", quantization="q1")
|
||
c = model_profile_hash("m", revision="r1", quantization="q2")
|
||
assert a != b
|
||
assert a != c
|
||
assert b != c
|
||
|
||
|
||
def test_conversation_hash_canonicalizes_message_order():
|
||
"""Message ordering matters — a different order is a different
|
||
conversation. Pin so anyone tempted to sort messages knows it would
|
||
invalidate cache."""
|
||
msgs1 = [{"role": "system", "content": "a"}, {"role": "user", "content": "b"}]
|
||
msgs2 = [{"role": "user", "content": "b"}, {"role": "system", "content": "a"}]
|
||
assert conversation_hash(msgs1) != conversation_hash(msgs2)
|
||
|
||
|
||
def test_governance_policy_hash_sensitive_to_keys():
|
||
a = governance_policy_hash({"k": 1})
|
||
b = governance_policy_hash({"k": 2})
|
||
c = governance_policy_hash({"j": 1})
|
||
assert a != b
|
||
assert a != c
|
||
|
||
|
||
def test_cache_key_combines_all_eight_dims():
|
||
"""If any of the 8 dims changes, cache_key changes. v9.8 invariant."""
|
||
base = ("src", "qh", "mh", "ch", "gh", "v9.8.0", "norm-v1", "tok-512-v1")
|
||
base_key = cache_key(*base)
|
||
for i in range(8):
|
||
mutated = list(base)
|
||
mutated[i] = mutated[i] + "_modified"
|
||
assert cache_key(*mutated) != base_key, (
|
||
f"dim {i} change did not bump cache_key — 8-dim invariant broken"
|
||
)
|
||
|
||
|
||
# ---------------------------------------------------------------- verifier_policy_hash
|
||
|
||
|
||
def test_verifier_policy_hash_only_hashes_verifier_subset():
|
||
"""Non-verifier fields (temperature, prompts, etc.) must NOT enter
|
||
verifier_policy_hash. The whole point of separating it from
|
||
governance_policy_hash is to surface verifier-rule changes
|
||
independently from prompt / sampling-knob changes.
|
||
"""
|
||
from aborist.qa.keys import verifier_policy_hash
|
||
|
||
base = {
|
||
"answer_mode": "claim_lattice_pointer",
|
||
"claim_lattice_min_citation_coverage": 0.30,
|
||
"temperature": 0.1, # non-verifier
|
||
"system_prompt": "blah", # non-verifier
|
||
"max_tokens": 512, # non-verifier
|
||
"grounding_reminder": "blah", # non-verifier
|
||
}
|
||
h_base = verifier_policy_hash(base)
|
||
# Changing a non-verifier field must NOT change the hash.
|
||
h_temp = verifier_policy_hash(dict(base, temperature=0.5))
|
||
h_prompt = verifier_policy_hash(dict(base, system_prompt="something else"))
|
||
h_tokens = verifier_policy_hash(dict(base, max_tokens=2048))
|
||
assert h_base == h_temp
|
||
assert h_base == h_prompt
|
||
assert h_base == h_tokens
|
||
|
||
|
||
def test_verifier_policy_hash_changes_when_verifier_field_changes():
|
||
"""Verifier-relevant fields MUST change the hash. Bumping any one
|
||
of these means the verifier rules differ and a new cache record
|
||
is required.
|
||
"""
|
||
from aborist.qa.keys import verifier_policy_hash
|
||
|
||
base = {
|
||
"answer_mode": "claim_lattice_pointer",
|
||
"claim_lattice_min_citation_coverage": 0.30,
|
||
"claim_lattice_max_pointers_per_claim": 2,
|
||
"claim_lattice_min_claim_content_tokens": 2,
|
||
}
|
||
h_base = verifier_policy_hash(base)
|
||
assert verifier_policy_hash(dict(base, answer_mode="quote")) != h_base
|
||
assert verifier_policy_hash(
|
||
dict(base, claim_lattice_min_citation_coverage=0.50)
|
||
) != h_base
|
||
assert verifier_policy_hash(
|
||
dict(base, claim_lattice_max_pointers_per_claim=3)
|
||
) != h_base
|
||
assert verifier_policy_hash(
|
||
dict(base, claim_lattice_min_claim_content_tokens=3)
|
||
) != h_base
|
||
|
||
|
||
def test_verifier_policy_hash_empty_policy_is_stable():
|
||
"""Empty policy → constant hash. Stable across runs."""
|
||
from aborist.qa.keys import verifier_policy_hash
|
||
assert verifier_policy_hash({}) == verifier_policy_hash({})
|
||
|
||
|
||
def test_cache_key_nine_dim_form_distinguishes_verifier_policy():
|
||
"""Adding the 9th dimension produces a distinct cache_key from
|
||
the 8-dim form. Records written under 8-dim form cannot be
|
||
retrieved under 9-dim form even when all common dims match —
|
||
that's the migration cost of adding the dimension."""
|
||
eight = ("src", "qh", "mh", "ch", "gh", "v9.8.0", "norm-v1", "tok-512-v1")
|
||
k_eight = cache_key(*eight)
|
||
k_nine = cache_key(*eight, "vh")
|
||
assert k_eight != k_nine
|
||
|
||
|
||
def test_cache_key_nine_dim_changes_with_verifier_hash():
|
||
"""Two records with identical 8 dims but different
|
||
verifier_policy_hash get distinct cache_keys."""
|
||
eight = ("src", "qh", "mh", "ch", "gh", "v9.8.0", "norm-v1", "tok-512-v1")
|
||
a = cache_key(*eight, "vh-1")
|
||
b = cache_key(*eight, "vh-2")
|
||
assert a != b
|
||
|
||
|
||
def test_cache_key_eight_dim_form_preserved_for_legacy_callers():
|
||
"""Calling cache_key with 8 args (or with the 9th = None) must
|
||
return the same hash a pre-2026-05-01 caller would have got.
|
||
Backward-compat is the legacy-INSERT path's lifeline."""
|
||
eight = ("src", "qh", "mh", "ch", "gh", "v9.8.0", "norm-v1", "tok-512-v1")
|
||
assert cache_key(*eight) == cache_key(*eight, None)
|