qa: dedup-mode policy + JIT fidelity — agents pick fast-cache vs audit-grade
Different agents have different value functions on the same providence
records. Today aborist bakes one canonicalization policy into
governance_policy_hash and calls it universal. This adds two knobs that
let agents express their preference without breaking provenance.
- aborist/qa/keys.py: `canonical_question(q, mode=...)` and
`question_hash(q, mode=...)` accept "strict" (NFC + ws-collapse only;
every variant gets its own hash) or "equivalence_class" (default —
additionally lowercase + trailing-punct strip + article strip).
New constants: QUESTION_DEDUP_MODES, FIDELITY_MODES, defaults.
- aborist/qa/query.py + aborist/qa/runner.py: each call computes the
primary cache_key under policy["question_dedup"]. New `fidelity`
parameter:
"strict" only primary cache_key checked
"equivalence_class" primary first; if miss AND alternate mode
produces a different cache_key, try alternate
Cross-silo fallback works because _ckey_for_mode rewrites
policy["question_dedup"] to the alternate mode before computing
governance_policy_hash — so the fallback ckey matches what an agent
under that mode would have written. Lookups can find each other's
records when fidelity permits. Result dict gains `lookup_path` ∈
{"strict", "equivalence_class", "strict_fallback",
"equivalence_class_fallback", "miss"}.
- aborist/cli.py: `--question-dedup` and `--fidelity` flags on the
`query` subcommand. Human render annotates cache_hit lines that
came from a fallback ckey ("cached via equivalence_class_fallback").
- tests/test_query.py: three new regressions
* strict-policy + strict-fidelity: variants get distinct cache_keys
* cross-silo fallback: strict-policy reads eq-class-policy records
via fidelity=equivalence_class
* strict-fidelity refuses fallback (audit-grade): cache miss even
when the alternate silo has a hit
- CLAUDE.md: dedup-mode and fidelity convention bullets updated.
This is V1 + V2 of the substrate move sketched in conversation: write
policy determines which silo a record lives in; read fidelity determines
how loosely an agent walks across silos. Records stay exactly-keyed
(provenance hard); routing is per-agent (preference soft). Same
verifier_no_diagnostics discipline holds — no soft signal enters the
hard chain.
442 tests pass.
This commit is contained in:
parent
1e8d2d2dba
commit
aefc032749
6 changed files with 431 additions and 90 deletions
|
|
@ -87,7 +87,8 @@ make chain-check-shards # audit-chain break count per shard (0 = intact)
|
|||
- **Chunker default = `tok-512-v1`.** Changing the default bumps `chunking_version` and **stales every prior cache record**. Add a new chunker as a new `name` instead.
|
||||
- **Canonicalization = `norm-v1`** (NFC, collapsed whitespace). Same rule.
|
||||
- **Schema = `v9.8.0`.** Same rule.
|
||||
- **`question_hash` canonicalization** drops trailing punctuation (ASCII `.?!,;:`, CJK `?!。、`, ellipsis `…`) AND standalone English articles (`the`, `a`, `an`) after lowercase + NFC + whitespace-collapse. So `"who is X"`, `"who is X?"`, `"Who Is X."`, `"who is the X"`, `"who is an X?"` all hash to the same value. Tokens are matched as exact lowercase strings, so substrings like `"thesis"` (contains `the`) stay untouched. Conservative on purpose — only ASCII English articles. Bumping `_QUESTION_TRAILING_STRIP` or `_QUESTION_ARTICLE_STRIP` orphans prior records whose canonical question contained newly-stripped tokens; they live as history but won't be re-hit on lookup.
|
||||
- **`question_hash` is dedup-mode-aware.** `question_hash(q, mode=...)` accepts `"strict"` (NFC + ws-collapse only — case-sensitive, punctuation-sensitive, article-sensitive; every variant gets its own hash) or `"equivalence_class"` (default — additionally lowercase + trailing-punct strip + standalone-article strip; collapses `"who is X"`, `"who is X?"`, `"Who Is X."`, `"who is the X"`, `"who is an X?"` to one hash). Mode lives in `policy["question_dedup"]` so it folds into `governance_policy_hash`; agents under different modes write records under different cache_keys.
|
||||
- **JIT lookup fidelity.** Per-call `fidelity` parameter on `query()`/`ask()` decouples lookup tolerance from write policy. `"strict"` checks only the primary cache_key (audit-grade — record reuse only on exact match). `"equivalence_class"` (default) tries primary first, then the alternate dedup mode's cache_key as a fallback so a fast-cache agent can reuse records written under either policy. Result includes `lookup_path` ∈ `{"strict", "equivalence_class", "strict_fallback", "equivalence_class_fallback", "miss"}`. Cross-silo fallback works because the helper rewrites `policy["question_dedup"]` to the alternate mode when computing the fallback ckey, so `governance_policy_hash` matches what an agent under that mode would have written. CLI: `--question-dedup`, `--fidelity` flags on `query` (and the `ask` API params).
|
||||
- **`audit_mode` is decided by the verifier, never asserted unconditionally.** Four layered strategies in `aborist/qa/verify.py`, tried in order; first to find evidence classifies the answer:
|
||||
1. **quote** — model wrapped claims in double quotes per system prompt. Sequential pairing: 1st & 2nd `"`, 3rd & 4th, etc. (NOT regex pairing — that captures inter-pair prose as a phantom span when the model writes `"title" prose "quote"`).
|
||||
2. **span** — bullet/sentence lines from the answer appear verbatim in context. Catches models that quote inline without `"..."` marks.
|
||||
|
|
|
|||
|
|
@ -343,6 +343,13 @@ def _cmd_query(args: argparse.Namespace) -> int:
|
|||
)
|
||||
single_db = None if shards_dir else args.db
|
||||
|
||||
# Apply per-call policy overrides (question_dedup) on top of the
|
||||
# default. fidelity is a function-level kwarg, not in the policy.
|
||||
from aborist.qa.query import DEFAULT_QUERY_POLICY
|
||||
call_policy = dict(DEFAULT_QUERY_POLICY)
|
||||
if getattr(args, "question_dedup", None):
|
||||
call_policy["question_dedup"] = args.question_dedup
|
||||
|
||||
result = query(
|
||||
question=args.question,
|
||||
qa_db=qa_db,
|
||||
|
|
@ -355,6 +362,8 @@ def _cmd_query(args: argparse.Namespace) -> int:
|
|||
top_k=args.top_k,
|
||||
over_fetch=args.over_fetch,
|
||||
max_context_chars=args.max_context_chars,
|
||||
policy=call_policy,
|
||||
fidelity=getattr(args, "fidelity", None),
|
||||
)
|
||||
|
||||
if args.json:
|
||||
|
|
@ -401,6 +410,12 @@ def _render_query_human(result: dict, question: str) -> str:
|
|||
total_ms = timings.get("total_ms")
|
||||
elapsed = f"{total_ms / 1000:.1f}s" if isinstance(total_ms, (int, float)) else "?"
|
||||
cache_status = "cached" if status == "cache_hit" else "fresh"
|
||||
lookup_path = result.get("lookup_path")
|
||||
# Annotate cache_hits when they came from a fallback ckey, not the
|
||||
# primary one — useful when an agent ran with fidelity=equivalence_class
|
||||
# and reused another agent's record.
|
||||
if lookup_path and lookup_path.endswith("_fallback"):
|
||||
cache_status = f"cached via {lookup_path}"
|
||||
|
||||
lines: list[str] = []
|
||||
lines.append(question)
|
||||
|
|
@ -2688,6 +2703,25 @@ def build_parser() -> argparse.ArgumentParser:
|
|||
"--max-context-chars", dest="max_context_chars", type=int, default=60000,
|
||||
help="cap on assembled context bytes (default 60000)",
|
||||
)
|
||||
query_cmd.add_argument(
|
||||
"--question-dedup", dest="question_dedup", default=None,
|
||||
choices=["strict", "equivalence_class"],
|
||||
help=(
|
||||
"write-time question canonicalization. 'equivalence_class' "
|
||||
"(default) collapses articles + trailing-punct + case so "
|
||||
"variants share cache_keys. 'strict' keeps every variant "
|
||||
"distinct (audit-grade)."
|
||||
),
|
||||
)
|
||||
query_cmd.add_argument(
|
||||
"--fidelity", dest="fidelity", default=None,
|
||||
choices=["strict", "equivalence_class"],
|
||||
help=(
|
||||
"lookup tolerance. 'equivalence_class' (default) tries the "
|
||||
"primary cache_key then falls back to the alternate dedup "
|
||||
"mode's cache_key. 'strict' refuses fallback."
|
||||
),
|
||||
)
|
||||
query_cmd.add_argument(
|
||||
"--qa-db", dest="qa_db", default=None,
|
||||
help=(
|
||||
|
|
|
|||
|
|
@ -35,47 +35,80 @@ def _canonical_json(obj) -> str:
|
|||
return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
|
||||
|
||||
|
||||
def canonical_question(question: str) -> str:
|
||||
"""Canonical equivalence-class form of `question` text.
|
||||
QUESTION_DEDUP_MODES = ("strict", "equivalence_class")
|
||||
DEFAULT_QUESTION_DEDUP = "equivalence_class"
|
||||
|
||||
Same four-step normalization that `question_hash` hashes, exposed
|
||||
so callers can dedup BEFORE hashing — e.g. inject the canonical
|
||||
form into the user message used for `conversation_hash`, while
|
||||
still sending the verbatim question to the LLM. Without this split
|
||||
`"who is batman"`, `"who is batman?"`, and `"who is the batman?"`
|
||||
all collapse on `question_hash` but each hits `conversation_hash`
|
||||
differently, missing cache.
|
||||
# Lookup-time fidelity. Decoupled from write-time `question_dedup`:
|
||||
# write determines under which `cache_key` a record lands; fidelity
|
||||
# determines which `cache_key`s a lookup will check.
|
||||
#
|
||||
# strict only the cache_key matching the agent's policy
|
||||
# is checked. No fallback. Audit-grade behavior.
|
||||
# equivalence_class primary cache_key checked first; if miss AND the
|
||||
# OTHER dedup mode produces a different cache_key,
|
||||
# the alternate is also checked. Lets a fast-cache
|
||||
# agent reuse records written under either mode.
|
||||
FIDELITY_MODES = ("strict", "equivalence_class")
|
||||
DEFAULT_FIDELITY = "equivalence_class"
|
||||
|
||||
|
||||
def canonical_question(
|
||||
question: str, *, mode: str = DEFAULT_QUESTION_DEDUP
|
||||
) -> str:
|
||||
"""Canonical form of `question` for the given dedup ``mode``.
|
||||
|
||||
Two modes:
|
||||
|
||||
- ``"equivalence_class"`` (default): four-step canonicalization —
|
||||
``canonicalize()`` (NFC + ws-collapse + strip ends), then
|
||||
lowercase, then trailing-punctuation strip, then standalone-article
|
||||
filter (``the``, ``a``, ``an``). All variants of "Who is THE
|
||||
Batman?" / "who is batman" / "who is X." collapse to one form.
|
||||
The default for chat-style agents that prefer fast cache hits.
|
||||
|
||||
- ``"strict"``: only ``canonicalize()`` — NFC + ws-collapse + strip
|
||||
ends. Case-sensitive, punctuation-sensitive, article-sensitive.
|
||||
Maximum granularity. The choice for audit-grade agents that want
|
||||
every distinct phrasing to get its own answer.
|
||||
|
||||
Exposed as a function so callers can dedup BEFORE hashing — e.g.
|
||||
inject the canonical form into the user message used for
|
||||
``conversation_hash``, while still sending the verbatim question to
|
||||
the LLM. Without this split ``"who is batman"``, ``"who is
|
||||
batman?"``, and ``"who is the batman?"`` collapse on
|
||||
``question_hash`` (under equivalence_class) but each hits
|
||||
``conversation_hash`` differently, missing cache.
|
||||
|
||||
The choice of mode flows through ``policy["question_dedup"]`` into
|
||||
``governance_policy_hash`` so two agents under different modes
|
||||
write records under different ``cache_key``s — they coexist in
|
||||
parallel namespaces, never collide.
|
||||
"""
|
||||
canon = canonicalize(question).lower().rstrip(_QUESTION_TRAILING_STRIP)
|
||||
if mode not in QUESTION_DEDUP_MODES:
|
||||
raise ValueError(
|
||||
f"question dedup mode must be one of {QUESTION_DEDUP_MODES}, got {mode!r}"
|
||||
)
|
||||
canon = canonicalize(question)
|
||||
if mode == "strict":
|
||||
return canon
|
||||
canon = canon.lower().rstrip(_QUESTION_TRAILING_STRIP)
|
||||
tokens = [t for t in canon.split() if t not in _QUESTION_ARTICLE_STRIP]
|
||||
return " ".join(tokens)
|
||||
|
||||
|
||||
def question_hash(question: str) -> str:
|
||||
"""SHA-256 of canonicalized + lowercased + punctuation-stripped +
|
||||
article-stripped question text.
|
||||
def question_hash(
|
||||
question: str, *, mode: str = DEFAULT_QUESTION_DEDUP
|
||||
) -> str:
|
||||
"""SHA-256 of the dedup-mode-canonicalized question.
|
||||
|
||||
Four-step canonicalization, in order:
|
||||
|
||||
1. ``canonicalize()`` — NFC, whitespace collapse, strip ends.
|
||||
2. ``.lower()`` — case-fold so "Who is X" == "who is X".
|
||||
3. ``rstrip(_QUESTION_TRAILING_STRIP)`` — trailing punctuation
|
||||
carries no semantic difference for retrieval. ``"who is X?"``
|
||||
and ``"who is X"`` map to the same cache_key.
|
||||
4. drop standalone articles (``the``, ``a``, ``an``). ``"who is the
|
||||
batman"`` and ``"who is batman"`` ask the same question — fox
|
||||
caught this on 2026-04-29 when the two forms produced different
|
||||
cache records. Articles are filtered as exact-match standalone
|
||||
tokens (split on whitespace), so ``"thesis"`` stays untouched.
|
||||
|
||||
Steps 3-4 are question-specific — chunk leaf hashes stay verbatim
|
||||
because they go through ``canonicalize()`` directly, not through
|
||||
this function. Bumping ``_QUESTION_TRAILING_STRIP`` or
|
||||
``_QUESTION_ARTICLE_STRIP`` orphans prior cache records whose
|
||||
See ``canonical_question`` for what each mode does. The hash is the
|
||||
SHA-256 of the canonical form. Bumping
|
||||
``_QUESTION_TRAILING_STRIP`` or ``_QUESTION_ARTICLE_STRIP`` (the
|
||||
equivalence-class strip sets) orphans prior cache records whose
|
||||
canonical question contained newly-stripped tokens; they live as
|
||||
history but won't be re-hit on lookup.
|
||||
|
||||
Equivalence class examples::
|
||||
Equivalence class examples (mode="equivalence_class")::
|
||||
|
||||
"who is X" |
|
||||
"who is X?" |
|
||||
|
|
@ -84,13 +117,15 @@ def question_hash(question: str) -> str:
|
|||
"who is a X" |
|
||||
"who is an X?" | (CJK question mark)
|
||||
|
||||
Strict mode (mode="strict") distinguishes all of those.
|
||||
|
||||
What's IN the trailing-strip set: ``.?!,;:`` (ASCII), ``?!。、``
|
||||
(CJK full-width), ``…`` (ellipsis). Pairs like ``"`` ``'`` ``)``
|
||||
``]`` ``}`` are NOT — naive one-sided stripping breaks balance.
|
||||
Apostrophes aren't either — ``X's`` is a different question from
|
||||
``X``.
|
||||
"""
|
||||
return _sha256(canonical_question(question))
|
||||
return _sha256(canonical_question(question, mode=mode))
|
||||
|
||||
|
||||
# Trailing punctuation that carries no semantic difference at the end
|
||||
|
|
|
|||
|
|
@ -53,6 +53,10 @@ from aborist.qa.concepts import (
|
|||
synonym_expand,
|
||||
)
|
||||
from aborist.qa.keys import (
|
||||
DEFAULT_FIDELITY,
|
||||
DEFAULT_QUESTION_DEDUP,
|
||||
FIDELITY_MODES,
|
||||
QUESTION_DEDUP_MODES,
|
||||
cache_key,
|
||||
canonical_question,
|
||||
conversation_hash,
|
||||
|
|
@ -608,9 +612,25 @@ def query(
|
|||
max_context_chars: int = 60000,
|
||||
policy: dict | None = None,
|
||||
chain: str = "private",
|
||||
fidelity: str | None = None,
|
||||
) -> dict:
|
||||
"""Answer `question` using the corpus. Cache to qa_db. Returns a result dict."""
|
||||
"""Answer `question` using the corpus. Cache to qa_db. Returns a result dict.
|
||||
|
||||
`fidelity` controls lookup tolerance — see ``FIDELITY_MODES`` in
|
||||
``aborist.qa.keys``. ``"strict"`` checks only the cache_key
|
||||
matching this call's ``policy["question_dedup"]``. ``"equivalence_class"``
|
||||
(default) tries the primary cache_key first, then the alternate
|
||||
dedup-mode cache_key as a fallback so a fast-cache agent can reuse
|
||||
a record written under either mode. Result includes ``lookup_path``
|
||||
naming which key matched (or ``"miss"`` when the LLM ran).
|
||||
"""
|
||||
policy = policy or DEFAULT_QUERY_POLICY
|
||||
if fidelity is None:
|
||||
fidelity = policy.get("fidelity", DEFAULT_FIDELITY)
|
||||
if fidelity not in FIDELITY_MODES:
|
||||
raise ValueError(
|
||||
f"fidelity must be one of {FIDELITY_MODES}, got {fidelity!r}"
|
||||
)
|
||||
t_start = time.monotonic()
|
||||
|
||||
# 1. Search.
|
||||
|
|
@ -708,13 +728,6 @@ def query(
|
|||
# user-turn instructions more reliably than system-turn rules under long
|
||||
# context). The sources + question land last so they are the most-recent
|
||||
# tokens before generation begins.
|
||||
#
|
||||
# Decoupled hash form vs LLM form: `messages` carries the verbatim
|
||||
# question (Hermes sees the user's natural phrasing); `canonical_messages`
|
||||
# substitutes the canonical question (no articles, no trailing
|
||||
# punctuation) so `conversation_hash` collapses on the equivalence class.
|
||||
# `who is batman`, `who is batman?`, `who is the batman?` → same chash.
|
||||
canonical_q = canonical_question(question)
|
||||
messages = [
|
||||
{"role": "system", "content": policy["system_prompt"]},
|
||||
]
|
||||
|
|
@ -728,37 +741,92 @@ def query(
|
|||
"content": f"Sources:\n\n{context}\n\n---\n\nQuestion: {question}",
|
||||
}
|
||||
)
|
||||
canonical_messages = list(messages[:-1]) + [
|
||||
context_root = _context_root([h.document_root for h in chosen])
|
||||
mhash = model_profile_hash(model_id, revision, quantization)
|
||||
|
||||
# Dedup-mode-aware cache_key build. For each mode we substitute the
|
||||
# mode's canonical question form into the user message used for
|
||||
# `conversation_hash` (LLM still sees the verbatim question), AND we
|
||||
# vary `policy["question_dedup"]` to match the mode so
|
||||
# `governance_policy_hash` matches what an agent under that mode
|
||||
# would have written. This makes cross-silo fallback work: a
|
||||
# strict-policy agent looking up with equivalence_class fidelity can
|
||||
# find a record written by an equivalence_class-policy agent.
|
||||
def _ckey_for_mode(mode: str) -> str:
|
||||
canon_q = canonical_question(question, mode=mode)
|
||||
canon_msgs = list(messages[:-1]) + [
|
||||
{
|
||||
"role": "user",
|
||||
"content": (
|
||||
f"Sources:\n\n{context}\n\n---\n\nQuestion: {canon_q}"
|
||||
),
|
||||
}
|
||||
]
|
||||
policy_variant = dict(policy, question_dedup=mode)
|
||||
return cache_key(
|
||||
context_root,
|
||||
question_hash(question, mode=mode),
|
||||
mhash,
|
||||
conversation_hash(canon_msgs),
|
||||
governance_policy_hash(policy_variant),
|
||||
SCHEMA_VERSION,
|
||||
CANONICALIZATION_VERSION,
|
||||
CHUNKING_VERSION,
|
||||
)
|
||||
|
||||
ghash = governance_policy_hash(policy) # for the legacy INSERT below
|
||||
|
||||
primary_dedup = policy.get("question_dedup", DEFAULT_QUESTION_DEDUP)
|
||||
if primary_dedup not in QUESTION_DEDUP_MODES:
|
||||
raise ValueError(
|
||||
f"policy['question_dedup'] must be one of {QUESTION_DEDUP_MODES}, "
|
||||
f"got {primary_dedup!r}"
|
||||
)
|
||||
# Re-derive the per-mode hashes for use in the INSERT below. The legacy
|
||||
# INSERT references qhash/chash by name; _ckey_for_mode already builds
|
||||
# them but doesn't expose the intermediates.
|
||||
qhash = question_hash(question, mode=primary_dedup)
|
||||
canonical_q_primary = canonical_question(question, mode=primary_dedup)
|
||||
canonical_messages_primary = list(messages[:-1]) + [
|
||||
{
|
||||
"role": "user",
|
||||
"content": f"Sources:\n\n{context}\n\n---\n\nQuestion: {canonical_q}",
|
||||
"content": (
|
||||
f"Sources:\n\n{context}\n\n---\n\nQuestion: {canonical_q_primary}"
|
||||
),
|
||||
}
|
||||
]
|
||||
context_root = _context_root([h.document_root for h in chosen])
|
||||
qhash = question_hash(question)
|
||||
mhash = model_profile_hash(model_id, revision, quantization)
|
||||
chash = conversation_hash(canonical_messages)
|
||||
ghash = governance_policy_hash(policy)
|
||||
ckey = cache_key(
|
||||
context_root,
|
||||
qhash,
|
||||
mhash,
|
||||
chash,
|
||||
ghash,
|
||||
SCHEMA_VERSION,
|
||||
CANONICALIZATION_VERSION,
|
||||
CHUNKING_VERSION,
|
||||
)
|
||||
chash = conversation_hash(canonical_messages_primary)
|
||||
primary_ckey = _ckey_for_mode(primary_dedup)
|
||||
ckey = primary_ckey # keep the legacy name in the rest of the function
|
||||
|
||||
qa_conn = connect(qa_db)
|
||||
try:
|
||||
# 4. Cache lookup.
|
||||
# 4. Cache lookup. Try the primary dedup-mode cache_key first.
|
||||
# If fidelity allows fallback AND the alternate dedup mode
|
||||
# produces a different cache_key, try that too — lets a
|
||||
# fast-cache agent reuse a record written under either mode.
|
||||
t_phase = time.monotonic()
|
||||
cached = qa_conn.execute(
|
||||
"SELECT * FROM providence_cache "
|
||||
"WHERE cache_key = ? AND falsification_state = 'live'",
|
||||
(ckey,),
|
||||
(primary_ckey,),
|
||||
).fetchone()
|
||||
hit_ckey = primary_ckey
|
||||
lookup_path = primary_dedup if cached is not None else None
|
||||
if cached is None and fidelity == "equivalence_class":
|
||||
other_mode = (
|
||||
"equivalence_class" if primary_dedup == "strict" else "strict"
|
||||
)
|
||||
other_ckey = _ckey_for_mode(other_mode)
|
||||
if other_ckey != primary_ckey:
|
||||
cached = qa_conn.execute(
|
||||
"SELECT * FROM providence_cache "
|
||||
"WHERE cache_key = ? AND falsification_state = 'live'",
|
||||
(other_ckey,),
|
||||
).fetchone()
|
||||
if cached is not None:
|
||||
hit_ckey = other_ckey
|
||||
lookup_path = f"{other_mode}_fallback"
|
||||
cache_lookup_ms = _ms_since(t_phase)
|
||||
if cached is not None:
|
||||
now = int(time.time())
|
||||
|
|
@ -767,12 +835,13 @@ def query(
|
|||
"UPDATE providence_cache "
|
||||
"SET hit_count = hit_count + 1, last_hit_at = ? "
|
||||
"WHERE cache_key = ?",
|
||||
(now, ckey),
|
||||
(now, hit_ckey),
|
||||
)
|
||||
return {
|
||||
"status": "cache_hit",
|
||||
"audit_mode": cached["audit_mode"],
|
||||
"cache_key": ckey,
|
||||
"cache_key": hit_ckey,
|
||||
"lookup_path": lookup_path,
|
||||
"context_root": context_root,
|
||||
"answer_text": cached["answer_text"],
|
||||
"sources": json.loads(cached["merkle_proof"])["sources"],
|
||||
|
|
@ -900,6 +969,7 @@ def query(
|
|||
"status": "cache_miss_then_written",
|
||||
"audit_mode": verdict["audit_mode"],
|
||||
"cache_key": ckey,
|
||||
"lookup_path": "miss",
|
||||
"context_root": context_root,
|
||||
"answer_text": answer_text,
|
||||
"sources": proof_obj["sources"],
|
||||
|
|
|
|||
|
|
@ -23,6 +23,10 @@ from aborist.compress import unpack_chunk
|
|||
from aborist.merkle import MerkleTree, proof_to_dict
|
||||
from aborist.qa.client import ChatClient
|
||||
from aborist.qa.keys import (
|
||||
DEFAULT_FIDELITY,
|
||||
DEFAULT_QUESTION_DEDUP,
|
||||
FIDELITY_MODES,
|
||||
QUESTION_DEDUP_MODES,
|
||||
cache_key,
|
||||
canonical_question,
|
||||
conversation_hash,
|
||||
|
|
@ -91,9 +95,24 @@ def ask(
|
|||
quantization: str = "",
|
||||
policy: dict | None = None,
|
||||
chain: str = "private",
|
||||
fidelity: str | None = None,
|
||||
) -> dict:
|
||||
"""Look up cached answer or run inference. Returns a result dict."""
|
||||
"""Look up cached answer or run inference. Returns a result dict.
|
||||
|
||||
See ``aborist.qa.query.query`` for `fidelity` semantics — it
|
||||
controls lookup tolerance: ``"strict"`` only checks the cache_key
|
||||
matching the call's ``policy["question_dedup"]``; the default
|
||||
``"equivalence_class"`` falls back to the alternate dedup mode's
|
||||
cache_key on miss so a fast-cache agent can reuse records written
|
||||
under either mode. Result includes ``lookup_path``.
|
||||
"""
|
||||
policy = policy or DEFAULT_POLICY
|
||||
if fidelity is None:
|
||||
fidelity = policy.get("fidelity", DEFAULT_FIDELITY)
|
||||
if fidelity not in FIDELITY_MODES:
|
||||
raise ValueError(
|
||||
f"fidelity must be one of {FIDELITY_MODES}, got {fidelity!r}"
|
||||
)
|
||||
t_start = time.monotonic()
|
||||
|
||||
doc = conn.execute(
|
||||
|
|
@ -126,13 +145,6 @@ def ask(
|
|||
# System sets the policy; a user-turn reminder restates the verbatim-quote
|
||||
# rule one message before the document arrives. Document + question land
|
||||
# last as the most-recent tokens before generation.
|
||||
#
|
||||
# Decoupled hash form vs LLM form: `messages` carries the verbatim
|
||||
# question; `canonical_messages` substitutes the canonical question
|
||||
# (no articles, no trailing punctuation) so `conversation_hash`
|
||||
# collapses on the question equivalence class. See aborist/qa/query.py
|
||||
# for the rationale.
|
||||
canonical_q = canonical_question(question)
|
||||
messages = [
|
||||
{"role": "system", "content": policy["system_prompt"]},
|
||||
]
|
||||
|
|
@ -149,37 +161,83 @@ def ask(
|
|||
),
|
||||
}
|
||||
)
|
||||
canonical_messages = list(messages[:-1]) + [
|
||||
|
||||
mhash = model_profile_hash(model_id, revision, quantization)
|
||||
|
||||
# Dedup-mode-aware cache_key. See aborist/qa/query.py for rationale —
|
||||
# policy_variant matches the alternate mode so governance_policy_hash
|
||||
# agrees with what an agent under that mode would have written,
|
||||
# enabling cross-silo fallback.
|
||||
def _ckey_for_mode(mode: str) -> str:
|
||||
canon_q = canonical_question(question, mode=mode)
|
||||
canon_msgs = list(messages[:-1]) + [
|
||||
{
|
||||
"role": "user",
|
||||
"content": (
|
||||
f"Document:\n\n{document_text}\n\n---\n\n"
|
||||
f"Question: {canon_q}"
|
||||
),
|
||||
}
|
||||
]
|
||||
policy_variant = dict(policy, question_dedup=mode)
|
||||
return cache_key(
|
||||
document_root,
|
||||
question_hash(question, mode=mode),
|
||||
mhash,
|
||||
conversation_hash(canon_msgs),
|
||||
governance_policy_hash(policy_variant),
|
||||
SCHEMA_VERSION,
|
||||
CANONICALIZATION_VERSION,
|
||||
doc["chunking_version"],
|
||||
)
|
||||
|
||||
ghash = governance_policy_hash(policy) # for the legacy INSERT below
|
||||
|
||||
primary_dedup = policy.get("question_dedup", DEFAULT_QUESTION_DEDUP)
|
||||
if primary_dedup not in QUESTION_DEDUP_MODES:
|
||||
raise ValueError(
|
||||
f"policy['question_dedup'] must be one of {QUESTION_DEDUP_MODES}, "
|
||||
f"got {primary_dedup!r}"
|
||||
)
|
||||
# Re-derive the per-mode hashes for use in the INSERT below. _ckey_for_mode
|
||||
# already builds them, but the legacy INSERT references qhash/chash by name.
|
||||
qhash = question_hash(question, mode=primary_dedup)
|
||||
canonical_q_primary = canonical_question(question, mode=primary_dedup)
|
||||
canonical_messages_primary = list(messages[:-1]) + [
|
||||
{
|
||||
"role": "user",
|
||||
"content": (
|
||||
f"Document:\n\n{document_text}\n\n---\n\n"
|
||||
f"Question: {canonical_q}"
|
||||
f"Question: {canonical_q_primary}"
|
||||
),
|
||||
}
|
||||
]
|
||||
|
||||
qhash = question_hash(question)
|
||||
mhash = model_profile_hash(model_id, revision, quantization)
|
||||
chash = conversation_hash(canonical_messages)
|
||||
ghash = governance_policy_hash(policy)
|
||||
ckey = cache_key(
|
||||
document_root,
|
||||
qhash,
|
||||
mhash,
|
||||
chash,
|
||||
ghash,
|
||||
SCHEMA_VERSION,
|
||||
CANONICALIZATION_VERSION,
|
||||
doc["chunking_version"],
|
||||
)
|
||||
chash = conversation_hash(canonical_messages_primary)
|
||||
primary_ckey = _ckey_for_mode(primary_dedup)
|
||||
ckey = primary_ckey # legacy name for the rest of the function
|
||||
|
||||
t_lookup = time.monotonic()
|
||||
cached = conn.execute(
|
||||
"SELECT * FROM providence_cache "
|
||||
"WHERE cache_key = ? AND falsification_state = 'live'",
|
||||
(ckey,),
|
||||
(primary_ckey,),
|
||||
).fetchone()
|
||||
hit_ckey = primary_ckey
|
||||
lookup_path = primary_dedup if cached is not None else None
|
||||
if cached is None and fidelity == "equivalence_class":
|
||||
other_mode = (
|
||||
"equivalence_class" if primary_dedup == "strict" else "strict"
|
||||
)
|
||||
other_ckey = _ckey_for_mode(other_mode)
|
||||
if other_ckey != primary_ckey:
|
||||
cached = conn.execute(
|
||||
"SELECT * FROM providence_cache "
|
||||
"WHERE cache_key = ? AND falsification_state = 'live'",
|
||||
(other_ckey,),
|
||||
).fetchone()
|
||||
if cached is not None:
|
||||
hit_ckey = other_ckey
|
||||
lookup_path = f"{other_mode}_fallback"
|
||||
cache_lookup_ms = _ms_since(t_lookup)
|
||||
if cached is not None:
|
||||
with transaction(conn):
|
||||
|
|
@ -188,12 +246,13 @@ def ask(
|
|||
"UPDATE providence_cache "
|
||||
"SET hit_count = hit_count + 1, last_hit_at = ? "
|
||||
"WHERE cache_key = ?",
|
||||
(now, ckey),
|
||||
(now, hit_ckey),
|
||||
)
|
||||
return {
|
||||
"status": "cache_hit",
|
||||
"audit_mode": cached["audit_mode"],
|
||||
"cache_key": ckey,
|
||||
"cache_key": hit_ckey,
|
||||
"lookup_path": lookup_path,
|
||||
"source_root": document_root,
|
||||
"answer_text": cached["answer_text"],
|
||||
"merkle_proof": json.loads(cached["merkle_proof"]),
|
||||
|
|
@ -302,6 +361,7 @@ def ask(
|
|||
"status": "cache_miss_then_written",
|
||||
"audit_mode": verdict["audit_mode"],
|
||||
"cache_key": ckey,
|
||||
"lookup_path": "miss",
|
||||
"source_root": document_root,
|
||||
"answer_text": answer_text,
|
||||
"merkle_proof": proof_obj,
|
||||
|
|
|
|||
|
|
@ -192,6 +192,147 @@ def test_query_question_equivalence_class_dedups_cache(tmp_path):
|
|||
assert "What is anarchism?" in user_text
|
||||
|
||||
|
||||
def test_query_strict_dedup_distinguishes_question_variants(tmp_path):
|
||||
"""policy['question_dedup']='strict' makes every variant get its
|
||||
own cache_key. 'Who is X?' and 'who is the X?' write separate
|
||||
records under strict policy."""
|
||||
main_db = tmp_path / "corpus.db"
|
||||
qa_db = tmp_path / "qa.db"
|
||||
conn = connect(main_db)
|
||||
try:
|
||||
ingest_source(conn, FakeSource(DOCS))
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
strict_policy = dict(query.__globals__["DEFAULT_QUERY_POLICY"])
|
||||
strict_policy["question_dedup"] = "strict"
|
||||
|
||||
r1 = query(
|
||||
question="What is anarchism?",
|
||||
qa_db=qa_db,
|
||||
chat_client=StubClient(answer="a"),
|
||||
model_id="m",
|
||||
single_db=main_db,
|
||||
policy=strict_policy,
|
||||
fidelity="strict",
|
||||
)
|
||||
r2 = query(
|
||||
question="what is anarchism",
|
||||
qa_db=qa_db,
|
||||
chat_client=StubClient(answer="b"),
|
||||
model_id="m",
|
||||
single_db=main_db,
|
||||
policy=strict_policy,
|
||||
fidelity="strict",
|
||||
)
|
||||
# Different cache_keys; both populated independently.
|
||||
assert r1["cache_key"] != r2["cache_key"]
|
||||
assert r1["status"] == r2["status"] == "cache_miss_then_written"
|
||||
assert r1["lookup_path"] == "miss"
|
||||
|
||||
|
||||
def test_query_equivalence_class_fidelity_falls_back_across_dedup_modes(tmp_path):
|
||||
"""A record written under equivalence_class policy gets reused by a
|
||||
later strict-policy lookup that asks for fidelity='equivalence_class'.
|
||||
Verifies the cross-silo fallback: strict ckey misses, alternate
|
||||
equivalence_class ckey hits, lookup_path reports the fallback."""
|
||||
main_db = tmp_path / "corpus.db"
|
||||
qa_db = tmp_path / "qa.db"
|
||||
conn = connect(main_db)
|
||||
try:
|
||||
ingest_source(conn, FakeSource(DOCS))
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
DEFAULT = query.__globals__["DEFAULT_QUERY_POLICY"]
|
||||
eq_policy = dict(DEFAULT)
|
||||
eq_policy["question_dedup"] = "equivalence_class"
|
||||
strict_policy = dict(DEFAULT)
|
||||
strict_policy["question_dedup"] = "strict"
|
||||
|
||||
# Agent A writes equivalence-class record for 'what is anarchism?'.
|
||||
captured: list = []
|
||||
|
||||
def _capture(messages, **kw):
|
||||
captured.append(messages)
|
||||
return "stub"
|
||||
|
||||
r1 = query(
|
||||
question="What is anarchism?",
|
||||
qa_db=qa_db,
|
||||
chat_client=StubClient(answer=_capture),
|
||||
model_id="m",
|
||||
single_db=main_db,
|
||||
policy=eq_policy,
|
||||
)
|
||||
assert r1["status"] == "cache_miss_then_written"
|
||||
|
||||
# Agent B (strict policy, equivalence_class fidelity) asks the same
|
||||
# question. Strict ckey misses; the eq_class fallback hits A's record.
|
||||
r2 = query(
|
||||
question="What is anarchism?",
|
||||
qa_db=qa_db,
|
||||
chat_client=StubClient(answer=_capture),
|
||||
model_id="m",
|
||||
single_db=main_db,
|
||||
policy=strict_policy,
|
||||
fidelity="equivalence_class",
|
||||
)
|
||||
assert r2["status"] == "cache_hit"
|
||||
assert r2["lookup_path"] == "equivalence_class_fallback"
|
||||
assert len(captured) == 1 # only A's call reached the LLM
|
||||
|
||||
|
||||
def test_query_strict_fidelity_does_not_fall_back(tmp_path):
|
||||
"""Audit-grade lookup: strict-fidelity refuses to read records from
|
||||
the other dedup mode's silo. Even if equivalence_class has a hit,
|
||||
strict fidelity reports cache miss."""
|
||||
main_db = tmp_path / "corpus.db"
|
||||
qa_db = tmp_path / "qa.db"
|
||||
conn = connect(main_db)
|
||||
try:
|
||||
ingest_source(conn, FakeSource(DOCS))
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
DEFAULT = query.__globals__["DEFAULT_QUERY_POLICY"]
|
||||
eq_policy = dict(DEFAULT)
|
||||
eq_policy["question_dedup"] = "equivalence_class"
|
||||
strict_policy = dict(DEFAULT)
|
||||
strict_policy["question_dedup"] = "strict"
|
||||
|
||||
# Agent A writes under equivalence_class.
|
||||
query(
|
||||
question="What is anarchism?",
|
||||
qa_db=qa_db,
|
||||
chat_client=StubClient(answer="a"),
|
||||
model_id="m",
|
||||
single_db=main_db,
|
||||
policy=eq_policy,
|
||||
)
|
||||
|
||||
# Agent B (strict policy + strict fidelity) asks same question.
|
||||
# Should NOT find A's record; runs LLM fresh.
|
||||
captured: list = []
|
||||
|
||||
def _capture(messages, **kw):
|
||||
captured.append(messages)
|
||||
return "stub"
|
||||
|
||||
r2 = query(
|
||||
question="What is anarchism?",
|
||||
qa_db=qa_db,
|
||||
chat_client=StubClient(answer=_capture),
|
||||
model_id="m",
|
||||
single_db=main_db,
|
||||
policy=strict_policy,
|
||||
fidelity="strict",
|
||||
)
|
||||
assert r2["status"] == "cache_miss_then_written"
|
||||
assert r2["lookup_path"] == "miss"
|
||||
assert len(captured) == 1
|
||||
|
||||
|
||||
def test_query_no_sources_when_empty_corpus(tmp_path):
|
||||
main_db = tmp_path / "empty.db"
|
||||
qa_db = tmp_path / "qa.db"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue