qa: 'make query Q=... BURN=1' busts matching cache before lookup
Per fox: while iterating on retrieval/verifier knobs, a flag on the query itself is more useful than a separate verb. One step reset: change a knob, re-query, see fresh result. Surface: aborist query --burn ... make query Q="..." BURN=1 make query-dry Q="..." BURN=1 (works on dry-run too) Behavior in `aborist.qa.query.query()`: - New `burn_existing: bool = False` parameter on query(). - After computing primary cache_key (per the active dedup mode) but BEFORE the cache lookup, if burn_existing: DELETE the live row that matches the primary cache_key & write one providence_burn audit event with reason "query --burn (test-ergonomic mid-query bust)". - The lookup then misses → fresh inference runs. Result reports `burned_existing: 0|1` so the caller sees whether anything got busted. - The equivalence-class fallback cache_key is deliberately NOT touched: prior alt-mode records stay as historic witnesses. Tests (3): cache populates → cache hits → BURN=1 forces fresh inference; --burn writes a providence_burn audit event; first-time query with --burn is a clean no-op (burned_existing=0). 356 passed, 1 skipped. Complementary to `aborist burn-kindergarten` (mass reset) — this is the surgical version. Both are local-only by design; for cross-peer invalidation use `make falsify` and let mesh sync broadcast.
This commit is contained in:
parent
4573dbcf4e
commit
e668a8355f
4 changed files with 181 additions and 6 deletions
12
Makefile
12
Makefile
|
|
@ -155,17 +155,17 @@ burn-kindergarten: bootstrap ## bust providence rows < SECONDS old [SECONDS=3600
|
|||
# Multi-source RAG query against the shard cluster.
|
||||
# Usage: make query Q="What is anarcho-capitalism?"
|
||||
QUERY_TOP_K ?= 8
|
||||
query: bootstrap ## ask the corpus a question [JSON=1 for raw record]; sources picked across shards
|
||||
query: bootstrap ## ask the corpus a question [JSON=1 BURN=1]; sources picked across shards
|
||||
@if [ -z "$$Q" ] && [ -z "$(Q)" ]; then \
|
||||
echo "usage: make query Q=\"your question\" [JSON=1]"; exit 2; \
|
||||
echo "usage: make query Q=\"your question\" [JSON=1 BURN=1]"; exit 2; \
|
||||
fi
|
||||
$(ABORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) $(if $(JSON),--json,) "$(Q)"
|
||||
$(ABORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) $(if $(JSON),--json,) $(if $(BURN),--burn,) "$(Q)"
|
||||
|
||||
query-dry: bootstrap ## like 'make query' but skip the LLM call (dry-run) [JSON=1 for raw]
|
||||
query-dry: bootstrap ## like 'make query' but skip the LLM call (dry-run) [JSON=1 BURN=1]
|
||||
@if [ -z "$$Q" ] && [ -z "$(Q)" ]; then \
|
||||
echo "usage: make query-dry Q=\"your question\" [JSON=1]"; exit 2; \
|
||||
echo "usage: make query-dry Q=\"your question\" [JSON=1 BURN=1]"; exit 2; \
|
||||
fi
|
||||
$(ABORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) --dry-run $(if $(JSON),--json,) "$(Q)"
|
||||
$(ABORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) --dry-run $(if $(JSON),--json,) $(if $(BURN),--burn,) "$(Q)"
|
||||
|
||||
verify-shards: bootstrap ## cross-shard Merkle round-trip on a random sample
|
||||
$(ABORIST) --shards-dir $(SHARDS_DIR) verify -n $(VERIFY_N)
|
||||
|
|
|
|||
|
|
@ -364,6 +364,7 @@ def _cmd_query(args: argparse.Namespace) -> int:
|
|||
max_context_chars=args.max_context_chars,
|
||||
policy=call_policy,
|
||||
fidelity=getattr(args, "fidelity", None),
|
||||
burn_existing=bool(getattr(args, "burn", False)),
|
||||
)
|
||||
|
||||
if args.json:
|
||||
|
|
@ -2830,6 +2831,15 @@ def build_parser() -> argparse.ArgumentParser:
|
|||
"--json", action="store_true",
|
||||
help="emit the raw record as indented JSON (default: human render)",
|
||||
)
|
||||
query_cmd.add_argument(
|
||||
"--burn", action="store_true",
|
||||
help=(
|
||||
"delete the matching live providence_cache row BEFORE lookup, "
|
||||
"forcing a fresh inference. Test-ergonomic — see new behavior "
|
||||
"without finding cache_keys by hand. Writes a providence_burn "
|
||||
"audit event."
|
||||
),
|
||||
)
|
||||
query_cmd.set_defaults(func=_cmd_query)
|
||||
|
||||
inspect_cmd = sub.add_parser(
|
||||
|
|
|
|||
|
|
@ -672,6 +672,7 @@ def query(
|
|||
policy: dict | None = None,
|
||||
chain: str = "private",
|
||||
fidelity: str | None = None,
|
||||
burn_existing: bool = False,
|
||||
) -> dict:
|
||||
"""Answer `question` using the corpus. Cache to qa_db. Returns a result dict.
|
||||
|
||||
|
|
@ -682,6 +683,16 @@ def query(
|
|||
dedup-mode cache_key as a fallback so a fast-cache agent can reuse
|
||||
a record written under either mode. Result includes ``lookup_path``
|
||||
naming which key matched (or ``"miss"`` when the LLM ran).
|
||||
|
||||
`burn_existing=True` deletes the matching live providence_cache row
|
||||
(under the primary dedup-mode cache_key) BEFORE the cache lookup,
|
||||
forcing a fresh inference. Each burn writes a ``providence_burn``
|
||||
audit event. Test-ergonomic: run `make query Q=... BURN=1` after
|
||||
tweaking a knob to see the new behavior without finding cache_keys
|
||||
by hand. Result includes ``burned_existing`` reporting how many
|
||||
rows were deleted (0 or 1 for the primary key; the equivalence-
|
||||
class fallback key is left alone so prior alt-mode records stay
|
||||
historic).
|
||||
"""
|
||||
policy = policy or DEFAULT_QUERY_POLICY
|
||||
if fidelity is None:
|
||||
|
|
@ -859,7 +870,43 @@ def query(
|
|||
ckey = primary_ckey # keep the legacy name in the rest of the function
|
||||
|
||||
qa_conn = connect(qa_db)
|
||||
burned_existing = 0
|
||||
try:
|
||||
# 3.5. Optional pre-lookup burn: deletes the matching live row
|
||||
# under the primary cache_key so the lookup misses & a fresh
|
||||
# inference runs. Test-ergonomic — pass `burn_existing=True`
|
||||
# (or `make query Q=... BURN=1`) after tweaking a knob to see
|
||||
# the new behavior. The equivalence-class fallback key is
|
||||
# deliberately NOT touched: prior alt-mode records stay as
|
||||
# historic witnesses.
|
||||
if burn_existing:
|
||||
existing = qa_conn.execute(
|
||||
"SELECT cache_key, audit_mode, n_verified, "
|
||||
" falsification_state, question_text "
|
||||
"FROM providence_cache WHERE cache_key = ?",
|
||||
(primary_ckey,),
|
||||
).fetchone()
|
||||
if existing is not None:
|
||||
with transaction(qa_conn):
|
||||
qa_conn.execute(
|
||||
"DELETE FROM providence_cache WHERE cache_key = ?",
|
||||
(primary_ckey,),
|
||||
)
|
||||
append_audit(
|
||||
qa_conn,
|
||||
event_type="providence_burn",
|
||||
subject_root=primary_ckey,
|
||||
body={
|
||||
"cache_key": primary_ckey,
|
||||
"burned_audit_mode": existing["audit_mode"],
|
||||
"burned_n_verified": int(existing["n_verified"] or 0),
|
||||
"burned_state": existing["falsification_state"],
|
||||
"question_text": existing["question_text"],
|
||||
"reason": "query --burn (test-ergonomic mid-query bust)",
|
||||
},
|
||||
)
|
||||
burned_existing = 1
|
||||
|
||||
# 4. Cache lookup. Try the primary dedup-mode cache_key first.
|
||||
# If fidelity allows fallback AND the alternate dedup mode
|
||||
# produces a different cache_key, try that too — lets a
|
||||
|
|
@ -901,6 +948,7 @@ def query(
|
|||
"audit_mode": cached["audit_mode"],
|
||||
"cache_key": hit_ckey,
|
||||
"lookup_path": lookup_path,
|
||||
"burned_existing": burned_existing,
|
||||
"context_root": context_root,
|
||||
"answer_text": cached["answer_text"],
|
||||
"sources": json.loads(cached["merkle_proof"])["sources"],
|
||||
|
|
@ -1029,6 +1077,7 @@ def query(
|
|||
"audit_mode": verdict["audit_mode"],
|
||||
"cache_key": ckey,
|
||||
"lookup_path": "miss",
|
||||
"burned_existing": burned_existing,
|
||||
"context_root": context_root,
|
||||
"answer_text": answer_text,
|
||||
"sources": proof_obj["sources"],
|
||||
|
|
|
|||
|
|
@ -424,6 +424,122 @@ def test_query_filter_one_token_query_still_synonym_expands(tmp_path):
|
|||
assert len(result["sources"]) == 1
|
||||
|
||||
|
||||
def test_query_burn_existing_forces_fresh_inference(tmp_path):
|
||||
"""Fox 2026-04-29: `make query Q=... BURN=1` busts any matching live
|
||||
cache record before lookup so a fresh inference runs.
|
||||
|
||||
Sequence:
|
||||
1. First query → cache_miss_then_written, populates cache
|
||||
2. Second query, NO burn → cache_hit (no new LLM call)
|
||||
3. Third query, BURN=True → cache_miss_then_written (cache busted),
|
||||
result reports burned_existing=1
|
||||
"""
|
||||
main_db = tmp_path / "corpus.db"
|
||||
qa_db = tmp_path / "qa.db"
|
||||
conn = connect(main_db)
|
||||
try:
|
||||
ingest_source(conn, FakeSource(DOCS))
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
client = StubClient(answer="anarchism is a thing")
|
||||
|
||||
r1 = query(
|
||||
question="What is anarchism?",
|
||||
qa_db=qa_db,
|
||||
chat_client=client,
|
||||
model_id="m",
|
||||
single_db=main_db,
|
||||
)
|
||||
assert r1["status"] == "cache_miss_then_written"
|
||||
assert r1["burned_existing"] == 0
|
||||
n_calls_after_first = len(client.calls)
|
||||
|
||||
r2 = query(
|
||||
question="What is anarchism?",
|
||||
qa_db=qa_db,
|
||||
chat_client=client,
|
||||
model_id="m",
|
||||
single_db=main_db,
|
||||
)
|
||||
assert r2["status"] == "cache_hit"
|
||||
assert r2["burned_existing"] == 0
|
||||
assert len(client.calls) == n_calls_after_first # no new LLM call
|
||||
|
||||
r3 = query(
|
||||
question="What is anarchism?",
|
||||
qa_db=qa_db,
|
||||
chat_client=client,
|
||||
model_id="m",
|
||||
single_db=main_db,
|
||||
burn_existing=True,
|
||||
)
|
||||
assert r3["status"] == "cache_miss_then_written"
|
||||
assert r3["burned_existing"] == 1
|
||||
assert len(client.calls) == n_calls_after_first + 1 # new LLM call after burn
|
||||
|
||||
|
||||
def test_query_burn_existing_writes_audit_event(tmp_path):
|
||||
"""Each --burn writes a providence_burn audit event so the chain
|
||||
records the bust. Verifies one event lands per burn."""
|
||||
from aborist.store import connect as _connect
|
||||
|
||||
main_db = tmp_path / "corpus.db"
|
||||
qa_db = tmp_path / "qa.db"
|
||||
conn = connect(main_db)
|
||||
try:
|
||||
ingest_source(conn, FakeSource(DOCS))
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
client = StubClient(answer="x")
|
||||
query(
|
||||
question="What is anarchism?",
|
||||
qa_db=qa_db, chat_client=client, model_id="m", single_db=main_db,
|
||||
)
|
||||
qc = _connect(qa_db)
|
||||
try:
|
||||
burns_before = qc.execute(
|
||||
"SELECT COUNT(*) FROM audit_events WHERE event_type='providence_burn'"
|
||||
).fetchone()[0]
|
||||
finally:
|
||||
qc.close()
|
||||
|
||||
query(
|
||||
question="What is anarchism?",
|
||||
qa_db=qa_db, chat_client=client, model_id="m", single_db=main_db,
|
||||
burn_existing=True,
|
||||
)
|
||||
qc = _connect(qa_db)
|
||||
try:
|
||||
burns_after = qc.execute(
|
||||
"SELECT COUNT(*) FROM audit_events WHERE event_type='providence_burn'"
|
||||
).fetchone()[0]
|
||||
finally:
|
||||
qc.close()
|
||||
assert burns_after == burns_before + 1
|
||||
|
||||
|
||||
def test_query_burn_existing_with_no_prior_record_is_noop(tmp_path):
|
||||
"""First-time query with --burn: nothing to burn → burned_existing=0,
|
||||
proceeds to fresh inference normally."""
|
||||
main_db = tmp_path / "corpus.db"
|
||||
qa_db = tmp_path / "qa.db"
|
||||
conn = connect(main_db)
|
||||
try:
|
||||
ingest_source(conn, FakeSource(DOCS))
|
||||
finally:
|
||||
conn.close()
|
||||
r = query(
|
||||
question="What is anarchism?",
|
||||
qa_db=qa_db, chat_client=StubClient(answer="x"),
|
||||
model_id="m", single_db=main_db,
|
||||
burn_existing=True,
|
||||
)
|
||||
assert r["status"] == "cache_miss_then_written"
|
||||
assert r["burned_existing"] == 0
|
||||
|
||||
|
||||
def test_query_per_source_cap_prevents_huge_doc_monopoly(tmp_path):
|
||||
"""Fox 2026-04-29 catch: a top-ranked huge document (e.g.
|
||||
List_of_Batman_comics, 80 KB+ bibliography) used to consume the
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue