docs: ticket — retrieval-keywords audit gap (provenance loop closure)
Per fox's '--retrieval-keywords' commit (2d6a86b), the keyword string
is nowhere in the audit chain — it influences cache_key only
indirectly via context_root + conversation_hash. Two runs with the
same question and different keywords that surface identical sources
are Merkle-indistinguishable; an audit replay can recover the chosen
documents but not the inputs that pulled them.
Doc-only ticket: design proposal for capturing keywords in the
run_dag retrieval stage and on the providence_cache row (Option A +
B), preserving the 8-dim cache_key invariant. Out of scope: folding
keywords into question_hash (Option C) — keywords are operator
metadata, not user intent.
Includes failure cases, three design options with trade-offs,
implementation sketch, scope boundaries, and a recommended cost/value
forecast (~1-2 hours, low risk, additive schema). Awaiting fox's
go/no-go before landing.
This commit is contained in:
parent
d9b05c586e
commit
73058b0e66
1 changed files with 239 additions and 0 deletions
239
docs/retrieval-keywords-audit-gap.md
Normal file
239
docs/retrieval-keywords-audit-gap.md
Normal file
|
|
@ -0,0 +1,239 @@
|
|||
# Retrieval-keywords audit gap — close the provenance loop on operator hints
|
||||
|
||||
**Date:** 2026-05-01
|
||||
**Scope:** Design proposal for capturing the `--retrieval-keywords` operator
|
||||
hint in the v9.8 audit chain so retrieval is fully reproducible from a
|
||||
providence record alone. Doc-only — no code in this commit.
|
||||
**Audience:** fox + future blackops shifts.
|
||||
**Hard constraint:** keywords stay operator-metadata, not part of the user's
|
||||
question. Cache-key dimensionality stays at 8.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
|
||||
`--retrieval-keywords` (commit `2d6a86b`) lets an operator augment FTS5
|
||||
retrieval with domain hints without polluting the LLM-facing question:
|
||||
|
||||
```
|
||||
make query Q="what tech may enable... thoughts..." \
|
||||
K="transcranial knowledge acquisition" BURN=1
|
||||
```
|
||||
|
||||
The keywords narrow OR-mode FTS5 to topical articles (Neurotechnology
|
||||
in this case) and lift the verdict from HYBRID to STRICT — a real
|
||||
operational win. **But the keywords themselves are nowhere in the
|
||||
audit chain.** They influence `cache_key` only indirectly via the
|
||||
chain `keywords → retrieval_query → chosen sources → context_root +
|
||||
conversation_hash`.
|
||||
|
||||
The provenance gap, by component:
|
||||
|
||||
| Component | Captures keywords? |
|
||||
|------------------------------|--------------------|
|
||||
| `cache_key` (8-dim) | indirect (via context_root + conversation_hash) |
|
||||
| `question_hash` | NO |
|
||||
| `governance_policy_hash` | NO |
|
||||
| `run_dag` retrieval stage | NO (hashes only `sources_summary`) |
|
||||
| `audit_events` | NO |
|
||||
| `providence_cache` row | NO (no column for it) |
|
||||
|
||||
### 1.1 Concrete failure cases
|
||||
|
||||
**Case A — undistinguishable runs.** Two queries with the same question
|
||||
but different keyword sets that happen to surface identical sources are
|
||||
Merkle-indistinguishable. Audit replay can recover "these documents
|
||||
were chosen" but not "and these were the keywords that pulled them in."
|
||||
|
||||
**Case B — non-reproducible audit.** An auditor receives a providence
|
||||
record with `cache_key=abc...`, `audit_mode=STRICT`. They re-run the
|
||||
question alone (without keywords) and get HYBRID with different
|
||||
sources. The record was correctly produced, but the audit can't
|
||||
reproduce the retrieval path without out-of-band knowledge of the
|
||||
keywords used.
|
||||
|
||||
**Case C — silent drift on schema bump.** If `_search_corpus` retrieval
|
||||
behavior changes (new stopwords, new rerank), prior records under
|
||||
identical (question, keywords) become irreproducible. The keywords
|
||||
aren't even queryable to know which records to re-run.
|
||||
|
||||
### 1.2 What gets logged today
|
||||
|
||||
The `run_dag` retrieval stage at `aborist/qa/dag.py:155` hashes only the
|
||||
`sources_summary` (output):
|
||||
|
||||
```python
|
||||
sources_summary = [
|
||||
{
|
||||
"document_root": s.get("document_root"),
|
||||
"source_role": s.get("source_role"),
|
||||
"score": s.get("score"),
|
||||
"chunk_idx": s.get("chunk_idx"),
|
||||
}
|
||||
for s in sources
|
||||
]
|
||||
retrieval_hash = _sha256_hex(_canonical_json(sources_summary))
|
||||
```
|
||||
|
||||
The retrieval query (`question + keywords`), `top_k`, `over_fetch`,
|
||||
and `max_context_chars` — every input that determined which sources
|
||||
appear in `sources_summary` — are absent.
|
||||
|
||||
---
|
||||
|
||||
## 2. Design choices
|
||||
|
||||
Three recovery options, ranked by my read of the trade-offs:
|
||||
|
||||
### 2.1 Option A — capture in run_dag retrieval stage (RECOMMENDED)
|
||||
|
||||
Extend the retrieval-stage hash to cover both inputs and outputs:
|
||||
|
||||
```python
|
||||
retrieval_inputs = {
|
||||
"question": question, # already in question_hash; redundancy is fine
|
||||
"retrieval_keywords": retrieval_keywords or "",
|
||||
"top_k": top_k,
|
||||
"over_fetch": over_fetch,
|
||||
"max_context_chars": max_context_chars,
|
||||
}
|
||||
retrieval_hash = _sha256_hex(_canonical_json({
|
||||
"inputs": retrieval_inputs,
|
||||
"outputs": sources_summary,
|
||||
}))
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- Audit chain reproduces retrieval inputs byte-for-byte.
|
||||
- Keywords visible in the per-run merkle proof.
|
||||
- No new `cache_key` dimensions; v9.8 invariant preserved.
|
||||
- Bumps prior `run_dag_root` values (greenfield-acceptable per
|
||||
fox's policy).
|
||||
|
||||
**Cons:**
|
||||
- `run_dag_root` churn on existing records.
|
||||
- Doesn't enable direct SQL query ("which records used keywords X?").
|
||||
|
||||
### 2.2 Option B — providence_cache column
|
||||
|
||||
Add a nullable `retrieval_keywords TEXT` column to the
|
||||
`providence_cache` table.
|
||||
|
||||
**Pros:**
|
||||
- Direct SQL query without parsing run_dag blobs.
|
||||
- Trivial migration (additive column, no rebuild).
|
||||
|
||||
**Cons:**
|
||||
- Storage-side only — not Merkle-bound. Anyone with shard write
|
||||
access could update the column without breaking the chain.
|
||||
- If keywords are also in the run_dag (Option A), the column is a
|
||||
cache for what's already in `run_dag_blob`. Acceptable redundancy.
|
||||
|
||||
### 2.3 Option C — fold into `question_hash`
|
||||
|
||||
Treat keywords as part of the question identity:
|
||||
|
||||
```python
|
||||
question_hash = sha256(canonical_question + "\x00" + retrieval_keywords)
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- `cache_key` distinguishes runs with different keywords cleanly,
|
||||
no indirect routing through `context_root`.
|
||||
- Audit-cleanest: one record per (question, keywords) tuple.
|
||||
|
||||
**Cons:**
|
||||
- Loses the "same question, different operator hints" framing.
|
||||
- Operators iterating on keyword sets get cache misses on every
|
||||
variation — defeats the session-only ergonomics that motivated
|
||||
the flag.
|
||||
- Conflates user intent with operator metadata; muddles the
|
||||
semantic distinction between `question` and `keywords`.
|
||||
|
||||
### 2.4 Recommendation
|
||||
|
||||
**A + B, no C.** Keywords are operator metadata, not user intent. Track
|
||||
them in the run_dag (Merkle-bound provenance) and on the providence
|
||||
row (direct queryability) without reframing what counts as "the
|
||||
question." The 8-dim `cache_key` invariant stays intact.
|
||||
|
||||
---
|
||||
|
||||
## 3. Implementation sketch (when scheduled)
|
||||
|
||||
1. **`aborist/qa/dag.py`** — `build_run_dag` gains `retrieval_inputs`
|
||||
parameter; embeds it into the retrieval-stage hash. Backward-compat:
|
||||
when `retrieval_inputs` is None, fall back to the current
|
||||
sources-summary-only hash.
|
||||
|
||||
2. **`aborist/qa/query.py`** — at the existing run_dag construction
|
||||
site, pass `{"question": ..., "retrieval_keywords": ...,
|
||||
"top_k": ..., "over_fetch": ..., "max_context_chars": ...}` as
|
||||
`retrieval_inputs`.
|
||||
|
||||
3. **`aborist/store.py`** — schema migration: add
|
||||
`retrieval_keywords TEXT` (nullable) to `providence_cache`. Mirror
|
||||
the existing `_rebuild_providence_cache_*` pattern only if a CHECK
|
||||
constraint or column-default constraint requires it (otherwise a
|
||||
plain `ALTER TABLE ... ADD COLUMN` suffices).
|
||||
|
||||
4. **`aborist/qa/query.py`** persist site — include `retrieval_keywords`
|
||||
in the INSERT.
|
||||
|
||||
5. **`aborist/qa/verify.py`** + relevant CLI render — surface keywords
|
||||
in `aborist providence` output and `aborist inspect` so an operator
|
||||
can see at a glance whether a record was retrieval-augmented.
|
||||
|
||||
6. **Tests:**
|
||||
- Unit: `build_run_dag` with retrieval_inputs produces a different
|
||||
`retrieval` stage hash than without (and stable across repeated
|
||||
calls with the same inputs).
|
||||
- Unit: providence row stores keywords; cache hit returns them.
|
||||
- Integration: same question, different keywords → different
|
||||
`run_dag_root` (since retrieval inputs differ), even when sources
|
||||
happen to overlap.
|
||||
- Bench: `bench/qa_sweep.py` adds a `retrieval_keywords` column
|
||||
so a sweep can A/B keyword-augmented runs against bare runs.
|
||||
|
||||
7. **Bumps:**
|
||||
- `run_dag_root` schema (existing run_dag blobs are still parseable
|
||||
but their root changes form). Greenfield-acceptable.
|
||||
- `providence_cache` schema (additive column, fully backward-
|
||||
compatible).
|
||||
- No `cache_key` dimension change. No `governance_policy_hash`
|
||||
bump. Existing cached records remain valid for lookup.
|
||||
|
||||
---
|
||||
|
||||
## 4. Out of scope
|
||||
|
||||
- Operator-supplied retrieval modifiers other than keywords (e.g., a
|
||||
`--source-allowlist` flag pinning retrieval to specific titles).
|
||||
Same audit-gap argument applies but design needs separate
|
||||
consideration since allow-listing changes which docs are eligible
|
||||
for FTS5 in the first place, not just how they're ranked.
|
||||
- Rebuilding existing run_dag_blobs to embed keywords retroactively.
|
||||
Greenfield assumption: prior records without keywords had no
|
||||
keywords, so reconstructing "what would the new hash be" returns
|
||||
the same value as today (`retrieval_inputs.retrieval_keywords =
|
||||
""`); no rewrite needed.
|
||||
- Whether to fold `top_k` / `over_fetch` / `max_context_chars` into
|
||||
`governance_policy_hash` instead of (or in addition to) the run_dag
|
||||
retrieval stage. They're already retrieval-knobs that don't enter
|
||||
cache_key today; keeping them confined to run_dag matches the
|
||||
keywords story. Revisit if retrieval-knob audit becomes a frequent
|
||||
request.
|
||||
|
||||
---
|
||||
|
||||
## 5. Status
|
||||
|
||||
**Proposal.** No code yet. Pinging fox for go/no-go before landing.
|
||||
|
||||
Forecast cost: ~1-2 hours of focused work (the dag.py change + schema
|
||||
migration + ~6 tests + bench column). Risk: low — additive schema
|
||||
change, additive run_dag input, no cache-key churn.
|
||||
|
||||
Forecast value: closes the operator-hint provenance loop. Necessary
|
||||
for audit-grade reproducibility once `--retrieval-keywords` becomes
|
||||
a regular operator practice.
|
||||
Loading…
Add table
Add a link
Reference in a new issue