arborist/docs/tickets/ticket-000001-retrieval-keywords-audit-gap.md

Ticket #000001 — Retrieval-keywords audit gap

Status: closed · landed 2026-05-02 (run-DAG binding scope; SQL column + audit-events scope deferred per §6 below) Opened: 2026-05-01 Closed: 2026-05-02 Scope: Design proposal for capturing the --retrieval-keywords operator hint in the v9.8 audit chain so retrieval is fully reproducible from a providence record alone. Doc-only — no code in this commit. Audience: fox + future blackops shifts. Hard constraint: keywords stay operator-metadata, not part of the user's question. Cache-key dimensionality stays at 8.

---

1. Problem statement

--retrieval-keywords (commit 2d6a86b) lets an operator augment FTS5 retrieval with domain hints without polluting the LLM-facing question:

make query Q="what tech may enable... thoughts..." \
           K="transcranial knowledge acquisition" BURN=1

The keywords narrow OR-mode FTS5 to topical articles (Neurotechnology in this case) and lift the verdict from HYBRID to STRICT — a real operational win. But the keywords themselves are nowhere in the audit chain. They influence cache_key only indirectly via the chain keywords → retrieval_query → chosen sources → context_root + conversation_hash.

The provenance gap, by component:

Component	Captures keywords?
cache_key (8-dim)	indirect (via context_root + conversation_hash)
question_hash	NO
governance_policy_hash	NO
run_dag retrieval stage	NO (hashes only sources_summary)
audit_events	NO
providence_cache row	NO (no column for it)

1.1 Concrete failure cases

Case A — undistinguishable runs. Two queries with the same question but different keyword sets that happen to surface identical sources are Merkle-indistinguishable. Audit replay can recover "these documents were chosen" but not "and these were the keywords that pulled them in."

Case B — non-reproducible audit. An auditor receives a providence record with cache_key=abc..., audit_mode=STRICT. They re-run the question alone (without keywords) and get HYBRID with different sources. The record was correctly produced, but the audit can't reproduce the retrieval path without out-of-band knowledge of the keywords used.

Case C — silent drift on schema bump. If _search_corpus retrieval behavior changes (new stopwords, new rerank), prior records under identical (question, keywords) become irreproducible. The keywords aren't even queryable to know which records to re-run.

1.2 What gets logged today

The run_dag retrieval stage at arborist/qa/dag.py:155 hashes only the sources_summary (output):

sources_summary = [
    {
        "document_root": s.get("document_root"),
        "source_role": s.get("source_role"),
        "score": s.get("score"),
        "chunk_idx": s.get("chunk_idx"),
    }
    for s in sources
]
retrieval_hash = _sha256_hex(_canonical_json(sources_summary))

The retrieval query (question + keywords), top_k, over_fetch, and max_context_chars — every input that determined which sources appear in sources_summary — are absent.

---

2. Design choices

Three recovery options, ranked by my read of the trade-offs:

2.1 Option A — capture in run_dag retrieval stage (RECOMMENDED)

Extend the retrieval-stage hash to cover both inputs and outputs:

retrieval_inputs = {
    "question": question,                       # already in question_hash; redundancy is fine
    "retrieval_keywords": retrieval_keywords or "",
    "top_k": top_k,
    "over_fetch": over_fetch,
    "max_context_chars": max_context_chars,
}
retrieval_hash = _sha256_hex(_canonical_json({
    "inputs": retrieval_inputs,
    "outputs": sources_summary,
}))

Pros:

• Audit chain reproduces retrieval inputs byte-for-byte.
• Keywords visible in the per-run merkle proof.
• No new cache_key dimensions; v9.8 invariant preserved.
• Bumps prior run_dag_root values (greenfield-acceptable per fox's policy).

Cons:

• run_dag_root churn on existing records.
• Doesn't enable direct SQL query ("which records used keywords X?").

2.2 Option B — providence_cache column

Add a nullable retrieval_keywords TEXT column to the providence_cache table.

Pros:

• Direct SQL query without parsing run_dag blobs.
• Trivial migration (additive column, no rebuild).

Cons:

• Storage-side only — not Merkle-bound. Anyone with shard write access could update the column without breaking the chain.
• If keywords are also in the run_dag (Option A), the column is a cache for what's already in run_dag_blob. Acceptable redundancy.

2.3 Option C — fold into question_hash

Treat keywords as part of the question identity:

question_hash = sha256(canonical_question + "\x00" + retrieval_keywords)

Pros:

• cache_key distinguishes runs with different keywords cleanly, no indirect routing through context_root.
• Audit-cleanest: one record per (question, keywords) tuple.

Cons:

• Loses the "same question, different operator hints" framing.
• Operators iterating on keyword sets get cache misses on every variation — defeats the session-only ergonomics that motivated the flag.
• Conflates user intent with operator metadata; muddles the semantic distinction between question and keywords.

2.4 Recommendation

A + B, no C. Keywords are operator metadata, not user intent. Track them in the run_dag (Merkle-bound provenance) and on the providence row (direct queryability) without reframing what counts as "the question." The 8-dim cache_key invariant stays intact.

---

3. Implementation sketch (when scheduled)

1. arborist/qa/dag.py — build_run_dag gains retrieval_inputs parameter; embeds it into the retrieval-stage hash. Backward-compat: when retrieval_inputs is None, fall back to the current sources-summary-only hash.

2. arborist/qa/query.py — at the existing run_dag construction site, pass {"question": ..., "retrieval_keywords": ..., "top_k": ..., "over_fetch": ..., "max_context_chars": ...} as retrieval_inputs.

3. arborist/store.py — schema migration: add retrieval_keywords TEXT (nullable) to providence_cache. Mirror the existing _rebuild_providence_cache_* pattern only if a CHECK constraint or column-default constraint requires it (otherwise a plain ALTER TABLE ... ADD COLUMN suffices).

4. arborist/qa/query.py persist site — include retrieval_keywords in the INSERT.

5. arborist/qa/verify.py + relevant CLI render — surface keywords in arborist providence output and arborist inspect so an operator can see at a glance whether a record was retrieval-augmented.

6. Tests:

   • Unit: build_run_dag with retrieval_inputs produces a different retrieval stage hash than without (and stable across repeated calls with the same inputs).
   • Unit: providence row stores keywords; cache hit returns them.
   • Integration: same question, different keywords → different run_dag_root (since retrieval inputs differ), even when sources happen to overlap.
   • Bench: bench/qa_sweep.py adds a retrieval_keywords column so a sweep can A/B keyword-augmented runs against bare runs.

7. Bumps:

   • run_dag_root schema (existing run_dag blobs are still parseable but their root changes form). Greenfield-acceptable.
   • providence_cache schema (additive column, fully backward- compatible).
   • No cache_key dimension change. No governance_policy_hash bump. Existing cached records remain valid for lookup.

---

4. Out of scope

• Operator-supplied retrieval modifiers other than keywords (e.g., a --source-allowlist flag pinning retrieval to specific titles). Same audit-gap argument applies but design needs separate consideration since allow-listing changes which docs are eligible for FTS5 in the first place, not just how they're ranked.
• Rebuilding existing run_dag_blobs to embed keywords retroactively. Greenfield assumption: prior records without keywords had no keywords, so reconstructing "what would the new hash be" returns the same value as today (retrieval_inputs.retrieval_keywords = ""); no rewrite needed.
• Whether to fold top_k / over_fetch / max_context_chars into governance_policy_hash instead of (or in addition to) the run_dag retrieval stage. They're already retrieval-knobs that don't enter cache_key today; keeping them confined to run_dag matches the keywords story. Revisit if retrieval-knob audit becomes a frequent request.

---

5. Status

Closed 2026-05-02. Run-DAG binding scope landed:

• New module arborist/qa/retrieval_plan.py — RetrievalPlan dataclass + retrieval_plan_hash function. Captures retrieval_keywords, top_k, over_fetch, max_context_chars, shard_ids. SHA-256 over canonical-JSON; deterministic per call.
• arborist/qa/dag.py:build_run_dag accepts retrieval_plan_hash parameter. When provided, the retrieval stage hash binds BOTH the plan (input) and sources_summary (output). When omitted, falls back to the historical sources-summary-only hash so pre-#000001 records keep their run_dag_root values stable.
• arborist/qa/query.py constructs the plan per call from the query-time inputs (keywords from --retrieval-keywords, top_k / over_fetch / max_context_chars from policy + args) plus the resolved shard set, computes the hash, and passes through to build_run_dag.

Two runs with identical sources but different retrieval keywords now produce different run_dag_root values — provenance closes the "how did retrieval choose these sources" gap.

3 new directive tests in tests/test_directives.py:

• test_d4_retrieval_plan_binding_landed (marker flipped from "absent" assertion to "present").
• Hash divergence asserted via differing retrieval_plan_hash arguments to build_run_dag.
• test_d4_retrieval_plan_hash_module_exists covers the RetrievalPlan dataclass + retrieval_plan_hash function shape.

Full suite: 712 passed.

6. Deferred — additive scope

Audit events (retrieval_plan_built / retrieval_result_selected) and the providence_cache.retrieval_keywords SQL column from the original Appendix A proposal stay deferred. The run-DAG binding satisfies the Merkle-AGI commitment requirement (Theorem T2: every causally relevant transformation input bound). Direct SQL queryability and audit-event-level capture are operator- ergonomic enhancements that earn their own tickets when bench evidence shows the run-DAG-blob path is too friction-heavy for real workflows. Per the five-step algorithm step 2: ship the minimum viable substrate; add ergonomic affordances only when they earn it.

cache_key impact stays as-was: the retrieval plan affects context_root and conversation_hash indirectly (different plan → different selected sources → different context fed to LLM → different conversation_hash). The plan does NOT enter question_hash or governance_policy_hash; the 8-dim cache_key invariant holds. Optional strict mode (where retrieval_plan_affects_cache_key=True would fold the plan hash in directly) stays in the original proposal as future work.

---

Appendix A — Architectural review (2026-05-01, Asia/Kuala_Lumpur)

Fox-supplied review expanding §2-§4 with axiomatic framing, CTI / Merkle-AGI / PROMETHEUS-Σ interpretations, a concrete implementation diff, and a strengthened test list. Captured verbatim. Treat this appendix as authoritative when it disagrees with §2-§4 above.

Yes: this is a real provenance gap.

The correct diagnosis is:

question text       → bound
conversation state  → bound
source/context root → bound
run DAG output      → partly bound
retrieval keywords  → not directly bound

So an auditor can currently recover:

"These sources were selected."

but not:

"These retrieval keywords / operator hints caused those sources to be selected."

That is incomplete provenance.

The proposed recommendation — 1 + 2, not 3 — is the right default.

---

A.1 Axiomatic framing

Define the relevant objects:

Q  = user question text
K  = retrieval keywords / operator search hints
R  = retrieval configuration: top_k, over_fetch, shard set, filters, max_context_chars
S  = retrieved source set / sources_summary
C  = assembled context root
A  = answer
D  = run DAG

Current approximate chain:

Q → keyword generation/search → S → C → A

But the committed audit chain mostly records:

Q, S, C, A

not:

K, R

So the missing edge is:

(K, R) → S

That is the provenance gap.

In CTI / Merkle-AGI language:

retrieval_keywords are parent clauses of retrieval_result

If a parent clause is not committed, the retrieval node is under-specified.

---

A.2 Why this matters

Two runs can have:

same Q
same S
same C
same A
different K

and still become Merkle-indistinguishable if the retrieval stage hashes only sources_summary.

Example:

Q = "has oceania always been at war with east asia"

K1 = ["Oceania", "East Asia", "war", "history"]
K2 = ["Nineteen Eighty-Four", "Oceania", "Eastasia", "always at war"]

If both eventually surface the same source set, the current DAG may not distinguish whether the system reached the source through literal geography routing or Orwell-reference routing.

That matters because retrieval intent is part of the reasoning trace.

It is especially important for your recent failure classes:

reference-frame routing
relation-warrant retrieval
operator hints
one-shot evidence maps
old-map avoidance

If you do not bind retrieval inputs, you cannot audit whether the system used the right map.

---

A.3 Do not fold keywords into question_hash by default

I agree with not doing fix 3 as the default.

Reason:

question_hash should identify the user's question,
not the operator's retrieval strategy.

Keywords are not always semantically part of the question. They may be:

retrieval expansion
operator hint
routing hypothesis
fallback search string
reference-frame expansion
relation-warrant expansion
debug override

If you fold K into question_hash, then these become treated as different questions:

Q + literal geography keywords
Q + Orwell reference keywords

That loses a useful distinction:

same user question,
different retrieval plan.

Better:

question_hash = identity of user question
retrieval_plan_hash = identity of search strategy

Keep them separate.

---

A.4 Best fix: add retrieval plan as first-class provenance

Implement 1 + 2, with one extra refinement.

A.4.1 Add retrieval input to run DAG

Current retrieval node appears to bind only the output:

retrieval_stage = H(sources_summary)

Change it to bind both input and output:

retrieval_stage = H({
  "question_hash": question_hash,
  "retrieval_keywords": retrieval_keywords,
  "retrieval_mode": retrieval_mode,
  "top_k": top_k,
  "over_fetch": over_fetch,
  "max_context_chars": max_context_chars,
  "filters": filters,
  "shards": shard_ids_or_roots,
  "sources_summary": sources_summary
})

Better still:

retrieval_plan_hash = H({
  "retrieval_keywords": retrieval_keywords,
  "retrieval_mode": retrieval_mode,
  "top_k": top_k,
  "over_fetch": over_fetch,
  "max_context_chars": max_context_chars,
  "filters": filters,
  "shards": shard_ids_or_roots,
  "ranking_policy_hash": ranking_policy_hash
})

retrieval_result_hash = H({
  "sources_summary": sources_summary,
  "context_root": context_root
})

retrieval_stage_hash = H({
  "question_hash": question_hash,
  "retrieval_plan_hash": retrieval_plan_hash,
  "retrieval_result_hash": retrieval_result_hash
})

This is cleaner because it separates:

plan → result

rather than mixing them.

---

A.4.2 Store keywords on providence_records

Add nullable columns:

ALTER TABLE providence_records ADD COLUMN retrieval_keywords TEXT;
ALTER TABLE providence_records ADD COLUMN retrieval_plan_hash TEXT;
ALTER TABLE providence_records ADD COLUMN retrieval_mode TEXT;

If using canonical JSON:

retrieval_keywords TEXT -- canonical JSON array

Example:

["Nineteen Eighty-Four", "Oceania", "Eastasia", "always at war"]

This makes inspection cheap:

No need to parse run_dag_blob just to see how retrieval was driven.

---

A.4.3 Also add retrieval plan to audit events

This is the refinement.

Add audit event:

retrieval_plan_built

Payload:

{
  "question_hash": "...",
  "retrieval_keywords": ["..."],
  "retrieval_mode": "reference_router",
  "top_k": 8,
  "over_fetch": 32,
  "max_context_chars": 60000,
  "retrieval_plan_hash": "..."
}

Then:

retrieval_result_selected

Payload:

{
  "retrieval_plan_hash": "...",
  "sources_summary_hash": "...",
  "context_root": "...",
  "source_count": 8
}

This gives full replay provenance.

The v9.8 spec already treats audit events as authoritative in the SQLite audit table, with hash-verifiable audit events and optional chaining. That makes audit events the right place to preserve retrieval-control history as well.

---

A.5 Should retrieval keywords affect cache identity?

Default: no.

But there is a nuance.

Case A — Same question, different keywords, same context root

Q same
K different
C same
A same

Cache reuse is probably acceptable because the answer was generated from the same source/context/model/policy state.

But provenance must record:

this cache hit occurred under retrieval_plan_hash = X

So on cache hit, you still need an audit event:

cache_hit_with_retrieval_plan

Otherwise the second run's retrieval plan disappears.

Case B — Same question, different keywords, different context root

Then cache key already changes if context_root or conversation/source context hash is part of cache identity.

Case C — Keywords imply semantic framing but context is same

This is tricky.

Example:

K1 = literal geography route
K2 = Orwell route
same sources happen to appear

If the prompt sent to the model does not include K, and the context is the same, cache reuse is technically consistent.

But if K influences prompt framing, then K must be part of conversation_hash or prompt_hash.

Rule:

If retrieval keywords are only retrieval-control metadata:
    store/hash in retrieval_plan, not question_hash/cache_key.

If retrieval keywords are shown to the model or affect answer instructions:
    include them in prompt_hash/conversation_hash.

If retrieval keywords are policy-level routing state:
    include retrieval_policy_hash or verifier_policy_hash.

This is the clean boundary.

---

A.6 CTI interpretation

In CTI, retrieval keywords are not just logs. They are upstream clauses.

For example:

C0: User asked Q.
C1: Router selected literal/geography frame.
C2: Retrieval keywords K were generated.
C3: Search returned sources S.
C4: Context C was assembled.
C5: Answer A was generated.

Current DAG commits C3–C5 better than C1–C2.

Fixing this means the retrieval stage becomes a real clause subgraph:

frame_node
→ retrieval_plan_node
→ retrieval_result_node
→ context_node

That is a stronger CTI graph.

---

A.7 Merkle-AGI interpretation

The Merkle-AGI axiom is:

Every causally relevant transformation input must be committed if the run is to be replay-auditable.

Retrieval keywords are causally relevant because they influence source selection.

Therefore, they must be committed somewhere.

They do not necessarily need to alter the user question hash.

Correct commitment structure:

question_hash
retrieval_plan_hash
retrieval_result_hash
context_root
answer_hash
verify_hash
final_label_hash

Then:

run_dag_root = H(all stage hashes)

This preserves both:

semantic identity of question

and:

operational identity of retrieval.

---

A.8 PROMETHEUS-Σ interpretation

PROMETHEUS-Σ should treat this as a policy and admissibility issue.

Add a policy field:

{
  "retrieval": {
    "bind_keywords_in_run_dag": true,
    "store_keywords_on_record": true,
    "keywords_affect_cache_key": false,
    "keywords_affect_prompt_hash_if_prompted": true
  }
}

This makes the distinction explicit.

---

A.9 Best implementation ticket

Ticket name

J — Retrieval Plan Provenance Binding

Scope

- Hash retrieval inputs into run DAG.
- Store retrieval_keywords and retrieval_plan_hash on providence_records.
- Emit audit events for retrieval plan and retrieval result.
- Do not fold keywords into question_hash.
- Do not alter cache_key unless keywords enter prompt/conversation state.

---

A.10 Concrete diff

arborist/qa/retrieval.py

Return a structured object:

@dataclass(frozen=True)
class RetrievalPlan:
    question_hash: str
    retrieval_keywords: list[str]
    retrieval_mode: str
    top_k: int
    over_fetch: int
    max_context_chars: int
    filters: dict
    shard_ids: list[str]
    ranking_policy_hash: str

    def canonical(self) -> dict:
        return {
            "question_hash": self.question_hash,
            "retrieval_keywords": self.retrieval_keywords,
            "retrieval_mode": self.retrieval_mode,
            "top_k": self.top_k,
            "over_fetch": self.over_fetch,
            "max_context_chars": self.max_context_chars,
            "filters": self.filters,
            "shard_ids": self.shard_ids,
            "ranking_policy_hash": self.ranking_policy_hash,
        }

arborist/qa/hash.py

def retrieval_plan_hash(plan: RetrievalPlan) -> str:
    return hash_json("arborist-retrieval-plan-v1", plan.canonical())

arborist/qa/dag.py

Change retrieval node from:

retrieval_hash = hash_json({"sources_summary": sources_summary})

to:

retrieval_hash = hash_json({
    "stage": "retrieval",
    "retrieval_plan_hash": retrieval_plan_hash,
    "retrieval_plan": retrieval_plan.canonical(),
    "sources_summary": sources_summary,
    "context_root": context_root,
})

Or, if you do not want full keywords in DAG blob for privacy:

retrieval_hash = hash_json({
    "stage": "retrieval",
    "retrieval_plan_hash": retrieval_plan_hash,
    "sources_summary_hash": hash_json("sources-summary-v1", sources_summary),
    "context_root": context_root,
})

and store the plan in DB privately.

SQLite migration

ALTER TABLE providence_records ADD COLUMN retrieval_keywords TEXT;
ALTER TABLE providence_records ADD COLUMN retrieval_plan_hash TEXT;
ALTER TABLE providence_records ADD COLUMN retrieval_mode TEXT;
ALTER TABLE providence_records ADD COLUMN ranking_policy_hash TEXT;

audit.py

Add:

retrieval_plan_built
retrieval_result_selected

---

A.11 Cache-key policy

Do not add retrieval keywords to question_hash.

Do not add retrieval keywords directly to cache key by default.

But add a future optional mode:

{
  "cache": {
    "retrieval_plan_affects_cache_key": false
  }
}

If set true, use:

cache_key_v2 = H({
  existing_cache_key_fields,
  retrieval_plan_hash
})

Default false.

Reason:

most of the time, cache identity should be answer-context identity,
not operator-search identity.

But high-assurance deployments may want stricter replay identity.

---

A.12 Tests to add

[ ] retrieval_keywords stored on providence record.
[ ] retrieval_plan_hash changes when keyword order/content changes.
[ ] retrieval_plan_hash changes when top_k changes.
[ ] retrieval_plan_hash changes when over_fetch changes.
[ ] retrieval_plan_hash changes when max_context_chars changes.
[ ] run_dag_root changes when retrieval_plan_hash changes.
[ ] two runs with same sources but different keywords have different run_dag_root.
[ ] question_hash unchanged when only retrieval_keywords change.
[ ] cache_key unchanged when only retrieval_keywords change under default policy.
[ ] cache_key changes when retrieval_plan_affects_cache_key = true.
[ ] audit_events include retrieval_plan_built.
[ ] cache hit still records current retrieval_plan_hash in audit event.

The critical one:

same sources, different keywords → different run_dag_root

This proves the provenance gap is closed.

---

A.13 Additive insight

This is not only a bug fix. It is a core architecture principle:

The map must be committed, not only the territory reached.

Source outputs are the territory reached. Retrieval keywords are the map used to get there.

Your recent "old maps" point applies directly. If you do not bind the map, you cannot later tell whether the system used a stale, literal, adversarial, or reference-aware route.

So the new invariant should be:

Every answer record must bind:
    question identity,
    retrieval plan,
    retrieval result,
    context root,
    model/policy identity,
    verifier identity,
    final claim/admissibility state.

That is the corrected PROMETHEUS-Σ / CTI / Merkle-AGI principle.

---

A.14 Final recommendation

Yes, land the fix, but land it as:

J — Retrieval Plan Provenance Binding

Implement:

1. Capture retrieval_keywords and retrieval config in run_dag retrieval stage.
2. Store retrieval_keywords + retrieval_plan_hash on providence_records.
3. Add audit events for retrieval_plan_built and retrieval_result_selected.
4. Do not fold keywords into question_hash.
5. Do not add keywords to cache_key by default.
6. Add optional strict mode where retrieval_plan_hash affects cache_key.

This preserves the semantic distinction:

question = what the user asked
retrieval plan = how the system searched
context root = what the system found

That is axiomatically cleaner, audit-complete, and compatible with the v9.8 provenance model.