diff --git a/docs/ticket-000001-retrieval-keywords-audit-gap.md b/docs/ticket-000001-retrieval-keywords-audit-gap.md index bc3baa9..b3ba91a 100644 --- a/docs/ticket-000001-retrieval-keywords-audit-gap.md +++ b/docs/ticket-000001-retrieval-keywords-audit-gap.md @@ -238,3 +238,721 @@ change, additive run_dag input, no cache-key churn. Forecast value: closes the operator-hint provenance loop. Necessary for audit-grade reproducibility once `--retrieval-keywords` becomes a regular operator practice. + +--- + +## Appendix A — Architectural review (2026-05-01, Asia/Kuala_Lumpur) + +> Fox-supplied review expanding §2-§4 with axiomatic framing, +> CTI / Merkle-AGI / PROMETHEUS-Σ interpretations, a concrete +> implementation diff, and a strengthened test list. Captured +> verbatim. Treat this appendix as authoritative when it disagrees +> with §2-§4 above. + +Yes: this is a **real provenance gap**. + +The correct diagnosis is: + +```text +question text → bound +conversation state → bound +source/context root → bound +run DAG output → partly bound +retrieval keywords → not directly bound +``` + +So an auditor can currently recover: + +```text +"These sources were selected." +``` + +but not: + +```text +"These retrieval keywords / operator hints caused those sources to be selected." +``` + +That is incomplete provenance. + +The proposed recommendation — **1 + 2, not 3** — is the right default. + +--- + +### A.1 Axiomatic framing + +Define the relevant objects: + +```text +Q = user question text +K = retrieval keywords / operator search hints +R = retrieval configuration: top_k, over_fetch, shard set, filters, max_context_chars +S = retrieved source set / sources_summary +C = assembled context root +A = answer +D = run DAG +``` + +Current approximate chain: + +```text +Q → keyword generation/search → S → C → A +``` + +But the committed audit chain mostly records: + +```text +Q, S, C, A +``` + +not: + +```text +K, R +``` + +So the missing edge is: + +```text +(K, R) → S +``` + +That is the provenance gap. + +In CTI / Merkle-AGI language: + +```text +retrieval_keywords are parent clauses of retrieval_result +``` + +If a parent clause is not committed, the retrieval node is under-specified. + +--- + +### A.2 Why this matters + +Two runs can have: + +```text +same Q +same S +same C +same A +different K +``` + +and still become Merkle-indistinguishable if the retrieval stage hashes only `sources_summary`. + +Example: + +```text +Q = "has oceania always been at war with east asia" + +K1 = ["Oceania", "East Asia", "war", "history"] +K2 = ["Nineteen Eighty-Four", "Oceania", "Eastasia", "always at war"] +``` + +If both eventually surface the same source set, the current DAG may not distinguish whether the system reached the source through literal geography routing or Orwell-reference routing. + +That matters because retrieval intent is part of the reasoning trace. + +It is especially important for your recent failure classes: + +```text +reference-frame routing +relation-warrant retrieval +operator hints +one-shot evidence maps +old-map avoidance +``` + +If you do not bind retrieval inputs, you cannot audit whether the system used the right map. + +--- + +### A.3 Do not fold keywords into `question_hash` by default + +I agree with **not** doing fix 3 as the default. + +Reason: + +```text +question_hash should identify the user's question, +not the operator's retrieval strategy. +``` + +Keywords are not always semantically part of the question. They may be: + +```text +retrieval expansion +operator hint +routing hypothesis +fallback search string +reference-frame expansion +relation-warrant expansion +debug override +``` + +If you fold `K` into `question_hash`, then these become treated as different questions: + +```text +Q + literal geography keywords +Q + Orwell reference keywords +``` + +That loses a useful distinction: + +```text +same user question, +different retrieval plan. +``` + +Better: + +```text +question_hash = identity of user question +retrieval_plan_hash = identity of search strategy +``` + +Keep them separate. + +--- + +### A.4 Best fix: add retrieval plan as first-class provenance + +Implement **1 + 2**, with one extra refinement. + +#### A.4.1 Add retrieval input to run DAG + +Current retrieval node appears to bind only the output: + +```text +retrieval_stage = H(sources_summary) +``` + +Change it to bind both input and output: + +```text +retrieval_stage = H({ + "question_hash": question_hash, + "retrieval_keywords": retrieval_keywords, + "retrieval_mode": retrieval_mode, + "top_k": top_k, + "over_fetch": over_fetch, + "max_context_chars": max_context_chars, + "filters": filters, + "shards": shard_ids_or_roots, + "sources_summary": sources_summary +}) +``` + +Better still: + +```text +retrieval_plan_hash = H({ + "retrieval_keywords": retrieval_keywords, + "retrieval_mode": retrieval_mode, + "top_k": top_k, + "over_fetch": over_fetch, + "max_context_chars": max_context_chars, + "filters": filters, + "shards": shard_ids_or_roots, + "ranking_policy_hash": ranking_policy_hash +}) + +retrieval_result_hash = H({ + "sources_summary": sources_summary, + "context_root": context_root +}) + +retrieval_stage_hash = H({ + "question_hash": question_hash, + "retrieval_plan_hash": retrieval_plan_hash, + "retrieval_result_hash": retrieval_result_hash +}) +``` + +This is cleaner because it separates: + +```text +plan → result +``` + +rather than mixing them. + +--- + +#### A.4.2 Store keywords on `providence_records` + +Add nullable columns: + +```sql +ALTER TABLE providence_records ADD COLUMN retrieval_keywords TEXT; +ALTER TABLE providence_records ADD COLUMN retrieval_plan_hash TEXT; +ALTER TABLE providence_records ADD COLUMN retrieval_mode TEXT; +``` + +If using canonical JSON: + +```sql +retrieval_keywords TEXT -- canonical JSON array +``` + +Example: + +```json +["Nineteen Eighty-Four", "Oceania", "Eastasia", "always at war"] +``` + +This makes inspection cheap: + +```text +No need to parse run_dag_blob just to see how retrieval was driven. +``` + +--- + +#### A.4.3 Also add retrieval plan to audit events + +This is the refinement. + +Add audit event: + +```text +retrieval_plan_built +``` + +Payload: + +```json +{ + "question_hash": "...", + "retrieval_keywords": ["..."], + "retrieval_mode": "reference_router", + "top_k": 8, + "over_fetch": 32, + "max_context_chars": 60000, + "retrieval_plan_hash": "..." +} +``` + +Then: + +```text +retrieval_result_selected +``` + +Payload: + +```json +{ + "retrieval_plan_hash": "...", + "sources_summary_hash": "...", + "context_root": "...", + "source_count": 8 +} +``` + +This gives full replay provenance. + +The v9.8 spec already treats audit events as authoritative in the SQLite audit table, with hash-verifiable audit events and optional chaining. That makes audit events the right place to preserve retrieval-control history as well. + +--- + +### A.5 Should retrieval keywords affect cache identity? + +Default: **no**. + +But there is a nuance. + +#### Case A — Same question, different keywords, same context root + +```text +Q same +K different +C same +A same +``` + +Cache reuse is probably acceptable because the answer was generated from the same source/context/model/policy state. + +But provenance must record: + +```text +this cache hit occurred under retrieval_plan_hash = X +``` + +So on cache hit, you still need an audit event: + +```text +cache_hit_with_retrieval_plan +``` + +Otherwise the second run's retrieval plan disappears. + +#### Case B — Same question, different keywords, different context root + +Then cache key already changes if `context_root` or conversation/source context hash is part of cache identity. + +#### Case C — Keywords imply semantic framing but context is same + +This is tricky. + +Example: + +```text +K1 = literal geography route +K2 = Orwell route +same sources happen to appear +``` + +If the prompt sent to the model does **not** include K, and the context is the same, cache reuse is technically consistent. + +But if K influences prompt framing, then K must be part of `conversation_hash` or `prompt_hash`. + +Rule: + +```text +If retrieval keywords are only retrieval-control metadata: + store/hash in retrieval_plan, not question_hash/cache_key. + +If retrieval keywords are shown to the model or affect answer instructions: + include them in prompt_hash/conversation_hash. + +If retrieval keywords are policy-level routing state: + include retrieval_policy_hash or verifier_policy_hash. +``` + +This is the clean boundary. + +--- + +### A.6 CTI interpretation + +In CTI, retrieval keywords are not just logs. They are **upstream clauses**. + +For example: + +```text +C0: User asked Q. +C1: Router selected literal/geography frame. +C2: Retrieval keywords K were generated. +C3: Search returned sources S. +C4: Context C was assembled. +C5: Answer A was generated. +``` + +Current DAG commits C3–C5 better than C1–C2. + +Fixing this means the retrieval stage becomes a real clause subgraph: + +```text +frame_node +→ retrieval_plan_node +→ retrieval_result_node +→ context_node +``` + +That is a stronger CTI graph. + +--- + +### A.7 Merkle-AGI interpretation + +The Merkle-AGI axiom is: + +> Every causally relevant transformation input must be committed if the run is to be replay-auditable. + +Retrieval keywords are causally relevant because they influence source selection. + +Therefore, they must be committed somewhere. + +They do **not** necessarily need to alter the user question hash. + +Correct commitment structure: + +```text +question_hash +retrieval_plan_hash +retrieval_result_hash +context_root +answer_hash +verify_hash +final_label_hash +``` + +Then: + +```text +run_dag_root = H(all stage hashes) +``` + +This preserves both: + +```text +semantic identity of question +``` + +and: + +```text +operational identity of retrieval. +``` + +--- + +### A.8 PROMETHEUS-Σ interpretation + +PROMETHEUS-Σ should treat this as a **policy and admissibility issue**. + +Add a policy field: + +```json +{ + "retrieval": { + "bind_keywords_in_run_dag": true, + "store_keywords_on_record": true, + "keywords_affect_cache_key": false, + "keywords_affect_prompt_hash_if_prompted": true + } +} +``` + +This makes the distinction explicit. + +--- + +### A.9 Best implementation ticket + +#### Ticket name + +```text +J — Retrieval Plan Provenance Binding +``` + +#### Scope + +```text +- Hash retrieval inputs into run DAG. +- Store retrieval_keywords and retrieval_plan_hash on providence_records. +- Emit audit events for retrieval plan and retrieval result. +- Do not fold keywords into question_hash. +- Do not alter cache_key unless keywords enter prompt/conversation state. +``` + +--- + +### A.10 Concrete diff + +#### `aborist/qa/retrieval.py` + +Return a structured object: + +```python +@dataclass(frozen=True) +class RetrievalPlan: + question_hash: str + retrieval_keywords: list[str] + retrieval_mode: str + top_k: int + over_fetch: int + max_context_chars: int + filters: dict + shard_ids: list[str] + ranking_policy_hash: str + + def canonical(self) -> dict: + return { + "question_hash": self.question_hash, + "retrieval_keywords": self.retrieval_keywords, + "retrieval_mode": self.retrieval_mode, + "top_k": self.top_k, + "over_fetch": self.over_fetch, + "max_context_chars": self.max_context_chars, + "filters": self.filters, + "shard_ids": self.shard_ids, + "ranking_policy_hash": self.ranking_policy_hash, + } +``` + +#### `aborist/qa/hash.py` + +```python +def retrieval_plan_hash(plan: RetrievalPlan) -> str: + return hash_json("aborist-retrieval-plan-v1", plan.canonical()) +``` + +#### `aborist/qa/dag.py` + +Change retrieval node from: + +```python +retrieval_hash = hash_json({"sources_summary": sources_summary}) +``` + +to: + +```python +retrieval_hash = hash_json({ + "stage": "retrieval", + "retrieval_plan_hash": retrieval_plan_hash, + "retrieval_plan": retrieval_plan.canonical(), + "sources_summary": sources_summary, + "context_root": context_root, +}) +``` + +Or, if you do not want full keywords in DAG blob for privacy: + +```python +retrieval_hash = hash_json({ + "stage": "retrieval", + "retrieval_plan_hash": retrieval_plan_hash, + "sources_summary_hash": hash_json("sources-summary-v1", sources_summary), + "context_root": context_root, +}) +``` + +and store the plan in DB privately. + +#### SQLite migration + +```sql +ALTER TABLE providence_records ADD COLUMN retrieval_keywords TEXT; +ALTER TABLE providence_records ADD COLUMN retrieval_plan_hash TEXT; +ALTER TABLE providence_records ADD COLUMN retrieval_mode TEXT; +ALTER TABLE providence_records ADD COLUMN ranking_policy_hash TEXT; +``` + +#### `audit.py` + +Add: + +```text +retrieval_plan_built +retrieval_result_selected +``` + +--- + +### A.11 Cache-key policy + +Do **not** add retrieval keywords to `question_hash`. + +Do **not** add retrieval keywords directly to cache key by default. + +But add a future optional mode: + +```json +{ + "cache": { + "retrieval_plan_affects_cache_key": false + } +} +``` + +If set true, use: + +```text +cache_key_v2 = H({ + existing_cache_key_fields, + retrieval_plan_hash +}) +``` + +Default false. + +Reason: + +```text +most of the time, cache identity should be answer-context identity, +not operator-search identity. +``` + +But high-assurance deployments may want stricter replay identity. + +--- + +### A.12 Tests to add + +```text +[ ] retrieval_keywords stored on providence record. +[ ] retrieval_plan_hash changes when keyword order/content changes. +[ ] retrieval_plan_hash changes when top_k changes. +[ ] retrieval_plan_hash changes when over_fetch changes. +[ ] retrieval_plan_hash changes when max_context_chars changes. +[ ] run_dag_root changes when retrieval_plan_hash changes. +[ ] two runs with same sources but different keywords have different run_dag_root. +[ ] question_hash unchanged when only retrieval_keywords change. +[ ] cache_key unchanged when only retrieval_keywords change under default policy. +[ ] cache_key changes when retrieval_plan_affects_cache_key = true. +[ ] audit_events include retrieval_plan_built. +[ ] cache hit still records current retrieval_plan_hash in audit event. +``` + +The critical one: + +```text +same sources, different keywords → different run_dag_root +``` + +This proves the provenance gap is closed. + +--- + +### A.13 Additive insight + +This is not only a bug fix. It is a core architecture principle: + +> **The map must be committed, not only the territory reached.** + +Source outputs are the territory reached. +Retrieval keywords are the map used to get there. + +Your recent "old maps" point applies directly. If you do not bind the map, you cannot later tell whether the system used a stale, literal, adversarial, or reference-aware route. + +So the new invariant should be: + +```text +Every answer record must bind: + question identity, + retrieval plan, + retrieval result, + context root, + model/policy identity, + verifier identity, + final claim/admissibility state. +``` + +That is the corrected PROMETHEUS-Σ / CTI / Merkle-AGI principle. + +--- + +### A.14 Final recommendation + +Yes, land the fix, but land it as: + +```text +J — Retrieval Plan Provenance Binding +``` + +Implement: + +```text +1. Capture retrieval_keywords and retrieval config in run_dag retrieval stage. +2. Store retrieval_keywords + retrieval_plan_hash on providence_records. +3. Add audit events for retrieval_plan_built and retrieval_result_selected. +4. Do not fold keywords into question_hash. +5. Do not add keywords to cache_key by default. +6. Add optional strict mode where retrieval_plan_hash affects cache_key. +``` + +This preserves the semantic distinction: + +```text +question = what the user asked +retrieval plan = how the system searched +context root = what the system found +``` + +That is axiomatically cleaner, audit-complete, and compatible with the v9.8 provenance model.