docs(ticket-000001): append architectural review (Asia/Kuala_Lumpur, 2026-05-01)

Fox-supplied review expanding §2-§4 with axiomatic framing, CTI / Merkle-AGI / PROMETHEUS-Σ interpretations, a concrete implementation diff, and a strengthened test list. Captured verbatim as Appendix A. Key refinements over the §2-§4 sketch: - Splits retrieval-stage hash into retrieval_plan_hash + retrieval_result_hash + retrieval_stage_hash (plan vs result separation, axiomatically cleaner than embedding plan in the retrieval-stage payload directly). - Adds two new audit events: retrieval_plan_built and retrieval_result_selected. Surfaces the retrieval inputs in the audit chain (not just the run_dag), so cache-hits also commit the plan via cache_hit_with_retrieval_plan. - Distinguishes three keyword-handling cases for cache identity (Case A/B/C) and lays out a clean boundary rule: keywords stay in retrieval_plan unless they reach the model (then prompt_hash) or affect routing policy (then retrieval_policy_hash). - Promotes 'the map must be committed, not only the territory reached' as a core principle. Operator-supplied maps (keywords, routing modes, ranking policies) become first-class clauses upstream of retrieval_result. - Lists 12 concrete tests including the critical 'same sources, different keywords → different run_dag_root' assertion that proves the gap is closed. Header note: Appendix A is authoritative when it disagrees with §2-§4 above. The original §2-§4 sketch stays in place for chronological context but is superseded by the appendix.
2026-05-01 12:22:16 -04:00 · 2026-05-01 12:22:16 -04:00 · 292e49e18f
commit 292e49e18f
parent 5990834ce8
1 changed files with 718 additions and 0 deletions
--- a/docs/ticket-000001-retrieval-keywords-audit-gap.md
+++ b/docs/ticket-000001-retrieval-keywords-audit-gap.md
@ -238,3 +238,721 @@ change, additive run_dag input, no cache-key churn.
 Forecast value: closes the operator-hint provenance loop. Necessary
 for audit-grade reproducibility once `--retrieval-keywords` becomes
 a regular operator practice.
+
+---
+
+## Appendix A — Architectural review (2026-05-01, Asia/Kuala_Lumpur)
+
+> Fox-supplied review expanding §2-§4 with axiomatic framing,
+> CTI / Merkle-AGI / PROMETHEUS-Σ interpretations, a concrete
+> implementation diff, and a strengthened test list. Captured
+> verbatim. Treat this appendix as authoritative when it disagrees
+> with §2-§4 above.
+
+Yes: this is a **real provenance gap**.
+
+The correct diagnosis is:
+
+```text
+question text       → bound
+conversation state  → bound
+source/context root → bound
+run DAG output      → partly bound
+retrieval keywords  → not directly bound
+```
+
+So an auditor can currently recover:
+
+```text
+"These sources were selected."
+```
+
+but not:
+
+```text
+"These retrieval keywords / operator hints caused those sources to be selected."
+```
+
+That is incomplete provenance.
+
+The proposed recommendation — **1 + 2, not 3** — is the right default.
+
+---
+
+### A.1 Axiomatic framing
+
+Define the relevant objects:
+
+```text
+Q  = user question text
+K  = retrieval keywords / operator search hints
+R  = retrieval configuration: top_k, over_fetch, shard set, filters, max_context_chars
+S  = retrieved source set / sources_summary
+C  = assembled context root
+A  = answer
+D  = run DAG
+```
+
+Current approximate chain:
+
+```text
+Q → keyword generation/search → S → C → A
+```
+
+But the committed audit chain mostly records:
+
+```text
+Q, S, C, A
+```
+
+not:
+
+```text
+K, R
+```
+
+So the missing edge is:
+
+```text
+(K, R) → S
+```
+
+That is the provenance gap.
+
+In CTI / Merkle-AGI language:
+
+```text
+retrieval_keywords are parent clauses of retrieval_result
+```
+
+If a parent clause is not committed, the retrieval node is under-specified.
+
+---
+
+### A.2 Why this matters
+
+Two runs can have:
+
+```text
+same Q
+same S
+same C
+same A
+different K
+```
+
+and still become Merkle-indistinguishable if the retrieval stage hashes only `sources_summary`.
+
+Example:
+
+```text
+Q = "has oceania always been at war with east asia"
+
+K1 = ["Oceania", "East Asia", "war", "history"]
+K2 = ["Nineteen Eighty-Four", "Oceania", "Eastasia", "always at war"]
+```
+
+If both eventually surface the same source set, the current DAG may not distinguish whether the system reached the source through literal geography routing or Orwell-reference routing.
+
+That matters because retrieval intent is part of the reasoning trace.
+
+It is especially important for your recent failure classes:
+
+```text
+reference-frame routing
+relation-warrant retrieval
+operator hints
+one-shot evidence maps
+old-map avoidance
+```
+
+If you do not bind retrieval inputs, you cannot audit whether the system used the right map.
+
+---
+
+### A.3 Do not fold keywords into `question_hash` by default
+
+I agree with **not** doing fix 3 as the default.
+
+Reason:
+
+```text
+question_hash should identify the user's question,
+not the operator's retrieval strategy.
+```
+
+Keywords are not always semantically part of the question. They may be:
+
+```text
+retrieval expansion
+operator hint
+routing hypothesis
+fallback search string
+reference-frame expansion
+relation-warrant expansion
+debug override
+```
+
+If you fold `K` into `question_hash`, then these become treated as different questions:
+
+```text
+Q + literal geography keywords
+Q + Orwell reference keywords
+```
+
+That loses a useful distinction:
+
+```text
+same user question,
+different retrieval plan.
+```
+
+Better:
+
+```text
+question_hash = identity of user question
+retrieval_plan_hash = identity of search strategy
+```
+
+Keep them separate.
+
+---
+
+### A.4 Best fix: add retrieval plan as first-class provenance
+
+Implement **1 + 2**, with one extra refinement.
+
+#### A.4.1 Add retrieval input to run DAG
+
+Current retrieval node appears to bind only the output:
+
+```text
+retrieval_stage = H(sources_summary)
+```
+
+Change it to bind both input and output:
+
+```text
+retrieval_stage = H({
+  "question_hash": question_hash,
+  "retrieval_keywords": retrieval_keywords,
+  "retrieval_mode": retrieval_mode,
+  "top_k": top_k,
+  "over_fetch": over_fetch,
+  "max_context_chars": max_context_chars,
+  "filters": filters,
+  "shards": shard_ids_or_roots,
+  "sources_summary": sources_summary
+})
+```
+
+Better still:
+
+```text
+retrieval_plan_hash = H({
+  "retrieval_keywords": retrieval_keywords,
+  "retrieval_mode": retrieval_mode,
+  "top_k": top_k,
+  "over_fetch": over_fetch,
+  "max_context_chars": max_context_chars,
+  "filters": filters,
+  "shards": shard_ids_or_roots,
+  "ranking_policy_hash": ranking_policy_hash
+})
+
+retrieval_result_hash = H({
+  "sources_summary": sources_summary,
+  "context_root": context_root
+})
+
+retrieval_stage_hash = H({
+  "question_hash": question_hash,
+  "retrieval_plan_hash": retrieval_plan_hash,
+  "retrieval_result_hash": retrieval_result_hash
+})
+```
+
+This is cleaner because it separates:
+
+```text
+plan → result
+```
+
+rather than mixing them.
+
+---
+
+#### A.4.2 Store keywords on `providence_records`
+
+Add nullable columns:
+
+```sql
+ALTER TABLE providence_records ADD COLUMN retrieval_keywords TEXT;
+ALTER TABLE providence_records ADD COLUMN retrieval_plan_hash TEXT;
+ALTER TABLE providence_records ADD COLUMN retrieval_mode TEXT;
+```
+
+If using canonical JSON:
+
+```sql
+retrieval_keywords TEXT -- canonical JSON array
+```
+
+Example:
+
+```json
+["Nineteen Eighty-Four", "Oceania", "Eastasia", "always at war"]
+```
+
+This makes inspection cheap:
+
+```text
+No need to parse run_dag_blob just to see how retrieval was driven.
+```
+
+---
+
+#### A.4.3 Also add retrieval plan to audit events
+
+This is the refinement.
+
+Add audit event:
+
+```text
+retrieval_plan_built
+```
+
+Payload:
+
+```json
+{
+  "question_hash": "...",
+  "retrieval_keywords": ["..."],
+  "retrieval_mode": "reference_router",
+  "top_k": 8,
+  "over_fetch": 32,
+  "max_context_chars": 60000,
+  "retrieval_plan_hash": "..."
+}
+```
+
+Then:
+
+```text
+retrieval_result_selected
+```
+
+Payload:
+
+```json
+{
+  "retrieval_plan_hash": "...",
+  "sources_summary_hash": "...",
+  "context_root": "...",
+  "source_count": 8
+}
+```
+
+This gives full replay provenance.
+
+The v9.8 spec already treats audit events as authoritative in the SQLite audit table, with hash-verifiable audit events and optional chaining. That makes audit events the right place to preserve retrieval-control history as well.
+
+---
+
+### A.5 Should retrieval keywords affect cache identity?
+
+Default: **no**.
+
+But there is a nuance.
+
+#### Case A — Same question, different keywords, same context root
+
+```text
+Q same
+K different
+C same
+A same
+```
+
+Cache reuse is probably acceptable because the answer was generated from the same source/context/model/policy state.
+
+But provenance must record:
+
+```text
+this cache hit occurred under retrieval_plan_hash = X
+```
+
+So on cache hit, you still need an audit event:
+
+```text
+cache_hit_with_retrieval_plan
+```
+
+Otherwise the second run's retrieval plan disappears.
+
+#### Case B — Same question, different keywords, different context root
+
+Then cache key already changes if `context_root` or conversation/source context hash is part of cache identity.
+
+#### Case C — Keywords imply semantic framing but context is same
+
+This is tricky.
+
+Example:
+
+```text
+K1 = literal geography route
+K2 = Orwell route
+same sources happen to appear
+```
+
+If the prompt sent to the model does **not** include K, and the context is the same, cache reuse is technically consistent.
+
+But if K influences prompt framing, then K must be part of `conversation_hash` or `prompt_hash`.
+
+Rule:
+
+```text
+If retrieval keywords are only retrieval-control metadata:
+    store/hash in retrieval_plan, not question_hash/cache_key.
+
+If retrieval keywords are shown to the model or affect answer instructions:
+    include them in prompt_hash/conversation_hash.
+
+If retrieval keywords are policy-level routing state:
+    include retrieval_policy_hash or verifier_policy_hash.
+```
+
+This is the clean boundary.
+
+---
+
+### A.6 CTI interpretation
+
+In CTI, retrieval keywords are not just logs. They are **upstream clauses**.
+
+For example:
+
+```text
+C0: User asked Q.
+C1: Router selected literal/geography frame.
+C2: Retrieval keywords K were generated.
+C3: Search returned sources S.
+C4: Context C was assembled.
+C5: Answer A was generated.
+```
+
+Current DAG commits C3–C5 better than C1–C2.
+
+Fixing this means the retrieval stage becomes a real clause subgraph:
+
+```text
+frame_node
+→ retrieval_plan_node
+→ retrieval_result_node
+→ context_node
+```
+
+That is a stronger CTI graph.
+
+---
+
+### A.7 Merkle-AGI interpretation
+
+The Merkle-AGI axiom is:
+
+> Every causally relevant transformation input must be committed if the run is to be replay-auditable.
+
+Retrieval keywords are causally relevant because they influence source selection.
+
+Therefore, they must be committed somewhere.
+
+They do **not** necessarily need to alter the user question hash.
+
+Correct commitment structure:
+
+```text
+question_hash
+retrieval_plan_hash
+retrieval_result_hash
+context_root
+answer_hash
+verify_hash
+final_label_hash
+```
+
+Then:
+
+```text
+run_dag_root = H(all stage hashes)
+```
+
+This preserves both:
+
+```text
+semantic identity of question
+```
+
+and:
+
+```text
+operational identity of retrieval.
+```
+
+---
+
+### A.8 PROMETHEUS-Σ interpretation
+
+PROMETHEUS-Σ should treat this as a **policy and admissibility issue**.
+
+Add a policy field:
+
+```json
+{
+  "retrieval": {
+    "bind_keywords_in_run_dag": true,
+    "store_keywords_on_record": true,
+    "keywords_affect_cache_key": false,
+    "keywords_affect_prompt_hash_if_prompted": true
+  }
+}
+```
+
+This makes the distinction explicit.
+
+---
+
+### A.9 Best implementation ticket
+
+#### Ticket name
+
+```text
+J — Retrieval Plan Provenance Binding
+```
+
+#### Scope
+
+```text
+- Hash retrieval inputs into run DAG.
+- Store retrieval_keywords and retrieval_plan_hash on providence_records.
+- Emit audit events for retrieval plan and retrieval result.
+- Do not fold keywords into question_hash.
+- Do not alter cache_key unless keywords enter prompt/conversation state.
+```
+
+---
+
+### A.10 Concrete diff
+
+#### `aborist/qa/retrieval.py`
+
+Return a structured object:
+
+```python
+@dataclass(frozen=True)
+class RetrievalPlan:
+    question_hash: str
+    retrieval_keywords: list[str]
+    retrieval_mode: str
+    top_k: int
+    over_fetch: int
+    max_context_chars: int
+    filters: dict
+    shard_ids: list[str]
+    ranking_policy_hash: str
+
+    def canonical(self) -> dict:
+        return {
+            "question_hash": self.question_hash,
+            "retrieval_keywords": self.retrieval_keywords,
+            "retrieval_mode": self.retrieval_mode,
+            "top_k": self.top_k,
+            "over_fetch": self.over_fetch,
+            "max_context_chars": self.max_context_chars,
+            "filters": self.filters,
+            "shard_ids": self.shard_ids,
+            "ranking_policy_hash": self.ranking_policy_hash,
+        }
+```
+
+#### `aborist/qa/hash.py`
+
+```python
+def retrieval_plan_hash(plan: RetrievalPlan) -> str:
+    return hash_json("aborist-retrieval-plan-v1", plan.canonical())
+```
+
+#### `aborist/qa/dag.py`
+
+Change retrieval node from:
+
+```python
+retrieval_hash = hash_json({"sources_summary": sources_summary})
+```
+
+to:
+
+```python
+retrieval_hash = hash_json({
+    "stage": "retrieval",
+    "retrieval_plan_hash": retrieval_plan_hash,
+    "retrieval_plan": retrieval_plan.canonical(),
+    "sources_summary": sources_summary,
+    "context_root": context_root,
+})
+```
+
+Or, if you do not want full keywords in DAG blob for privacy:
+
+```python
+retrieval_hash = hash_json({
+    "stage": "retrieval",
+    "retrieval_plan_hash": retrieval_plan_hash,
+    "sources_summary_hash": hash_json("sources-summary-v1", sources_summary),
+    "context_root": context_root,
+})
+```
+
+and store the plan in DB privately.
+
+#### SQLite migration
+
+```sql
+ALTER TABLE providence_records ADD COLUMN retrieval_keywords TEXT;
+ALTER TABLE providence_records ADD COLUMN retrieval_plan_hash TEXT;
+ALTER TABLE providence_records ADD COLUMN retrieval_mode TEXT;
+ALTER TABLE providence_records ADD COLUMN ranking_policy_hash TEXT;
+```
+
+#### `audit.py`
+
+Add:
+
+```text
+retrieval_plan_built
+retrieval_result_selected
+```
+
+---
+
+### A.11 Cache-key policy
+
+Do **not** add retrieval keywords to `question_hash`.
+
+Do **not** add retrieval keywords directly to cache key by default.
+
+But add a future optional mode:
+
+```json
+{
+  "cache": {
+    "retrieval_plan_affects_cache_key": false
+  }
+}
+```
+
+If set true, use:
+
+```text
+cache_key_v2 = H({
+  existing_cache_key_fields,
+  retrieval_plan_hash
+})
+```
+
+Default false.
+
+Reason:
+
+```text
+most of the time, cache identity should be answer-context identity,
+not operator-search identity.
+```
+
+But high-assurance deployments may want stricter replay identity.
+
+---
+
+### A.12 Tests to add
+
+```text
+[ ] retrieval_keywords stored on providence record.
+[ ] retrieval_plan_hash changes when keyword order/content changes.
+[ ] retrieval_plan_hash changes when top_k changes.
+[ ] retrieval_plan_hash changes when over_fetch changes.
+[ ] retrieval_plan_hash changes when max_context_chars changes.
+[ ] run_dag_root changes when retrieval_plan_hash changes.
+[ ] two runs with same sources but different keywords have different run_dag_root.
+[ ] question_hash unchanged when only retrieval_keywords change.
+[ ] cache_key unchanged when only retrieval_keywords change under default policy.
+[ ] cache_key changes when retrieval_plan_affects_cache_key = true.
+[ ] audit_events include retrieval_plan_built.
+[ ] cache hit still records current retrieval_plan_hash in audit event.
+```
+
+The critical one:
+
+```text
+same sources, different keywords → different run_dag_root
+```
+
+This proves the provenance gap is closed.
+
+---
+
+### A.13 Additive insight
+
+This is not only a bug fix. It is a core architecture principle:
+
+> **The map must be committed, not only the territory reached.**
+
+Source outputs are the territory reached.
+Retrieval keywords are the map used to get there.
+
+Your recent "old maps" point applies directly. If you do not bind the map, you cannot later tell whether the system used a stale, literal, adversarial, or reference-aware route.
+
+So the new invariant should be:
+
+```text
+Every answer record must bind:
+    question identity,
+    retrieval plan,
+    retrieval result,
+    context root,
+    model/policy identity,
+    verifier identity,
+    final claim/admissibility state.
+```
+
+That is the corrected PROMETHEUS-Σ / CTI / Merkle-AGI principle.
+
+---
+
+### A.14 Final recommendation
+
+Yes, land the fix, but land it as:
+
+```text
+J — Retrieval Plan Provenance Binding
+```
+
+Implement:
+
+```text
+1. Capture retrieval_keywords and retrieval config in run_dag retrieval stage.
+2. Store retrieval_keywords + retrieval_plan_hash on providence_records.
+3. Add audit events for retrieval_plan_built and retrieval_result_selected.
+4. Do not fold keywords into question_hash.
+5. Do not add keywords to cache_key by default.
+6. Add optional strict mode where retrieval_plan_hash affects cache_key.
+```
+
+This preserves the semantic distinction:
+
+```text
+question = what the user asked
+retrieval plan = how the system searched
+context root = what the system found
+```
+
+That is axiomatically cleaner, audit-complete, and compatible with the v9.8 provenance model.