session: thread conversation_history through providence_query (multi-turn LLM context)

The verifier was assuming single-turn — a session pronoun follow-up
("who created it?" after "what is permaculture?") hit the LLM with
no antecedent in the question, so the model couldn't ground.
Retrieval keywords already flowed ancestor titles → FTS5 (the
Permaculture article was in context), but the LLM saw just "who
created it?" with no subject. → UNGROUNDED.

Wire (verifier-blind for retrieval; cache_key-aware for chat):

- providence_query + run_query: new ``conversation_history`` param
  (list of OpenAI-shape {role, content} dicts). When set, messages
  become [system, ...history..., user(EVIDENCE+QUESTION)] instead
  of [system, user]. Folded into the canonical-messages hash so
  ``conversation_hash`` reflects the chat path — each distinct
  branch gets its own cache_key (fix verified: the same question
  with vs without history now produces different cache_keys).

- CLI session REPL: builds history from path_to_root of
  current_bates (which IS the parent at the moment of the new turn).
  Oldest-first. Strips claim-lattice pointer-line excerpts from
  prior answers via _session_strip_pointers — the [E#] bindings are
  stale across turns (each turn mints a fresh evidence_map) and
  the spotlight excerpts inflate the prompt without antecedent
  value. The bare prose claims remain.

- Off-by-one fix: was stripping path[1:] under the assumption that
  the current node was the just-added turn; in the REPL, current is
  the PARENT before the new turn is minted, so path-to-root IS the
  history. Drop the [1:].

Live test: "what is permaculture?" → STRICT → "who created it?"
→ STRICT (was UNGROUNDED). The model cites the Permaculture and
David Holmgren articles correctly for the pronoun question.

33 verify+session+providence tests pass. The byte-identity test
for run_query without history is preserved — history defaults to
None and skips the extension entirely.

arborist/cli.py

@@ -5,6 +5,7 @@ from __future__ import annotations
 import argparse
 import json
 import os
+import re
 import sys
 from pathlib import Path
 
@@ -7517,6 +7518,35 @@ def _ignore(fn, *a, **kw):
         pass
 
 
+_POINTER_LINE_RE = re.compile(
+    r'^\s*\[E\d+(?:\s*\|[^\]]*?)?\]\s*$',
+    flags=re.MULTILINE,
+)
+_INLINE_POINTER_RE = re.compile(r'\[E\d+(?:\s*\|[^\]]*?)?\]')
+
+
+def _session_strip_pointers(answer_text: str) -> str:
+    """Strip claim-lattice pointer markers from a prior turn's answer
+    before threading it through as LLM conversation history. The
+    spotlight-rendered ``[E# | <title> | <chunk_prefix>: "..."]``
+    blocks (which sit on their own line per claim) get removed; bare
+    ``[E#]`` mid-prose markers also drop. Leaves the prose claims
+    intact so the model sees: "Spider-Man is a superhero. Created by
+    Stan Lee." without stale pointer ids that don't match the new
+    turn's evidence_map.
+    """
+    if not answer_text:
+        return ""
+    # Drop standalone pointer lines first (they're the bulk of the
+    # answer_text size for claim_lattice mode).
+    s = _POINTER_LINE_RE.sub("", answer_text)
+    # Then drop any remaining inline [E#] / [E# | …] markers.
+    s = _INLINE_POINTER_RE.sub("", s)
+    # Collapse multi-blank-lines.
+    s = re.sub(r"\n{3,}", "\n\n", s).strip()
+    return s
+
+
 def _cmd_session(args: argparse.Namespace) -> int:
     """Merkle-rooted multi-turn Q&A REPL with forks.
 
@@ -7829,16 +7859,40 @@ def _session_repl(args: argparse.Namespace, sess) -> int:
                 print(f"unknown command: {line}  (try /help)")
                 continue
 
-            # Non-command line → new question. Read cited titles
+            # Non-command line → new question. Two pieces of session
-            # directly from ancestor node rows (already materialized at
+            # context flow into the call:
-            # insert time) and pass as retrieval_keywords. Verifier-
+            #
-            # blind; same discipline as the legacy --retrieval-keywords
+            # 1. retrieval_keywords: ancestor cited titles → widen
-            # flag (#000001).
+            #    FTS5 retrieval. Verifier-blind; same discipline as
+            #    the legacy --retrieval-keywords flag (#000001).
+            # 2. conversation_history: ancestor (user/assistant)
+            #    pairs → give the LLM antecedent context for pronoun
+            #    questions ("who created it?" after "what is
+            #    permaculture?"). Folds into conversation_hash so each
+            #    distinct chat path gets its own cache_key.
             seen_titles: list[str] = []
-            for n in sess.path_to_root(sess.current_bates)[1:]:
+            history: list[dict] = []
+            # current_bates is the PARENT of the soon-to-be-minted
+            # node, so its path-to-root IS the conversation history.
+            # Walk oldest-first so the LLM reads in chronological
+            # order; skip the synthetic root (seq=0).
+            ancestors = sess.path_to_root(sess.current_bates)
+            for n in reversed(ancestors):
+                if n.seq == 0:
+                    continue  # skip synthetic root
                 for t in n.cited_titles:
                     if t and t not in seen_titles:
                         seen_titles.append(t)
+                history.append({"role": "user", "content": n.question})
+                # Strip claim-lattice pointer-line excerpts before
+                # passing prior answers to the model — the [E#] bindings
+                # are stale (each turn has a fresh evidence_map) and the
+                # spotlight excerpts inflate the prompt without adding
+                # antecedent value. The bare claim prose remains.
+                history.append({
+                    "role": "assistant",
+                    "content": _session_strip_pointers(n.answer_text),
+                })
             policy_local = dict(policy)
             if seen_titles:
                 policy_local["retrieval_keywords"] = " ".join(seen_titles[:8])
@@ -7851,6 +7905,7 @@ def _session_repl(args: argparse.Namespace, sess) -> int:
                     model_id=model,
                     burn_existing=bool(args.burn),
                     top_k=args.top_k,
+                    conversation_history=history or None,
                 )
             except Exception as e:
                 print(f"query failed: {e}")

arborist/qa/corpus_query.py

@@ -135,6 +135,7 @@ def run_query(
     warrant_check_enabled: bool = False,
     policy: dict | None = None,
     precomputed_hits: list | None = None,
+    conversation_history: list[dict] | None = None,
 ) -> dict:
     """End-to-end claim-lattice query over any Corpus.
 
@@ -385,11 +386,17 @@ def run_query(
         f"QUESTION: {question}\n\n"
         f"{CLAIM_LATTICE_GROUNDING_REMINDER}"
     )
-    messages = [
+    # Optional conversation_history is inserted between system and the
-        {"role": "system", "content": sys_prompt},
+    # current user turn — gives the model antecedent context for
-        {"role": "user", "content": user_prompt},
+    # pronoun questions ("who created it?" after "what is permaculture?").
-    ]
+    # The history's per-turn user/assistant content stays grounded in
-    prompt_chars = len(sys_prompt) + len(user_prompt)
+    # the CURRENT turn's evidence_map: the model is told which pointer
+    # ids resolve via the new EVIDENCE block, not the prior turns'.
+    messages = [{"role": "system", "content": sys_prompt}]
+    if conversation_history:
+        messages.extend(conversation_history)
+    messages.append({"role": "user", "content": user_prompt})
+    prompt_chars = sum(len(m["content"]) for m in messages)
 
     # 4. LLM call.
     ts = _time.time()
@@ -469,6 +476,7 @@ def run_query(
             "document_root": ev.source_root,
             "document_uri": ev.document_uri,
             "title": ev.title,
+            "chunk_idx": ev.chunk_idx,
             "source_role": ev.source_role,
             "used": ev.source_root in used_doc_roots,
             "used_pointer_ids": sorted(set(

arborist/qa/providence_query.py

@@ -62,23 +62,35 @@ _DEFAULT_CHUNKING_VERSION = "tok-512-v1"
 
 
 def _build_canonical_messages_for_hash(
-    question: str, evidence_text: str
+    question: str, evidence_text: str,
+    conversation_history: list[dict] | None = None,
 ) -> list[dict]:
     """Mirror the message shape run_query passes to chat_completion.
 
     Must stay in sync with arborist/qa/corpus_query.py:run_query —
     same system prompt, same user-payload template. The byte-identity
     test (tests/test_run_query_byte_identity.py) catches drift.
+
+    ``conversation_history``: optional list of preceding turn
+    messages (user / assistant pairs). When set, they're inserted
+    between the system prompt and the current EVIDENCE+QUESTION user
+    turn so the model has antecedent context for pronouns ("who
+    created it?" after "what is permaculture?"). Folds into
+    ``conversation_hash`` so each unique chat history gets its own
+    cache_key.
     """
     user = (
         f"EVIDENCE:\n\n{evidence_text}\n\n"
         f"QUESTION: {question}\n\n"
         f"{CLAIM_LATTICE_GROUNDING_REMINDER}"
     )
-    return [
+    msgs: list[dict] = [
         {"role": "system", "content": CLAIM_LATTICE_SYSTEM_PROMPT},
-        {"role": "user", "content": user},
     ]
+    if conversation_history:
+        msgs.extend(conversation_history)
+    msgs.append({"role": "user", "content": user})
+    return msgs
 
 
 def _compute_cache_key(
@@ -282,6 +294,7 @@ def providence_query(
     temperature: float = 0.1,
     max_tokens: int = 512,
     progress: Any = None,
+    conversation_history: list[dict] | None = None,
 ) -> dict[str, Any]:
     """Cache-aware claim-lattice query.
 
@@ -459,7 +472,10 @@ def providence_query(
     qhash = question_hash(question, mode="equivalence_class")
     mhash = model_profile_hash(model_id)
     chash = conversation_hash(
-        _build_canonical_messages_for_hash(question, evidence_repr)
+        _build_canonical_messages_for_hash(
+            question, evidence_repr,
+            conversation_history=conversation_history,
+        )
     )
     ghash = governance_policy_hash(policy)
     schema_version = policy.get("schema_version", _DEFAULT_SCHEMA_VERSION)
@@ -556,6 +572,7 @@ def providence_query(
             temperature=temperature, max_tokens=max_tokens,
             policy=policy,
             precomputed_hits=hits,
+            conversation_history=conversation_history,
         )
         progress.emit(
             "llm.done",