session: Merkle-rooted multi-turn Q&A REPL with Bates ledger
`arborist session` is an interactive multi-turn Q&A REPL where every turn (or fork) mints one node in a per-session SQLite-backed tree. Each node carries a stable Bates id (`<sid>-<6-digit>`) and folds into a Merkle subtree-hash chain; the root node's subtree_hash is the session_root. Tree shape lets: - **Forks** happen implicitly: `/cd <bates>` to a prior node, ask again → sibling under that parent. Branch points (≥2 children) surfaced by `/branches`. - **Page-refresh caching** stay cheap: a client tracking (bates → subtree_hash, body) only refetches subtrees whose hash changed. Sibling subtrees that didn't change are byte-identical → cache-equivalent. Same property git pack-protocol and IPFS MFS use. - **Audit-chain verification** be per-session and independent: each session db has its own session_audit_events with event_hash = sha256(prev_hash || canonical_body). `make session-chain-check` walks all sessions; 0 breaks each = intact. Wire: - arborist/qa/session.py — Session class, Bates minting, Merkle recompute on O(depth) insert, audit chain, helpers (list, render, resolve <bates|seq|label>). - arborist/cli.py — `session` subcommand: REPL + --list / --tree / --chain-check / --gc / --json flags. Ancestor-titles → retrieval keywords (parsed from cited-pointer lines in answer_text) flow down the branch via policy["retrieval_keywords"]. - arborist/qa/providence_query.py — honor policy["retrieval_keywords"]: augment FTS5 retrieval query without touching cache_key (mirrors legacy --retrieval-keywords discipline, #000001). - Makefile — `make session [SID=...]`, `make session-list`, `make session-tree SID=...`, `make session-chain-check`, `make session-gc SESSION_KEEP=N`. - docs/sessions.md — schema, Merkle conventions (portability for non-Python consumers), REPL command reference. - tests/test_session.py — 15 tests: create, resume, add_node, fork via cd, branches, root determinism, audit chain (intact + tampered), resolve, list, render, sibling-invariance of subtree_hash. Storage: ~/.arborist/sessions/<sid>.db (self-contained — no FK into main store). Answers live in providence_cache keyed by cache_key; session only carries conversation shape. Cache hits stay live across sessions. Bounded growth via --gc. Phase 1 scope: tree + Merkle + Bates + retrieval-keyword flow. NOT in Phase 1: LLM-side conversation_history (threading prior Q&A into the LLM prompt + conversation_hash). A bare-pronoun follow-up ("who created him?") gets the right retrieval today but the LLM may still UNGROUNDED because it sees only the new question as user message. Folding conversation_history into the prompt + cache_key's conversation_hash dimension is the natural Phase 2. 197 tests pass.
This commit is contained in:
parent
21d2d2774a
commit
508fec6975
6 changed files with 1541 additions and 9 deletions
205
docs/sessions.md
Normal file
205
docs/sessions.md
Normal file
|
|
@ -0,0 +1,205 @@
|
|||
# Sessions — Merkle-rooted conversation trees
|
||||
|
||||
Multi-turn Q&A with branching, where the tree itself is Merkle-rooted.
|
||||
Each turn (or fork) mints a node with a stable Bates id and folds into
|
||||
a per-session subtree-hash chain; the root node's `subtree_hash` is the
|
||||
`session_root`, a single hash that pins the entire conversation state.
|
||||
|
||||
## On-disk shape
|
||||
|
||||
One SQLite file per session at `~/.arborist/sessions/<sid>.db`:
|
||||
|
||||
```sql
|
||||
CREATE TABLE session_meta (
|
||||
session_id TEXT PRIMARY KEY,
|
||||
root_bates TEXT NOT NULL, -- always seq=0
|
||||
session_root TEXT NOT NULL, -- subtree_hash of root node
|
||||
current_bates TEXT NOT NULL, -- REPL "you are here" pointer
|
||||
next_seq INTEGER NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE nodes (
|
||||
bates TEXT PRIMARY KEY, -- "<sid>-<6-digit>"
|
||||
seq INTEGER NOT NULL, -- monotonic per session
|
||||
parent_bates TEXT REFERENCES nodes(bates),
|
||||
question TEXT NOT NULL,
|
||||
cache_key TEXT NOT NULL, -- → providence_cache
|
||||
audit_mode TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
label TEXT, -- optional human name
|
||||
node_hash TEXT NOT NULL, -- sha256 of canonical body
|
||||
subtree_hash TEXT NOT NULL -- Merkle hash over self + children
|
||||
);
|
||||
|
||||
CREATE TABLE session_audit_events (
|
||||
seq INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
event_type TEXT NOT NULL, -- session_init | node_added
|
||||
bates TEXT,
|
||||
prev_root TEXT NOT NULL,
|
||||
session_root TEXT NOT NULL,
|
||||
body TEXT NOT NULL, -- canonical JSON
|
||||
event_hash TEXT NOT NULL, -- sha256(prev_hash || body)
|
||||
created_at TEXT NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
The session db is **self-contained**: no foreign keys into the main
|
||||
store. Answers live in `~/.arborist/qa.db:providence_cache` keyed by
|
||||
`cache_key`; the session only carries the **shape** of the conversation
|
||||
(parent links, Bates, hashes). Cache hits across sessions stay live.
|
||||
|
||||
## Bates discipline
|
||||
|
||||
`<sid>-<6-digit-seq>`. `sid` is `YYYYMMDD-HHMMSS-<4-hex>`; `seq` is
|
||||
monotonic per session, never reassigned, never re-used. `seq=0` is a
|
||||
synthetic root with empty question — collapsed in tree rendering, kept
|
||||
so all real questions descend from a single node.
|
||||
|
||||
## Merkle hashes
|
||||
|
||||
Two derived hashes per node:
|
||||
|
||||
```
|
||||
node_hash = sha256(canonical_json({
|
||||
parent_node_hash, bates, question, cache_key,
|
||||
audit_mode, created_at, label,
|
||||
}))
|
||||
|
||||
subtree_hash = HashCombine(
|
||||
leaf_hash(node_hash),
|
||||
children_root,
|
||||
)
|
||||
```
|
||||
|
||||
Where `children_root` is built bottom-up by `HashCombine` over the
|
||||
children's `subtree_hash` values, sorted by hash (deterministic),
|
||||
with the standard arborist odd-element rule (self-duplicate).
|
||||
|
||||
Sorting children by `subtree_hash` (content-addressed) keeps the
|
||||
combine order independent of insertion timing — two sessions that
|
||||
land the same nodes in different order produce the same root.
|
||||
|
||||
## Insert path is O(depth)
|
||||
|
||||
When a new node `N` is inserted under parent `P`:
|
||||
|
||||
1. Compute `N.node_hash`.
|
||||
2. `N.subtree_hash = leaf_hash(N.node_hash)` (new node is a leaf).
|
||||
3. Walk `N → P → P.parent → … → root` recomputing each ancestor's
|
||||
`subtree_hash`.
|
||||
4. Update `session_meta.session_root` to root's new `subtree_hash`.
|
||||
5. Append one `session_audit_events` row whose `event_hash` chains
|
||||
off the previous event.
|
||||
|
||||
A sibling subtree that didn't change keeps its `subtree_hash`. This
|
||||
is the property that makes page-refresh caching cheap.
|
||||
|
||||
## Caching for page refresh
|
||||
|
||||
A UI client tracks `(bates → subtree_hash, body)` locally. On refresh:
|
||||
|
||||
1. Client sends current `session_root`.
|
||||
2. Server returns either "still current" or the new root + a list of
|
||||
`subtree_hash` deltas along the changed paths.
|
||||
3. Client walks top-down, fetching only nodes whose `subtree_hash`
|
||||
differs from cache.
|
||||
|
||||
Same shape as git's pack-protocol or IPFS MFS: content-addressed
|
||||
sibling subtrees are byte-identical → cache-equivalent.
|
||||
|
||||
## Audit chain
|
||||
|
||||
`session_audit_events` is a hash-chain mirror of `arborist.audit_events`,
|
||||
scoped per session. Every state change emits one event whose `event_hash`
|
||||
is `sha256(prev_event_hash || canonical_body)`. Verify with
|
||||
`arborist session --chain-check [--sid SID]` (or
|
||||
`make session-chain-check`).
|
||||
|
||||
`event_hash` and `session_root` are independent integrity signals:
|
||||
`session_root` answers "what does the tree look like now?";
|
||||
`event_hash` answers "is every state change in order intact?". Both
|
||||
moves on every insert, but for different reasons.
|
||||
|
||||
## Forking
|
||||
|
||||
There is no `/fork` verb. Forking is implicit: `/cd <bates>` moves the
|
||||
"current" pointer to any node in the tree; the next question becomes a
|
||||
child of that node. Branch points are nodes with ≥2 children — listed
|
||||
by `/branches` or in the tree header.
|
||||
|
||||
## Retrieval-side context flow
|
||||
|
||||
When a turn is asked under parent P, the REPL gathers the source titles
|
||||
the LLM **cited** in P and P's ancestors (parsed from each ancestor's
|
||||
stored `answer_text` for `[E# | <title> | …]` pointers) and passes them
|
||||
into `providence_query` via `policy["retrieval_keywords"]`. The keyword
|
||||
string augments FTS5 retrieval ONLY — it never enters `cache_key`,
|
||||
`question_hash`, or `governance_policy_hash` (mirrors the discipline of
|
||||
the legacy `--retrieval-keywords` flag, see ticket #000001).
|
||||
|
||||
This makes pronoun follow-ups ("who created him?") still surface the
|
||||
prior turn's primary source. **It does not solve the LLM-side antecedent
|
||||
problem** — the model receives only the new question as the user
|
||||
message, so a bare pronoun question may still fail to ground. Threading
|
||||
prior Q&A pairs as `conversation_history` into the LLM prompt (+ folding
|
||||
that history into `conversation_hash`) is a future enhancement.
|
||||
|
||||
## CLI
|
||||
|
||||
```
|
||||
arborist session # new session, REPL
|
||||
arborist session --sid <id> # resume
|
||||
arborist session --list [--json] # list sessions
|
||||
arborist session --tree --sid <id> # print tree
|
||||
arborist session --chain-check # verify audit chain (all)
|
||||
arborist session --gc N # keep N most recent
|
||||
```
|
||||
|
||||
REPL commands:
|
||||
|
||||
| command | effect |
|
||||
|---|---|
|
||||
| `/tree` | print the tree |
|
||||
| `/branches` | list branch points (Bates with ≥2 children) |
|
||||
| `/cd <bates\|seq\|label>` | move current pointer |
|
||||
| `/back` | cd to parent of current |
|
||||
| `/label <name>` | name current node |
|
||||
| `/show <ref>` | dump stored info for a node |
|
||||
| `/sid` | print session id |
|
||||
| `/root` | print session_root |
|
||||
| `/help` | help |
|
||||
| `/quit` | exit (or Ctrl-D) |
|
||||
| anything else | new question, child of current |
|
||||
|
||||
Make targets:
|
||||
|
||||
```
|
||||
make session [SID=...] # interactive REPL
|
||||
make session-list # list all
|
||||
make session-tree SID=... # print one tree
|
||||
make session-chain-check # verify audit chain
|
||||
make session-gc # keep SESSION_KEEP=100 (default) most recent
|
||||
```
|
||||
|
||||
## Bounded growth
|
||||
|
||||
`make session-gc` (or `arborist session --gc N`) keeps the N most-recent
|
||||
sessions, deletes the rest. Each session is a single SQLite file →
|
||||
trivially deletable, shareable, archivable.
|
||||
|
||||
## Portability
|
||||
|
||||
The on-disk schema has no FK dependency on the main arborist store.
|
||||
A non-Python consumer that wants to read a session db needs only:
|
||||
|
||||
1. SQLite3 with FTS5 (not required — sessions don't use FTS).
|
||||
2. The canonical-JSON convention: sorted keys, no whitespace, UTF-8.
|
||||
3. Three hash primitives:
|
||||
- `sha256(bytes) → hex`
|
||||
- `leaf_hash(bytes) = sha256(0x00 || bytes)`
|
||||
- `hash_combine(L, R) = sha256(0x03 || L || R)`
|
||||
|
||||
Those are the same conventions as the proxy.unturf.com Go merkle and
|
||||
the rest of arborist (see `arborist/merkle.py`).
|
||||
Loading…
Add table
Add a link
Reference in a new issue