arborist/docs/sessions.md

Sessions — Merkle-rooted conversation trees

Multi-turn Q&A with branching, where the tree itself is Merkle-rooted. Each turn (or fork) mints a node with a stable Bates id and folds into a per-session subtree-hash chain; the root node's subtree_hash is the session_root, a single hash that pins the entire conversation state.

On-disk shape

One SQLite file per session at ~/.arborist/sessions/<sid>.db:

CREATE TABLE session_meta (
    session_id      TEXT PRIMARY KEY,
    root_bates      TEXT NOT NULL,        -- always seq=0
    session_root    TEXT NOT NULL,        -- subtree_hash of root node
    current_bates   TEXT NOT NULL,        -- REPL "you are here" pointer
    next_seq        INTEGER NOT NULL,
    created_at      TEXT NOT NULL,
    updated_at      TEXT NOT NULL
);

CREATE TABLE nodes (
    bates           TEXT PRIMARY KEY,     -- "<sid>-<6-digit>"
    seq             INTEGER NOT NULL,     -- monotonic per session
    parent_bates    TEXT REFERENCES nodes(bates),
    question        TEXT NOT NULL,
    cache_key       TEXT NOT NULL,        -- → providence_cache
    audit_mode      TEXT NOT NULL,
    created_at      TEXT NOT NULL,
    label           TEXT,                 -- optional human name
    node_hash       TEXT NOT NULL,        -- sha256 of canonical body
    subtree_hash    TEXT NOT NULL         -- Merkle hash over self + children
);

CREATE TABLE session_audit_events (
    seq             INTEGER PRIMARY KEY AUTOINCREMENT,
    event_type      TEXT NOT NULL,        -- session_init | node_added
    bates           TEXT,
    prev_root       TEXT NOT NULL,
    session_root    TEXT NOT NULL,
    body            TEXT NOT NULL,        -- canonical JSON
    event_hash      TEXT NOT NULL,        -- sha256(prev_hash || body)
    created_at      TEXT NOT NULL
);

The session db is self-contained: no foreign keys into the main store. Answers live in ~/.arborist/qa.db:providence_cache keyed by cache_key; the session only carries the shape of the conversation (parent links, Bates, hashes). Cache hits across sessions stay live.

Bates discipline

<sid>-<6-digit-seq>. sid is YYYYMMDD-HHMMSS-<4-hex>; seq is monotonic per session, never reassigned, never re-used. seq=0 is a synthetic root with empty question — collapsed in tree rendering, kept so all real questions descend from a single node.

Merkle hashes

Two derived hashes per node:

node_hash = sha256(canonical_json({
    parent_node_hash, bates, question, cache_key,
    audit_mode, created_at, label,
}))

subtree_hash = HashCombine(
    leaf_hash(node_hash),
    children_root,
)

Where children_root is built bottom-up by HashCombine over the children's subtree_hash values, sorted by hash (deterministic), with the standard arborist odd-element rule (self-duplicate).

Sorting children by subtree_hash (content-addressed) keeps the combine order independent of insertion timing — two sessions that land the same nodes in different order produce the same root.

Insert path is O(depth)

When a new node N is inserted under parent P:

1. Compute N.node_hash.
2. N.subtree_hash = leaf_hash(N.node_hash) (new node is a leaf).
3. Walk N → P → P.parent → … → root recomputing each ancestor's subtree_hash.
4. Update session_meta.session_root to root's new subtree_hash.
5. Append one session_audit_events row whose event_hash chains off the previous event.

A sibling subtree that didn't change keeps its subtree_hash. This is the property that makes page-refresh caching cheap.

Caching for page refresh

A UI client tracks (bates → subtree_hash, body) locally. On refresh:

1. Client sends current session_root.
2. Server returns either "still current" or the new root + a list of subtree_hash deltas along the changed paths.
3. Client walks top-down, fetching only nodes whose subtree_hash differs from cache.

Same shape as git's pack-protocol or IPFS MFS: content-addressed sibling subtrees are byte-identical → cache-equivalent.

Audit chain

session_audit_events is a hash-chain mirror of arborist.audit_events, scoped per session. Every state change emits one event whose event_hash is sha256(prev_event_hash || canonical_body). Verify with arborist session --chain-check [--sid SID] (or make session-chain-check).

event_hash and session_root are independent integrity signals: session_root answers "what does the tree look like now?"; event_hash answers "is every state change in order intact?". Both moves on every insert, but for different reasons.

Forking

There is no /fork verb. Forking is implicit: /cd <bates> moves the "current" pointer to any node in the tree; the next question becomes a child of that node. Branch points are nodes with ≥2 children — listed by /branches or in the tree header.

Retrieval-side context flow

When a turn is asked under parent P, the REPL gathers the source titles the LLM cited in P and P's ancestors (parsed from each ancestor's stored answer_text for [E# | <title> | …] pointers) and passes them into providence_query via policy["retrieval_keywords"]. The keyword string augments FTS5 retrieval ONLY — it never enters cache_key, question_hash, or governance_policy_hash (mirrors the discipline of the legacy --retrieval-keywords flag, see ticket #000001).

This makes pronoun follow-ups ("who created him?") still surface the prior turn's primary source. It does not solve the LLM-side antecedent problem — the model receives only the new question as the user message, so a bare pronoun question may still fail to ground. Threading prior Q&A pairs as conversation_history into the LLM prompt (+ folding that history into conversation_hash) is a future enhancement.

CLI

arborist session                        # new session, REPL
arborist session --sid <id>             # resume
arborist session --list [--json]        # list sessions
arborist session --tree --sid <id>      # print tree
arborist session --chain-check          # verify audit chain (all)
arborist session --gc N                 # keep N most recent

REPL commands:

command	effect
/tree	print the tree
/branches	list branch points (Bates with ≥2 children)
/cd <bates|seq|label>	move current pointer
/back	cd to parent of current
/label <name>	name current node
/show <ref>	dump stored info for a node
/sid	print session id
/root	print session_root
/help	help
/quit	exit (or Ctrl-D)
anything else	new question, child of current

Make targets:

make session [SID=...]      # interactive REPL
make session-list           # list all
make session-tree SID=...   # print one tree
make session-chain-check    # verify audit chain
make session-gc             # keep SESSION_KEEP=100 (default) most recent

Bounded growth

make session-gc (or arborist session --gc N) keeps the N most-recent sessions, deletes the rest. Each session is a single SQLite file → trivially deletable, shareable, archivable.

Portability

The on-disk schema has no FK dependency on the main arborist store. A non-Python consumer that wants to read a session db needs only:

1. SQLite3 with FTS5 (not required — sessions don't use FTS).
2. The canonical-JSON convention: sorted keys, no whitespace, UTF-8.
3. Three hash primitives:
   • sha256(bytes) → hex
   • leaf_hash(bytes) = sha256(0x00 || bytes)
   • hash_combine(L, R) = sha256(0x03 || L || R)

Those are the same conventions as the proxy.unturf.com Go merkle and the rest of arborist (see arborist/merkle.py).