arborist/docs/sessions.md

Sessions — single-shard Merkle conversation forest

Multi-turn Q&A with branching. One SQLite shard at ~/.arborist/sessions.db holds every turn from every session a user has ever started. Sessions are viewports onto a shared forest — nodes can fork across sessions (yesterday's turn → today's child) via a single parent_bates pointer.

Privacy: this layer is local-private for now. Gossip-trust sharing across users is future work; the shard's contract today is "private journal of everything the operator asked, plus the answers."

On-disk shape

CREATE TABLE sessions (
    sid             TEXT PRIMARY KEY,
    root_bates      TEXT NOT NULL,        -- always seq=0, empty question
    current_bates   TEXT NOT NULL,        -- REPL "you are here"
    next_seq        INTEGER NOT NULL,
    created_at      TEXT NOT NULL,
    updated_at      TEXT NOT NULL,
    label           TEXT
);

CREATE TABLE nodes (
    bates              TEXT PRIMARY KEY,  -- "<sid>-<6-digit>"
    sid                TEXT NOT NULL,     -- session that minted this node
    seq                INTEGER NOT NULL,
    parent_bates       TEXT REFERENCES nodes(bates),  -- can cross sids
    question           TEXT NOT NULL,
    answer_text        TEXT NOT NULL,     -- materialized at insert time
    cache_key          TEXT NOT NULL,     -- → providence_cache
    audit_mode         TEXT NOT NULL,
    cited_titles_json  TEXT NOT NULL,     -- parsed from [E# | <title> | …]
    n_cited_sources    INTEGER NOT NULL,
    created_at         TEXT NOT NULL,
    label              TEXT,
    node_hash          TEXT NOT NULL,     -- content-addressed
    subtree_hash       TEXT NOT NULL      -- Merkle over self + all descendants
);

CREATE VIRTUAL TABLE nodes_fts USING fts5(   -- search across all sessions
    question, answer_text, cited_titles
);

CREATE TABLE audit_events (                  -- one global hash chain
    seq             INTEGER PRIMARY KEY AUTOINCREMENT,
    event_type      TEXT NOT NULL,
    sid             TEXT,
    bates           TEXT,
    prev_event_hash TEXT,
    event_hash      TEXT NOT NULL,           -- sha256(prev || canonical body)
    body            TEXT NOT NULL,
    created_at      TEXT NOT NULL
);

Three integrity surfaces:

• node_hash — content-addressed, never changes after insert.
• subtree_hash — recomputed when a node gains a child (anywhere in the forest); ripples up the parent chain regardless of session boundaries.
• audit_events.event_hash — single global chain; tampering with any state-change record breaks it.

Bates discipline

<sid>-<6-digit-seq>. sid is timestamp+random, never reused; seq is per-sid monotonic, never reassigned. seq=0 is a synthetic root with empty question (collapsed in /tree rendering, kept so every real turn has a parent).

Merkle properties

node_hash = sha256(canonical_json({
    parent_node_hash, bates, question, cache_key,
    audit_mode, created_at, label,
}))

subtree_hash = HashCombine(
    leaf_hash(node_hash),
    HashCombine over sorted-by-hash children_subtree_hashes,
)

Children sorted by subtree_hash (content-addressed) keep the combine order deterministic — two stores that land the same nodes in different order produce the same root.

Insert path is O(depth). On insert of N under P:

1. Compute N.node_hash.
2. N.subtree_hash = leaf_hash(N.node_hash).
3. Walk N → P → … → root (crossing sessions if parent_bates does) recomputing each ancestor's subtree_hash.
4. Append audit_events row.

Sibling subtrees that didn't change keep their subtree_hash. That's the property page-refresh caching exploits — a UI client tracks (bates → subtree_hash, body) and only refetches paths whose hash differs.

Cross-session forks

Forking is implicit. Any turn from any session is /find-able and /cd-able. After /cd <bates>, the next non-command question lands as a child of that node. If the node belongs to a different session, the new child inherits the original parent's node_hash via Merkle — that session's subtree_hash ripples up to include the cross-session child.

In /tree rendering, cross-session children are tagged [from <sid>]. (With local-only storage that's always self today; gossip-shared sessions will surface external sids the same way.)

CLI

arborist session                          # new session, REPL
arborist session --sid <id>               # resume
arborist session --list [--json]          # all sessions
arborist session --tree --sid <id>        # one session's tree
arborist session --find "spider man"      # FTS5 across ALL sessions
arborist session --chain-check            # verify global audit chain
arborist session --sessions-db PATH       # override shard location

REPL commands:

command	effect
/tree	print this session's tree
/branches	list branch-point Bates (≥2 children)
/find <query>	FTS5 search across ALL sessions
/cd <bates|seq|label>	move current pointer (cross-session OK)
/back	cd to parent of current
/label <name>	name current node
/show [ref]	dump stored info for a node
/sid	print session id
/root	print session_root
/help	help
/quit	exit (or Ctrl-D)
anything else	new question, child of current

Makefile:

make session [SID=...]      # interactive REPL (defaults LLM=qwen)
make session LLM=hermes     # use Hermes-3-8B instead
make session-list           # list sessions
make session-tree SID=...   # one tree
make session-find Q="..."   # search [LIMIT=N JSON=1]
make session-chain-check    # verify audit chain

Retrieval flow down a branch

When you ask a follow-up turn under parent P, the REPL gathers cited titles from P and all ancestors (already materialized in nodes.cited_titles_json at insert time) and passes them to providence_query via policy["retrieval_keywords"]. The keyword string augments FTS5 retrieval only — it never enters cache_key / question_hash / governance_policy_hash. Same discipline as the legacy --retrieval-keywords flag (#000001).

This makes pronoun follow-ups ("who created him?") surface the prior turn's primary source. It does not thread prior Q&A into the LLM prompt — that's a separate enhancement (would fold into conversation_hash).

Portability

The shard's schema has no FK dependency on the main arborist store. A non-Python consumer reading it needs:

1. SQLite3 with FTS5 (for the search index).
2. The canonical-JSON convention: sorted keys, no whitespace, UTF-8.
3. Three hash primitives:
   • sha256(bytes) → hex
   • leaf_hash(bytes) = sha256(0x00 || bytes)
   • hash_combine(L, R) = sha256(0x03 || L || R)

Same conventions as proxy.unturf.com Go merkle and the rest of arborist (see arborist/merkle.py).