arborist/docs/sessions.md

# Sessions — single-shard Merkle conversation forest

Multi-turn Q&A with branching. **One** SQLite shard at
`~/.arborist/sessions.db` holds every turn from every session a user
has ever started. Sessions are *viewports* onto a shared forest —
nodes can fork across sessions (yesterday's turn → today's child) via
a single `parent_bates` pointer.

Privacy: this layer is **local-private** for now. Gossip-trust
sharing across users is future work; the shard's contract today is
"private journal of everything the operator asked, plus the answers."

## On-disk shape

```sql
CREATE TABLE sessions (
    sid             TEXT PRIMARY KEY,
    root_bates      TEXT NOT NULL,        -- always seq=0, empty question
    current_bates   TEXT NOT NULL,        -- REPL "you are here"
    next_seq        INTEGER NOT NULL,
    created_at      TEXT NOT NULL,
    updated_at      TEXT NOT NULL,
    label           TEXT
);

CREATE TABLE nodes (
    bates              TEXT PRIMARY KEY,  -- "<sid>-<6-digit>"
    sid                TEXT NOT NULL,     -- session that minted this node
    seq                INTEGER NOT NULL,
    parent_bates       TEXT REFERENCES nodes(bates),  -- can cross sids
    question           TEXT NOT NULL,
    answer_text        TEXT NOT NULL,     -- materialized at insert time
    cache_key          TEXT NOT NULL,     -- → providence_cache
    audit_mode         TEXT NOT NULL,
    cited_titles_json  TEXT NOT NULL,     -- parsed from [E# | <title> | …]
    n_cited_sources    INTEGER NOT NULL,
    created_at         TEXT NOT NULL,
    label              TEXT,
    node_hash          TEXT NOT NULL,     -- content-addressed
    subtree_hash       TEXT NOT NULL      -- Merkle over self + all descendants
);

CREATE VIRTUAL TABLE nodes_fts USING fts5(   -- search across all sessions
    question, answer_text, cited_titles
);

CREATE TABLE audit_events (                  -- one global hash chain
    seq             INTEGER PRIMARY KEY AUTOINCREMENT,
    event_type      TEXT NOT NULL,
    sid             TEXT,
    bates           TEXT,
    prev_event_hash TEXT,
    event_hash      TEXT NOT NULL,           -- sha256(prev || canonical body)
    body            TEXT NOT NULL,
    created_at      TEXT NOT NULL
);
```

Three integrity surfaces:
- **`node_hash`** — content-addressed, never changes after insert.
- **`subtree_hash`** — recomputed when a node gains a child (anywhere
  in the forest); ripples up the parent chain regardless of session
  boundaries.
- **`audit_events.event_hash`** — single global chain; tampering with
  any state-change record breaks it.

## Bates discipline

`<sid>-<6-digit-seq>`. `sid` is timestamp+random, never reused; `seq`
is per-sid monotonic, never reassigned. seq=0 is a synthetic root with
empty question (collapsed in `/tree` rendering, kept so every real
turn has a parent).

## Merkle properties

```
node_hash = sha256(canonical_json({
    parent_node_hash, bates, question, cache_key,
    audit_mode, created_at, label,
}))

subtree_hash = HashCombine(
    leaf_hash(node_hash),
    HashCombine over sorted-by-hash children_subtree_hashes,
)
```

Children sorted by `subtree_hash` (content-addressed) keep the combine
order deterministic — two stores that land the same nodes in different
order produce the same root.

Insert path is O(depth). On insert of N under P:
1. Compute `N.node_hash`.
2. `N.subtree_hash = leaf_hash(N.node_hash)`.
3. Walk `N → P → … → root` (crossing sessions if parent_bates does)
   recomputing each ancestor's `subtree_hash`.
4. Append `audit_events` row.

Sibling subtrees that didn't change keep their `subtree_hash`. That's
the property page-refresh caching exploits — a UI client tracks
`(bates → subtree_hash, body)` and only refetches paths whose hash
differs.

## Cross-session forks

Forking is implicit. Any turn from any session is `/find`-able and
`/cd`-able. After `/cd <bates>`, the next non-command question lands
as a child of that node. If the node belongs to a different session,
the new child inherits the original parent's `node_hash` via Merkle —
that session's `subtree_hash` ripples up to include the cross-session
child.

In `/tree` rendering, cross-session children are tagged
`[from <sid>]`. (With local-only storage that's always self today;
gossip-shared sessions will surface external sids the same way.)

## CLI

```
arborist session                          # new session, REPL
arborist session --sid <id>               # resume
arborist session --list [--json]          # all sessions
arborist session --tree --sid <id>        # one session's tree
arborist session --find "spider man"      # FTS5 across ALL sessions
arborist session --chain-check            # verify global audit chain
arborist session --sessions-db PATH       # override shard location
```

REPL commands:

| command | effect |
|---|---|
| `/tree` | print this session's tree |
| `/branches` | list branch-point Bates (≥2 children) |
| `/find <query>` | FTS5 search across ALL sessions |
| `/cd <bates\|seq\|label>` | move current pointer (cross-session OK) |
| `/back` | cd to parent of current |
| `/label <name>` | name current node |
| `/show [ref]` | dump stored info for a node |
| `/sid` | print session id |
| `/root` | print session_root |
| `/help` | help |
| `/quit` | exit (or Ctrl-D) |
| anything else | new question, child of current |

Makefile:

```
make session [SID=...]      # interactive REPL (defaults LLM=qwen)
make session LLM=hermes     # use Hermes-3-8B instead
make session-list           # list sessions
make session-tree SID=...   # one tree
make session-find Q="..."   # search [LIMIT=N JSON=1]
make session-chain-check    # verify audit chain
```

## Retrieval flow down a branch

When you ask a follow-up turn under parent P, the REPL gathers cited
titles from P and all ancestors (already materialized in
`nodes.cited_titles_json` at insert time) and passes them to
`providence_query` via `policy["retrieval_keywords"]`. The keyword
string augments FTS5 retrieval **only** — it never enters
`cache_key` / `question_hash` / `governance_policy_hash`. Same
discipline as the legacy `--retrieval-keywords` flag (#000001).

This makes pronoun follow-ups ("who created him?") surface the prior
turn's primary source. It does **not** thread prior Q&A into the LLM
prompt — that's a separate enhancement (would fold into
`conversation_hash`).

## Portability

The shard's schema has no FK dependency on the main arborist store.
A non-Python consumer reading it needs:

1. SQLite3 with FTS5 (for the search index).
2. The canonical-JSON convention: sorted keys, no whitespace, UTF-8.
3. Three hash primitives:
   - `sha256(bytes) → hex`
   - `leaf_hash(bytes) = sha256(0x00 || bytes)`
   - `hash_combine(L, R) = sha256(0x03 || L || R)`

Same conventions as `proxy.unturf.com` Go merkle and the rest of
arborist (see `arborist/merkle.py`).