engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

session: Merkle-rooted multi-turn Q&A REPL with Bates ledger

Browse source 
`arborist session` is an interactive multi-turn Q&A REPL where every
turn (or fork) mints one node in a per-session SQLite-backed tree.
Each node carries a stable Bates id (`<sid>-<6-digit>`) and folds into
a Merkle subtree-hash chain; the root node's subtree_hash is the
session_root.

Tree shape lets:

- **Forks** happen implicitly: `/cd <bates>` to a prior node, ask
  again → sibling under that parent. Branch points (≥2 children)
  surfaced by `/branches`.
- **Page-refresh caching** stay cheap: a client tracking
  (bates → subtree_hash, body) only refetches subtrees whose hash
  changed. Sibling subtrees that didn't change are byte-identical
  → cache-equivalent. Same property git pack-protocol and IPFS MFS
  use.
- **Audit-chain verification** be per-session and independent: each
  session db has its own session_audit_events with event_hash =
  sha256(prev_hash || canonical_body). `make session-chain-check`
  walks all sessions; 0 breaks each = intact.

Wire:

- arborist/qa/session.py — Session class, Bates minting, Merkle
  recompute on O(depth) insert, audit chain, helpers (list, render,
  resolve <bates|seq|label>).
- arborist/cli.py — `session` subcommand: REPL + --list / --tree
  / --chain-check / --gc / --json flags. Ancestor-titles → retrieval
  keywords (parsed from cited-pointer lines in answer_text) flow down
  the branch via policy["retrieval_keywords"].
- arborist/qa/providence_query.py — honor policy["retrieval_keywords"]:
  augment FTS5 retrieval query without touching cache_key (mirrors
  legacy --retrieval-keywords discipline, #000001).
- Makefile — `make session [SID=...]`, `make session-list`,
  `make session-tree SID=...`, `make session-chain-check`,
  `make session-gc SESSION_KEEP=N`.
- docs/sessions.md — schema, Merkle conventions (portability for
  non-Python consumers), REPL command reference.
- tests/test_session.py — 15 tests: create, resume, add_node, fork
  via cd, branches, root determinism, audit chain (intact + tampered),
  resolve, list, render, sibling-invariance of subtree_hash.

Storage: ~/.arborist/sessions/<sid>.db (self-contained — no FK into
main store). Answers live in providence_cache keyed by cache_key;
session only carries conversation shape. Cache hits stay live across
sessions. Bounded growth via --gc.

Phase 1 scope: tree + Merkle + Bates + retrieval-keyword flow.
NOT in Phase 1: LLM-side conversation_history (threading prior Q&A
into the LLM prompt + conversation_hash). A bare-pronoun follow-up
("who created him?") gets the right retrieval today but the LLM may
still UNGROUNDED because it sees only the new question as user
message. Folding conversation_history into the prompt + cache_key's
conversation_hash dimension is the natural Phase 2.

197 tests pass.
This commit is contained in:
russell@unturf.com 2026-06-01 16:58:08 -04:00
parent 21d2d2774a
commit 508fec6975
No known key found for this signature in database
6 changed files with 1541 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

24
Makefile
View file

@ -207,6 +207,30 @@ query: bootstrap ## ask the corpus a question [JSON=1 BURN=1 LLM=qwen|hermes REP
$(if $(LLM_ENDPOINT),@echo "# llm: $(LLM_ENDPOINT) / $(LLM_MODEL)" >&2,)
$(ARBORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) $(if $(LLM_ENDPOINT),--endpoint $(LLM_ENDPOINT) --model $(LLM_MODEL),) $(if $(JSON),--json,) $(if $(BURN),--burn,) $(if $(REPAIR),--repair,) $(if $(REPROMPTS),--repair-reprompts $(REPROMPTS),) $(if $(ANSWER_MODE),--answer-mode $(ANSWER_MODE),) --user-payload-layout $(LAYOUT) $(if $(K),--retrieval-keywords "$(K)",) $(if $(BROAD),--apply-quantifier-caps,) $(if $(REJECT_BROAD),--reject-broad,) $(if $(ALLOW_BROAD),--allow-broad,) $(if $(WITNESS),--witness,) $(if $(XLANG),--crosslang-guard,) $(if $(XLANG_MT),--crosslang-translate,) "$(Q)"
# Merkle-rooted multi-turn Q&A REPL. Each turn (or fork) mints one
# tree node with a Bates id + a session-wide subtree_hash. Forking is
# implicit: /cd to a parent node and ask again → sibling under that
# parent. See arborist/qa/session.py + docs/sessions.md.
session: bootstrap ## interactive multi-turn Q&A REPL [SID=... LLM=qwen|hermes ANSWER_MODE=... BURN=1]
$(if $(LLM_ENDPOINT),@echo "# llm: $(LLM_ENDPOINT) / $(LLM_MODEL)" >&2,)
$(ARBORIST) --shards-dir $(SHARDS_DIR) session $(if $(SID),--sid $(SID),) --top-k $(QUERY_TOP_K) $(if $(LLM_ENDPOINT),--endpoint $(LLM_ENDPOINT) --model $(LLM_MODEL),) $(if $(ANSWER_MODE),--answer-mode $(ANSWER_MODE),) $(if $(BURN),--burn,)
session-list: bootstrap ## list all session dbs in ~/.arborist/sessions/
$(ARBORIST) session --list $(if $(JSON),--json,)
session-tree: bootstrap ## print one session's tree [SID=...]
@if [ -z "$$SID" ] && [ -z "$(SID)" ]; then \
echo "usage: make session-tree SID=<session-id> [JSON=1]"; exit 2; \
fi
$(ARBORIST) session --sid $(SID) --tree $(if $(JSON),--json,)
session-chain-check: bootstrap ## verify session audit-chain integrity [SID=... (default: all)]
$(ARBORIST) session --chain-check $(if $(SID),--sid $(SID),)
SESSION_KEEP ?= 100
session-gc: bootstrap ## delete old session dbs, keep $(SESSION_KEEP) most recent [SESSION_KEEP=N]
$(ARBORIST) session --gc $(SESSION_KEEP)
query-dry: bootstrap ## like 'make query' but skip the LLM call (dry-run) [JSON=1 BURN=1 ANSWER_MODE=... BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 XLANG=1 XLANG_MT=1]
@if [ -z "$$Q" ] && [ -z "$(Q)" ]; then \
echo "usage: make query-dry Q=\"your question\" [JSON=1 BURN=1 ANSWER_MODE=claim_lattice|claim_lattice_pointer|quote BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 XLANG=1 XLANG_MT=1]"; exit 2; \