arborist/docs/mesh.md

Mesh — multiplayer arborist

Arborist on a single laptop is a content-addressed forest of documents. Arborist across many laptops is a gossip-able forest. Two peers that ingest the same Wikipedia dump compute bit-identical document_roots, so any peer can verify another peer's claim by re-deriving the hash locally. Mesh is wire-and-trust scaffolding wrapped around that fact.

This file documents how groups manage data across each other. It pairs with diagrams in docs/diagrams/.

Status: cryptographic foundation, state machine, and CLI ship today. HTTP wire (mesh sync, mesh serve) is the next milestone. Protocol contract below pins what wire will implement.

Default off

No code path touches a network unless a peer flips meta.mesh.enabled = 1. Lookups, ingests, distillations, and Q&A run identically with or without mesh. Initialization is two commands and is reversible.

arborist mesh init --group myteam   # mint Ed25519 + X25519 keys; create epoch 0
arborist mesh enable                # flip gating flag on
arborist mesh status                # always-safe inspection
arborist mesh disable               # flag back off; keys + history stay on disk

Identity stack

Each peer carries:

• one Ed25519 keypair for signing membership ops and gossip envelopes
• one X25519 keypair for ECDH key agreement (used to wrap epoch secrets)
• one 8-hex member_id (random by default; pinnable via --member-id)

Group state, replicated across every peer that participates:

• mesh_roster — per-epoch (member_id, sign_pub, dh_pub, role) tuples
• mesh_epochs — per-epoch lifecycle (started_at, audit linkage, secret envelope)
• audit_events — append-only Merkle chain of every state-changing op

Public keys are shared. Private keys never leave a peer's SQLite file.

Epochs — a counter that bumps on every roster change

init_identity() writes epoch 0 with the founder as sole admin. Every subsequent add, kick, or rotate writes a new epoch row, generates a fresh 32-byte symmetric secret, and wraps that secret to every member of a new roster.

Old epochs are not deleted. Their rosters and audit events stay on disk so that historical signatures remain verifiable by anyone, forever. Cores never evict; mesh state never evicts either.

Per-member secret envelope

A single epoch carries one symmetric secret. That one secret is wrapped N times, once per member, by ECDH between a rotator's X25519 priv and each member's X25519 pub, then sealed with ChaCha20-Poly1305 using member_id as AAD.

Decryption is local: a peer derives the same shared secret by ECDH between their own X25519 priv and a rotator's X25519 pub, then unwraps their own slot. An evicted peer simply has no slot, so unwrap raises ValueError — they are opaque to gossip from epoch+1 onward by design.

Code: arborist/mesh/state.py::_wrap_secret_for_members and unwrap_secret_for_self.

What flows between peers

Arborist is content-addressed, so identifiers are short and bodies are optional. A typical sync round looks like:

1. alice signs and sends ANNOUNCE_ROOT(document_root, source_uri, chunking_version, schema_version) for everything new since last sync. Same for derivations, providence_cache rows, and falsifications.
2. bob verifies alice's signature against her epoch-N sign_pub, then checks his local store. Anything he already has is dedup'd by document_root and dropped on the floor.
3. On miss, bob sends REQUEST_BODY(root). alice replies with DELIVER_BODY(bytes, merkle_proof). bob verifies the proof against the announced root before inserting.
4. Audit chains merge by prev_event_hash. A gossip insert that does not extend a chain consistently is rejected — mesh never silently forks history.

The five gossip message types:

message	meaning	encrypted?	signed?
ANNOUNCE_ROOT	"I have these document_roots"	optional	always
ANNOUNCE_DERIVATION	"core C derives from surfaces S₁..Sₙ via distiller D"	optional	always
ANNOUNCE_PROVIDENCE	"I cached an answer at this 8-dim cache_key with this audit_mode"	optional	always
ANNOUNCE_FALSIFICATION	"this cache_key is wrong (witness signature attached)"	optional	always
REQUEST_BODY / DELIVER_BODY	pull bytes on local miss; recipient verifies Merkle proof before insert	optional	always

Soft hashes (embeddings, TF-IDF scores) never enter wire. Only hard SHA-256 commitments cross the network. Per CLAUDE.md's soft/hard hash split.

Group operations — operator decision tree

A group has two roles: admin and member. Admins may add and kick. Any current member may rotate the secret on the same roster. Self-kick is rejected; an admin who wants out runs mesh disable instead.

Add a member (admin only)

# bob mints his keys and shares pubs (out-of-band: signal, in person, signed file).
arborist mesh init --group myteam --member-id bob   # bob's machine

# alice (admin) enrolls bob at her machine:
arborist mesh add --member-id bob \
    --sign-pub <bob_sign_pub_hex> \
    --dh-pub   <bob_dh_pub_hex>
# epoch bumps. fresh secret wrapped to alice + bob.

Bob can decrypt epoch+1 forward. Bob cannot reach prior epochs — by design, joining doesn't grant retroactive access.

Kick a member (admin only)

arborist mesh kick --member-id dave --reason "left team 2026-04-28"

Epoch bumps. New secret wrapped to everyone except dave. Dave's prior signatures stay verifiable forever (his roster row at older epochs is preserved). Any AEAD-protected gossip from epoch+1 onward is opaque to him.

Cannot leave a roster with no admins. Promote first, then kick.

Rotate the secret (any member)

arborist mesh rotate --reason "scheduled monthly hygiene"

Same roster, fresh secret. Use on suspected secret leak when no identifiable bad actor exists, or as periodic key hygiene.

When to use which

situation	command
starting a new federation	mesh init then mesh enable
onboarding a teammate	mesh add (admin)
someone left the team	mesh kick (admin)
rumor of secret compromise but everyone's keys still trusted	mesh rotate
stepping away yourself	mesh disable (own peer only)

Forward-secrecy guarantees in plain language

• Joining is non-retroactive. A new member cannot decrypt anything from before they joined. Wrapping per-epoch makes this automatic.
• Eviction is forward-only. A kicked member keeps everything they already had. They lose access to gossip from the next epoch on.
• Past signatures remain verifiable forever. Roster history is immutable on disk, so anyone can audit who said what at what epoch even after rotations.
• Rotation costs O(N). One ECDH + one AEAD per member per rotate. Linear in roster size, not document count.

Trust boundary, threat model

Mesh defends against:

• Roster impersonation. Every membership op is Ed25519-signed by an admin and chained via audit_events.event_hash = sha256(prev || body). A peer cannot fake a mesh_epoch_rotate without the admin's signing key.
• Gossip tampering. Every announcement is signed; recipients verify before insert. A peer that doesn't carry an epoch's secret cannot produce a valid AEAD payload for that epoch.
• Eviction bypass. Evicted members have no slot in the post-rotation envelope. They cannot derive the new secret from any prior secret.
• Forking. Audit chain prev_event_hash linkage rejects gossip inserts that do not extend a chain consistently.

Mesh does not defend against:

• Out-of-band key distribution. A new member's pubs must reach the enrolling admin via a trusted channel (signal, in person, signed file). Mesh has no built-in introduction protocol.
• Compromise of an admin's signing key. With that key an attacker can forge any membership op. Mitigation: rotate admins regularly, keep admin count small.
• Traffic analysis. Wire layer ships TLS but does not pad or mix.

Related code

• arborist/mesh/__init__.py — module entrypoint, public API
• arborist/mesh/crypto.py — Ed25519 + X25519 + ChaCha20-Poly1305 wrappers
• arborist/mesh/state.py — DB writes + audit chain hooks; epoch rotation
• arborist/mesh/members.py — add_member, kick_member, scheduled_rotate
• arborist/cli.py — _cmd_mesh_* argparse handlers
• arborist/store.py — mesh_identity, mesh_roster, mesh_epochs schema

Related docs

• README.md — top-level overview, ingest flow, single-peer Q&A
• CLAUDE.md — schema invariants, conventions, audit chain rules
• ~/git/proxy.unturf.com/pkg/verified/merkle.go — fox's Go merkle reference. Arborist's Python port mirrors the conventions.

Rendering the diagrams

make docs    # renders docs/diagrams/*.dot to .png via graphviz

Or directly: dot -Tpng docs/diagrams/<file>.dot -o /tmp/out.png.