arborist/docs/mesh.md

# Mesh — multiplayer arborist

Arborist on a single laptop is a content-addressed forest of documents.
Arborist across many laptops is a *gossip-able* forest. Two peers that
ingest the same Wikipedia dump compute bit-identical `document_root`s,
so any peer can verify another peer's claim by re-deriving the hash
locally. Mesh is wire-and-trust scaffolding wrapped around that fact.

This file documents how groups manage data across each other. It pairs
with diagrams in `docs/diagrams/`.

> Status: cryptographic foundation, state machine, and CLI ship today.
> HTTP wire (`mesh sync`, `mesh serve`) is the next milestone. Protocol
> contract below pins what wire will implement.

## Default off

No code path touches a network unless a peer flips `meta.mesh.enabled = 1`.
Lookups, ingests, distillations, and Q&A run identically with or without
mesh. Initialization is two commands and is reversible.

```
arborist mesh init --group myteam   # mint Ed25519 + X25519 keys; create epoch 0
arborist mesh enable                # flip gating flag on
arborist mesh status                # always-safe inspection
arborist mesh disable               # flag back off; keys + history stay on disk
```

## Identity stack

![identity stack](diagrams/mesh-identity-stack.png)

Each peer carries:

- one **Ed25519** keypair for signing membership ops and gossip envelopes
- one **X25519** keypair for ECDH key agreement (used to wrap epoch secrets)
- one **8-hex `member_id`** (random by default; pinnable via `--member-id`)

Group state, replicated across every peer that participates:

- `mesh_roster` — per-epoch (member_id, sign_pub, dh_pub, role) tuples
- `mesh_epochs` — per-epoch lifecycle (started_at, audit linkage, secret envelope)
- `audit_events` — append-only Merkle chain of every state-changing op

Public keys are shared. Private keys never leave a peer's SQLite file.

## Epochs — a counter that bumps on every roster change

![epoch lifecycle](diagrams/mesh-epoch-lifecycle.png)

`init_identity()` writes epoch 0 with the founder as sole admin.
Every subsequent `add`, `kick`, or `rotate` writes a new epoch row,
generates a fresh 32-byte symmetric secret, and wraps that secret to
every member of a *new* roster.

Old epochs are not deleted. Their rosters and audit events stay on
disk so that historical signatures remain verifiable by anyone, forever.
Cores never evict; mesh state never evicts either.

## Per-member secret envelope

![secret envelope](diagrams/mesh-secret-envelope.png)

A single epoch carries one symmetric secret. That one secret is wrapped
N times, once per member, by ECDH between a rotator's X25519 priv and
each member's X25519 pub, then sealed with ChaCha20-Poly1305 using
`member_id` as AAD.

Decryption is local: a peer derives the same shared secret by ECDH
between their own X25519 priv and a rotator's X25519 pub, then unwraps
their own slot. An evicted peer simply has no slot, so unwrap raises
`ValueError` — they are opaque to gossip from epoch+1 onward by design.

Code: `arborist/mesh/state.py::_wrap_secret_for_members` and
`unwrap_secret_for_self`.

## What flows between peers

![data flow](diagrams/mesh-data-flow.png)

Arborist is content-addressed, so identifiers are short and bodies are
optional. A typical sync round looks like:

1. **alice** signs and sends `ANNOUNCE_ROOT(document_root, source_uri,
   chunking_version, schema_version)` for everything new since last
   sync. Same for derivations, providence_cache rows, and falsifications.
2. **bob** verifies alice's signature against her epoch-N `sign_pub`,
   then checks his local store. Anything he already has is dedup'd by
   `document_root` and dropped on the floor.
3. On miss, bob sends `REQUEST_BODY(root)`. alice replies with
   `DELIVER_BODY(bytes, merkle_proof)`. bob verifies the proof against
   the announced root before inserting.
4. Audit chains merge by `prev_event_hash`. A gossip insert that does
   not extend a chain consistently is rejected — mesh never silently
   forks history.

The five gossip message types:

| message | meaning | encrypted? | signed? |
|---|---|---|---|
| `ANNOUNCE_ROOT` | "I have these document_roots" | optional | always |
| `ANNOUNCE_DERIVATION` | "core C derives from surfaces S₁..Sₙ via distiller D" | optional | always |
| `ANNOUNCE_PROVIDENCE` | "I cached an answer at this 8-dim cache_key with this audit_mode" | optional | always |
| `ANNOUNCE_FALSIFICATION` | "this cache_key is wrong (witness signature attached)" | optional | always |
| `REQUEST_BODY` / `DELIVER_BODY` | pull bytes on local miss; recipient verifies Merkle proof before insert | optional | always |

Soft hashes (embeddings, TF-IDF scores) never enter wire. Only hard
SHA-256 commitments cross the network. Per `CLAUDE.md`'s soft/hard hash
split.

## Group operations — operator decision tree

![operator decisions](diagrams/mesh-group-decisions.png)

A group has two roles: `admin` and `member`. Admins may add and kick.
Any current member may rotate the secret on the same roster. Self-kick
is rejected; an admin who wants out runs `mesh disable` instead.

### Add a member (admin only)

```
# bob mints his keys and shares pubs (out-of-band: signal, in person, signed file).
arborist mesh init --group myteam --member-id bob   # bob's machine

# alice (admin) enrolls bob at her machine:
arborist mesh add --member-id bob \
    --sign-pub <bob_sign_pub_hex> \
    --dh-pub   <bob_dh_pub_hex>
# epoch bumps. fresh secret wrapped to alice + bob.
```

Bob can decrypt epoch+1 forward. Bob *cannot* reach prior epochs — by
design, joining doesn't grant retroactive access.

### Kick a member (admin only)

```
arborist mesh kick --member-id dave --reason "left team 2026-04-28"
```

Epoch bumps. New secret wrapped to everyone *except* dave. Dave's prior
signatures stay verifiable forever (his roster row at older epochs is
preserved). Any AEAD-protected gossip from epoch+1 onward is opaque to
him.

Cannot leave a roster with no admins. Promote first, then kick.

### Rotate the secret (any member)

```
arborist mesh rotate --reason "scheduled monthly hygiene"
```

Same roster, fresh secret. Use on suspected secret leak when no
identifiable bad actor exists, or as periodic key hygiene.

### When to use which

| situation | command |
|---|---|
| starting a new federation | `mesh init` then `mesh enable` |
| onboarding a teammate | `mesh add` (admin) |
| someone left the team | `mesh kick` (admin) |
| rumor of secret compromise but everyone's keys still trusted | `mesh rotate` |
| stepping away yourself | `mesh disable` (own peer only) |

## Forward-secrecy guarantees in plain language

- **Joining is non-retroactive.** A new member cannot decrypt anything
  from before they joined. Wrapping per-epoch makes this automatic.
- **Eviction is forward-only.** A kicked member keeps everything they
  already had. They lose access to gossip from the next epoch on.
- **Past signatures remain verifiable forever.** Roster history is
  immutable on disk, so anyone can audit who said what at what epoch
  even after rotations.
- **Rotation costs O(N).** One ECDH + one AEAD per member per rotate.
  Linear in roster size, not document count.

## Trust boundary, threat model

Mesh defends against:

- **Roster impersonation.** Every membership op is Ed25519-signed by an
  admin and chained via `audit_events.event_hash = sha256(prev || body)`.
  A peer cannot fake a `mesh_epoch_rotate` without the admin's signing key.
- **Gossip tampering.** Every announcement is signed; recipients verify
  before insert. A peer that doesn't carry an epoch's secret cannot
  produce a valid AEAD payload for that epoch.
- **Eviction bypass.** Evicted members have no slot in the post-rotation
  envelope. They cannot derive the new secret from any prior secret.
- **Forking.** Audit chain `prev_event_hash` linkage rejects gossip
  inserts that do not extend a chain consistently.

Mesh does not defend against:

- **Out-of-band key distribution.** A new member's pubs must reach the
  enrolling admin via a trusted channel (signal, in person, signed file).
  Mesh has no built-in introduction protocol.
- **Compromise of an admin's signing key.** With that key an attacker
  can forge any membership op. Mitigation: rotate admins regularly,
  keep admin count small.
- **Traffic analysis.** Wire layer ships TLS but does not pad or mix.

## Related code

- `arborist/mesh/__init__.py` — module entrypoint, public API
- `arborist/mesh/crypto.py` — Ed25519 + X25519 + ChaCha20-Poly1305 wrappers
- `arborist/mesh/state.py` — DB writes + audit chain hooks; epoch rotation
- `arborist/mesh/members.py` — `add_member`, `kick_member`, `scheduled_rotate`
- `arborist/cli.py` — `_cmd_mesh_*` argparse handlers
- `arborist/store.py` — `mesh_identity`, `mesh_roster`, `mesh_epochs` schema

## Related docs

- `README.md` — top-level overview, ingest flow, single-peer Q&A
- `CLAUDE.md` — schema invariants, conventions, audit chain rules
- `~/git/proxy.unturf.com/pkg/verified/merkle.go` — fox's Go merkle
  reference. Arborist's Python port mirrors the conventions.

## Rendering the diagrams

```
make docs    # renders docs/diagrams/*.dot to .png via graphviz
```

Or directly: `dot -Tpng docs/diagrams/<file>.dot -o /tmp/out.png`.