# Mesh deploy — two-host runbook

This is the operator runbook for standing up an aborist mesh between two
real hosts. The protocol is documented in `docs/mesh.md`; this file
covers what to type, in what order, with what to verify at each step.

> Every step assumes both peers have aborist installed and have already
> ingested some local corpus. If you only want to test the wire on
> localhost-loopback, see `tests/test_mesh_wire_e2e.py` instead.

## Naming

Two hosts in this runbook:

- **`alice`** — admin, founder of group `myteam`. Runs `mesh serve`.
- **`bob`** — joining member. Runs `mesh sync` outbound to alice.

Substitute hostnames / IPs / ports per your environment.

## 0. Pre-flight on each host

```sh
# Confirm aborist + extras + tests:
make bootstrap
make test
.venv/bin/aborist --version
```

Both hosts should be on the same git commit. Mismatched
`schema_version` / `chunking_version` / `canonicalization_version`
will be rejected by the v9.8 admissibility check; aborist refuses
silent corruption.

## 1. Initialize mesh on each peer

```sh
# alice — first peer becomes founder of epoch 0:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh init --group myteam --member-id alice
.venv/bin/aborist --db ~/.aborist/aborist.db mesh enable
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status

# bob — initializes a separate identity in the same group name:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh init --group myteam --member-id bob
.venv/bin/aborist --db ~/.aborist/aborist.db mesh enable
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status
```

At this point alice and bob both think they're the sole member of
`myteam`. The next step adds bob to alice's roster — and vice versa —
so both rosters carry both pubkeys at the same epoch.

## 2. Exchange pubkeys (out-of-band)

bob's public keys must reach alice via a trusted channel. Mesh has no
built-in introduction protocol; signal, in-person hand-off, or a
signed file all work.

```sh
# On bob:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status | jq '.identity'
# -> { "member_id": "bob", "sign_pub_hex": "...", "dh_pub_hex": "..." }
```

Send the two hex strings to alice. Verify the channel out of band.

Same direction in reverse so bob can enroll alice:

```sh
# On alice:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status | jq '.identity'
```

## 3. Mutual enrollment

```sh
# On alice (admin) — enroll bob at alice's epoch:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh add \
    --member-id bob \
    --sign-pub <bob_sign_pub_hex> \
    --dh-pub   <bob_dh_pub_hex>

# On bob — enroll alice at bob's epoch (so bob's local roster knows
# alice's pubkey for signature verification when alice's announces
# arrive):
.venv/bin/aborist --db ~/.aborist/aborist.db mesh add \
    --member-id alice \
    --sign-pub <alice_sign_pub_hex> \
    --dh-pub   <alice_dh_pub_hex>
```

Each `mesh add` bumps the local epoch and writes a `mesh_epoch_rotate`
audit event. Verify:

```sh
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status
.venv/bin/aborist --db ~/.aborist/aborist.db mesh members
```

Both peers should now show 2 members at epoch 1.

## 4. Start alice's gossip server

```sh
# On alice — bind on a routable interface:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh serve \
    --host 0.0.0.0 --port 8400
# Stdout: {"status": "serving", "url": "http://0.0.0.0:8400", ...}
```

Leave this running. Open the firewall to allow bob to reach
`alice:8400`. Production deployments should put a TLS terminator
(Caddy / nginx) in front; the wire speaks plain HTTP — TLS is the
operator's choice.

Verify reachability from bob:

```sh
# On bob:
curl http://alice.example.com:8400/mesh/info | jq
# -> { "v": 1, "member_id": "alice", "group_name": "myteam",
#      "current_epoch": 1, "sign_pub_hex": "..." }
```

If `member_id` matches what bob has in his roster for "alice", and the
`sign_pub_hex` matches what bob enrolled, you're good. If they don't
match, bob is talking to the wrong peer or alice's enrollment was wrong;
do not push gossip until the discrepancy is resolved.

## 5. Bob announces his local roots to alice

```sh
# On bob:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh sync \
    --peer http://alice.example.com:8400 \
    --limit 100
```

Output: JSON status, count of acked vs errored announces.

Each accepted announce writes one `mesh_received` event into alice's
audit chain. Verify on alice:

```sh
sqlite3 ~/.aborist/aborist.db \
    "SELECT COUNT(*) FROM audit_events WHERE event_type='mesh_received'"
```

Cross-check chain integrity:

```sh
make chain-check
```

Should return `0` chain breaks.

## 6. Make sync bidirectional

The current `mesh sync` only pushes outbound. To get alice's roots
onto bob, run `mesh sync` from alice pointed at bob:

```sh
# On alice (in a new terminal — keep mesh serve running):
.venv/bin/aborist --db ~/.aborist/aborist.db mesh sync \
    --peer http://bob.example.com:8400 --limit 100
```

This requires bob to also be running `mesh serve`. Symmetric setup is
the standard mode.

## 7. Pulling missing bodies (when shipped)

Today only ANNOUNCE_ROOT is exercised by `mesh sync`. The wire layer
already implements REQUEST_BODY/DELIVER_BODY with Merkle verification
on the client side; the CLI hookup (`mesh pull --root <hex> --peer
<url>`) is on the queue. Once shipped, the pull-on-miss workflow is:

```sh
# bob discovers via announces that alice has document X he doesn't:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh pull \
    --root <document_root_hex> \
    --peer http://alice.example.com:8400
# bob's local store now has X (Merkle-verified at receive).
```

## 8. Eviction protocol

If bob leaves the team, alice (admin) kicks. This bumps alice's epoch
and rewraps the next epoch secret to the post-bob roster:

```sh
# On alice:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh kick \
    --member-id bob \
    --reason "left team 2026-04-28"
```

Bob's prior signatures stay verifiable forever (his roster row at
older epochs is preserved on disk). Any AEAD-protected gossip from
epoch+1 onward is opaque to him — that's the eviction guarantee.

Important: kicking only mutates alice's local state. If bob's host is
still running `mesh serve`, alice should also stop sending to him (or
he'll keep accepting alice's announces under the OLD epoch she's no
longer broadcasting from). Operationally, kicks should pair with
either: (a) dropping bob's URL from alice's sync targets, OR (b)
announcing the kick to remaining members so they update their
rosters.

## 9. Operational checks

Every chain-write op (`mesh init`, `mesh add`, `mesh kick`, `mesh
rotate`, every received announce) extends the local audit chain. The
fast probe is:

```sh
make chain-check         # single db
make chain-check-shards  # every db in $(SHARDS_DIR)
```

Both should always print `0`. Non-zero = operator must investigate
before further operations; data has diverged from the audit trail.

## 10. Reset / teardown

To take a peer fully out of the mesh (irreversible — fresh keys
needed to rejoin):

```sh
.venv/bin/aborist --db ~/.aborist/aborist.db mesh disable
# Identity + roster history stay on disk for forensics. Re-enable
# requires another peer to re-add this member at a fresh epoch.
```

For full local nuke (developers only — destroys identity + chain):

```sh
sqlite3 ~/.aborist/aborist.db <<'SQL'
DELETE FROM mesh_identity;
DELETE FROM mesh_roster;
DELETE FROM mesh_epochs;
SQL
```

## What's not yet shipped

These features are documented in the protocol but not wired into the
CLI yet:

- **`mesh pull <root>`** — fetch a body on cache miss.
- **Per-peer audit-chain merge** — receiver tracks each sender's last
  known event_hash and rejects fork events. Today the receiver verifies
  signatures only; chain-of-claims tracking is deferred.
- **AEAD body encryption** — wire signs envelope bytes for integrity;
  body confidentiality (encrypting against the per-epoch shared
  secret) is optional in the contract and not yet wired.

When these land, this runbook will gain steps for them.