docs: mesh-deploy runbook for two-host gossip setup

Step-by-step operator guide for standing up an aborist mesh between
two real hosts: pre-flight, init/enable on both peers, out-of-band
pubkey exchange, mutual enrollment, mesh serve, mesh sync, audit
chain integrity checks, eviction protocol, reset/teardown.

Calls out what's not yet wired (mesh pull, per-peer chain merge,
AEAD body encryption) so operators don't expect features that ship
later. Pairs with docs/mesh.md (protocol contract) and the e2e tests
in tests/test_mesh_wire_e2e.py.
This commit is contained in:
russell@unturf.com 2026-04-28 17:24:37 -04:00
parent 71507f9327
commit e868b95530
No known key found for this signature in database

257
docs/mesh-deploy.md Normal file
View file

@ -0,0 +1,257 @@
# Mesh deploy — two-host runbook
This is the operator runbook for standing up an aborist mesh between two
real hosts. The protocol is documented in `docs/mesh.md`; this file
covers what to type, in what order, with what to verify at each step.
> Every step assumes both peers have aborist installed and have already
> ingested some local corpus. If you only want to test the wire on
> localhost-loopback, see `tests/test_mesh_wire_e2e.py` instead.
## Naming
Two hosts in this runbook:
- **`alice`** — admin, founder of group `myteam`. Runs `mesh serve`.
- **`bob`** — joining member. Runs `mesh sync` outbound to alice.
Substitute hostnames / IPs / ports per your environment.
## 0. Pre-flight on each host
```sh
# Confirm aborist + extras + tests:
make bootstrap
make test
.venv/bin/aborist --version
```
Both hosts should be on the same git commit. Mismatched
`schema_version` / `chunking_version` / `canonicalization_version`
will be rejected by the v9.8 admissibility check; aborist refuses
silent corruption.
## 1. Initialize mesh on each peer
```sh
# alice — first peer becomes founder of epoch 0:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh init --group myteam --member-id alice
.venv/bin/aborist --db ~/.aborist/aborist.db mesh enable
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status
# bob — initializes a separate identity in the same group name:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh init --group myteam --member-id bob
.venv/bin/aborist --db ~/.aborist/aborist.db mesh enable
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status
```
At this point alice and bob both think they're the sole member of
`myteam`. The next step adds bob to alice's roster — and vice versa —
so both rosters carry both pubkeys at the same epoch.
## 2. Exchange pubkeys (out-of-band)
bob's public keys must reach alice via a trusted channel. Mesh has no
built-in introduction protocol; signal, in-person hand-off, or a
signed file all work.
```sh
# On bob:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status | jq '.identity'
# -> { "member_id": "bob", "sign_pub_hex": "...", "dh_pub_hex": "..." }
```
Send the two hex strings to alice. Verify the channel out of band.
Same direction in reverse so bob can enroll alice:
```sh
# On alice:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status | jq '.identity'
```
## 3. Mutual enrollment
```sh
# On alice (admin) — enroll bob at alice's epoch:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh add \
--member-id bob \
--sign-pub <bob_sign_pub_hex> \
--dh-pub <bob_dh_pub_hex>
# On bob — enroll alice at bob's epoch (so bob's local roster knows
# alice's pubkey for signature verification when alice's announces
# arrive):
.venv/bin/aborist --db ~/.aborist/aborist.db mesh add \
--member-id alice \
--sign-pub <alice_sign_pub_hex> \
--dh-pub <alice_dh_pub_hex>
```
Each `mesh add` bumps the local epoch and writes a `mesh_epoch_rotate`
audit event. Verify:
```sh
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status
.venv/bin/aborist --db ~/.aborist/aborist.db mesh members
```
Both peers should now show 2 members at epoch 1.
## 4. Start alice's gossip server
```sh
# On alice — bind on a routable interface:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh serve \
--host 0.0.0.0 --port 8400
# Stdout: {"status": "serving", "url": "http://0.0.0.0:8400", ...}
```
Leave this running. Open the firewall to allow bob to reach
`alice:8400`. Production deployments should put a TLS terminator
(Caddy / nginx) in front; the wire speaks plain HTTP — TLS is the
operator's choice.
Verify reachability from bob:
```sh
# On bob:
curl http://alice.example.com:8400/mesh/info | jq
# -> { "v": 1, "member_id": "alice", "group_name": "myteam",
# "current_epoch": 1, "sign_pub_hex": "..." }
```
If `member_id` matches what bob has in his roster for "alice", and the
`sign_pub_hex` matches what bob enrolled, you're good. If they don't
match, bob is talking to the wrong peer or alice's enrollment was wrong;
do not push gossip until the discrepancy is resolved.
## 5. Bob announces his local roots to alice
```sh
# On bob:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh sync \
--peer http://alice.example.com:8400 \
--limit 100
```
Output: JSON status, count of acked vs errored announces.
Each accepted announce writes one `mesh_received` event into alice's
audit chain. Verify on alice:
```sh
sqlite3 ~/.aborist/aborist.db \
"SELECT COUNT(*) FROM audit_events WHERE event_type='mesh_received'"
```
Cross-check chain integrity:
```sh
make chain-check
```
Should return `0` chain breaks.
## 6. Make sync bidirectional
The current `mesh sync` only pushes outbound. To get alice's roots
onto bob, run `mesh sync` from alice pointed at bob:
```sh
# On alice (in a new terminal — keep mesh serve running):
.venv/bin/aborist --db ~/.aborist/aborist.db mesh sync \
--peer http://bob.example.com:8400 --limit 100
```
This requires bob to also be running `mesh serve`. Symmetric setup is
the standard mode.
## 7. Pulling missing bodies (when shipped)
Today only ANNOUNCE_ROOT is exercised by `mesh sync`. The wire layer
already implements REQUEST_BODY/DELIVER_BODY with Merkle verification
on the client side; the CLI hookup (`mesh pull --root <hex> --peer
<url>`) is on the queue. Once shipped, the pull-on-miss workflow is:
```sh
# bob discovers via announces that alice has document X he doesn't:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh pull \
--root <document_root_hex> \
--peer http://alice.example.com:8400
# bob's local store now has X (Merkle-verified at receive).
```
## 8. Eviction protocol
If bob leaves the team, alice (admin) kicks. This bumps alice's epoch
and rewraps the next epoch secret to the post-bob roster:
```sh
# On alice:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh kick \
--member-id bob \
--reason "left team 2026-04-28"
```
Bob's prior signatures stay verifiable forever (his roster row at
older epochs is preserved on disk). Any AEAD-protected gossip from
epoch+1 onward is opaque to him — that's the eviction guarantee.
Important: kicking only mutates alice's local state. If bob's host is
still running `mesh serve`, alice should also stop sending to him (or
he'll keep accepting alice's announces under the OLD epoch she's no
longer broadcasting from). Operationally, kicks should pair with
either: (a) dropping bob's URL from alice's sync targets, OR (b)
announcing the kick to remaining members so they update their
rosters.
## 9. Operational checks
Every chain-write op (`mesh init`, `mesh add`, `mesh kick`, `mesh
rotate`, every received announce) extends the local audit chain. The
fast probe is:
```sh
make chain-check # single db
make chain-check-shards # every db in $(SHARDS_DIR)
```
Both should always print `0`. Non-zero = operator must investigate
before further operations; data has diverged from the audit trail.
## 10. Reset / teardown
To take a peer fully out of the mesh (irreversible — fresh keys
needed to rejoin):
```sh
.venv/bin/aborist --db ~/.aborist/aborist.db mesh disable
# Identity + roster history stay on disk for forensics. Re-enable
# requires another peer to re-add this member at a fresh epoch.
```
For full local nuke (developers only — destroys identity + chain):
```sh
sqlite3 ~/.aborist/aborist.db <<'SQL'
DELETE FROM mesh_identity;
DELETE FROM mesh_roster;
DELETE FROM mesh_epochs;
SQL
```
## What's not yet shipped
These features are documented in the protocol but not wired into the
CLI yet:
- **`mesh pull <root>`** — fetch a body on cache miss.
- **Per-peer audit-chain merge** — receiver tracks each sender's last
known event_hash and rejects fork events. Today the receiver verifies
signatures only; chain-of-claims tracking is deferred.
- **AEAD body encryption** — wire signs envelope bytes for integrity;
body confidentiality (encrypting against the per-epoch shared
secret) is optional in the contract and not yet wired.
When these land, this runbook will gain steps for them.