docs: mesh-deploy runbook for two-host gossip setup
Step-by-step operator guide for standing up an aborist mesh between two real hosts: pre-flight, init/enable on both peers, out-of-band pubkey exchange, mutual enrollment, mesh serve, mesh sync, audit chain integrity checks, eviction protocol, reset/teardown. Calls out what's not yet wired (mesh pull, per-peer chain merge, AEAD body encryption) so operators don't expect features that ship later. Pairs with docs/mesh.md (protocol contract) and the e2e tests in tests/test_mesh_wire_e2e.py.
This commit is contained in:
parent
71507f9327
commit
e868b95530
1 changed files with 257 additions and 0 deletions
257
docs/mesh-deploy.md
Normal file
257
docs/mesh-deploy.md
Normal file
|
|
@ -0,0 +1,257 @@
|
|||
# Mesh deploy — two-host runbook
|
||||
|
||||
This is the operator runbook for standing up an aborist mesh between two
|
||||
real hosts. The protocol is documented in `docs/mesh.md`; this file
|
||||
covers what to type, in what order, with what to verify at each step.
|
||||
|
||||
> Every step assumes both peers have aborist installed and have already
|
||||
> ingested some local corpus. If you only want to test the wire on
|
||||
> localhost-loopback, see `tests/test_mesh_wire_e2e.py` instead.
|
||||
|
||||
## Naming
|
||||
|
||||
Two hosts in this runbook:
|
||||
|
||||
- **`alice`** — admin, founder of group `myteam`. Runs `mesh serve`.
|
||||
- **`bob`** — joining member. Runs `mesh sync` outbound to alice.
|
||||
|
||||
Substitute hostnames / IPs / ports per your environment.
|
||||
|
||||
## 0. Pre-flight on each host
|
||||
|
||||
```sh
|
||||
# Confirm aborist + extras + tests:
|
||||
make bootstrap
|
||||
make test
|
||||
.venv/bin/aborist --version
|
||||
```
|
||||
|
||||
Both hosts should be on the same git commit. Mismatched
|
||||
`schema_version` / `chunking_version` / `canonicalization_version`
|
||||
will be rejected by the v9.8 admissibility check; aborist refuses
|
||||
silent corruption.
|
||||
|
||||
## 1. Initialize mesh on each peer
|
||||
|
||||
```sh
|
||||
# alice — first peer becomes founder of epoch 0:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh init --group myteam --member-id alice
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh enable
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status
|
||||
|
||||
# bob — initializes a separate identity in the same group name:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh init --group myteam --member-id bob
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh enable
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status
|
||||
```
|
||||
|
||||
At this point alice and bob both think they're the sole member of
|
||||
`myteam`. The next step adds bob to alice's roster — and vice versa —
|
||||
so both rosters carry both pubkeys at the same epoch.
|
||||
|
||||
## 2. Exchange pubkeys (out-of-band)
|
||||
|
||||
bob's public keys must reach alice via a trusted channel. Mesh has no
|
||||
built-in introduction protocol; signal, in-person hand-off, or a
|
||||
signed file all work.
|
||||
|
||||
```sh
|
||||
# On bob:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status | jq '.identity'
|
||||
# -> { "member_id": "bob", "sign_pub_hex": "...", "dh_pub_hex": "..." }
|
||||
```
|
||||
|
||||
Send the two hex strings to alice. Verify the channel out of band.
|
||||
|
||||
Same direction in reverse so bob can enroll alice:
|
||||
|
||||
```sh
|
||||
# On alice:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status | jq '.identity'
|
||||
```
|
||||
|
||||
## 3. Mutual enrollment
|
||||
|
||||
```sh
|
||||
# On alice (admin) — enroll bob at alice's epoch:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh add \
|
||||
--member-id bob \
|
||||
--sign-pub <bob_sign_pub_hex> \
|
||||
--dh-pub <bob_dh_pub_hex>
|
||||
|
||||
# On bob — enroll alice at bob's epoch (so bob's local roster knows
|
||||
# alice's pubkey for signature verification when alice's announces
|
||||
# arrive):
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh add \
|
||||
--member-id alice \
|
||||
--sign-pub <alice_sign_pub_hex> \
|
||||
--dh-pub <alice_dh_pub_hex>
|
||||
```
|
||||
|
||||
Each `mesh add` bumps the local epoch and writes a `mesh_epoch_rotate`
|
||||
audit event. Verify:
|
||||
|
||||
```sh
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh members
|
||||
```
|
||||
|
||||
Both peers should now show 2 members at epoch 1.
|
||||
|
||||
## 4. Start alice's gossip server
|
||||
|
||||
```sh
|
||||
# On alice — bind on a routable interface:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh serve \
|
||||
--host 0.0.0.0 --port 8400
|
||||
# Stdout: {"status": "serving", "url": "http://0.0.0.0:8400", ...}
|
||||
```
|
||||
|
||||
Leave this running. Open the firewall to allow bob to reach
|
||||
`alice:8400`. Production deployments should put a TLS terminator
|
||||
(Caddy / nginx) in front; the wire speaks plain HTTP — TLS is the
|
||||
operator's choice.
|
||||
|
||||
Verify reachability from bob:
|
||||
|
||||
```sh
|
||||
# On bob:
|
||||
curl http://alice.example.com:8400/mesh/info | jq
|
||||
# -> { "v": 1, "member_id": "alice", "group_name": "myteam",
|
||||
# "current_epoch": 1, "sign_pub_hex": "..." }
|
||||
```
|
||||
|
||||
If `member_id` matches what bob has in his roster for "alice", and the
|
||||
`sign_pub_hex` matches what bob enrolled, you're good. If they don't
|
||||
match, bob is talking to the wrong peer or alice's enrollment was wrong;
|
||||
do not push gossip until the discrepancy is resolved.
|
||||
|
||||
## 5. Bob announces his local roots to alice
|
||||
|
||||
```sh
|
||||
# On bob:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh sync \
|
||||
--peer http://alice.example.com:8400 \
|
||||
--limit 100
|
||||
```
|
||||
|
||||
Output: JSON status, count of acked vs errored announces.
|
||||
|
||||
Each accepted announce writes one `mesh_received` event into alice's
|
||||
audit chain. Verify on alice:
|
||||
|
||||
```sh
|
||||
sqlite3 ~/.aborist/aborist.db \
|
||||
"SELECT COUNT(*) FROM audit_events WHERE event_type='mesh_received'"
|
||||
```
|
||||
|
||||
Cross-check chain integrity:
|
||||
|
||||
```sh
|
||||
make chain-check
|
||||
```
|
||||
|
||||
Should return `0` chain breaks.
|
||||
|
||||
## 6. Make sync bidirectional
|
||||
|
||||
The current `mesh sync` only pushes outbound. To get alice's roots
|
||||
onto bob, run `mesh sync` from alice pointed at bob:
|
||||
|
||||
```sh
|
||||
# On alice (in a new terminal — keep mesh serve running):
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh sync \
|
||||
--peer http://bob.example.com:8400 --limit 100
|
||||
```
|
||||
|
||||
This requires bob to also be running `mesh serve`. Symmetric setup is
|
||||
the standard mode.
|
||||
|
||||
## 7. Pulling missing bodies (when shipped)
|
||||
|
||||
Today only ANNOUNCE_ROOT is exercised by `mesh sync`. The wire layer
|
||||
already implements REQUEST_BODY/DELIVER_BODY with Merkle verification
|
||||
on the client side; the CLI hookup (`mesh pull --root <hex> --peer
|
||||
<url>`) is on the queue. Once shipped, the pull-on-miss workflow is:
|
||||
|
||||
```sh
|
||||
# bob discovers via announces that alice has document X he doesn't:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh pull \
|
||||
--root <document_root_hex> \
|
||||
--peer http://alice.example.com:8400
|
||||
# bob's local store now has X (Merkle-verified at receive).
|
||||
```
|
||||
|
||||
## 8. Eviction protocol
|
||||
|
||||
If bob leaves the team, alice (admin) kicks. This bumps alice's epoch
|
||||
and rewraps the next epoch secret to the post-bob roster:
|
||||
|
||||
```sh
|
||||
# On alice:
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh kick \
|
||||
--member-id bob \
|
||||
--reason "left team 2026-04-28"
|
||||
```
|
||||
|
||||
Bob's prior signatures stay verifiable forever (his roster row at
|
||||
older epochs is preserved on disk). Any AEAD-protected gossip from
|
||||
epoch+1 onward is opaque to him — that's the eviction guarantee.
|
||||
|
||||
Important: kicking only mutates alice's local state. If bob's host is
|
||||
still running `mesh serve`, alice should also stop sending to him (or
|
||||
he'll keep accepting alice's announces under the OLD epoch she's no
|
||||
longer broadcasting from). Operationally, kicks should pair with
|
||||
either: (a) dropping bob's URL from alice's sync targets, OR (b)
|
||||
announcing the kick to remaining members so they update their
|
||||
rosters.
|
||||
|
||||
## 9. Operational checks
|
||||
|
||||
Every chain-write op (`mesh init`, `mesh add`, `mesh kick`, `mesh
|
||||
rotate`, every received announce) extends the local audit chain. The
|
||||
fast probe is:
|
||||
|
||||
```sh
|
||||
make chain-check # single db
|
||||
make chain-check-shards # every db in $(SHARDS_DIR)
|
||||
```
|
||||
|
||||
Both should always print `0`. Non-zero = operator must investigate
|
||||
before further operations; data has diverged from the audit trail.
|
||||
|
||||
## 10. Reset / teardown
|
||||
|
||||
To take a peer fully out of the mesh (irreversible — fresh keys
|
||||
needed to rejoin):
|
||||
|
||||
```sh
|
||||
.venv/bin/aborist --db ~/.aborist/aborist.db mesh disable
|
||||
# Identity + roster history stay on disk for forensics. Re-enable
|
||||
# requires another peer to re-add this member at a fresh epoch.
|
||||
```
|
||||
|
||||
For full local nuke (developers only — destroys identity + chain):
|
||||
|
||||
```sh
|
||||
sqlite3 ~/.aborist/aborist.db <<'SQL'
|
||||
DELETE FROM mesh_identity;
|
||||
DELETE FROM mesh_roster;
|
||||
DELETE FROM mesh_epochs;
|
||||
SQL
|
||||
```
|
||||
|
||||
## What's not yet shipped
|
||||
|
||||
These features are documented in the protocol but not wired into the
|
||||
CLI yet:
|
||||
|
||||
- **`mesh pull <root>`** — fetch a body on cache miss.
|
||||
- **Per-peer audit-chain merge** — receiver tracks each sender's last
|
||||
known event_hash and rejects fork events. Today the receiver verifies
|
||||
signatures only; chain-of-claims tracking is deferred.
|
||||
- **AEAD body encryption** — wire signs envelope bytes for integrity;
|
||||
body confidentiality (encrypting against the per-epoch shared
|
||||
secret) is optional in the contract and not yet wired.
|
||||
|
||||
When these land, this runbook will gain steps for them.
|
||||
Loading…
Add table
Add a link
Reference in a new issue