feat: #000061 cold-pack distribution tier (boto3 S3-compat + DVD-R safe-fit)

Ship arborist corpus state to new peers and DVD-R archival via
point-in-time tar.zst packs. One artifact serves both channels —
bucket+CDN delivery and physical-media archival.

Bucket holds packs only. Pack key = hash_leaf(manifest_bytes), so same
chunk set on two writers produces the same pack_hash and upload is
idempotent. Each pack pins the corpus snapshot_root it covers in audit
+ result body — packs are delayed snapshots, not live mirrors;
falsifications between repacks produce new pack_hashes.

stream_packs runs streaming zstd over tarfile, peeking compressed-buffer
size after each chunk via FLUSH_BLOCK (preserves dictionary). Default
cap 4_400_000_000 — 4.4 GB DVD-R safe-fit, ~6.5% buffer below the
4.7 GB marketing capacity to absorb ISO9660 overhead, growisofs
lead-in/lead-out, media variance, and drive-edge refusal. Each disc
fills to ~4.4 GB recorded data, not the ~1.5 GB an uncompressed cap
produced.

One backend class (S3CompatibleBackend via boto3 + endpoint_url) covers
AWS S3, DO Spaces, R2, B2, GCS S3-interop, MinIO. Optional dep
[object-store] = boto3>=1.34; dev extras pull moto for the wire test.
Voyeur: credentials via AWS_ACCESS_KEY_ID/_SECRET_ACCESS_KEY env or
~/.aws/credentials, never printed; only endpoint URL + bucket name
surface in logs.

CLI: arborist cold {pack,unpack,stats}. Makefile: cold-pack,
cold-pack-dvd (local-dir output for growisofs), cold-unpack, cold-stats.

Sizing for current shards (14.1M chunks, ~17 GB compressed): ~4 packs
at the default cap, ~\$0.34/mo DO Spaces storage, ~\$0.0001/fresh-peer
hydrate.

Always-on raw-UTF-8 leaf store (per ticket "Hard invariants") deferred
— packs-only for now, backfill later.

2557 passed, 28 skipped, 1 xfailed.
This commit is contained in:
russell@unturf.com 2026-05-25 20:23:44 -04:00
parent 06e6c7a918
commit 727cb1bd96
No known key found for this signature in database
11 changed files with 1911 additions and 4 deletions

View file

@ -40,7 +40,8 @@ SEARCH_Q ?= computer
bootstrap-math bootstrap-nli bootstrap-nli-only bench-nli-shadow export-nli-onnx bench-nli-backends judge-self-test control-ab control-sweep rapl-access rapl-access-revoke clean clean-db clean-data help \
textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \
crawl-textbooks crawl-textbooks-stats textbook textbook-list bench-jaggedness \
monitor-poll monitor-graph monitor-access
monitor-poll monitor-graph monitor-access \
bootstrap-object-store cold-pack cold-pack-dvd cold-unpack cold-stats
all: bootstrap fetch-cur ingest-cur verify stats ## bootstrap → fetch cur → ingest cur → verify → stats
@ -846,6 +847,42 @@ search: bootstrap ## keyword search; override SEARCH_Q (or pass Q=...)
stats: bootstrap ## counts: documents, chunks, edges, audit chain
$(ARBORIST) --db $(DB) stats
# --- cold-object-store tier (ticket #000061) ---------------------------------
#
# Push chunk bodies to an S3-compatible bucket so the corpus can grow past
# one machine while the Merkle tree stays intact. One backend covers AWS S3,
# DO Spaces, R2, B2, GCS (S3 interop), MinIO — different `endpoint_url`.
#
# Required env (set in your shell, never hard-code into Makefile vars):
# AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY — standard boto3 discovery
# ARBORIST_COLD_ENDPOINT_URL — e.g. https://nyc3.digitaloceanspaces.com
# ARBORIST_COLD_BUCKET — e.g. arborist-corpus
# ARBORIST_COLD_REGION — optional (DO Spaces is region-agnostic)
#
# Operation Voyeur: credentials NEVER appear on argv, in logs, in audit body.
# Only endpoint URL + bucket name surface; keys live in env / ~/.aws/credentials.
bootstrap-object-store: bootstrap ## install [object-store] (boto3) into the venv
$(PIP) install -e '.[object-store]'
cold-pack: bootstrap ## bundle hot chunks into tar.zst pack(s) on bucket (≤4.4 GB DVD-R safe-fit each by default)
$(ARBORIST) --db $(DB) cold pack \
$(if $(MAX_CHUNKS),--max-chunks $(MAX_CHUNKS),) \
$(if $(MAX_PACK_BYTES),--max-pack-bytes $(MAX_PACK_BYTES),) \
$(if $(LOCAL_DIR),--local-dir $(LOCAL_DIR),)
cold-pack-dvd: bootstrap ## write packs to LOCAL_DIR for burning (no S3 upload); each pack ≤4.7 GB
@if [ -z "$(LOCAL_DIR)" ]; then echo "LOCAL_DIR=<dir> required"; exit 2; fi
$(ARBORIST) --db $(DB) cold pack --no-push --local-dir $(LOCAL_DIR)
@echo ">> packs written to $(LOCAL_DIR) — burn each .tar.zst with growisofs:"
@echo ">> growisofs -dvd-compat -Z /dev/sr0 $(LOCAL_DIR)/arborist-pack-<hash>.tar.zst"
cold-unpack: bootstrap ## pull pack PACK=<hash> and restore chunks locally
@if [ -z "$(PACK)" ]; then echo "PACK=<pack_hash> required"; exit 2; fi
$(ARBORIST) --db $(DB) cold unpack $(PACK)
cold-stats: bootstrap ## bucket summary: chunk count, pack count, endpoint
$(ARBORIST) --db $(DB) cold stats
test: bootstrap ## run pytest suite (excludes opt-in crawler tests)
$(VENV)/bin/pytest -q --ignore=tests/crawler -n auto