feat: #000061 cold-pack distribution tier (boto3 S3-compat + DVD-R safe-fit)
Ship arborist corpus state to new peers and DVD-R archival via
point-in-time tar.zst packs. One artifact serves both channels —
bucket+CDN delivery and physical-media archival.
Bucket holds packs only. Pack key = hash_leaf(manifest_bytes), so same
chunk set on two writers produces the same pack_hash and upload is
idempotent. Each pack pins the corpus snapshot_root it covers in audit
+ result body — packs are delayed snapshots, not live mirrors;
falsifications between repacks produce new pack_hashes.
stream_packs runs streaming zstd over tarfile, peeking compressed-buffer
size after each chunk via FLUSH_BLOCK (preserves dictionary). Default
cap 4_400_000_000 — 4.4 GB DVD-R safe-fit, ~6.5% buffer below the
4.7 GB marketing capacity to absorb ISO9660 overhead, growisofs
lead-in/lead-out, media variance, and drive-edge refusal. Each disc
fills to ~4.4 GB recorded data, not the ~1.5 GB an uncompressed cap
produced.
One backend class (S3CompatibleBackend via boto3 + endpoint_url) covers
AWS S3, DO Spaces, R2, B2, GCS S3-interop, MinIO. Optional dep
[object-store] = boto3>=1.34; dev extras pull moto for the wire test.
Voyeur: credentials via AWS_ACCESS_KEY_ID/_SECRET_ACCESS_KEY env or
~/.aws/credentials, never printed; only endpoint URL + bucket name
surface in logs.
CLI: arborist cold {pack,unpack,stats}. Makefile: cold-pack,
cold-pack-dvd (local-dir output for growisofs), cold-unpack, cold-stats.
Sizing for current shards (14.1M chunks, ~17 GB compressed): ~4 packs
at the default cap, ~\$0.34/mo DO Spaces storage, ~\$0.0001/fresh-peer
hydrate.
Always-on raw-UTF-8 leaf store (per ticket "Hard invariants") deferred
— packs-only for now, backfill later.
2557 passed, 28 skipped, 1 xfailed.
This commit is contained in:
parent
06e6c7a918
commit
727cb1bd96
11 changed files with 1911 additions and 4 deletions
39
Makefile
39
Makefile
|
|
@ -40,7 +40,8 @@ SEARCH_Q ?= computer
|
|||
bootstrap-math bootstrap-nli bootstrap-nli-only bench-nli-shadow export-nli-onnx bench-nli-backends judge-self-test control-ab control-sweep rapl-access rapl-access-revoke clean clean-db clean-data help \
|
||||
textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \
|
||||
crawl-textbooks crawl-textbooks-stats textbook textbook-list bench-jaggedness \
|
||||
monitor-poll monitor-graph monitor-access
|
||||
monitor-poll monitor-graph monitor-access \
|
||||
bootstrap-object-store cold-pack cold-pack-dvd cold-unpack cold-stats
|
||||
|
||||
all: bootstrap fetch-cur ingest-cur verify stats ## bootstrap → fetch cur → ingest cur → verify → stats
|
||||
|
||||
|
|
@ -846,6 +847,42 @@ search: bootstrap ## keyword search; override SEARCH_Q (or pass Q=...)
|
|||
stats: bootstrap ## counts: documents, chunks, edges, audit chain
|
||||
$(ARBORIST) --db $(DB) stats
|
||||
|
||||
# --- cold-object-store tier (ticket #000061) ---------------------------------
|
||||
#
|
||||
# Push chunk bodies to an S3-compatible bucket so the corpus can grow past
|
||||
# one machine while the Merkle tree stays intact. One backend covers AWS S3,
|
||||
# DO Spaces, R2, B2, GCS (S3 interop), MinIO — different `endpoint_url`.
|
||||
#
|
||||
# Required env (set in your shell, never hard-code into Makefile vars):
|
||||
# AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY — standard boto3 discovery
|
||||
# ARBORIST_COLD_ENDPOINT_URL — e.g. https://nyc3.digitaloceanspaces.com
|
||||
# ARBORIST_COLD_BUCKET — e.g. arborist-corpus
|
||||
# ARBORIST_COLD_REGION — optional (DO Spaces is region-agnostic)
|
||||
#
|
||||
# Operation Voyeur: credentials NEVER appear on argv, in logs, in audit body.
|
||||
# Only endpoint URL + bucket name surface; keys live in env / ~/.aws/credentials.
|
||||
bootstrap-object-store: bootstrap ## install [object-store] (boto3) into the venv
|
||||
$(PIP) install -e '.[object-store]'
|
||||
|
||||
cold-pack: bootstrap ## bundle hot chunks into tar.zst pack(s) on bucket (≤4.4 GB DVD-R safe-fit each by default)
|
||||
$(ARBORIST) --db $(DB) cold pack \
|
||||
$(if $(MAX_CHUNKS),--max-chunks $(MAX_CHUNKS),) \
|
||||
$(if $(MAX_PACK_BYTES),--max-pack-bytes $(MAX_PACK_BYTES),) \
|
||||
$(if $(LOCAL_DIR),--local-dir $(LOCAL_DIR),)
|
||||
|
||||
cold-pack-dvd: bootstrap ## write packs to LOCAL_DIR for burning (no S3 upload); each pack ≤4.7 GB
|
||||
@if [ -z "$(LOCAL_DIR)" ]; then echo "LOCAL_DIR=<dir> required"; exit 2; fi
|
||||
$(ARBORIST) --db $(DB) cold pack --no-push --local-dir $(LOCAL_DIR)
|
||||
@echo ">> packs written to $(LOCAL_DIR) — burn each .tar.zst with growisofs:"
|
||||
@echo ">> growisofs -dvd-compat -Z /dev/sr0 $(LOCAL_DIR)/arborist-pack-<hash>.tar.zst"
|
||||
|
||||
cold-unpack: bootstrap ## pull pack PACK=<hash> and restore chunks locally
|
||||
@if [ -z "$(PACK)" ]; then echo "PACK=<pack_hash> required"; exit 2; fi
|
||||
$(ARBORIST) --db $(DB) cold unpack $(PACK)
|
||||
|
||||
cold-stats: bootstrap ## bucket summary: chunk count, pack count, endpoint
|
||||
$(ARBORIST) --db $(DB) cold stats
|
||||
|
||||
test: bootstrap ## run pytest suite (excludes opt-in crawler tests)
|
||||
$(VENV)/bin/pytest -q --ignore=tests/crawler -n auto
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue