russell@unturf.com
727cb1bd96
Ship arborist corpus state to new peers and DVD-R archival via
point-in-time tar.zst packs. One artifact serves both channels —
bucket+CDN delivery and physical-media archival.
Bucket holds packs only. Pack key = hash_leaf(manifest_bytes), so same
chunk set on two writers produces the same pack_hash and upload is
idempotent. Each pack pins the corpus snapshot_root it covers in audit
+ result body — packs are delayed snapshots, not live mirrors;
falsifications between repacks produce new pack_hashes.
stream_packs runs streaming zstd over tarfile, peeking compressed-buffer
size after each chunk via FLUSH_BLOCK (preserves dictionary). Default
cap 4_400_000_000 — 4.4 GB DVD-R safe-fit, ~6.5% buffer below the
4.7 GB marketing capacity to absorb ISO9660 overhead, growisofs
lead-in/lead-out, media variance, and drive-edge refusal. Each disc
fills to ~4.4 GB recorded data, not the ~1.5 GB an uncompressed cap
produced.
One backend class (S3CompatibleBackend via boto3 + endpoint_url) covers
AWS S3, DO Spaces, R2, B2, GCS S3-interop, MinIO. Optional dep
[object-store] = boto3>=1.34; dev extras pull moto for the wire test.
Voyeur: credentials via AWS_ACCESS_KEY_ID/_SECRET_ACCESS_KEY env or
~/.aws/credentials, never printed; only endpoint URL + bucket name
surface in logs.
CLI: arborist cold {pack,unpack,stats}. Makefile: cold-pack,
cold-pack-dvd (local-dir output for growisofs), cold-unpack, cold-stats.
Sizing for current shards (14.1M chunks, ~17 GB compressed): ~4 packs
at the default cap, ~\$0.34/mo DO Spaces storage, ~\$0.0001/fresh-peer
hydrate.
Always-on raw-UTF-8 leaf store (per ticket "Hard invariants") deferred
— packs-only for now, backfill later.
2557 passed, 28 skipped, 1 xfailed.
69 lines
2.1 KiB
Python
69 lines
2.1 KiB
Python
"""On-the-wire boto3 backend test (#000061) — moto-mocked S3.
|
|
|
|
Verifies that S3CompatibleBackend correctly issues PUT / GET / HEAD /
|
|
LIST against an S3 API for pack-shaped payloads. The rest of the cold-
|
|
pack surface lives in tests/test_cold_object.py and runs against the
|
|
in-process MemoryBackend.
|
|
|
|
Gated on boto3 + moto. The default suite skips this file.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
boto3 = pytest.importorskip("boto3")
|
|
moto = pytest.importorskip("moto")
|
|
|
|
from arborist.cold_object import PACK_PREFIX, S3CompatibleBackend, pack_key
|
|
from arborist.merkle import hash_leaf
|
|
|
|
|
|
@pytest.fixture
|
|
def moto_s3():
|
|
"""Spin up an in-memory S3 endpoint via moto, yield a fresh bucket."""
|
|
from moto import mock_aws
|
|
|
|
with mock_aws():
|
|
client = boto3.client("s3", region_name="us-east-1")
|
|
client.create_bucket(Bucket="arborist-test")
|
|
yield {
|
|
"bucket": "arborist-test",
|
|
"region": "us-east-1",
|
|
}
|
|
|
|
|
|
def test_s3_backend_put_get_head_list(moto_s3):
|
|
# moto mocks the AWS endpoint when endpoint_url is None or AWS-shaped.
|
|
# Pointing at the default AWS URL keeps boto3 happy inside the mock.
|
|
backend = S3CompatibleBackend(
|
|
endpoint_url="https://s3.amazonaws.com",
|
|
bucket=moto_s3["bucket"],
|
|
region=moto_s3["region"],
|
|
)
|
|
|
|
pack_body = b"fake pack body padded out " * 200
|
|
pack_hash = hash_leaf(pack_body).hex()
|
|
key = pack_key(pack_hash)
|
|
|
|
assert not backend.head(key)
|
|
backend.put_pack(pack_hash, pack_body)
|
|
assert backend.head(key)
|
|
assert backend.get_pack(pack_hash) == pack_body
|
|
|
|
keys = list(backend.list_keys(PACK_PREFIX))
|
|
assert keys == [key]
|
|
|
|
|
|
def test_s3_backend_identity_carries_endpoint_and_bucket_only(moto_s3):
|
|
backend = S3CompatibleBackend(
|
|
endpoint_url="https://nyc3.digitaloceanspaces.com",
|
|
bucket="arborist-do-test",
|
|
region="us-east-1",
|
|
)
|
|
identity = backend.identity
|
|
body = identity.to_audit_body()
|
|
assert body["endpoint_url"] == "https://nyc3.digitaloceanspaces.com"
|
|
assert body["bucket"] == "arborist-do-test"
|
|
assert "access_key" not in body
|
|
assert "secret" not in body
|