#000061: fold 3 gaps from Dav1d review (manifest/latest, license_class, cold_pending)
Dav1d's reviews of #000061 (Response A + Response B/FINAL in ~/Downloads, 2026-05-26) flagged a long list of items — most already shipped in the SPV-split work. Three were genuine gaps worth folding into #000061 before close: Gap 1: manifest/latest pointer for new-peer discovery. A fresh peer doing `cold list` got a list of metadata-pack hashes but no obvious "which one is current for shard X." Added get_latest_pointer + update_latest_pointer to the backend ABC. push_pack writes manifest/latest.json on every successful metadata pack push (read-modify-write keyed by snapshot_root). Mutable pointer; content addressing of the packs themselves preserves the trust root. Last-writer-wins on contention. Gap 2: license_class field + producer-side refuse for public buckets. Maps documents.source_type to a license bucket (wikipedia_cur / textbook_tex → public_redistributable; html / grok / vcs → unknown; anything else → unknown). Strictness order: public < unknown < private. compute_shard_license_class() walks DISTINCT source_type in documents. push_pack now refuses to upload if the shard's strictest license is more restrictive than the operator's allow_license_class (default: public_redistributable). The metadata pack's manifest carries _license_class so consumers / auditors can see the producer's classification without inspecting source documents. ValueError on refusal — the bucket ACL is the operator's call, but arborist refuses to participate in a licensing/membership leak unless explicitly opted in. Gap 3: cold_pending table for resumable uploads. Killed mid-upload, push_pack left orphan multi-GB tempfiles in /tmp with no DB trace. Added schema: CREATE TABLE cold_pending ( tempfile_path TEXT PRIMARY KEY, pack_hash TEXT NOT NULL, kind TEXT NOT NULL, backend_endpoint TEXT NOT NULL, backend_bucket TEXT NOT NULL, object_key TEXT NOT NULL, started_at INTEGER NOT NULL, state TEXT NOT NULL DEFAULT 'pending' ); push_pack INSERTs a row before each upload + DELETEs on success. A killed process leaves the row pointing at the orphan tempfile; a recovery script (future) reads cold_pending, checks bucket for the object, either deletes the row + tempfile (success was just unreported) or re-uploads from the tempfile if it still exists. Matches the same pattern as the audit chain — explicit state rows beat inferring from chunks.content IS NULL. Sibling tickets opened for the larger items the reviews flagged (scaffold-only, no code; opening them captures the design in the log without proliferating, per CLAUDE.md): - #000063 Cold-object private-ciphertext mode (mesh-keyed object keys for non-public corpora on public-read buckets). Needs mesh group-key ABI + real non-public corpus before code. - #000064 Cold-object operations toolkit (verify / diff / doctor / repair-fts / gc-plan CLI + expanded audit-event taxonomy). Bundled so the audit-event vocabulary gets one design pass. 5 new tests: test_gap2_license_gate_refuses_unknown_class_to_public_bucket test_gap2_license_class_in_metadata_manifest test_gap1_latest_pointer_resolves_metadata_pack_per_snapshot test_gap3_cold_pending_clears_on_successful_upload test_gap3_cold_pending_records_inflight_upload 26 cold-object + 7 evict tests pass (33/33 green incl. boto3 wire). Next ID bumped to 000065. Live v3 SPV corpus run (bmq47x6t3) completed cleanly during this work. Will report sizing + memory profile in the next message.
This commit is contained in:
parent
eba08beb61
commit
576cb0eeaf
8 changed files with 667 additions and 4 deletions
|
|
@ -26,8 +26,19 @@ from arborist.evict import (
|
|||
hydrate_from_metadata_pack,
|
||||
pull_chunk_pack,
|
||||
pull_metadata_pack,
|
||||
push_pack,
|
||||
push_pack as _push_pack,
|
||||
)
|
||||
|
||||
|
||||
def push_pack(*args, **kwargs):
|
||||
"""Test wrapper for evict.push_pack that opts into 'unknown'
|
||||
license_class. FakeSource uses source_type='html' which classifies
|
||||
as 'unknown', and the Gap-2 license gate refuses unknown-class
|
||||
pushes to the bucket by default. Tests here exercise pack
|
||||
semantics, not the license gate — that gets its own test below.
|
||||
"""
|
||||
kwargs.setdefault("allow_license_class", "unknown")
|
||||
return _push_pack(*args, **kwargs)
|
||||
from arborist.ingest import ingest_source
|
||||
from arborist.merkle import hash_leaf
|
||||
from arborist.source import Source
|
||||
|
|
@ -664,6 +675,112 @@ def test_push_pack_is_deterministic_across_runs(tmp_path):
|
|||
conn.close()
|
||||
|
||||
|
||||
def test_gap2_license_gate_refuses_unknown_class_to_public_bucket(tmp_path):
|
||||
"""Gap 2: push_pack refuses to ship a shard whose strictest source
|
||||
license_class is more restrictive than the operator's
|
||||
allow_license_class. Default allow=public_redistributable; html-source
|
||||
shards are 'unknown' by default → push must raise."""
|
||||
db = tmp_path / "license.db"
|
||||
conn = connect(db)
|
||||
backend = MemoryBackend()
|
||||
try:
|
||||
ingest_source(conn, FakeSource([_doc("html://x", LONG)]))
|
||||
# Default allow_license_class='public_redistributable' refuses.
|
||||
with pytest.raises(ValueError, match="license_class='unknown'"):
|
||||
_push_pack(conn, backend)
|
||||
# Explicit opt-in: allow_license_class='unknown' passes.
|
||||
result = _push_pack(conn, backend, allow_license_class="unknown")
|
||||
assert result["status"] == "pushed"
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_gap2_license_class_in_metadata_manifest(tmp_path):
|
||||
"""Gap 2: the metadata pack manifest carries _license_class so
|
||||
consumers + auditors can see the producer's classification."""
|
||||
db = tmp_path / "lc-manifest.db"
|
||||
conn = connect(db)
|
||||
backend = MemoryBackend()
|
||||
try:
|
||||
ingest_source(conn, FakeSource([_doc("html://x", LONG)]))
|
||||
result = push_pack(conn, backend) # uses test wrapper, allow='unknown'
|
||||
meta_manifest = backend.get_pack_manifest(
|
||||
result["metadata_pack_hash"], kind="metadata"
|
||||
)
|
||||
parsed = parse_manifest(meta_manifest)
|
||||
assert parsed.license_class == "unknown"
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_gap1_latest_pointer_resolves_metadata_pack_per_snapshot(tmp_path):
|
||||
"""Gap 1: after push, manifest/latest.json points to the current
|
||||
metadata pack for each snapshot_root. Fresh peer can resolve the
|
||||
metadata pack hash without enumerating every object."""
|
||||
db = tmp_path / "latest.db"
|
||||
conn = connect(db)
|
||||
backend = MemoryBackend()
|
||||
try:
|
||||
ingest_source(conn, FakeSource([_doc("html://x", LONG)]))
|
||||
result = push_pack(conn, backend)
|
||||
snapshot_root = result["snapshot_root"]
|
||||
metadata_hash = result["metadata_pack_hash"]
|
||||
|
||||
# Backend has the latest pointer.
|
||||
pointer = backend.get_latest_pointer()
|
||||
assert snapshot_root in pointer
|
||||
entry = pointer[snapshot_root]
|
||||
assert entry["metadata_pack_hash"] == metadata_hash
|
||||
assert entry["snapshot_doc_count"] == result["snapshot_doc_count"]
|
||||
assert entry["license_class"] == "unknown"
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_gap3_cold_pending_clears_on_successful_upload(tmp_path):
|
||||
"""Gap 3: cold_pending rows track in-flight uploads. On success,
|
||||
they're cleared. After a clean push, cold_pending is empty."""
|
||||
db = tmp_path / "pending.db"
|
||||
conn = connect(db)
|
||||
backend = MemoryBackend()
|
||||
try:
|
||||
ingest_source(conn, FakeSource([_doc("html://x", LONG)]))
|
||||
push_pack(conn, backend)
|
||||
# Successful push leaves no orphan pending rows.
|
||||
pending = conn.execute("SELECT COUNT(*) FROM cold_pending").fetchone()[0]
|
||||
assert pending == 0
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_gap3_cold_pending_records_inflight_upload(tmp_path):
|
||||
"""Gap 3: when the backend put_pack_file fails mid-flight, the
|
||||
cold_pending row stays so a recovery tool can find the orphan
|
||||
tempfile and clean it up."""
|
||||
db = tmp_path / "pending-fail.db"
|
||||
conn = connect(db)
|
||||
try:
|
||||
ingest_source(conn, FakeSource([_doc("html://x", LONG)]))
|
||||
|
||||
class FailingBackend(MemoryBackend):
|
||||
def put_file(self, key, path, *, content_type="application/octet-stream"):
|
||||
raise RuntimeError("simulated upload failure")
|
||||
|
||||
backend = FailingBackend()
|
||||
with pytest.raises(RuntimeError, match="simulated upload"):
|
||||
push_pack(conn, backend)
|
||||
# The row should exist tracking what was almost uploaded.
|
||||
pending_rows = conn.execute(
|
||||
"SELECT kind, pack_hash, object_key FROM cold_pending"
|
||||
).fetchall()
|
||||
assert len(pending_rows) >= 1
|
||||
# The recorded row identifies enough to either resume or clean up.
|
||||
assert pending_rows[0]["pack_hash"]
|
||||
assert pending_rows[0]["object_key"].startswith("packs/")
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_default_max_pack_bytes_is_dvdr_safe_fit():
|
||||
"""Default cap is 4.4 GB — DVD-R safe-fit, with ~6.5% buffer below the
|
||||
4.7 GB media spec to absorb ISO9660 overhead, media manufacturing
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue