#000061: fold 3 gaps from Dav1d review (manifest/latest, license_class, cold_pending)

Dav1d's reviews of #000061 (Response A + Response B/FINAL in
~/Downloads, 2026-05-26) flagged a long list of items — most already
shipped in the SPV-split work. Three were genuine gaps worth folding
into #000061 before close:

Gap 1: manifest/latest pointer for new-peer discovery.

  A fresh peer doing `cold list` got a list of metadata-pack hashes
  but no obvious "which one is current for shard X." Added
  get_latest_pointer + update_latest_pointer to the backend ABC.
  push_pack writes manifest/latest.json on every successful metadata
  pack push (read-modify-write keyed by snapshot_root). Mutable
  pointer; content addressing of the packs themselves preserves the
  trust root. Last-writer-wins on contention.

Gap 2: license_class field + producer-side refuse for public buckets.

  Maps documents.source_type to a license bucket (wikipedia_cur /
  textbook_tex → public_redistributable; html / grok / vcs → unknown;
  anything else → unknown). Strictness order: public < unknown <
  private. compute_shard_license_class() walks DISTINCT source_type
  in documents. push_pack now refuses to upload if the shard's
  strictest license is more restrictive than the operator's
  allow_license_class (default: public_redistributable). The
  metadata pack's manifest carries _license_class so consumers /
  auditors can see the producer's classification without inspecting
  source documents. ValueError on refusal — the bucket ACL is the
  operator's call, but arborist refuses to participate in a
  licensing/membership leak unless explicitly opted in.

Gap 3: cold_pending table for resumable uploads.

  Killed mid-upload, push_pack left orphan multi-GB tempfiles in
  /tmp with no DB trace. Added schema:

    CREATE TABLE cold_pending (
        tempfile_path TEXT PRIMARY KEY,
        pack_hash TEXT NOT NULL,
        kind TEXT NOT NULL,
        backend_endpoint TEXT NOT NULL,
        backend_bucket TEXT NOT NULL,
        object_key TEXT NOT NULL,
        started_at INTEGER NOT NULL,
        state TEXT NOT NULL DEFAULT 'pending'
    );

  push_pack INSERTs a row before each upload + DELETEs on success.
  A killed process leaves the row pointing at the orphan tempfile;
  a recovery script (future) reads cold_pending, checks bucket for
  the object, either deletes the row + tempfile (success was just
  unreported) or re-uploads from the tempfile if it still exists.
  Matches the same pattern as the audit chain — explicit state
  rows beat inferring from chunks.content IS NULL.

Sibling tickets opened for the larger items the reviews flagged
(scaffold-only, no code; opening them captures the design in the
log without proliferating, per CLAUDE.md):

- #000063 Cold-object private-ciphertext mode (mesh-keyed object
  keys for non-public corpora on public-read buckets). Needs mesh
  group-key ABI + real non-public corpus before code.

- #000064 Cold-object operations toolkit (verify / diff / doctor /
  repair-fts / gc-plan CLI + expanded audit-event taxonomy).
  Bundled so the audit-event vocabulary gets one design pass.

5 new tests:
  test_gap2_license_gate_refuses_unknown_class_to_public_bucket
  test_gap2_license_class_in_metadata_manifest
  test_gap1_latest_pointer_resolves_metadata_pack_per_snapshot
  test_gap3_cold_pending_clears_on_successful_upload
  test_gap3_cold_pending_records_inflight_upload

26 cold-object + 7 evict tests pass (33/33 green incl. boto3 wire).

Next ID bumped to 000065.

Live v3 SPV corpus run (bmq47x6t3) completed cleanly during this work.
Will report sizing + memory profile in the next message.
This commit is contained in:
russell@unturf.com 2026-05-26 10:50:57 -04:00
parent eba08beb61
commit 576cb0eeaf
No known key found for this signature in database
8 changed files with 667 additions and 4 deletions

View file

@ -26,8 +26,19 @@ from arborist.evict import (
hydrate_from_metadata_pack,
pull_chunk_pack,
pull_metadata_pack,
push_pack,
push_pack as _push_pack,
)
def push_pack(*args, **kwargs):
"""Test wrapper for evict.push_pack that opts into 'unknown'
license_class. FakeSource uses source_type='html' which classifies
as 'unknown', and the Gap-2 license gate refuses unknown-class
pushes to the bucket by default. Tests here exercise pack
semantics, not the license gate — that gets its own test below.
"""
kwargs.setdefault("allow_license_class", "unknown")
return _push_pack(*args, **kwargs)
from arborist.ingest import ingest_source
from arborist.merkle import hash_leaf
from arborist.source import Source
@ -664,6 +675,112 @@ def test_push_pack_is_deterministic_across_runs(tmp_path):
conn.close()
def test_gap2_license_gate_refuses_unknown_class_to_public_bucket(tmp_path):
"""Gap 2: push_pack refuses to ship a shard whose strictest source
license_class is more restrictive than the operator's
allow_license_class. Default allow=public_redistributable; html-source
shards are 'unknown' by default → push must raise."""
db = tmp_path / "license.db"
conn = connect(db)
backend = MemoryBackend()
try:
ingest_source(conn, FakeSource([_doc("html://x", LONG)]))
# Default allow_license_class='public_redistributable' refuses.
with pytest.raises(ValueError, match="license_class='unknown'"):
_push_pack(conn, backend)
# Explicit opt-in: allow_license_class='unknown' passes.
result = _push_pack(conn, backend, allow_license_class="unknown")
assert result["status"] == "pushed"
finally:
conn.close()
def test_gap2_license_class_in_metadata_manifest(tmp_path):
"""Gap 2: the metadata pack manifest carries _license_class so
consumers + auditors can see the producer's classification."""
db = tmp_path / "lc-manifest.db"
conn = connect(db)
backend = MemoryBackend()
try:
ingest_source(conn, FakeSource([_doc("html://x", LONG)]))
result = push_pack(conn, backend) # uses test wrapper, allow='unknown'
meta_manifest = backend.get_pack_manifest(
result["metadata_pack_hash"], kind="metadata"
)
parsed = parse_manifest(meta_manifest)
assert parsed.license_class == "unknown"
finally:
conn.close()
def test_gap1_latest_pointer_resolves_metadata_pack_per_snapshot(tmp_path):
"""Gap 1: after push, manifest/latest.json points to the current
metadata pack for each snapshot_root. Fresh peer can resolve the
metadata pack hash without enumerating every object."""
db = tmp_path / "latest.db"
conn = connect(db)
backend = MemoryBackend()
try:
ingest_source(conn, FakeSource([_doc("html://x", LONG)]))
result = push_pack(conn, backend)
snapshot_root = result["snapshot_root"]
metadata_hash = result["metadata_pack_hash"]
# Backend has the latest pointer.
pointer = backend.get_latest_pointer()
assert snapshot_root in pointer
entry = pointer[snapshot_root]
assert entry["metadata_pack_hash"] == metadata_hash
assert entry["snapshot_doc_count"] == result["snapshot_doc_count"]
assert entry["license_class"] == "unknown"
finally:
conn.close()
def test_gap3_cold_pending_clears_on_successful_upload(tmp_path):
"""Gap 3: cold_pending rows track in-flight uploads. On success,
they're cleared. After a clean push, cold_pending is empty."""
db = tmp_path / "pending.db"
conn = connect(db)
backend = MemoryBackend()
try:
ingest_source(conn, FakeSource([_doc("html://x", LONG)]))
push_pack(conn, backend)
# Successful push leaves no orphan pending rows.
pending = conn.execute("SELECT COUNT(*) FROM cold_pending").fetchone()[0]
assert pending == 0
finally:
conn.close()
def test_gap3_cold_pending_records_inflight_upload(tmp_path):
"""Gap 3: when the backend put_pack_file fails mid-flight, the
cold_pending row stays so a recovery tool can find the orphan
tempfile and clean it up."""
db = tmp_path / "pending-fail.db"
conn = connect(db)
try:
ingest_source(conn, FakeSource([_doc("html://x", LONG)]))
class FailingBackend(MemoryBackend):
def put_file(self, key, path, *, content_type="application/octet-stream"):
raise RuntimeError("simulated upload failure")
backend = FailingBackend()
with pytest.raises(RuntimeError, match="simulated upload"):
push_pack(conn, backend)
# The row should exist tracking what was almost uploaded.
pending_rows = conn.execute(
"SELECT kind, pack_hash, object_key FROM cold_pending"
).fetchall()
assert len(pending_rows) >= 1
# The recorded row identifies enough to either resume or clean up.
assert pending_rows[0]["pack_hash"]
assert pending_rows[0]["object_key"].startswith("packs/")
finally:
conn.close()
def test_default_max_pack_bytes_is_dvdr_safe_fit():
"""Default cap is 4.4 GB — DVD-R safe-fit, with ~6.5% buffer below the
4.7 GB media spec to absorb ISO9660 overhead, media manufacturing