Dav1d's reviews of #000061 (Response A + Response B/FINAL in
~/Downloads, 2026-05-26) flagged a long list of items — most already
shipped in the SPV-split work. Three were genuine gaps worth folding
into #000061 before close:
Gap 1: manifest/latest pointer for new-peer discovery.
A fresh peer doing `cold list` got a list of metadata-pack hashes
but no obvious "which one is current for shard X." Added
get_latest_pointer + update_latest_pointer to the backend ABC.
push_pack writes manifest/latest.json on every successful metadata
pack push (read-modify-write keyed by snapshot_root). Mutable
pointer; content addressing of the packs themselves preserves the
trust root. Last-writer-wins on contention.
Gap 2: license_class field + producer-side refuse for public buckets.
Maps documents.source_type to a license bucket (wikipedia_cur /
textbook_tex → public_redistributable; html / grok / vcs → unknown;
anything else → unknown). Strictness order: public < unknown <
private. compute_shard_license_class() walks DISTINCT source_type
in documents. push_pack now refuses to upload if the shard's
strictest license is more restrictive than the operator's
allow_license_class (default: public_redistributable). The
metadata pack's manifest carries _license_class so consumers /
auditors can see the producer's classification without inspecting
source documents. ValueError on refusal — the bucket ACL is the
operator's call, but arborist refuses to participate in a
licensing/membership leak unless explicitly opted in.
Gap 3: cold_pending table for resumable uploads.
Killed mid-upload, push_pack left orphan multi-GB tempfiles in
/tmp with no DB trace. Added schema:
CREATE TABLE cold_pending (
tempfile_path TEXT PRIMARY KEY,
pack_hash TEXT NOT NULL,
kind TEXT NOT NULL,
backend_endpoint TEXT NOT NULL,
backend_bucket TEXT NOT NULL,
object_key TEXT NOT NULL,
started_at INTEGER NOT NULL,
state TEXT NOT NULL DEFAULT 'pending'
);
push_pack INSERTs a row before each upload + DELETEs on success.
A killed process leaves the row pointing at the orphan tempfile;
a recovery script (future) reads cold_pending, checks bucket for
the object, either deletes the row + tempfile (success was just
unreported) or re-uploads from the tempfile if it still exists.
Matches the same pattern as the audit chain — explicit state
rows beat inferring from chunks.content IS NULL.
Sibling tickets opened for the larger items the reviews flagged
(scaffold-only, no code; opening them captures the design in the
log without proliferating, per CLAUDE.md):
- #000063 Cold-object private-ciphertext mode (mesh-keyed object
keys for non-public corpora on public-read buckets). Needs mesh
group-key ABI + real non-public corpus before code.
- #000064 Cold-object operations toolkit (verify / diff / doctor /
repair-fts / gc-plan CLI + expanded audit-event taxonomy).
Bundled so the audit-event vocabulary gets one design pass.
5 new tests:
test_gap2_license_gate_refuses_unknown_class_to_public_bucket
test_gap2_license_class_in_metadata_manifest
test_gap1_latest_pointer_resolves_metadata_pack_per_snapshot
test_gap3_cold_pending_clears_on_successful_upload
test_gap3_cold_pending_records_inflight_upload
26 cold-object + 7 evict tests pass (33/33 green incl. boto3 wire).
Next ID bumped to 000065.
Live v3 SPV corpus run (bmq47x6t3) completed cleanly during this work.
Will report sizing + memory profile in the next message.
Bidirectional sync: producer always emits both kinds; consumer chooses
how much to pull.
packs/<hash>.metadata.tar.zst — one per shard (~0.6 GB compressed)
packs/<hash>.metadata.manifest.ndjson
packs/<hash>.chunks.tar.zst — N per shard (each ≤ 4.4 GB cap)
packs/<hash>.chunks.manifest.ndjson
Each artifact is independently content-addressed by its own manifest
hash. The metadata pack's manifest carries _chunk_pack_hashes — every
chunk pack covering this shard's content — so a "full" consumer can
iterate them. Chunks packs are anonymous from the consumer side
(reachable only via the metadata pack's reference list).
Consumer sync modes:
arborist cold unpack <metadata_hash> # default: just-enough
arborist cold unpack <metadata_hash> --full # also pulls chunks
just-enough: pull only the metadata pack. Schema fully restored; every
chunks row has content=NULL. Node is immediately queryable for
metadata operations (documents, edges, audit chain, Merkle); chunk-body
queries return null until a future JIT-fetch path fills them on cache
miss.
full: after the metadata pack lands, iterate _chunk_pack_hashes and pull
every chunk pack. Final state: full corpus offline-queryable.
This is fox's original SPV-wallet framing — was right from day one.
Key API changes:
pack_key(hash, *, kind="chunks", manifest=False) — kind in bucket path
stream_packs(chunks, *, max_compressed_bytes, ...) -> Iterator[FilePack]
now emits chunk-only packs (no extra_members)
build_metadata_pack(table_files, *, snapshot_root, chunk_pack_hashes, ...)
single FilePack with v3 metadata manifest
pull_metadata_pack(conn, backend, hash) -> dict
pull_chunk_pack(conn, backend, hash) -> dict
hydrate_from_metadata_pack(conn, backend, metadata_hash, *, mode) -> dict
push_pack orchestrates: dump tables → stream chunk packs (collect hashes)
→ build metadata pack with chunk_pack_hashes → upload all. Returns
{metadata_pack_hash, chunk_pack_hashes, packs: [...]}.
Manifest format v3:
metadata pack:
{"_format_version": 3}
{"_kind": "metadata"}
{"_snapshot_root": "..."}
{"_chunk_pack_hashes": [...]}
{"table_file": "tables/X.jsonl", "hash": "...", "size": N} per table
chunks pack:
{"_format_version": 3}
{"_kind": "chunks"}
{"leaf_hash": "...", "size": N} per chunk
pack_hash for each = hash_leaf(manifest_bytes); content-addressed at
both layers. Two writers with the same shard state produce identical
metadata_pack_hash AND identical chunk_pack_hashes.
Cap-and-split applies only to chunk packs (chunks are bounded N).
Metadata pack is one file per shard; if a single table file is larger
than the cap, that's noted as future row-level split work.
cold list now surfaces kind per artifact (metadata vs chunks), plus
table_count + chunk_pack_hashes (for metadata packs) or chunk_count
(for chunks packs). Operators can quickly find the metadata pack hash
to feed `cold unpack`.
28 cold-object + evict + boto3 tests pass (3 new SPV-shape tests + 1
v3 manifest-shape test + tampered-metadata-pack test).
Caught a real defect on the live v2 corpus run: pack_hash was computed
from the chunk-only manifest, so two packs with the same chunk set but
different table content collided on pack_hash. Observed live: v1 packs
and v2 packs for the same shard produced the SAME pack_hash:
v1 (chunks-only) b3c427baaf9b40e3c33ace8438b4254e3abdc7b38bdcb7b1a8012588cb32848a
v2 (tables + chunks) b3c427baaf9b40e3c33ace8438b4254e3abdc7b38bdcb7b1a8012588cb32848a
↑ identical, different bytes
Consequences if shipped: bucket overwrites swap whose tables you get,
mesh peers A and B disagree on what "pack X" is while both claim to
have it, content-addressing story is broken for the metadata half.
Fix: manifest format v2 prepends a `{"_format_version": 2}` header,
then one row per shipped table:
{"table_file": "tables/audit_events.jsonl", "hash": "...", "size": N}
{"table_file": "tables/chunks.jsonl", "hash": "...", "size": N}
...
{"leaf_hash": "...", "size": N} # chunks come after
{"leaf_hash": "...", "size": N}
pack_hash = hash_leaf(manifest_bytes) now binds the full pack content.
Two writers with the same shard state produce the same pack_hash; same
chunks + different tables produce different pack_hashes.
Implementation:
- hash_file_leaf(path) — streaming sha256 with leaf 0x00 prefix, 1 MB
reads, used to fingerprint multi-GB tables/<name>.jsonl files without
loading them into RAM.
- stream_packs' extra_members now takes (member_name, source_path,
content_hash) tuples. push_pack computes the hash with hash_file_leaf
after the dump finishes.
- _finalize_pack writes the format header + table refs (sorted by name
for determinism) before chunk entries.
- parse_manifest returns a ParsedManifest (format_version, tables[],
chunks[]) instead of just list[PackEntry]. Backward-compatible with
v1 manifests (no header → format_version=1, tables=()).
- pull_pack pulls the manifest aside during tar walk, then after
extraction verifies each tables/<name>.jsonl file via hash_file_leaf
against the manifest's content_hash. Mismatch → ValueError, no rows
reach the live schema.
New tests:
- test_v2_pack_hash_binds_table_contents — asserts manifest format +
table refs + per-table hash & size.
- test_pull_pack_rejects_tampered_table — corrupts one table's bytes
in a real pack, confirms pull_pack raises on hash mismatch.
27 cold-object + evict tests pass.
Live v2 corpus run showed ~5 GB RSS per worker — RssAnon dominant, so
process heap, not mmap. Traced to two unfixed memory pits in the edges
fan-in dump path:
1. SQLite ORDER BY on edges (22M rows, no covering index for the v2
sort order dst_uri+edge_type+anchor) allocates a multi-GB in-memory
sort area before spilling. Adding:
CREATE INDEX IF NOT EXISTS idx_edges_dst_uri_type_anchor
ON edges(dst_uri, edge_type, anchor, dst_root, src_root)
means the ORDER BY walks the index in order — no in-memory sort.
First create takes ~30-60 s on a 22M-row shard; idempotent on
subsequent dumps. Disk cost ~1 GB per shard (4 shards × 1 GB ≈
2-3 % corpus footprint increase). Worth it.
2. Python groupby accumulator: src_roots = [row[4] for row in group]
materializes the entire src_root list per destination. For
en.wikipedia.org/wiki/* destinations with millions of inbound links,
this list is itself ~GB-sized. Switch to bounded batches:
_FAN_IN_BATCH = 10_000 # max src_roots per fan-in JSON row
A destination with N inbound links splits into ceil(N / batch) rows.
Restore path (INSERT OR IGNORE) handles multi-row destinations
correctly because PK includes src_root — accidental duplicates
collapse cleanly.
New regression test test_edges_fan_in_batches_huge_destinations builds
an edges table with FAN_IN_BATCH+137 rows pointing at one dst_uri,
verifies the dump produces the expected number of split rows and the
restore reconstructs all N edges with no loss or duplication.
25 cold-object + evict tests pass.
v1 packs (chunks-only) were under-engineered: a new peer landing on
v1 packs would have chunk bodies indexed by leaf_hash but no documents
table, no audit chain, no merkle interior, no edges — couldn't actually
hydrate. fox: "isn't what I wanted you under engineered..."
v2 packs ship every load-bearing shard table alongside chunk bodies in
the same tar.zst:
manifest.jsonl # chunk catalog (unchanged)
tables/documents.jsonl # array-per-line columnar JSONL
tables/chunks.jsonl # without content column
tables/merkle_nodes.jsonl
tables/edges.jsonl # FAN-IN restructured
tables/audit_events.jsonl
tables/derivations.jsonl
tables/concept_relations.jsonl
tables/concept_token_idf.jsonl
tables/providence_cache.jsonl
tables/citation_aliases.jsonl
tables/term_aliases.jsonl
tables/snapshots.jsonl
tables/document_http_meta.jsonl
blobs/<hash[:2]>/<hash[2:]> # raw UTF-8 chunk bodies
Two compression strategies inside the pack:
1. Array-per-line JSONL ({"_columns": [...]} header line + ["v1","v2",...]
data lines) drops ~30% of uncompressed bytes vs object-per-row JSONL.
zstd recovers most of that on its own, but smaller uncompressed
footprint also speeds up stream-restore.
2. Edges fan-in restructure at pack-build time: 22M rows of
(src_root, edge_type, dst_root, dst_uri, anchor) → ~500k unique
(dst_uri, edge_type, anchor, dst_root) groups with src_roots as an
array. ~5-10x compressed savings on the dominant table. Reverses on
unpack into the per-edge live schema. Live queries unchanged.
NOT shipped (per-peer state): mesh_*, selfmodel_*, capital_ledger,
memory_*, controller_events, fork_score_branches, adapter_loss_reports,
falsifications, schema_meta, meta. NOT shipped (rebuildable): chunks_fts*,
documents_fts* — restored from chunks.content + documents.title on
unpack.
push_pack no longer appends `cold_pack_pushed` to the audit chain.
That event leaked into the next push's audit_events.jsonl dump and
broke the "two writers at the same corpus state produce identical
pack_hash" determinism property. The bucket/disc file IS the receipt;
the snapshot_root pinned inside the pack metadata binds it to a corpus
state. No load-bearing consumer of the audit row.
pull_pack restored to handle both v1 (chunks-only) and v2 (tables +
chunks) packs. For v2 it extracts tables/*.jsonl to a temp dir,
calls restore_shard_metadata (which INSERT OR IGNOREs into the live
schema and expands edges back to per-edge rows), then fills chunk
content for every leaf_hash in blobs/. Idempotent against populated
DBs (INSERT OR IGNORE all the way down). Self-cleaning temp dir.
Sizing measured 2026-05-26: ~2.1 GB per shard pack compressed (chunk
content 1.78 GB + metadata ~0.3 GB), ~8.5 GB total across 4 shards.
~20% more than v1 chunks-only for self-sufficient hydration.
24 cold-object + evict tests pass (+1 new test_push_pack_v2_hydrates_fresh_empty_db
that builds a pack from a populated DB and unpacks into a completely
empty DB to verify all tables restored). Full suite: 2558 passed,
28 skipped, 1 xfailed.
New peers landing on a public bucket need to enumerate pack_hashes
before they can `arborist cold unpack <hash>`. `cold stats` only gives
a count; `cold list` returns the per-pack details:
- pack_hash (taken from the key)
- compressed_bytes (one HEAD round-trip via new object_size method)
- chunk_count (parsed from the small manifest sidecar; skippable
via --no-manifest for huge-bucket fast listing)
`cold stats` also now reports total bucket footprint (sums HEAD sizes).
Backend ABC gains `object_size(key) -> int | None` so both subcommands
get sizes without paying egress for the body. S3 impl uses HEAD;
MemoryBackend reads from the dict.
Verified live against DO Spaces NYC3: list-empty → push tiny pack →
list-with-manifest (pack_hash, size=5311 B, chunk_count=5) → list
--no-manifest → cold stats → cleanup.
23 passed in tests/test_cold_object.py + tests/test_evict.py.
Three improvements after the first DO Spaces smoke + bench:
1. ORDER BY c.leaf_hash on the chunk-selection SQL. Two writers running
cold pack against the same DB at the same snapshot now produce the
same pack_hashes — chunk-to-pack assignment is a function of (chunk
set, cap) and nothing else. Prerequisite for parallel per-shard pack
workers and for two replicas to converge on byte-identical bucket
state. Costs ~25% on build wall (real-bench 31s → 40s on 100k chunks)
due to sort over the leaf_hash index + documents JOIN; worth it.
New test pins the determinism property.
2. boto3 multipart upload via TransferConfig (8 MB threshold + 8 MB
parts + 10-way concurrency) on every put. Required anyway for packs
> 5 GB (DO Spaces single-PUT limit). Measured 5.5 MB/s → 9.0 MB/s
on 121 MB pack to DO Spaces NYC3 (1.6x; ceiling is closer to network
than to boto3 serialization).
3. Stream the SQL cursor in push_pack instead of fetchall(). At 14M
chunks × ~700 bytes/row the prior fetchall materialized ~10 GB of
Python heap before stream_packs ever ran. Cursor iteration bounds
memory by the in-progress pack (~few hundred MB at the 4.4 GB cap).
Full corpus extrapolation revises ~125 min (single-PUT) → ~104 min
(multipart, sequential per-shard). Real wins live in parallel per-shard
pack workers — deferred; the determinism work landed here is the
prerequisite.
22 passed in tests/test_cold_object.py + tests/test_evict.py.
Ship arborist corpus state to new peers and DVD-R archival via
point-in-time tar.zst packs. One artifact serves both channels —
bucket+CDN delivery and physical-media archival.
Bucket holds packs only. Pack key = hash_leaf(manifest_bytes), so same
chunk set on two writers produces the same pack_hash and upload is
idempotent. Each pack pins the corpus snapshot_root it covers in audit
+ result body — packs are delayed snapshots, not live mirrors;
falsifications between repacks produce new pack_hashes.
stream_packs runs streaming zstd over tarfile, peeking compressed-buffer
size after each chunk via FLUSH_BLOCK (preserves dictionary). Default
cap 4_400_000_000 — 4.4 GB DVD-R safe-fit, ~6.5% buffer below the
4.7 GB marketing capacity to absorb ISO9660 overhead, growisofs
lead-in/lead-out, media variance, and drive-edge refusal. Each disc
fills to ~4.4 GB recorded data, not the ~1.5 GB an uncompressed cap
produced.
One backend class (S3CompatibleBackend via boto3 + endpoint_url) covers
AWS S3, DO Spaces, R2, B2, GCS S3-interop, MinIO. Optional dep
[object-store] = boto3>=1.34; dev extras pull moto for the wire test.
Voyeur: credentials via AWS_ACCESS_KEY_ID/_SECRET_ACCESS_KEY env or
~/.aws/credentials, never printed; only endpoint URL + bucket name
surface in logs.
CLI: arborist cold {pack,unpack,stats}. Makefile: cold-pack,
cold-pack-dvd (local-dir output for growisofs), cold-unpack, cold-stats.
Sizing for current shards (14.1M chunks, ~17 GB compressed): ~4 packs
at the default cap, ~\$0.34/mo DO Spaces storage, ~\$0.0001/fresh-peer
hydrate.
Always-on raw-UTF-8 leaf store (per ticket "Hard invariants") deferred
— packs-only for now, backfill later.
2557 passed, 28 skipped, 1 xfailed.
