#000061: pack format v2 — self-sufficient new-peer hydration

v1 packs (chunks-only) were under-engineered: a new peer landing on
v1 packs would have chunk bodies indexed by leaf_hash but no documents
table, no audit chain, no merkle interior, no edges — couldn't actually
hydrate. fox: "isn't what I wanted you under engineered..."

v2 packs ship every load-bearing shard table alongside chunk bodies in
the same tar.zst:

  manifest.jsonl                          # chunk catalog (unchanged)
  tables/documents.jsonl                  # array-per-line columnar JSONL
  tables/chunks.jsonl                     # without content column
  tables/merkle_nodes.jsonl
  tables/edges.jsonl                      # FAN-IN restructured
  tables/audit_events.jsonl
  tables/derivations.jsonl
  tables/concept_relations.jsonl
  tables/concept_token_idf.jsonl
  tables/providence_cache.jsonl
  tables/citation_aliases.jsonl
  tables/term_aliases.jsonl
  tables/snapshots.jsonl
  tables/document_http_meta.jsonl
  blobs/<hash[:2]>/<hash[2:]>             # raw UTF-8 chunk bodies

Two compression strategies inside the pack:

1. Array-per-line JSONL ({"_columns": [...]} header line + ["v1","v2",...]
   data lines) drops ~30% of uncompressed bytes vs object-per-row JSONL.
   zstd recovers most of that on its own, but smaller uncompressed
   footprint also speeds up stream-restore.

2. Edges fan-in restructure at pack-build time: 22M rows of
   (src_root, edge_type, dst_root, dst_uri, anchor) → ~500k unique
   (dst_uri, edge_type, anchor, dst_root) groups with src_roots as an
   array. ~5-10x compressed savings on the dominant table. Reverses on
   unpack into the per-edge live schema. Live queries unchanged.

NOT shipped (per-peer state): mesh_*, selfmodel_*, capital_ledger,
memory_*, controller_events, fork_score_branches, adapter_loss_reports,
falsifications, schema_meta, meta. NOT shipped (rebuildable): chunks_fts*,
documents_fts* — restored from chunks.content + documents.title on
unpack.

push_pack no longer appends `cold_pack_pushed` to the audit chain.
That event leaked into the next push's audit_events.jsonl dump and
broke the "two writers at the same corpus state produce identical
pack_hash" determinism property. The bucket/disc file IS the receipt;
the snapshot_root pinned inside the pack metadata binds it to a corpus
state. No load-bearing consumer of the audit row.

pull_pack restored to handle both v1 (chunks-only) and v2 (tables +
chunks) packs. For v2 it extracts tables/*.jsonl to a temp dir,
calls restore_shard_metadata (which INSERT OR IGNOREs into the live
schema and expands edges back to per-edge rows), then fills chunk
content for every leaf_hash in blobs/. Idempotent against populated
DBs (INSERT OR IGNORE all the way down). Self-cleaning temp dir.

Sizing measured 2026-05-26: ~2.1 GB per shard pack compressed (chunk
content 1.78 GB + metadata ~0.3 GB), ~8.5 GB total across 4 shards.
~20% more than v1 chunks-only for self-sufficient hydration.

24 cold-object + evict tests pass (+1 new test_push_pack_v2_hydrates_fresh_empty_db
that builds a pack from a populated DB and unpacks into a completely
empty DB to verify all tables restored). Full suite: 2558 passed,
28 skipped, 1 xfailed.

tests/test_cold_object.py

@@ -299,6 +299,81 @@ def test_push_pack_local_dir_writes_files(tmp_path):
         conn.close()
 
 
+def test_push_pack_v2_hydrates_fresh_empty_db(tmp_path):
+    """Pack format v2 self-sufficiency test: build a pack from a populated
+    DB, then unpack it into a FRESH empty DB and verify every table is
+    restored (not just chunks content). This is the real new-peer
+    hydration path."""
+    src_db = tmp_path / "src.db"
+    rx_db = tmp_path / "rx.db"
+    conn_src = connect(src_db)
+    backend = MemoryBackend()
+    try:
+        # Populate the source DB with a few docs so multiple tables get
+        # rows: documents, chunks, merkle_nodes, audit_events at minimum.
+        # Distinct content per doc — same content dedupes by document_root.
+        docs = [
+            _doc(f"html://hydrate{i}", LONG + f"\n\nDoc-specific marker {i}." * 5)
+            for i in range(3)
+        ]
+        ingest_source(conn_src, FakeSource(docs))
+
+        src_docs = conn_src.execute(
+            "SELECT COUNT(*) FROM documents"
+        ).fetchone()[0]
+        src_chunks = conn_src.execute(
+            "SELECT COUNT(*) FROM chunks"
+        ).fetchone()[0]
+        src_merkle = conn_src.execute(
+            "SELECT COUNT(*) FROM merkle_nodes"
+        ).fetchone()[0]
+        src_audit = conn_src.execute(
+            "SELECT COUNT(*) FROM audit_events"
+        ).fetchone()[0]
+        assert src_docs == 3 and src_chunks > 0 and src_merkle > 0 and src_audit > 0
+
+        result = push_pack(conn_src, backend)
+        assert result["status"] == "pushed"
+        pack_hash = result["packs"][0]["pack_hash"]
+    finally:
+        conn_src.close()
+
+    # Brand-new empty receiving DB. No ingest, nothing — just `connect()`
+    # which creates the schema.
+    conn_rx = connect(rx_db)
+    try:
+        # Confirm rx is empty before unpack.
+        assert conn_rx.execute(
+            "SELECT COUNT(*) FROM documents"
+        ).fetchone()[0] == 0
+
+        pull_result = pull_pack(conn_rx, backend, pack_hash)
+        assert pull_result["status"] == "pulled"
+        assert pull_result["pack_format"] == "v2"
+        # Every table that was non-empty in src should have rows now in rx.
+        assert pull_result["tables_restored"]["documents"] == src_docs
+        assert pull_result["tables_restored"]["chunks"] == src_chunks
+        assert pull_result["tables_restored"]["merkle_nodes"] == src_merkle
+        assert pull_result["tables_restored"]["audit_events"] == src_audit
+        # Chunk bodies restored.
+        rx_hot = conn_rx.execute(
+            "SELECT COUNT(*) FROM chunks WHERE tier='hot' AND content IS NOT NULL"
+        ).fetchone()[0]
+        assert rx_hot == src_chunks
+        # FTS5 rows rebuilt.
+        rx_fts = conn_rx.execute("SELECT COUNT(*) FROM chunks_fts").fetchone()[0]
+        assert rx_fts == src_chunks
+        # All three documents reachable by URI.
+        for i in range(3):
+            row = conn_rx.execute(
+                "SELECT title FROM documents WHERE document_uri=?",
+                (f"html://hydrate{i}",),
+            ).fetchone()
+            assert row is not None, f"document html://hydrate{i} not restored"
+    finally:
+        conn_rx.close()
+
+
 def test_push_pack_no_push_writes_locally_only(tmp_path):
     """push_to_bucket=False skips bucket writes, requires local_dir."""
     db = tmp_path / "local-only.db"