engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000061: pack format v2 — self-sufficient new-peer hydration

Browse source 
v1 packs (chunks-only) were under-engineered: a new peer landing on
v1 packs would have chunk bodies indexed by leaf_hash but no documents
table, no audit chain, no merkle interior, no edges — couldn't actually
hydrate. fox: "isn't what I wanted you under engineered..."

v2 packs ship every load-bearing shard table alongside chunk bodies in
the same tar.zst:

  manifest.jsonl                          # chunk catalog (unchanged)
  tables/documents.jsonl                  # array-per-line columnar JSONL
  tables/chunks.jsonl                     # without content column
  tables/merkle_nodes.jsonl
  tables/edges.jsonl                      # FAN-IN restructured
  tables/audit_events.jsonl
  tables/derivations.jsonl
  tables/concept_relations.jsonl
  tables/concept_token_idf.jsonl
  tables/providence_cache.jsonl
  tables/citation_aliases.jsonl
  tables/term_aliases.jsonl
  tables/snapshots.jsonl
  tables/document_http_meta.jsonl
  blobs/<hash[:2]>/<hash[2:]>             # raw UTF-8 chunk bodies

Two compression strategies inside the pack:

1. Array-per-line JSONL ({"_columns": [...]} header line + ["v1","v2",...]
   data lines) drops ~30% of uncompressed bytes vs object-per-row JSONL.
   zstd recovers most of that on its own, but smaller uncompressed
   footprint also speeds up stream-restore.

2. Edges fan-in restructure at pack-build time: 22M rows of
   (src_root, edge_type, dst_root, dst_uri, anchor) → ~500k unique
   (dst_uri, edge_type, anchor, dst_root) groups with src_roots as an
   array. ~5-10x compressed savings on the dominant table. Reverses on
   unpack into the per-edge live schema. Live queries unchanged.

NOT shipped (per-peer state): mesh_*, selfmodel_*, capital_ledger,
memory_*, controller_events, fork_score_branches, adapter_loss_reports,
falsifications, schema_meta, meta. NOT shipped (rebuildable): chunks_fts*,
documents_fts* — restored from chunks.content + documents.title on
unpack.

push_pack no longer appends `cold_pack_pushed` to the audit chain.
That event leaked into the next push's audit_events.jsonl dump and
broke the "two writers at the same corpus state produce identical
pack_hash" determinism property. The bucket/disc file IS the receipt;
the snapshot_root pinned inside the pack metadata binds it to a corpus
state. No load-bearing consumer of the audit row.

pull_pack restored to handle both v1 (chunks-only) and v2 (tables +
chunks) packs. For v2 it extracts tables/*.jsonl to a temp dir,
calls restore_shard_metadata (which INSERT OR IGNOREs into the live
schema and expands edges back to per-edge rows), then fills chunk
content for every leaf_hash in blobs/. Idempotent against populated
DBs (INSERT OR IGNORE all the way down). Self-cleaning temp dir.

Sizing measured 2026-05-26: ~2.1 GB per shard pack compressed (chunk
content 1.78 GB + metadata ~0.3 GB), ~8.5 GB total across 4 shards.
~20% more than v1 chunks-only for self-sufficient hydration.

24 cold-object + evict tests pass (+1 new test_push_pack_v2_hydrates_fresh_empty_db
that builds a pack from a populated DB and unpacks into a completely
empty DB to verify all tables restored). Full suite: 2558 passed,
28 skipped, 1 xfailed.
This commit is contained in:
russell@unturf.com 2026-05-25 22:21:45 -04:00
parent 57f89894e2
commit 50324b4d7a
No known key found for this signature in database
6 changed files with 654 additions and 53 deletions
Download patch file Download diff file Expand all files Collapse all files

51
docs/cold-object-store.md
View file

@ -34,6 +34,31 @@ backup consumers download packs whole.
No `blobs/` prefix, no per-chunk objects. (One pack ↔ one disc ↔ one
bucket object.)
Pack contents (v2 format, self-sufficient for new-peer hydration):
```
manifest.ndjson # chunk leaf_hash + size catalog
tables/documents.jsonl # array-per-line, sorted
tables/chunks.jsonl # without content column
tables/merkle_nodes.jsonl
tables/edges.jsonl # FAN-IN restructured
tables/audit_events.jsonl
tables/derivations.jsonl
tables/concept_relations.jsonl
tables/concept_token_idf.jsonl
tables/providence_cache.jsonl
tables/citation_aliases.jsonl
tables/term_aliases.jsonl
tables/snapshots.jsonl
tables/document_http_meta.jsonl
blobs/<hash[:2]>/<hash[2:]> # raw UTF-8 chunk bodies
```
Not shipped (per-peer or rebuildable):
`mesh_*`, `selfmodel_*`, `capital_ledger`, `controller_events`,
`fork_score_branches`, `memory_*`, `adapter_loss_reports`,
`falsifications`, `schema_meta`, `meta`, `chunks_fts*`,
`documents_fts*`.
2. **`pack_hash = hash_leaf(manifest_bytes)`.** The manifest is sorted
by `leaf_hash` and deduped before hashing, so input order and
accidental duplicates don't move the hash. Two writers producing the
@ -215,16 +240,24 @@ For larger media:
## Cost model (DO Spaces, current corpus)
Numbers from the live shard estimator (4 shards × ~3.5M chunks each,
14.1M chunks total, ~17 GB compressed; streaming cap fills each pack
to ~4.4 GB compressed):
Pack format v2 (self-sufficient for new-peer hydration). Numbers measured
2026-05-26 against the live 4-shard corpus (14.1M total chunks; the
1.56M hot-content chunks per shard go into packs; metadata is added on
top via the v2 dump path):
| Path | Count | Storage | Cost |
|-----------------------------|-------------|----------|---------------------|
| Bucket pack storage | ~4 packs | ~17 GB | $0.34/mo (@ $0.02/GB) |
| Full-corpus hydrate (CDN) | ~4 GETs | — | ~$0.00002 in requests |
| Egress (in-region) | 0 | — | $0 |
| Egress (CDN to public) | 17 GB / peer | — | $0.17 per fresh peer (@ $0.01/GB) |
| Path | Count | Storage | Cost |
|-------------------------------|-------------|----------|-------------------------------|
| v2 pack storage (per shard) | 1 pack | ~2.1 GB | — |
| v2 pack storage (all 4) | 4 packs | ~8.5 GB | $0.17/mo (@ $0.02/GB) |
| Full-corpus hydrate (CDN) | ~4 GETs | — | ~$0.00002 in requests |
| Egress (in-region) | 0 | — | $0 |
| Egress (CDN to public) | 8.5 GB/peer | — | $0.09 per fresh peer (@ $0.01/GB) |
The v1 chunks-only format produced ~1.78 GB per shard (7.1 GB total).
v2 adds ~0.3–0.4 GB per shard for the metadata tables (chunks-meta,
documents, audit_events, merkle_nodes, edges fan-in restructured, plus
small tables). Trade: ~20 % more storage for a self-sufficient pack
that a fresh peer can unpack into a working shard with no other inputs.
Repacking after a falsification event costs the same as the initial
pack — one full corpus serialization per event-batched run, gated by