cold pack: --jit-blobs mode for online JIT consumer flow

Replaces the batched chunk-pack phase with per-chunk content-addressed
blob uploads to `blobs/<hash[:2]>/<hash[2:]>`. The metadata pack still
ships (small, fast to restore), but consumers no longer have to pull
multi-GB chunk packs to get queryable: `cold unpack --mode just-enough`
+ `ARBORIST_JIT_CHUNKS=1` fetches single chunks on cache miss.

Producer (`_stream_jit_blobs` in evict.py):
- ThreadPoolExecutor with bounded queue (workers*4) keeps memory flat
  across millions of chunks
- HEAD-checks object_size for idempotent re-upload
- Mutually exclusive with chunk packs — manifest's `chunk_pack_hashes`
  is empty in JIT mode (consumer reads that as "JIT-only")

Consumer (`hydrate_doc_jit` in cold_clone.py + `_maybe_jit_hydrate` in
qa/query.py):
- Detects both content shapes that need JIT: NULL (Tier B raw-clone) and
  zeroblob placeholders (just-enough pack restore, per #53). Discriminator
  is first-byte = NUL — zstd-framed bodies start with 0x28, plain UTF-8
  prose never has leading NUL.
- Same placeholder filter applied to chunk-read sites in qa/query.py so
  partial hydrate doesn't surface zero-bytes content into the LLM context.

Test (`TestJitBlobsPackMode` in tests/test_cold_unpack_routed.py):
- End-to-end push → just-enough hydrate → JIT-fetch → content matches
  original byte-for-byte through `unpack_chunk`.

Docs (cold-object-store.md):
- Hard-invariant #1 updated: bucket holds packs by default; `blobs/`
  and `clones/` are opt-in prefixes for the JIT and Tier-A flows.
- New "Three consumer modes" section: full-pack vs JIT-blobs vs raw-clone
  comparison table + operator decision tree.
This commit is contained in:
russell@unturf.com 2026-05-30 07:19:01 -04:00
parent 9c747ad862
commit d43714a503
No known key found for this signature in database
6 changed files with 273 additions and 7 deletions

View file

@ -24,15 +24,24 @@ backup consumers download packs whole.
## Hard invariants
1. **Bucket holds packs only.** Layout:
1. **Bucket holds packs by default; `blobs/` is opt-in for JIT.** Layout:
```
<bucket>/packs/<pack_hash>.tar.zst # pack body
<bucket>/packs/<pack_hash>.manifest.ndjson # pack contents sidecar
<bucket>/blobs/<hash[:2]>/<hash[2:]> # per-chunk body — opt-in,
# populated only by
# `cold pack --jit-blobs`
# (online-JIT consumer flow)
<bucket>/clones/<snapshot_id>/<NNN>.db # raw shard clones — opt-in,
# populated only by
# `cold stream-snapshot`
# (Tier A raw-clone flow)
```
No `blobs/` prefix, no per-chunk objects. (One pack ↔ one disc ↔ one
bucket object.)
The classic pack flow (DVD-burn channel) keeps one pack ↔ one disc ↔
one bucket object. The two opt-in prefixes (`blobs/`, `clones/`) light
up additional consumer flows; see "Three consumer modes" below.
Pack contents (v2 format, self-sufficient for new-peer hydration):
```
@ -144,6 +153,50 @@ Pack files are byte-identical between channels. A DVD burned from one
local-dir pack and a CDN-fetched pack of the same content collide on
`sha256sum`.
## Three consumer modes
A fresh peer has three ways to hydrate from the bucket; producer mode
decides which the bucket supports.
| mode | producer | consumer | local disk | bucket bytes pulled | first-query latency |
|------|---------------------|-------------------------------------------|-------------------|---------------------|---------------------|
| **full pack** | `cold pack` (default) | `cold unpack --mode full` | full corpus (~35 GB) | metadata + every chunk pack | seconds (all data local) |
| **JIT-blobs** | `cold pack --jit-blobs` | `cold unpack --mode just-enough` + queries with `ARBORIST_JIT_CHUNKS=1` | metadata-only (~27 GB at FTS+schema floor) | metadata + N×blob per query | one round-trip per chunk on first hit, cached after |
| **raw clone (Tier A)** | `cold stream-snapshot` | `cold clone` | full corpus (~35 GB) | raw `.db` files via SQLite Backup API | seconds (all data local) |
**full-pack mode** is the canonical path: hardened, fully tested, FTS
restorable or rebuildable. Use for genesis recovery of a self-hosting
peer, DVD-archival workflows, and any consumer that wants the full
corpus offline-queryable.
**JIT-blobs mode** trades steady-state download for fastest time-to-
queryable. The metadata pack is small; chunk bodies stream in as
queries access them. Good for ephemeral nodes, demo VMs, edge servers
that only serve a subset of the corpus. The `blobs/<hash[:2]>/<hash[2:]>`
prefix is content-addressed: a chunk that exists in two snapshots costs
one bucket object, and producer reuploads are free idempotent no-ops.
Failure mode is graceful — `hydrate_doc_jit` skips missing blobs and
hash-mismatches rather than crashing the query.
**raw-clone mode** is the simplest. `cold stream-snapshot` runs the
SQLite Backup API over each shard to a tempfile and PUTs the bytes; the
consumer DOWNLOAD the `.db` files and is queryable immediately. No
compression, no restore phase. Larger bucket footprint than the pack
flow but skips the slow metadata-restore step entirely.
```bash
# Producer (JIT-blobs mode):
arborist cold pack --jit-blobs --shards-dir ~/.arborist/shards
# Consumer:
arborist cold unpack --mode just-enough <metadata_pack_hash> \
--hydrate-shards-dir ~/.arborist/shards --hydrate-M 4
export ARBORIST_JIT_CHUNKS=1
arborist query --shards-dir ~/.arborist/shards "what is X?"
# → first hit on each chunk fetches blobs/<hash> from the bucket and
# caches it back into the local row; subsequent reads are free.
```
## DO Spaces quickstart
```bash