cold pack: --jit-blobs mode for online JIT consumer flow
Replaces the batched chunk-pack phase with per-chunk content-addressed blob uploads to `blobs/<hash[:2]>/<hash[2:]>`. The metadata pack still ships (small, fast to restore), but consumers no longer have to pull multi-GB chunk packs to get queryable: `cold unpack --mode just-enough` + `ARBORIST_JIT_CHUNKS=1` fetches single chunks on cache miss. Producer (`_stream_jit_blobs` in evict.py): - ThreadPoolExecutor with bounded queue (workers*4) keeps memory flat across millions of chunks - HEAD-checks object_size for idempotent re-upload - Mutually exclusive with chunk packs — manifest's `chunk_pack_hashes` is empty in JIT mode (consumer reads that as "JIT-only") Consumer (`hydrate_doc_jit` in cold_clone.py + `_maybe_jit_hydrate` in qa/query.py): - Detects both content shapes that need JIT: NULL (Tier B raw-clone) and zeroblob placeholders (just-enough pack restore, per #53). Discriminator is first-byte = NUL — zstd-framed bodies start with 0x28, plain UTF-8 prose never has leading NUL. - Same placeholder filter applied to chunk-read sites in qa/query.py so partial hydrate doesn't surface zero-bytes content into the LLM context. Test (`TestJitBlobsPackMode` in tests/test_cold_unpack_routed.py): - End-to-end push → just-enough hydrate → JIT-fetch → content matches original byte-for-byte through `unpack_chunk`. Docs (cold-object-store.md): - Hard-invariant #1 updated: bucket holds packs by default; `blobs/` and `clones/` are opt-in prefixes for the JIT and Tier-A flows. - New "Three consumer modes" section: full-pack vs JIT-blobs vs raw-clone comparison table + operator decision tree.
This commit is contained in:
parent
9c747ad862
commit
d43714a503
6 changed files with 273 additions and 7 deletions
|
|
@ -24,15 +24,24 @@ backup consumers download packs whole.
|
|||
|
||||
## Hard invariants
|
||||
|
||||
1. **Bucket holds packs only.** Layout:
|
||||
1. **Bucket holds packs by default; `blobs/` is opt-in for JIT.** Layout:
|
||||
|
||||
```
|
||||
<bucket>/packs/<pack_hash>.tar.zst # pack body
|
||||
<bucket>/packs/<pack_hash>.manifest.ndjson # pack contents sidecar
|
||||
<bucket>/blobs/<hash[:2]>/<hash[2:]> # per-chunk body — opt-in,
|
||||
# populated only by
|
||||
# `cold pack --jit-blobs`
|
||||
# (online-JIT consumer flow)
|
||||
<bucket>/clones/<snapshot_id>/<NNN>.db # raw shard clones — opt-in,
|
||||
# populated only by
|
||||
# `cold stream-snapshot`
|
||||
# (Tier A raw-clone flow)
|
||||
```
|
||||
|
||||
No `blobs/` prefix, no per-chunk objects. (One pack ↔ one disc ↔ one
|
||||
bucket object.)
|
||||
The classic pack flow (DVD-burn channel) keeps one pack ↔ one disc ↔
|
||||
one bucket object. The two opt-in prefixes (`blobs/`, `clones/`) light
|
||||
up additional consumer flows; see "Three consumer modes" below.
|
||||
|
||||
Pack contents (v2 format, self-sufficient for new-peer hydration):
|
||||
```
|
||||
|
|
@ -144,6 +153,50 @@ Pack files are byte-identical between channels. A DVD burned from one
|
|||
local-dir pack and a CDN-fetched pack of the same content collide on
|
||||
`sha256sum`.
|
||||
|
||||
## Three consumer modes
|
||||
|
||||
A fresh peer has three ways to hydrate from the bucket; producer mode
|
||||
decides which the bucket supports.
|
||||
|
||||
| mode | producer | consumer | local disk | bucket bytes pulled | first-query latency |
|
||||
|------|---------------------|-------------------------------------------|-------------------|---------------------|---------------------|
|
||||
| **full pack** | `cold pack` (default) | `cold unpack --mode full` | full corpus (~35 GB) | metadata + every chunk pack | seconds (all data local) |
|
||||
| **JIT-blobs** | `cold pack --jit-blobs` | `cold unpack --mode just-enough` + queries with `ARBORIST_JIT_CHUNKS=1` | metadata-only (~27 GB at FTS+schema floor) | metadata + N×blob per query | one round-trip per chunk on first hit, cached after |
|
||||
| **raw clone (Tier A)** | `cold stream-snapshot` | `cold clone` | full corpus (~35 GB) | raw `.db` files via SQLite Backup API | seconds (all data local) |
|
||||
|
||||
**full-pack mode** is the canonical path: hardened, fully tested, FTS
|
||||
restorable or rebuildable. Use for genesis recovery of a self-hosting
|
||||
peer, DVD-archival workflows, and any consumer that wants the full
|
||||
corpus offline-queryable.
|
||||
|
||||
**JIT-blobs mode** trades steady-state download for fastest time-to-
|
||||
queryable. The metadata pack is small; chunk bodies stream in as
|
||||
queries access them. Good for ephemeral nodes, demo VMs, edge servers
|
||||
that only serve a subset of the corpus. The `blobs/<hash[:2]>/<hash[2:]>`
|
||||
prefix is content-addressed: a chunk that exists in two snapshots costs
|
||||
one bucket object, and producer reuploads are free idempotent no-ops.
|
||||
Failure mode is graceful — `hydrate_doc_jit` skips missing blobs and
|
||||
hash-mismatches rather than crashing the query.
|
||||
|
||||
**raw-clone mode** is the simplest. `cold stream-snapshot` runs the
|
||||
SQLite Backup API over each shard to a tempfile and PUTs the bytes; the
|
||||
consumer DOWNLOAD the `.db` files and is queryable immediately. No
|
||||
compression, no restore phase. Larger bucket footprint than the pack
|
||||
flow but skips the slow metadata-restore step entirely.
|
||||
|
||||
```bash
|
||||
# Producer (JIT-blobs mode):
|
||||
arborist cold pack --jit-blobs --shards-dir ~/.arborist/shards
|
||||
|
||||
# Consumer:
|
||||
arborist cold unpack --mode just-enough <metadata_pack_hash> \
|
||||
--hydrate-shards-dir ~/.arborist/shards --hydrate-M 4
|
||||
export ARBORIST_JIT_CHUNKS=1
|
||||
arborist query --shards-dir ~/.arborist/shards "what is X?"
|
||||
# → first hit on each chunk fetches blobs/<hash> from the bucket and
|
||||
# caches it back into the local row; subsequent reads are free.
|
||||
```
|
||||
|
||||
## DO Spaces quickstart
|
||||
|
||||
```bash
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue