engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

make: HYDRATE_MODE flag on cold-hydrate (just-enough vs full)

Browse source 
Two genesis paths the SPV-wallet design supports, now both exposed
via the same make target:

  make cold-hydrate HYDRATE_DIR=… HYDRATE_M=4 HYDRATE_MODE=full
    pulls metadata packs + every referenced chunk pack. Full corpus
    offline-queryable post-hydrate.

  make cold-hydrate HYDRATE_DIR=… HYDRATE_M=4 HYDRATE_MODE=just-enough
    pulls only metadata packs (~4 GB on a 37 GB corpus → ~10×
    bandwidth reduction). chunks rows land with content=NULL, every
    chunk-body query misses local cache and falls through to a
    future JIT-fetch path (cache miss → CDN / mesh / re-pull). The
    SPV-wallet headline shape.

Defaults to full for backward compatibility. The mode-flag validation
guards against typos at recipe time (invalid mode → exit 2 before
firing parallel boto3 traffic).

Bench target: measure both paths back-to-back on 3090. Just-enough
should bound the network phase only (no chunk-body fill, no FTS5
rebuild); full adds the chunk-restore phase that #000067's bench-
max round just optimized.
This commit is contained in:
russell@unturf.com 2026-05-26 18:12:25 -04:00
parent 676a6b5a94
commit 939d3ced78
No known key found for this signature in database
1 changed files with 10 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

14
Makefile
View file

@ -938,23 +938,29 @@ cold-pack-all: bootstrap ## parallel cold-pack, RAM-aware (override with COLD_PA
@echo ">> all shards packed; final bucket state:"
@$(ARBORIST) cold stats
cold-hydrate: bootstrap ## genesis a fresh peer from cloud, M-aware: pull every metadata pack from bucket in parallel and route into HYDRATE_DIR/00N.db × HYDRATE_M
cold-hydrate: bootstrap ## genesis a fresh peer from cloud, M-aware: pull every metadata pack and route into HYDRATE_DIR/00N.db × HYDRATE_M. HYDRATE_MODE=full (default) | just-enough
@if [ -z "$(HYDRATE_DIR)" ]; then echo "HYDRATE_DIR=<dir> required"; exit 2; fi
@if [ -z "$(HYDRATE_M)" ]; then echo "HYDRATE_M=<int> required (matches #000065 canonical M)"; exit 2; fi
@mkdir -p $(HYDRATE_DIR)
@$(COLD_PACK_PICK_JOBS); \
@MODE="$${HYDRATE_MODE:-full}"; \
case "$$MODE" in \
full) MODE_FLAG="--full" ;; \
just-enough) MODE_FLAG="--just-enough" ;; \
*) echo "HYDRATE_MODE=$$MODE invalid (full | just-enough)"; exit 2 ;; \
esac; \
$(COLD_PACK_PICK_JOBS); \
echo ">> discovering metadata packs in bucket"; \
hashes=$$($(ARBORIST) cold list --no-manifest \
| $(VENV)/bin/python -c \
"import sys, json; d=json.load(sys.stdin); print(' '.join(p['pack_hash'] for p in d['packs'] if p['kind']=='metadata'))"); \
if [ -z "$$hashes" ]; then echo "no metadata packs in bucket"; exit 3; fi; \
echo ">> $$JOBS-way parallel pull of metadata packs (M=$(HYDRATE_M))"; \
echo ">> $$JOBS-way parallel pull of metadata packs (M=$(HYDRATE_M), mode=$$MODE)"; \
printf "%s\n" $$hashes | \
xargs -n 1 -P $$JOBS -I {} \
$(ARBORIST) cold unpack {} \
--hydrate-shards-dir $(HYDRATE_DIR) \
--hydrate-M $(HYDRATE_M) \
--full
$$MODE_FLAG
@echo ">> hydration complete:"
@ls -lh $(HYDRATE_DIR)/*.db 2>/dev/null