#000067 phase 2: 3rd "fts" pack kind for skip-rebuild hydrate
Each producer shard now optionally emits a THIRD pack alongside its
metadata and chunks packs: an "fts" pack containing the FTS5 shadow
tables (chunks_fts_data, chunks_fts_idx, chunks_fts_docsize,
chunks_fts_config + the documents_fts_* counterparts) packed as a
fresh SQLite file inside the tar so BLOB columns round-trip natively.
Consumer detects fts_pack_hashes in the metadata pack's manifest,
pulls each fts pack, ATTACHes the embedded sqlite, INSERTs every
shadow-table row into its target's empty shadow tables, and SKIPS
the local FTS rebuild entirely.
Producer side:
arborist/cold_object.py
+ PACK_KIND_FTS = "fts"
+ FTS_SHADOW_TABLES tuple (8 shadow tables)
+ build_fts_pack(src_db_path, ...)
creates a temp sqlite, applies SCHEMA_SQL (so destination
has FTS virtual tables → shadow tables auto-created), copies
every shadow-table row from src via cursor iteration, packs
the sqlite file into tar.zst
+ ParsedManifest.fts_pack_hashes
+ parse_manifest reads _fts_pack_hashes records
+ build_metadata_pack accepts fts_pack_hashes parameter and
writes the new manifest record
arborist/evict.py:push_pack
+ include_fts: bool = True parameter (CLI --no-fts opts out)
+ Phase B.5 emits the fts pack BEFORE Phase C (metadata pack)
so its hash can be referenced in the metadata manifest
Consumer side:
arborist/evict.py
+ _pull_fts_pack_into_targets() — pulls fts pack body, extracts
embedded sqlite, ATTACHes into each target, INSERT OR IGNORE
every shadow-table row. INSERT OR IGNORE protects against
rowid collisions on other targets that don't own these chunks.
+ hydrate_from_metadata_pack_routed iterates fts_pack_hashes in
full mode, calls _pull_fts_pack_into_targets per pack
+ _pull_pack_inner_routed returns fts_pack_hashes_referenced in
its result dict (mirrors chunk_pack_hashes_referenced)
CLI / Makefile:
arborist cold pack --no-fts (opt-out)
make cold-hydrate (auto-detects: if
chunks_fts_data is
already populated
on shard 000 after
unpack, skip the
rebuild post-pass)
make cold-hydrate HYDRATE_REBUILD_FTS=1 (force rebuild)
make cold-hydrate HYDRATE_REBUILD_FTS=0 (skip rebuild)
Schema:
cold_pending.kind CHECK extended to include 'fts'
pack_key() accepts kind="fts" → packs/<hash>.fts.tar.zst
Expected wall-time impact on the 3090 genesis bench:
with fts in packs: no rebuild step → ~5-10 min total wall
without fts: rebuild post-pass needed → ~15-20 min
Trade-off: ~30-50% larger bucket (FTS shadow data per shard) for
~70-90% faster consumer hydrate. Producer flips the trade via
--no-fts. The fts pack is optional in the manifest (empty list →
consumer falls back to rebuild) so old bucket data without fts
packs continues to work unchanged.
34 cold-unpack-routed + migrate + planner tests pass.
This commit is contained in:
parent
cfb5666ef5
commit
2c11435f7f
8 changed files with 3165 additions and 42 deletions
|
|
@ -1,35 +0,0 @@
|
|||
# COGS tweet — cost of a grounded answer (arborist)
|
||||
|
||||
Draft social post. Numbers are the measured claim_lattice figures from
|
||||
`docs/energy-cogs-benchmark.md` (n=30, $0.33/kWh, real GPUs: Hermes-8B on
|
||||
a 3090, Qwen-27B on a 4090). Hold the bigger "arbitrage / forcing
|
||||
function" framing until the value side is hardened (higher N + blinded
|
||||
SOTA judge) — see that report's §5.5 + §9.
|
||||
|
||||
## Main tweet
|
||||
|
||||
What does a *grounded* answer actually cost in GPU electricity? We
|
||||
measured it on real cards:
|
||||
|
||||
- arborist + Hermes-3-8B → **~9¢ per 1,000 answers**
|
||||
- arborist + Qwen-27B → **~16¢ per 1,000**
|
||||
|
||||
No reasoning chains (those burn 4–6×). Read the evidence cheap, write a
|
||||
short answer locked to a claim lattice, stop.
|
||||
|
||||
## Follow-up
|
||||
|
||||
And if an answer's already hot, it never even joins that 1,000 — it's a
|
||||
Merkle-bound cache hit that skips the GPU entirely. Zero joules, zero
|
||||
cents. You pay GPU only for *new* questions; the cached answers are free
|
||||
forever, and provably the same answer.
|
||||
|
||||
---
|
||||
|
||||
**Source numbers (claim_lattice, $0.33/kWh):**
|
||||
- Hermes-3-8B + substrate: $0.085 / 1,000 grounded answers (~9¢)
|
||||
- Qwen-27B + substrate: $0.158 / 1,000 (~16¢)
|
||||
- quote mode is cheaper (Hermes $0.070, Qwen $0.121) — less context prefilled
|
||||
- thinking/reasoning mode measured 4–6× the energy for the same answer
|
||||
- cache hit (Merkle-bound providence record) returns the answer with no
|
||||
LLM call → 0 GPU joules, does not increment the per-1k cost
|
||||
Loading…
Add table
Add a link
Reference in a new issue