# Ticket #000051 — Federated vecpack distribution (gossip the embedding backfill)

**Status:** open · awaiting go/no-go — doc-only scaffold. Removes the
"who runs the days-long embed job" friction from #000050's gate by
making `chunk_vecs` a *distributable artifact*: backfill once on any
box with CPU (a cloud node, a Prometheus-Σ sweep box — #000037 §3.1),
publish a **vecpack** over the existing mesh wire layer, every peer
pulls + bulk-loads it. The transformer never runs on a laptop.
**Opened:** 2026-05-12
**Scope:** A serialize/verify/load path for per-shard embedding sets,
plus a mesh payload type carrying them. A vecpack is
`(shard_root, vec_backend_version, [(leaf_hash, embedding_blob), …])`
— content-addressed by `leaf_hash` (the chunk's hard hash), keyed in
transit on `leaf_hash` not `chunk_id` (`chunk_id` is a shard-local
rowid; `leaf_hash` is portable). A receiving node maps
`leaf_hash → local chunk_id` against its own `chunks` table and
bulk-INSERTs into `chunk_vecs` — sub-millisecond per chunk, the cheap
side of the #000039 §14.6 asymmetry. Storage cost is real (~384 B/chunk
at int8sym, ~2.4 GB for the 6.24M-chunk wiki set, ~6% over the shards)
but it is a *download + disk* cost, never a *compute* cost, on the
receiver.
**Audience:** fox + future blackops shifts + anyone running a public
arborist node that wants the semantic layer without an accelerator.
**Hard constraint:** vecpack payloads are **soft data** — embeddings
are `audit_mode=UNGROUNDED`, never in the proof path (#000039 §4,
CLAUDE.md "soft hash vs hard hash"). A wrong or adversarial gossiped
embedding can only nudge retrieval recall on the receiver; it cannot
corrupt a `cache_key`, a `document_root`, or an `audit_event_hash`.
**So vecpacks do not need Merkle-proof-grade verification** — a cheap
structural sanity gate is sufficient and correct (see §3). The `cache_key`
8-dim invariant is untouched; the vec config still folds into
`governance_policy_hash` per #000050 (this ticket distributes the
vectors, #000050 makes a QA answer that *consulted* them
policy-hash-honest).

---

## 1. Why this is a separate ticket

#000050's prerequisite #1 — "a corpus backfill exists" — was written
as a batch job *somebody runs*, with fox's laptop implicitly the
somebody. It is not: the embed pass is ~10–100× heavier than the rest
of ingest (#000039 §14.6) and a full-corpus backfill is hours-on-idle
/ days-on-contended. fox does not have an accelerator and should never
need one. The fix is distribution, not faster local embedding:

```
backfill once, anywhere with CPU
  → publish a vecpack per shard over mesh
  → every peer pulls + bulk-loads it
  → laptop pays download + sqlite INSERTs only
```

That is a distinct mechanism from #000050's RRF fusion (which is about
*using* `chunk_vecs` once present) and from #000039's backend (which
is about *building* it locally), so it gets its own ticket per repo
convention rather than bloating either.

This is also the concrete answer to the whitepaper §1 claim that "the
embedding pass runs off the device" — gossip is *how* it reaches the
device. The two strengthen each other.

## 2. Why vec data gossips well (the trust-model argument)

An embedding is a pure function of `(chunk content bytes, embedder
config)`. Two consequences:

1. **Content-addressable.** `embedding = f(leaf_content,
   vec_backend_version)`. A vecpack keyed on `(leaf_hash,
   vec_backend_version)` is the natural unit — a peer either has the
   chunk (knows its `leaf_hash`) or doesn't; if it does, the embedding
   slots straight in. No `chunk_id` alignment problem (those are
   shard-local rowids and must NOT be trusted across DBs even when two
   shards were built from the same dump).
2. **Soft by construction → cheap to verify.** Embeddings never enter
   a proof. The worst a bad vecpack does is degrade *recall* on the
   receiver — and even that is bounded, because the post-retrieval
   filter chain (title-relevance, rivalry exclusion, Rule-8
   title-mismatch, and the verifier itself) still runs on every
   candidate vec surfaces. So a vecpack does **not** need a Merkle
   inclusion proof or a signed-by-the-corpus-author guarantee. It
   needs: (a) the chunks it references actually exist locally with
   matching `leaf_hash`, (b) each blob is the right length for the
   declared `(dim, quant)`, (c) each decoded vector has a finite,
   sane norm, (d) `vec_backend_version` matches the receiver's
   `vec_meta` (or the receiver is creating `vec_meta` fresh from this
   pack). That is a cheap structural gate, not cryptographic
   verification. (Contrast: cache records gossiped over mesh *are*
   proof-bearing and *do* carry the full audit-chain machinery —
   vecpacks deliberately don't, and that's the right call.)

This is the same hard/soft discipline as #000035 (φ_PRG bytes pinned)
and #000036 (soft channels bounded, never confused with hard proof):
the layer that's safe to distribute loosely is exactly the layer that
can't hurt you if it's wrong.

## 3. Vecpack format (sketch)

```
vecpack-v1:
    shard_root            # the shard this pack embeds (sanity-bind, not a proof)
    vec_backend_version   # e.g. vec-v1-bge-small-en-v1.5-384int8sym-l2norm-flat
    dim                   # 384
    quant                 # "int8sym" | "float32"
    metric                # "l2norm"  (cosine-equivalent for unit vectors)
    count                 # number of (leaf_hash, blob) entries
    entries: [ (leaf_hash: 32 bytes, embedding_blob: dim*bytes_per_elem) , … ]
```

Wire it as a mesh payload (`arborist/mesh/wire.py` `WireEnvelope`
already does signed/AEAD-wrapped bodies — a vecpack is just another
body type; the signature authenticates the *sender*, not the
*correctness* of the embeddings, which is fine because they're soft).
Large packs chunk into multiple envelopes keyed by `(shard_root,
vec_backend_version, part_i, n_parts)`.

Receiver algorithm:

```
on vecpack(shard_root, vec_backend_version, dim, quant, metric, entries):
    if local shard_root != shard_root:                  -> reject (wrong shard)
    if chunk_vecs exists and vec_meta.backend_version != vec_backend_version:
                                                        -> reject (rebuild needed) or stage separately
    if not chunk_vecs: ensure_chunk_vecs_table(quant=quant)  # creates vec_meta from this pack
    for (leaf_hash, blob) in entries:
        cid = SELECT chunk_id FROM chunks WHERE leaf_hash = ?   # local rowid
        if cid is None:            continue                      # don't have this chunk; skip
        if len(blob) != dim * bytes_per_elem(quant): skip + count_bad
        if not finite_sane_norm(decode(blob, quant)): skip + count_bad
        stage (cid, wrap(blob, quant))                           # vec_f32(?) / vec_int8(?)
    bulk delete-then-insert staged rows into chunk_vecs
    emit status: {received, loaded, skipped_missing_chunk, skipped_bad, backend_version}
```

Idempotent: re-loading the same vecpack is a no-op (delete-then-insert
on the same `chunk_id` with the same blob). Incremental: a partial
vecpack (some shards, some chunks) loads what it can; the rest stay
FTS5-only until another pack arrives — the mixed-fleet case #000050 §3
already has to handle.

## 4. Producer side

```
arborist vecpack export --shard <db> [--out path] [--max-bytes N (split)]
    # SELECT leaf_hash, embedding FROM chunks JOIN chunk_vecs USING(chunk_id)
    # → vecpack-v1 file(s); records vec_backend_version + dim + quant from vec_meta
arborist vecpack import --shard <db> --in path
    # the receiver algorithm above; prints the status dict
```

Mesh-native variant (when mesh is enabled): a node advertises which
`(shard_root, vec_backend_version)` packs it serves; a peer requests
the ones it lacks; transfer rides the existing wire layer. The
Prometheus-Σ unconscious-sweep task (#000037 §3.1) is the natural
producer — it backfills `chunk_vecs` off-peak *and* exports the pack
*and* answers pull requests, all as one low-priority background role.

**GPU producer (the fast path).** `bge-small-en-v1.5` is a 33M-param
model; on a CUDA box (a 4090 or similar) with batched inference (batch
256–512) embedding throughput is ~10³–10⁴ chunks/s — tokenization /
data-feeding bound, not matmul bound — so the full 6.24M-chunk wiki
set is **minutes of GPU time, not days of CPU**. The producer is just
`arborist embed --quant int8` run with a CUDA `Embedder`: `fastembed`'s
`TextEmbedding(..., providers=["CUDAExecutionProvider"])`, or
`sentence-transformers` `.encode(..., device="cuda", batch_size=512)`
— both drop into `default_embedder()`'s pluggable callable (#000039
§14.6). int8 quantization stays on the CPU side in `_to_blob` (cheap,
post-embed) so the GPU only produces float32 vectors; the rest of the
pipeline is byte-for-byte the same. The CUDA stack (`torch` /
`onnxruntime-gpu`) lives **only on the producer box** — an optional
toolchain in a sibling repo / extra, never in arborist's
`python3.12 + sqlite3` core; a fresh laptop checkout needs only the
`[vec]` extra to *import* a pack. Determinism note: a GPU backfill is
**not** byte-identical to a CPU one (matmul order differs) — irrelevant
here, embeddings are soft data (`UNGROUNDED`, never proof path) and two
independent backfills are interchangeable; pin the model revision +
quant recipe in `vec_backend_version`, not the output bytes (the same
"don't byte-KAT the embedder" call the #000050 review made — contrast
#000035's φ_PRG, which *is* byte-KAT'd because it's hard-channel).

## 5. Out of scope

- The local embed backend + `arborist embed` itself (#000039 — done).
- RRF hybrid fusion in `query.py` (#000050).
- A GPU embedder (orthogonal — #000039 §14.6; a faster producer, same
  vecpack output).
- Strong/cryptographic verification of embedding *correctness* (a peer
  can re-embed a sample and spot-check if it wants higher assurance,
  but that's an optional belt-and-suspenders, not part of v1 — the
  soft-data argument in §2 is why it isn't required).
- A content-distribution network / torrent-style fan-out for very
  large packs (if the 2.4 GB wiki pack ever needs it; v1 is
  point-to-point over mesh + a plain file export).

## 6. Acceptance criteria

1. `vecpack-v1` serialize/parse round-trips (`leaf_hash`-keyed,
   `vec_backend_version`/`dim`/`quant`/`metric` in the header).
2. `arborist vecpack export` / `import` work file-to-file across two
   independently-built copies of the same shard (chunk_ids differ,
   leaf_hashes match → embeddings land correctly).
3. The receiver's structural sanity gate rejects: wrong `shard_root`,
   `backend_version` mismatch against existing `vec_meta`, wrong blob
   length, non-finite vectors — each with a distinct status field, no
   silent acceptance.
4. Import is idempotent and incremental (partial packs load what they
   can; missing chunks counted, not errored).
5. Mesh payload type wired into `arborist/mesh/wire.py` (or a thin
   sibling), signed-by-sender, AEAD-wrapped like other bodies; large
   packs split/reassemble by part index.
6. Tests: round-trip; cross-shard leaf_hash mapping; each sanity-gate
   rejection path; idempotent re-import; mesh envelope carry; the
   soft-data boundary (a deliberately-corrupted blob is rejected by
   the gate, and even if it weren't, it's `UNGROUNDED` and the
   post-retrieval filters still run — i.e. a unit test asserting a bad
   vec hit can't reach the proof path).
7. Docs: a short section in `docs/mesh.md` (vecpack as a payload type)
   + a pointer from #000050 §2 prereq #1 ("backfill once, distribute
   via #000051").

## 6a. Review amendments — 2026-05-13 (Dav1d de-novo review)

Fold these into the v1 spec before implementation. They do **not**
change the architecture (soft sidecar, `leaf_hash`-keyed, cheap
validation, never proof path) — they sharpen it.

1. **Narrower "harmless" wording.** A bad/adversarial vecpack cannot
   ground a claim, mutate hard roots, or forge an audit proof *if the
   hard/soft boundary holds* — but it is **not** "only a recall nudge".
   It can also bias candidate ranking, waste top-K on junk (hiding good
   candidates), inflate latency/context, and skew bench conclusions —
   i.e. a *denial-of-retrieval* surface. So: no proof-grade
   embedding-*correctness* verification; **yes** structural,
   compatibility, provenance, trust-tier, and import-integrity checks.

2. **Manifest + roots (integrity, not correctness).** A vecpack carries
   a canonical-JSON **manifest** — `format_version` (`vecpack-v1`),
   `vecpack_id = sha256(canonical manifest)`, `shard_root`,
   `vec_backend_version`, full `backend_config` (see #3), `count`,
   `entry_root`, `part_count`, `compression`, `producer_id`,
   `created_at`. Each row has a deterministic
   `row_hash = H(leaf_hash || vec_backend_version || quant_recipe ||
   embedding_blob)`; `entry_root = MerkleRoot(row_hashes)`. This is
   **transit integrity** ("did the bytes arrive unchanged?"), *not*
   correctness ("was this embedding actually computed by the declared
   model?") — #000051 needs the former, never the latter. Audit events
   cite `manifest_root` / `entry_root`, not millions of row hashes.

3. **`vec_backend_version` must commit the full reproducible input.**
   "embedding = f(chunk bytes, embedder config)" is under-specified.
   `backend_config` (in the manifest, summarized by the version token)
   must pin: `embedder_name`, `embedder_revision` (digest, not just a
   tag), `tokenizer_revision`, `preprocessing_policy` (which prose
   normalization — `chunk_text_base@v1`), `pooling_policy`,
   `normalization_policy` (`l2_normalize@v1`), `quantization_recipe`
   (`int8sym127_global_clamp_round@v1` — *named*, not bare `int8`),
   `dimension`, `dtype`, `distance_metric`, `ann_index`. The compact
   token (`vec-v1-bge-small-en-v1.5-384int8sym127-l2norm-flat`) is a
   derived summary; the manifest fields are the auditable payload.
   (Producer profile CPU-vs-GPU is *not* pinned — outputs aren't
   byte-stable across it and don't need to be; soft data.)

4. **Trust tiers.** Tier 0 = local file / unknown producer (import OK
   for raw vec search if structural checks pass; `vecpack_trust =
   untrusted`; **no cached hybrid QA**). Tier 1 = integrity-checked
   mesh sender (sender authenticated, manifest/roots check out; raw +
   experimental hybrid; cached QA still gated on #000050 policy).
   Tier 2 = trusted producer (identity accepted by local policy;
   eligible for cached hybrid QA once #000050 governance-hash wiring is
   live). Tier 3 = locally generated (strongest operational trust;
   still soft data, still `UNGROUNDED`). #000050 decides which tiers a
   *cached* hybrid answer may rest on. Trust tier is **audit metadata,
   never proof identity** — a trusted vecpack saying "this chunk is
   semantically close" is still just a *candidate proposal*; the
   verifier must inspect the text.

5. **Streaming, resumable import — never load 2.4 GB into RAM.**
   Manifest-first validation → shard/backend compat check → disk-budget
   check → stream entries part-by-part → stage valid rows in batches
   (1k–10k rows) → commit in bounded transactions (WAL, `executemany`,
   no per-row commits) → record progress → resume-or-rollback after
   interruption (idempotent: re-import is a no-op).

6. **Bad-row thresholds + duplicate policy.** `bad_row_rate == 0` →
   clean; `0 < rate ≤ 0.01%` → import valid rows, warn;
   `rate > 1%` → reject / quarantine the pack. `missing_chunk` is fine
   for partial packs (record `coverage_ratio`). `duplicate_leaf_hash`:
   identical blob → skip silently; conflicting blob → reject the pack
   (or row, by policy). Thresholds are policy knobs.

7. **Exception matrix (deterministic status for every failure).**
   Wrong `shard_root` → `reject_wrong_shard_root`; missing `chunk_vecs`
   → `created_vec_meta`; existing `vec_meta` mismatch →
   `reject_backend_mismatch`; unsupported quant →
   `reject_unsupported_quant`; metric/quant incompat →
   `reject_metric_quant_mismatch`; bad blob length →
   `skipped_bad_length`; non-finite norm → `skipped_bad_norm`; norm
   out of sane range → `skipped_bad_norm_range`; missing local chunk →
   `skipped_missing_chunk`; dup identical → `duplicate_identical`; dup
   conflict → `duplicate_conflict`; mesh sig fail →
   `reject_signature_fail`; AEAD fail → `reject_aead_fail`; missing
   part → `incomplete_parts`; part-hash mismatch →
   `reject_part_hash_mismatch`; disk budget exceeded →
   `reject_insufficient_disk`; interrupted → `import_interrupted`;
   bad-row rate over threshold → `reject_bad_row_rate`; unknown
   producer → `untrusted_producer`. **Cache impact is `None` on every
   row** — vecpack import never touches `providence_cache`.

8. **Soft audit events.** Import/export/mesh-transfer write soft
   provenance events: `vecpack_export_{started,completed}`,
   `vecpack_import_{started,verified,completed,rejected,interrupted}`,
   `vecpack_part_{received,rejected}`,
   `vecpack_mesh_{announced,requested,served}` — each with
   `{vecpack_id, manifest_root, shard_root, vec_backend_version,
   trust_tier, counts:{received,loaded,skipped,bad}}`. Soft events,
   not proof events.

9. **CLI surface.** Add to §4's `export`/`import`:
   `arborist vecpack inspect --in path` (manifest only);
   `vecpack verify --shard <db> --in path` (all structural checks, no
   insert); `vecpack import --dry-run` (report expected status);
   `vecpack import --replace` (explicit rewrite, same backend only —
   never silently overwrite a *different* backend's vectors);
   `vecpack import --trust-policy {reject-unknown,untrusted-ok,trusted-only}`.

10. **SQD discipline now, π* module later.** Don't block on a
    `vecpack@v1` π* canonicalizer — but the manifest must already be
    canonical bytes: sorted keys, explicit scalar types, no implicit
    defaults in hash-bearing fields, versioned quant recipe, stable
    `manifest_root` / `entry_root`. Formalize as
    `pi_star_ref ∈ {vecpack_manifest@v1, vecpack_entry@v1,
    vec_backend_config@v1}` later.

11. **Metadata #000050 consumes** (for run-DAG, audit, replay — never
    proof): `{vecpack_id, vecpack_manifest_root, vec_backend_version,
    coverage_ratio, trust_tier, import_event_hash}`.

**Test additions** (append to §6.6):
`test_vecpack_manifest_canonical_hash_stable`,
`test_vecpack_export_import_cross_build_leaf_hash_mapping`,
`test_vecpack_import_rejects_{wrong_shard_root,backend_version_mismatch,metric_quant_mismatch}`,
`test_vecpack_import_skips_{missing_leaf_hash,bad_blob_length,nonfinite_vector}`,
`test_vecpack_import_rejects_duplicate_leaf_conflict`,
`test_vecpack_import_idempotent_same_pack`,
`test_vecpack_import_resume_after_interruption`,
`test_vecpack_import_respects_bad_row_threshold`,
`test_vecpack_import_rejects_insufficient_disk_budget`,
`test_vecpack_mesh_envelope_signature_failure_rejects`,
`test_vecpack_mesh_part_reassembly_requires_all_parts`,
`test_vecpack_bad_embedding_cannot_increment_verified_evidence_count`,
`test_vecpack_import_writes_soft_audit_event_only`,
`test_vecpack_import_does_not_modify_document_root_or_cache_key`.
The last two are the hard/soft-boundary proof.

**Implementation order:** (1) file `vecpack export`/`import` +
manifest/roots; (2) structural validation + soft audit events; (3)
cross-build `leaf_hash`-mapping tests; (4) streaming/resume; (5) mesh
payload transport; (6) #000050 integration via vecpack metadata.
Default safety throughout: imported vecpacks support **raw vec search**;
**cached hybrid QA stays blocked** until #000050's governance-hash
wiring + cache-write guard land.

## 7. References

- #000039 — the local vec backend (Phase 0 + Phase 1, closed
  2026-05-12); §14.6 (embed throughput — the asymmetry this ticket
  routes around), §4 (additive, never proof path).
- #000050 — RRF hybrid fusion; this ticket supplies prereq #1 (a
  corpus backfill) as a *distributable artifact* rather than a job
  fox runs locally.
- #000037 §3.1 — the Prometheus-Σ unconscious-sweep task: the natural
  vecpack *producer* (backfill off-peak, export, serve pulls).
- `arborist/mesh/wire.py` — the signed/AEAD wire envelope a vecpack
  payload rides on. `docs/mesh.md` — wire format + group-key state.
- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst`
  §1 — "the embedding pass runs off the device"; vecpack distribution
  is the mechanism behind that sentence.
- CLAUDE.md "soft hash vs hard hash" — why soft data (embeddings) is
  safe to gossip with a cheap structural gate instead of proof-grade
  verification. #000035 / #000036 — the same discipline applied to
  PRG bytes / soft channels.
