diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 3507931..00940f8 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -101,7 +101,8 @@ Newest first. Update on every open/close. | ID | Title | Status | Opened | Directive | |----------|------------------------------------------------|-----------------------|------------|-----------| -| #000050 | Vec RRF hybrid fusion (#000039 Phase 2) | open · awaiting go/no-go · doc-only scaffold; design in #000039 §4.2 (RRF) + §8 (the gate). Wire `VecBackend` as a 5th retrieval route in `query.py`, RRF-merged with the 4 FTS5 routes; UNGROUNDED hits, additive not replacement; folds the 5 vec hyperparams into `governance_policy_hash`. **Gated** on (a) a corpus backfill (`arborist embed --quant int8` — a one-time batch job, hours-on-idle/days-on-contended per #000039 §14.6) AND (b) a recall bench clearing the 5pp floor (vec-only beats FTS5-only ≥5pp on ≥1 semantic-allusion fixture; hybrid lifts STRICT-rate ≥5pp, no UNGROUNDED regression — else park, vec stays opt-in `--backend vec`). #000039 follow-up | 2026-05-12 | — | +| #000051 | Federated vecpack distribution (gossip the embedding backfill) | open · awaiting go/no-go · doc-only scaffold. Makes `chunk_vecs` a distributable artifact: backfill once on any CPU box (cloud / Prometheus-Σ sweep — #000037 §3.1), publish a **vecpack** `(shard_root, vec_backend_version, [(leaf_hash, embedding_blob)…])` over the mesh wire layer, every peer pulls + bulk-loads (sub-ms/chunk on the receiver — the laptop never runs the transformer). Keyed on `leaf_hash` (portable) not `chunk_id` (shard-local). Vecpacks are **soft data** — embeddings are `UNGROUNDED`, never proof path — so a cheap structural sanity gate (chunk exists locally w/ matching leaf_hash, right blob length for (dim,quant), finite norm, backend_version matches) suffices, no Merkle-proof-grade verification needed. Supplies #000050's prereq #1 ("a vecpack exists & is imported on the bench box", not "fox embedded the corpus locally"). The mechanism behind whitepaper §1's "the embedding pass runs off the device". #000039 / #000050 sibling | 2026-05-12 | — | +| #000050 | Vec RRF hybrid fusion (#000039 Phase 2) | open · awaiting go/no-go · doc-only scaffold; design in #000039 §4.2 (RRF) + §8 (the gate). Wire `VecBackend` as a 5th retrieval route in `query.py`, RRF-merged (route provenance carried) with the 4 FTS5 routes; UNGROUNDED hits, additive not replacement. Phase-2 sub-items now explicit: **accept-path-5** in `_filter_by_title_relevance` (low-title-overlap vec hits survive only via a stronger span-level warrant, never similarity-score alone — else the title gate drops exactly the semantic candidates vec exists for & the bench shows no lift); **six** vec config fields fold into `governance_policy_hash` (recipe-named quant `int8sym`) **+ a cache-write guard** blocking `providence_cache` persistence for vec/hybrid runs until that's wired; **run-DAG records the vec stage** (backend version, six fields, top_k, query-embedding hash, candidate chunk_ids+distances). **Gated** on (a) a corpus backfill **distributed via #000051** AND (b) a **four-condition** recall bench (A FTS5-only / B vec-only / C RRF hybrid / D candidate-union-no-RRF) clearing the 5pp floor incl. C-beats-D, on semantic-allusion + curated + **adversarial-semantic-neighbor** fixtures (else park, vec stays opt-in `--backend vec`; if C≈D ship the union, drop RRF). #000039 follow-up | 2026-05-12 | — | | #000049 | Attribution-aware grounding check (the recombination boundary) | open · awaiting go/no-go · doc-only; the home for #000048's deferred §2.3 — closing the 2 recombination over-grounds in `falsification-hard` (hard-003 Mercury / hard-005 Einstein) needs an attribution / dependency-parse or mini-NLI check, which is *not lexical* (#000048 §5). First decision = the discipline question: may a small fixed purpose-built NLI/entailment *model* influence `audit_mode`? (vs the "no LLM-as-judge" rule). Recommends Option 2.3 (do nothing — the 2 fixtures are a boundary marker) until real-traffic recombination-over-grounds show up, then Option 2.2 (`[nli]` extra, policy-gated, off-by-default, bench-gated) *if* fox rules a fixed NLI model is acceptable. #000048 follow-up | 2026-05-12 | — | | #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | **closed · 2026-05-12** — steps 2.1 + 2.4 landed 2026-05-11 (12 of 16 residual items: 4 HYBRID_ENTITY over-grounds + 8 Formulate mis-segments → `formulate-hard` 12/12, `falsification-hard` 10/12; each bench-gated, no STRICT-rate regression — 2.1's gate fired on 0 QA answers, 2.4's segmenter touched 7 of 450 lattice cells both verdict changes correct). Step 2.2 (single-clause-containment paraphrase check) attempted + reverted — catches the 2 recombination fixtures but also rejects legit cross-sentence summaries with no threshold separating the two; recombination-vs-summary isn't lexical (§5 "What we learned"). The attribution-aware path moved to **#000049** (fox 2026-05-12). 2 live-pack `expected_reason` updated HYBRID_ENTITY→UNGROUNDED; 12+ tests; `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`. #000046 follow-up; #000047 closed | 2026-05-11 | — | | #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — | @@ -154,4 +155,4 @@ Newest first. Update on every open/close. ## Next ID -`000051` +`000052` diff --git a/docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md b/docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md index eb5ff94..a4d6abf 100644 --- a/docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md +++ b/docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md @@ -47,56 +47,140 @@ than leaving #000039 open indefinitely. Phase 2 implementation does **not** open until both: -1. **A corpus backfill exists.** `arborist embed --quant int8` over - the shard set. Per #000039 §14.6 this is a one-time batch job — - hours on an idle box, days on a contended one (~4 chunks/s - measured on the dev box; ~50-200/s idle). Best run off-peak or - on a dedicated box; ~2.4 GB total at int8 (+6 % over the 38 GB - shards). The lazy `arborist embed` pass / a cron / a Prometheus-Σ - unconscious-sweep task (#000037 §3.1) is the home for this. -2. **A recall bench clears the gate.** Run the bench fixtures - (bench-emergent semantic-allusion shapes + the curated set) - with `--backend vec` vs `--backend fts5` vs the planned hybrid; - require **vec-only beats FTS5-only by ≥ 5pp on at least one - semantic-allusion fixture** AND **hybrid lifts STRICT-rate ≥ 5pp - over FTS5-only with no UNGROUNDED-rate regression** (the 5pp - signal floor per `docs/bench-maxing.md`). If hybrid lift is - sub-floor: park here — vec stays opt-in per-call (`--backend - vec`), hybrid does NOT become the default, and the ~2.4 GB tax - isn't paid for nothing. +1. **A corpus backfill exists** — *anywhere*, not necessarily where + the bench runs. `arborist embed --quant int8` over the shard set. + Per #000039 §14.6 this is a one-time batch job — hours on an idle + box, days on a contended one (~4 chunks/s measured on the dev box; + ~50-200/s idle), ~2.4 GB total at int8 (+6 % over the 38 GB + shards). The embed pass is ~10–100× heavier than the rest of + ingest, so it does **not** run on a laptop — it runs once on any + box with CPU (a cloud node, a Prometheus-Σ unconscious-sweep box + — #000037 §3.1) and the resulting `chunk_vecs` is **distributed as + a vecpack over mesh** so every peer pulls + bulk-loads it (sub-ms + per chunk on the receiver). That distribution mechanism is + **#000051** — this prereq is "a vecpack exists and has been + imported on the bench box", not "fox embedded the corpus locally". +2. **A recall bench clears the gate.** Run **four conditions** on the + bench fixtures (bench-emergent semantic-allusion shapes + the + curated set + the adversarial semantic-neighbor pack from §3 + below): **(A)** FTS5-only baseline, **(B)** vec-only, **(C)** FTS5 + + vec RRF hybrid, **(D)** FTS5 + vec candidate-union but the + *existing* reranker only (no RRF). D is the control that + distinguishes "RRF helped" from "more candidates helped" — if C + ≈ D, RRF is not earning its complexity. Gate (5pp signal floor per + `docs/bench-maxing.md`): + - **vec-only (B) beats FTS5-only (A) by ≥ 5pp recall@K on ≥ 1 + semantic-allusion fixture** *and* surfaces ≥ 1 correct candidate + absent from A's top-K. (B is a diagnostic, *not* required to + match A's overall STRICT-rate — vec is allowed to lose on exact + title / rare-token shapes; punishing it for not being FTS5 is + the wrong test.) + - **hybrid (C) lifts the proof-quality-label rate ≥ 5pp** over A on + the target fixtures (STRICT for quote/span/entity/paraphrase; + EVIDENCE-WARRANTED for lattice modes — "the current proof-quality + label family", not literally "STRICT") **with no statistically + meaningful regression** on the lexical/title/false-neighbor + fixtures and **no UNGROUNDED-rate regression**. + - **C must beat D** (or be no worse) — else ship the cheaper union, + not RRF. + If hybrid lift is sub-floor: park here — vec stays opt-in per-call + (`--backend vec`), hybrid does NOT become the default, and the + ~2.4 GB tax isn't paid for nothing. ## 3. Implementation sketch (when the gate clears) -- `arborist/qa/query.py`: after the four FTS5 routes produce ranked - hit lists and before the rerank/filter chain, add a fifth list - from `VecBackend.search(query, limit=K)`. Merge all five by RRF: - `score(doc) = Σ_route 1/(60 + rank_route(doc))`. The merged - ranking then feeds the existing body-coverage `sqrt` rerank, - title-token boost, `_filter_by_title_relevance`, rivalry - exclusion, stem-aware matching, per-source cap, wikitext-base - prose — unchanged. Vec hits with no title overlap survive via the - same accept-path-4 (phrase-match) logic the phrase route uses, or - add a fifth accept path (semantic-similarity above a threshold) — - decide at implementation time from the bench. -- Skip the vec route cleanly when `chunk_vecs` is absent/empty on a - shard (mixed-fleet: some shards backfilled, some not) — falls back - to FTS5-only for that shard. -- Governance: fold the 5 vec hyperparams (embedder name+version, - dim, quant, metric, ann) into `governance_policy_hash` (#000039 - §6). A Q&A answer whose context drew on vec hits then carries the - vec config in its cache_key's `governance_policy_hash` dim; - bumping the embedder stales prior vec-consulting cache rows on - lookup. The 8-dim invariant is untouched (existing dim, not a new - one). -- `arborist query` / the bench harness: a `--retrieval-backend - {fts5,hybrid}` knob (default `fts5` until the bench clears - `hybrid`, then flip the default — a fox call + a `governance_policy_hash` - bump). +**RRF merge.** `arborist/qa/query.py`: after the four FTS5 routes +produce ranked hit lists and before the rerank/filter chain, add a +fifth list from `VecBackend.search(query, limit=K)`. Merge all five by +RRF: `score(doc) = Σ_route 1/(60 + rank_route(doc))`, *before* the +accept-path filtering. Carry route provenance on each merged hit +(`routes: [...]`, `route_ranks: {body_bm25: 14, vec: 2}`, +`rrf_score`) — needed for the four-condition bench and for later +failure analysis. The merged ranking then feeds the existing +body-coverage `sqrt` rerank, title-token boost, +`_filter_by_title_relevance`, rivalry exclusion, stem-aware matching, +per-source cap, wikitext-base prose — unchanged. + +**Accept-path-5 (semantic-warrant).** This is the load-bearing +addition, not an afterthought: the existing title-relevance gate may +drop *exactly* the low-title-overlap semantic candidates vec exists to +find (the "what did Orwell call the country at war with Oceania?" → +*Nineteen Eighty-Four* / Eastasia case — right chunk, zero query/title +token overlap), so without a vec-safe accept path the bench can show +**no lift even when vec retrieval works**. Add a fifth accept path in +`_filter_by_title_relevance` that lets a vec hit survive low title +overlap **only if it earns a stronger span-level signal** — at least +one of: (a) the cited span contains a deterministic anchor from the +question or the generated claim; (b) a claim-lattice pointer resolves +to a span whose anchor-class warrant matches the claim; (c) an +entity/predicate/date/quote warrant passes on the span; (d) a +`concept_relations` synonym edge bridges a query anchor to a span +anchor; (e) the answer is explicitly exploratory / `UNGROUNDED` and +does **not** reach an EVIDENCE-WARRANTED / STRICT label. In short: vec +may bypass title overlap only by earning a stronger warrant, never by +embedding proximity alone. Decide the exact predicate set at +implementation time from the bench, but the *principle* — "vec hits +clear the gate via warrant strength, not similarity score" — is +non-negotiable. + +**Mixed fleet.** Skip the vec route cleanly when `chunk_vecs` is +absent/empty on a shard (some shards have an imported vecpack — #000051 +— some don't) — falls back to FTS5-only for that shard, with an audit +note `vec_route_unavailable` so a hybrid-labelled run never silently +ran FTS5-only. + +**Governance — wire it before any cached hybrid run, not "later".** +Fold the **six** vec config fields — `embedder_name`, +`embedder_version` (pinned revision/digest, separate from the name), +`quantization` (named *with its recipe*: `int8sym` = +symmetric-global-scale ×127/clamp/round, not bare `int8`), `dimension`, +`distance_metric`, `ann_index` — into `governance_policy_hash` +(#000039 §6), summarized by the compact `VEC_BACKEND_VERSION` token +(e.g. `vec-v1-bge-small-en-v1.5-384int8sym-l2norm-flat`; the token is +a *derived summary*, the six fields are the auditable payload). Until +that wiring lands, a **cache-write guard**: a QA run whose retrieval +backend is `vec`/`hybrid` may return an answer but **must not persist a +`providence_cache` record** (status `vec_experimental_no_cache`) — +because changing the embedder changes the candidates changes the +context changes the answer, and an un-versioned cached hybrid answer is +exactly the silent provenance drift the substrate exists to prevent. +The 8-dim `cache_key` invariant is untouched (existing dim, not a new +one). + +**Run-DAG metadata.** A hybrid run's per-run Merkle-DAG records the vec +retrieval stage: `route="vec"`, `VEC_BACKEND_VERSION`, the six config +fields, `top_k`, a `query_embedding_hash`, and the candidate +`(chunk_id, quantized_distance)` list (or a `candidate_root` over +them). `governance_policy_hash` says *which regime was allowed*; the +run-DAG says *what actually happened this run* — both must be visible +for replay (same lesson as the `--retrieval-keywords` audit gap, +#000001). + +**The knob.** `arborist query` / the bench harness: a +`--retrieval-backend {fts5,hybrid}` knob (default `fts5` until the +bench clears `hybrid`, then flipping the default is a fox call + a +`governance_policy_hash` bump). No vec-*only* QA path except +explicitly-experimental (vec-only is a retrieval diagnostic, not an +answer regime). + +**Adversarial semantic-neighbor fixtures.** Vec opens a new failure +class: a chunk that is *semantically* near the query but factually +wrong — same topic / wrong entity, same title family / wrong franchise +installment, same era / wrong date, a paraphrase that retrieves the +*background* source instead of the *primary* one (the "Mr. Burns" / +"which Back to the Future" shape, now reachable through a semantic side +door). Build a small adversarial pack for the §2 bench whose pass +condition is: hybrid must **not** raise the rate of confidently-wrong +warranted answers, and the title-relevance / Rule-8 hard checks must +still catch the mismatched vec hits. This pack is part of the gate, not +a nice-to-have. ## 4. Out of scope -- The corpus backfill itself (a batch op, not code — #000039 §14.6; - prerequisite #1 above). +- The corpus backfill itself (a batch op — #000039 §14.6) **and its + distribution** (vecpack export/import + mesh payload — **#000051**). + Prereq #1 above is "a vecpack exists and is imported on the bench + box", which #000051 delivers. - A GPU/accelerated embedder (drop-in via `default_embedder()`'s pluggable `Embedder` callable; orthogonal — #000039 §14.6). - int8 vs float32 default (settled: int8 is the production config @@ -108,24 +192,50 @@ Phase 2 implementation does **not** open until both: ## 5. Acceptance criteria -1. `arborist/qa/query.py` merges a fifth (vec) route via RRF, behind - a `--retrieval-backend` knob; FTS5-only behavior unchanged when - the knob says `fts5` or `chunk_vecs` is absent. -2. The 5 vec hyperparams fold into `governance_policy_hash`. -3. Bench data recorded showing the §2 gate cleared (or, if it - didn't, this ticket parks with the bench result documented and - `hybrid` left non-default). -4. Tests: hybrid-merge unit tests (RRF math; vec-absent fallback; - filter chain runs on vec hits); a bench-row in the QA-modes - journal. +1. `arborist/qa/query.py` merges a fifth (vec) route via RRF (with + route provenance on each hit), behind a `--retrieval-backend + {fts5,hybrid}` knob; FTS5-only behavior byte-unchanged when the + knob says `fts5` or `chunk_vecs` is absent (mixed-fleet fallback + emits the `vec_route_unavailable` audit note). +2. Accept-path-5 lands in `_filter_by_title_relevance`: a low-title- + overlap vec hit survives only via a stronger span-level warrant + (anchor / pointer-warrant / entity-predicate-date-quote / + concept-synonym / explicitly-`UNGROUNDED`), never via similarity + score alone. +3. The **six** vec config fields fold into `governance_policy_hash`, + summarized by `VEC_BACKEND_VERSION` (recipe-named quant, e.g. + `int8sym`); **and** a cache-write guard blocks `providence_cache` + persistence for `vec`/`hybrid` runs until that wiring is live + (status `vec_experimental_no_cache`). +4. A hybrid run's run-DAG records the vec retrieval stage (backend + version, six fields, `top_k`, query-embedding hash, candidate + chunk_ids + quantized distances / `candidate_root`). +5. Bench data recorded for all **four conditions** (A FTS5-only / B + vec-only / C RRF hybrid / D candidate-union-no-RRF) on the + semantic-allusion + curated + adversarial-neighbor fixtures, + showing the §2 gate cleared — including C-beats-D — (or, if it + didn't, this ticket parks with the result documented and `hybrid` + left non-default; if C ≈ D, ship the union, drop RRF). +6. Tests: RRF math + route provenance; vec-absent fallback + the + audit note; accept-path-5 (a known semantic-allusion hit survives + low title overlap *only* with a warrant, a bare-similarity hit + does not); cache-write guard fires for un-versioned hybrid; + filter chain + Rule-8 still reject adversarial semantic-neighbor + hits; a bench-row in the QA-modes journal. ## 6. References - #000039 — the parent (Phase 0 + Phase 1, closed 2026-05-12); §4 (additive not replacement), §4.2 (RRF design), §8 (the gate), §14 (ingest integration), §14.6 (embed throughput). +- **#000051 — federated vecpack distribution.** Supplies prereq #1 + (a corpus backfill) as a *distributable artifact* — backfill once + on any CPU box, gossip the `chunk_vecs` over mesh — so the laptop + never embeds. Read it for why vec data is safe to gossip with a + cheap structural gate (soft data, never proof path). - `docs/bench-maxing.md` — the 5pp signal floor + bench discipline. - `docs/benchmarks.md` — bench harness orientation. - #000037 §3.1 — the Prometheus-Σ unconscious-sweep task that's the - natural home for the lazy embed pass that backfills `chunk_vecs`. -- CLAUDE.md "Retrieval pipeline" — the four FTS5 routes vec joins. + natural home for the embed backfill *and* the vecpack producer. +- CLAUDE.md "Retrieval pipeline" — the four FTS5 routes vec joins; + "soft hash vs hard hash" — why vec stays out of the proof path. diff --git a/docs/tickets/ticket-000051-federated-vecpack-distribution.md b/docs/tickets/ticket-000051-federated-vecpack-distribution.md new file mode 100644 index 0000000..e5698ad --- /dev/null +++ b/docs/tickets/ticket-000051-federated-vecpack-distribution.md @@ -0,0 +1,216 @@ +# Ticket #000051 — Federated vecpack distribution (gossip the embedding backfill) + +**Status:** open · awaiting go/no-go — doc-only scaffold. Removes the +"who runs the days-long embed job" friction from #000050's gate by +making `chunk_vecs` a *distributable artifact*: backfill once on any +box with CPU (a cloud node, a Prometheus-Σ sweep box — #000037 §3.1), +publish a **vecpack** over the existing mesh wire layer, every peer +pulls + bulk-loads it. The transformer never runs on a laptop. +**Opened:** 2026-05-12 +**Scope:** A serialize/verify/load path for per-shard embedding sets, +plus a mesh payload type carrying them. A vecpack is +`(shard_root, vec_backend_version, [(leaf_hash, embedding_blob), …])` +— content-addressed by `leaf_hash` (the chunk's hard hash), keyed in +transit on `leaf_hash` not `chunk_id` (`chunk_id` is a shard-local +rowid; `leaf_hash` is portable). A receiving node maps +`leaf_hash → local chunk_id` against its own `chunks` table and +bulk-INSERTs into `chunk_vecs` — sub-millisecond per chunk, the cheap +side of the #000039 §14.6 asymmetry. Storage cost is real (~384 B/chunk +at int8sym, ~2.4 GB for the 6.24M-chunk wiki set, ~6% over the shards) +but it is a *download + disk* cost, never a *compute* cost, on the +receiver. +**Audience:** fox + future blackops shifts + anyone running a public +arborist node that wants the semantic layer without an accelerator. +**Hard constraint:** vecpack payloads are **soft data** — embeddings +are `audit_mode=UNGROUNDED`, never in the proof path (#000039 §4, +CLAUDE.md "soft hash vs hard hash"). A wrong or adversarial gossiped +embedding can only nudge retrieval recall on the receiver; it cannot +corrupt a `cache_key`, a `document_root`, or an `audit_event_hash`. +**So vecpacks do not need Merkle-proof-grade verification** — a cheap +structural sanity gate is sufficient and correct (see §3). The `cache_key` +8-dim invariant is untouched; the vec config still folds into +`governance_policy_hash` per #000050 (this ticket distributes the +vectors, #000050 makes a QA answer that *consulted* them +policy-hash-honest). + +--- + +## 1. Why this is a separate ticket + +#000050's prerequisite #1 — "a corpus backfill exists" — was written +as a batch job *somebody runs*, with fox's laptop implicitly the +somebody. It is not: the embed pass is ~10–100× heavier than the rest +of ingest (#000039 §14.6) and a full-corpus backfill is hours-on-idle +/ days-on-contended. fox does not have an accelerator and should never +need one. The fix is distribution, not faster local embedding: + +``` +backfill once, anywhere with CPU + → publish a vecpack per shard over mesh + → every peer pulls + bulk-loads it + → laptop pays download + sqlite INSERTs only +``` + +That is a distinct mechanism from #000050's RRF fusion (which is about +*using* `chunk_vecs` once present) and from #000039's backend (which +is about *building* it locally), so it gets its own ticket per repo +convention rather than bloating either. + +This is also the concrete answer to the whitepaper §1 claim that "the +embedding pass runs off the device" — gossip is *how* it reaches the +device. The two strengthen each other. + +## 2. Why vec data gossips well (the trust-model argument) + +An embedding is a pure function of `(chunk content bytes, embedder +config)`. Two consequences: + +1. **Content-addressable.** `embedding = f(leaf_content, + vec_backend_version)`. A vecpack keyed on `(leaf_hash, + vec_backend_version)` is the natural unit — a peer either has the + chunk (knows its `leaf_hash`) or doesn't; if it does, the embedding + slots straight in. No `chunk_id` alignment problem (those are + shard-local rowids and must NOT be trusted across DBs even when two + shards were built from the same dump). +2. **Soft by construction → cheap to verify.** Embeddings never enter + a proof. The worst a bad vecpack does is degrade *recall* on the + receiver — and even that is bounded, because the post-retrieval + filter chain (title-relevance, rivalry exclusion, Rule-8 + title-mismatch, and the verifier itself) still runs on every + candidate vec surfaces. So a vecpack does **not** need a Merkle + inclusion proof or a signed-by-the-corpus-author guarantee. It + needs: (a) the chunks it references actually exist locally with + matching `leaf_hash`, (b) each blob is the right length for the + declared `(dim, quant)`, (c) each decoded vector has a finite, + sane norm, (d) `vec_backend_version` matches the receiver's + `vec_meta` (or the receiver is creating `vec_meta` fresh from this + pack). That is a cheap structural gate, not cryptographic + verification. (Contrast: cache records gossiped over mesh *are* + proof-bearing and *do* carry the full audit-chain machinery — + vecpacks deliberately don't, and that's the right call.) + +This is the same hard/soft discipline as #000035 (φ_PRG bytes pinned) +and #000036 (soft channels bounded, never confused with hard proof): +the layer that's safe to distribute loosely is exactly the layer that +can't hurt you if it's wrong. + +## 3. Vecpack format (sketch) + +``` +vecpack-v1: + shard_root # the shard this pack embeds (sanity-bind, not a proof) + vec_backend_version # e.g. vec-v1-bge-small-en-v1.5-384int8sym-l2norm-flat + dim # 384 + quant # "int8sym" | "float32" + metric # "l2norm" (cosine-equivalent for unit vectors) + count # number of (leaf_hash, blob) entries + entries: [ (leaf_hash: 32 bytes, embedding_blob: dim*bytes_per_elem) , … ] +``` + +Wire it as a mesh payload (`arborist/mesh/wire.py` `WireEnvelope` +already does signed/AEAD-wrapped bodies — a vecpack is just another +body type; the signature authenticates the *sender*, not the +*correctness* of the embeddings, which is fine because they're soft). +Large packs chunk into multiple envelopes keyed by `(shard_root, +vec_backend_version, part_i, n_parts)`. + +Receiver algorithm: + +``` +on vecpack(shard_root, vec_backend_version, dim, quant, metric, entries): + if local shard_root != shard_root: -> reject (wrong shard) + if chunk_vecs exists and vec_meta.backend_version != vec_backend_version: + -> reject (rebuild needed) or stage separately + if not chunk_vecs: ensure_chunk_vecs_table(quant=quant) # creates vec_meta from this pack + for (leaf_hash, blob) in entries: + cid = SELECT chunk_id FROM chunks WHERE leaf_hash = ? # local rowid + if cid is None: continue # don't have this chunk; skip + if len(blob) != dim * bytes_per_elem(quant): skip + count_bad + if not finite_sane_norm(decode(blob, quant)): skip + count_bad + stage (cid, wrap(blob, quant)) # vec_f32(?) / vec_int8(?) + bulk delete-then-insert staged rows into chunk_vecs + emit status: {received, loaded, skipped_missing_chunk, skipped_bad, backend_version} +``` + +Idempotent: re-loading the same vecpack is a no-op (delete-then-insert +on the same `chunk_id` with the same blob). Incremental: a partial +vecpack (some shards, some chunks) loads what it can; the rest stay +FTS5-only until another pack arrives — the mixed-fleet case #000050 §3 +already has to handle. + +## 4. Producer side + +``` +arborist vecpack export --shard [--out path] [--max-bytes N (split)] + # SELECT leaf_hash, embedding FROM chunks JOIN chunk_vecs USING(chunk_id) + # → vecpack-v1 file(s); records vec_backend_version + dim + quant from vec_meta +arborist vecpack import --shard --in path + # the receiver algorithm above; prints the status dict +``` + +Mesh-native variant (when mesh is enabled): a node advertises which +`(shard_root, vec_backend_version)` packs it serves; a peer requests +the ones it lacks; transfer rides the existing wire layer. The +Prometheus-Σ unconscious-sweep task (#000037 §3.1) is the natural +producer — it backfills `chunk_vecs` off-peak *and* exports the pack +*and* answers pull requests, all as one low-priority background role. + +## 5. Out of scope + +- The local embed backend + `arborist embed` itself (#000039 — done). +- RRF hybrid fusion in `query.py` (#000050). +- A GPU embedder (orthogonal — #000039 §14.6; a faster producer, same + vecpack output). +- Strong/cryptographic verification of embedding *correctness* (a peer + can re-embed a sample and spot-check if it wants higher assurance, + but that's an optional belt-and-suspenders, not part of v1 — the + soft-data argument in §2 is why it isn't required). +- A content-distribution network / torrent-style fan-out for very + large packs (if the 2.4 GB wiki pack ever needs it; v1 is + point-to-point over mesh + a plain file export). + +## 6. Acceptance criteria + +1. `vecpack-v1` serialize/parse round-trips (`leaf_hash`-keyed, + `vec_backend_version`/`dim`/`quant`/`metric` in the header). +2. `arborist vecpack export` / `import` work file-to-file across two + independently-built copies of the same shard (chunk_ids differ, + leaf_hashes match → embeddings land correctly). +3. The receiver's structural sanity gate rejects: wrong `shard_root`, + `backend_version` mismatch against existing `vec_meta`, wrong blob + length, non-finite vectors — each with a distinct status field, no + silent acceptance. +4. Import is idempotent and incremental (partial packs load what they + can; missing chunks counted, not errored). +5. Mesh payload type wired into `arborist/mesh/wire.py` (or a thin + sibling), signed-by-sender, AEAD-wrapped like other bodies; large + packs split/reassemble by part index. +6. Tests: round-trip; cross-shard leaf_hash mapping; each sanity-gate + rejection path; idempotent re-import; mesh envelope carry; the + soft-data boundary (a deliberately-corrupted blob is rejected by + the gate, and even if it weren't, it's `UNGROUNDED` and the + post-retrieval filters still run — i.e. a unit test asserting a bad + vec hit can't reach the proof path). +7. Docs: a short section in `docs/mesh.md` (vecpack as a payload type) + + a pointer from #000050 §2 prereq #1 ("backfill once, distribute + via #000051"). + +## 7. References + +- #000039 — the local vec backend (Phase 0 + Phase 1, closed + 2026-05-12); §14.6 (embed throughput — the asymmetry this ticket + routes around), §4 (additive, never proof path). +- #000050 — RRF hybrid fusion; this ticket supplies prereq #1 (a + corpus backfill) as a *distributable artifact* rather than a job + fox runs locally. +- #000037 §3.1 — the Prometheus-Σ unconscious-sweep task: the natural + vecpack *producer* (backfill off-peak, export, serve pulls). +- `arborist/mesh/wire.py` — the signed/AEAD wire envelope a vecpack + payload rides on. `docs/mesh.md` — wire format + group-key state. +- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` + §1 — "the embedding pass runs off the device"; vecpack distribution + is the mechanism behind that sentence. +- CLAUDE.md "soft hash vs hard hash" — why soft data (embeddings) is + safe to gossip with a cheap structural gate instead of proof-grade + verification. #000035 / #000036 — the same discipline applied to + PRG bytes / soft channels.