#000051: gossip the embedding backfill over mesh — backfill once on any CPU box, publish a vecpack (leaf_hash-keyed, soft data, cheap structural gate, never proof path), peers pull + bulk-load. The laptop never runs the transformer; this is the mechanism behind whitepaper §1's "the embedding pass runs off the device". Supplies #000050's backfill prereq as a distributable artifact. #000050: fold the Dav1dPrometheus-review high-value items into Phase-2 scope — accept-path-5 (vec hits clear the title gate via span-level warrant, not similarity score, so the gate doesn't drop the semantic candidates vec exists for), six vec config fields into governance_policy_hash + a cache-write guard until wired, run-DAG records the vec stage, four-condition bench (A/B/C/D, C-beats-D) on semantic-allusion + curated + adversarial-semantic-neighbor fixtures. Backfill prereq now routes through #000051. Next ID 000051 -> 000052.
This commit is contained in:
parent
b5da0473a1
commit
07b8ceb638
3 changed files with 385 additions and 58 deletions
|
|
@ -101,7 +101,8 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000050 | Vec RRF hybrid fusion (#000039 Phase 2) | open · awaiting go/no-go · doc-only scaffold; design in #000039 §4.2 (RRF) + §8 (the gate). Wire `VecBackend` as a 5th retrieval route in `query.py`, RRF-merged with the 4 FTS5 routes; UNGROUNDED hits, additive not replacement; folds the 5 vec hyperparams into `governance_policy_hash`. **Gated** on (a) a corpus backfill (`arborist embed --quant int8` — a one-time batch job, hours-on-idle/days-on-contended per #000039 §14.6) AND (b) a recall bench clearing the 5pp floor (vec-only beats FTS5-only ≥5pp on ≥1 semantic-allusion fixture; hybrid lifts STRICT-rate ≥5pp, no UNGROUNDED regression — else park, vec stays opt-in `--backend vec`). #000039 follow-up | 2026-05-12 | — |
|
||||
| #000051 | Federated vecpack distribution (gossip the embedding backfill) | open · awaiting go/no-go · doc-only scaffold. Makes `chunk_vecs` a distributable artifact: backfill once on any CPU box (cloud / Prometheus-Σ sweep — #000037 §3.1), publish a **vecpack** `(shard_root, vec_backend_version, [(leaf_hash, embedding_blob)…])` over the mesh wire layer, every peer pulls + bulk-loads (sub-ms/chunk on the receiver — the laptop never runs the transformer). Keyed on `leaf_hash` (portable) not `chunk_id` (shard-local). Vecpacks are **soft data** — embeddings are `UNGROUNDED`, never proof path — so a cheap structural sanity gate (chunk exists locally w/ matching leaf_hash, right blob length for (dim,quant), finite norm, backend_version matches) suffices, no Merkle-proof-grade verification needed. Supplies #000050's prereq #1 ("a vecpack exists & is imported on the bench box", not "fox embedded the corpus locally"). The mechanism behind whitepaper §1's "the embedding pass runs off the device". #000039 / #000050 sibling | 2026-05-12 | — |
|
||||
| #000050 | Vec RRF hybrid fusion (#000039 Phase 2) | open · awaiting go/no-go · doc-only scaffold; design in #000039 §4.2 (RRF) + §8 (the gate). Wire `VecBackend` as a 5th retrieval route in `query.py`, RRF-merged (route provenance carried) with the 4 FTS5 routes; UNGROUNDED hits, additive not replacement. Phase-2 sub-items now explicit: **accept-path-5** in `_filter_by_title_relevance` (low-title-overlap vec hits survive only via a stronger span-level warrant, never similarity-score alone — else the title gate drops exactly the semantic candidates vec exists for & the bench shows no lift); **six** vec config fields fold into `governance_policy_hash` (recipe-named quant `int8sym`) **+ a cache-write guard** blocking `providence_cache` persistence for vec/hybrid runs until that's wired; **run-DAG records the vec stage** (backend version, six fields, top_k, query-embedding hash, candidate chunk_ids+distances). **Gated** on (a) a corpus backfill **distributed via #000051** AND (b) a **four-condition** recall bench (A FTS5-only / B vec-only / C RRF hybrid / D candidate-union-no-RRF) clearing the 5pp floor incl. C-beats-D, on semantic-allusion + curated + **adversarial-semantic-neighbor** fixtures (else park, vec stays opt-in `--backend vec`; if C≈D ship the union, drop RRF). #000039 follow-up | 2026-05-12 | — |
|
||||
| #000049 | Attribution-aware grounding check (the recombination boundary) | open · awaiting go/no-go · doc-only; the home for #000048's deferred §2.3 — closing the 2 recombination over-grounds in `falsification-hard` (hard-003 Mercury / hard-005 Einstein) needs an attribution / dependency-parse or mini-NLI check, which is *not lexical* (#000048 §5). First decision = the discipline question: may a small fixed purpose-built NLI/entailment *model* influence `audit_mode`? (vs the "no LLM-as-judge" rule). Recommends Option 2.3 (do nothing — the 2 fixtures are a boundary marker) until real-traffic recombination-over-grounds show up, then Option 2.2 (`[nli]` extra, policy-gated, off-by-default, bench-gated) *if* fox rules a fixed NLI model is acceptable. #000048 follow-up | 2026-05-12 | — |
|
||||
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | **closed · 2026-05-12** — steps 2.1 + 2.4 landed 2026-05-11 (12 of 16 residual items: 4 HYBRID_ENTITY over-grounds + 8 Formulate mis-segments → `formulate-hard` 12/12, `falsification-hard` 10/12; each bench-gated, no STRICT-rate regression — 2.1's gate fired on 0 QA answers, 2.4's segmenter touched 7 of 450 lattice cells both verdict changes correct). Step 2.2 (single-clause-containment paraphrase check) attempted + reverted — catches the 2 recombination fixtures but also rejects legit cross-sentence summaries with no threshold separating the two; recombination-vs-summary isn't lexical (§5 "What we learned"). The attribution-aware path moved to **#000049** (fox 2026-05-12). 2 live-pack `expected_reason` updated HYBRID_ENTITY→UNGROUNDED; 12+ tests; `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`. #000046 follow-up; #000047 closed | 2026-05-11 | — |
|
||||
| #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — |
|
||||
|
|
@ -154,4 +155,4 @@ Newest first. Update on every open/close.
|
|||
|
||||
## Next ID
|
||||
|
||||
`000051`
|
||||
`000052`
|
||||
|
|
|
|||
|
|
@ -47,56 +47,140 @@ than leaving #000039 open indefinitely.
|
|||
|
||||
Phase 2 implementation does **not** open until both:
|
||||
|
||||
1. **A corpus backfill exists.** `arborist embed --quant int8` over
|
||||
the shard set. Per #000039 §14.6 this is a one-time batch job —
|
||||
hours on an idle box, days on a contended one (~4 chunks/s
|
||||
measured on the dev box; ~50-200/s idle). Best run off-peak or
|
||||
on a dedicated box; ~2.4 GB total at int8 (+6 % over the 38 GB
|
||||
shards). The lazy `arborist embed` pass / a cron / a Prometheus-Σ
|
||||
unconscious-sweep task (#000037 §3.1) is the home for this.
|
||||
2. **A recall bench clears the gate.** Run the bench fixtures
|
||||
(bench-emergent semantic-allusion shapes + the curated set)
|
||||
with `--backend vec` vs `--backend fts5` vs the planned hybrid;
|
||||
require **vec-only beats FTS5-only by ≥ 5pp on at least one
|
||||
semantic-allusion fixture** AND **hybrid lifts STRICT-rate ≥ 5pp
|
||||
over FTS5-only with no UNGROUNDED-rate regression** (the 5pp
|
||||
signal floor per `docs/bench-maxing.md`). If hybrid lift is
|
||||
sub-floor: park here — vec stays opt-in per-call (`--backend
|
||||
vec`), hybrid does NOT become the default, and the ~2.4 GB tax
|
||||
isn't paid for nothing.
|
||||
1. **A corpus backfill exists** — *anywhere*, not necessarily where
|
||||
the bench runs. `arborist embed --quant int8` over the shard set.
|
||||
Per #000039 §14.6 this is a one-time batch job — hours on an idle
|
||||
box, days on a contended one (~4 chunks/s measured on the dev box;
|
||||
~50-200/s idle), ~2.4 GB total at int8 (+6 % over the 38 GB
|
||||
shards). The embed pass is ~10–100× heavier than the rest of
|
||||
ingest, so it does **not** run on a laptop — it runs once on any
|
||||
box with CPU (a cloud node, a Prometheus-Σ unconscious-sweep box
|
||||
— #000037 §3.1) and the resulting `chunk_vecs` is **distributed as
|
||||
a vecpack over mesh** so every peer pulls + bulk-loads it (sub-ms
|
||||
per chunk on the receiver). That distribution mechanism is
|
||||
**#000051** — this prereq is "a vecpack exists and has been
|
||||
imported on the bench box", not "fox embedded the corpus locally".
|
||||
2. **A recall bench clears the gate.** Run **four conditions** on the
|
||||
bench fixtures (bench-emergent semantic-allusion shapes + the
|
||||
curated set + the adversarial semantic-neighbor pack from §3
|
||||
below): **(A)** FTS5-only baseline, **(B)** vec-only, **(C)** FTS5
|
||||
+ vec RRF hybrid, **(D)** FTS5 + vec candidate-union but the
|
||||
*existing* reranker only (no RRF). D is the control that
|
||||
distinguishes "RRF helped" from "more candidates helped" — if C
|
||||
≈ D, RRF is not earning its complexity. Gate (5pp signal floor per
|
||||
`docs/bench-maxing.md`):
|
||||
- **vec-only (B) beats FTS5-only (A) by ≥ 5pp recall@K on ≥ 1
|
||||
semantic-allusion fixture** *and* surfaces ≥ 1 correct candidate
|
||||
absent from A's top-K. (B is a diagnostic, *not* required to
|
||||
match A's overall STRICT-rate — vec is allowed to lose on exact
|
||||
title / rare-token shapes; punishing it for not being FTS5 is
|
||||
the wrong test.)
|
||||
- **hybrid (C) lifts the proof-quality-label rate ≥ 5pp** over A on
|
||||
the target fixtures (STRICT for quote/span/entity/paraphrase;
|
||||
EVIDENCE-WARRANTED for lattice modes — "the current proof-quality
|
||||
label family", not literally "STRICT") **with no statistically
|
||||
meaningful regression** on the lexical/title/false-neighbor
|
||||
fixtures and **no UNGROUNDED-rate regression**.
|
||||
- **C must beat D** (or be no worse) — else ship the cheaper union,
|
||||
not RRF.
|
||||
If hybrid lift is sub-floor: park here — vec stays opt-in per-call
|
||||
(`--backend vec`), hybrid does NOT become the default, and the
|
||||
~2.4 GB tax isn't paid for nothing.
|
||||
|
||||
## 3. Implementation sketch (when the gate clears)
|
||||
|
||||
- `arborist/qa/query.py`: after the four FTS5 routes produce ranked
|
||||
hit lists and before the rerank/filter chain, add a fifth list
|
||||
from `VecBackend.search(query, limit=K)`. Merge all five by RRF:
|
||||
`score(doc) = Σ_route 1/(60 + rank_route(doc))`. The merged
|
||||
ranking then feeds the existing body-coverage `sqrt` rerank,
|
||||
title-token boost, `_filter_by_title_relevance`, rivalry
|
||||
exclusion, stem-aware matching, per-source cap, wikitext-base
|
||||
prose — unchanged. Vec hits with no title overlap survive via the
|
||||
same accept-path-4 (phrase-match) logic the phrase route uses, or
|
||||
add a fifth accept path (semantic-similarity above a threshold) —
|
||||
decide at implementation time from the bench.
|
||||
- Skip the vec route cleanly when `chunk_vecs` is absent/empty on a
|
||||
shard (mixed-fleet: some shards backfilled, some not) — falls back
|
||||
to FTS5-only for that shard.
|
||||
- Governance: fold the 5 vec hyperparams (embedder name+version,
|
||||
dim, quant, metric, ann) into `governance_policy_hash` (#000039
|
||||
§6). A Q&A answer whose context drew on vec hits then carries the
|
||||
vec config in its cache_key's `governance_policy_hash` dim;
|
||||
bumping the embedder stales prior vec-consulting cache rows on
|
||||
lookup. The 8-dim invariant is untouched (existing dim, not a new
|
||||
one).
|
||||
- `arborist query` / the bench harness: a `--retrieval-backend
|
||||
{fts5,hybrid}` knob (default `fts5` until the bench clears
|
||||
`hybrid`, then flip the default — a fox call + a `governance_policy_hash`
|
||||
bump).
|
||||
**RRF merge.** `arborist/qa/query.py`: after the four FTS5 routes
|
||||
produce ranked hit lists and before the rerank/filter chain, add a
|
||||
fifth list from `VecBackend.search(query, limit=K)`. Merge all five by
|
||||
RRF: `score(doc) = Σ_route 1/(60 + rank_route(doc))`, *before* the
|
||||
accept-path filtering. Carry route provenance on each merged hit
|
||||
(`routes: [...]`, `route_ranks: {body_bm25: 14, vec: 2}`,
|
||||
`rrf_score`) — needed for the four-condition bench and for later
|
||||
failure analysis. The merged ranking then feeds the existing
|
||||
body-coverage `sqrt` rerank, title-token boost,
|
||||
`_filter_by_title_relevance`, rivalry exclusion, stem-aware matching,
|
||||
per-source cap, wikitext-base prose — unchanged.
|
||||
|
||||
**Accept-path-5 (semantic-warrant).** This is the load-bearing
|
||||
addition, not an afterthought: the existing title-relevance gate may
|
||||
drop *exactly* the low-title-overlap semantic candidates vec exists to
|
||||
find (the "what did Orwell call the country at war with Oceania?" →
|
||||
*Nineteen Eighty-Four* / Eastasia case — right chunk, zero query/title
|
||||
token overlap), so without a vec-safe accept path the bench can show
|
||||
**no lift even when vec retrieval works**. Add a fifth accept path in
|
||||
`_filter_by_title_relevance` that lets a vec hit survive low title
|
||||
overlap **only if it earns a stronger span-level signal** — at least
|
||||
one of: (a) the cited span contains a deterministic anchor from the
|
||||
question or the generated claim; (b) a claim-lattice pointer resolves
|
||||
to a span whose anchor-class warrant matches the claim; (c) an
|
||||
entity/predicate/date/quote warrant passes on the span; (d) a
|
||||
`concept_relations` synonym edge bridges a query anchor to a span
|
||||
anchor; (e) the answer is explicitly exploratory / `UNGROUNDED` and
|
||||
does **not** reach an EVIDENCE-WARRANTED / STRICT label. In short: vec
|
||||
may bypass title overlap only by earning a stronger warrant, never by
|
||||
embedding proximity alone. Decide the exact predicate set at
|
||||
implementation time from the bench, but the *principle* — "vec hits
|
||||
clear the gate via warrant strength, not similarity score" — is
|
||||
non-negotiable.
|
||||
|
||||
**Mixed fleet.** Skip the vec route cleanly when `chunk_vecs` is
|
||||
absent/empty on a shard (some shards have an imported vecpack — #000051
|
||||
— some don't) — falls back to FTS5-only for that shard, with an audit
|
||||
note `vec_route_unavailable` so a hybrid-labelled run never silently
|
||||
ran FTS5-only.
|
||||
|
||||
**Governance — wire it before any cached hybrid run, not "later".**
|
||||
Fold the **six** vec config fields — `embedder_name`,
|
||||
`embedder_version` (pinned revision/digest, separate from the name),
|
||||
`quantization` (named *with its recipe*: `int8sym` =
|
||||
symmetric-global-scale ×127/clamp/round, not bare `int8`), `dimension`,
|
||||
`distance_metric`, `ann_index` — into `governance_policy_hash`
|
||||
(#000039 §6), summarized by the compact `VEC_BACKEND_VERSION` token
|
||||
(e.g. `vec-v1-bge-small-en-v1.5-384int8sym-l2norm-flat`; the token is
|
||||
a *derived summary*, the six fields are the auditable payload). Until
|
||||
that wiring lands, a **cache-write guard**: a QA run whose retrieval
|
||||
backend is `vec`/`hybrid` may return an answer but **must not persist a
|
||||
`providence_cache` record** (status `vec_experimental_no_cache`) —
|
||||
because changing the embedder changes the candidates changes the
|
||||
context changes the answer, and an un-versioned cached hybrid answer is
|
||||
exactly the silent provenance drift the substrate exists to prevent.
|
||||
The 8-dim `cache_key` invariant is untouched (existing dim, not a new
|
||||
one).
|
||||
|
||||
**Run-DAG metadata.** A hybrid run's per-run Merkle-DAG records the vec
|
||||
retrieval stage: `route="vec"`, `VEC_BACKEND_VERSION`, the six config
|
||||
fields, `top_k`, a `query_embedding_hash`, and the candidate
|
||||
`(chunk_id, quantized_distance)` list (or a `candidate_root` over
|
||||
them). `governance_policy_hash` says *which regime was allowed*; the
|
||||
run-DAG says *what actually happened this run* — both must be visible
|
||||
for replay (same lesson as the `--retrieval-keywords` audit gap,
|
||||
#000001).
|
||||
|
||||
**The knob.** `arborist query` / the bench harness: a
|
||||
`--retrieval-backend {fts5,hybrid}` knob (default `fts5` until the
|
||||
bench clears `hybrid`, then flipping the default is a fox call + a
|
||||
`governance_policy_hash` bump). No vec-*only* QA path except
|
||||
explicitly-experimental (vec-only is a retrieval diagnostic, not an
|
||||
answer regime).
|
||||
|
||||
**Adversarial semantic-neighbor fixtures.** Vec opens a new failure
|
||||
class: a chunk that is *semantically* near the query but factually
|
||||
wrong — same topic / wrong entity, same title family / wrong franchise
|
||||
installment, same era / wrong date, a paraphrase that retrieves the
|
||||
*background* source instead of the *primary* one (the "Mr. Burns" /
|
||||
"which Back to the Future" shape, now reachable through a semantic side
|
||||
door). Build a small adversarial pack for the §2 bench whose pass
|
||||
condition is: hybrid must **not** raise the rate of confidently-wrong
|
||||
warranted answers, and the title-relevance / Rule-8 hard checks must
|
||||
still catch the mismatched vec hits. This pack is part of the gate, not
|
||||
a nice-to-have.
|
||||
|
||||
## 4. Out of scope
|
||||
|
||||
- The corpus backfill itself (a batch op, not code — #000039 §14.6;
|
||||
prerequisite #1 above).
|
||||
- The corpus backfill itself (a batch op — #000039 §14.6) **and its
|
||||
distribution** (vecpack export/import + mesh payload — **#000051**).
|
||||
Prereq #1 above is "a vecpack exists and is imported on the bench
|
||||
box", which #000051 delivers.
|
||||
- A GPU/accelerated embedder (drop-in via `default_embedder()`'s
|
||||
pluggable `Embedder` callable; orthogonal — #000039 §14.6).
|
||||
- int8 vs float32 default (settled: int8 is the production config
|
||||
|
|
@ -108,24 +192,50 @@ Phase 2 implementation does **not** open until both:
|
|||
|
||||
## 5. Acceptance criteria
|
||||
|
||||
1. `arborist/qa/query.py` merges a fifth (vec) route via RRF, behind
|
||||
a `--retrieval-backend` knob; FTS5-only behavior unchanged when
|
||||
the knob says `fts5` or `chunk_vecs` is absent.
|
||||
2. The 5 vec hyperparams fold into `governance_policy_hash`.
|
||||
3. Bench data recorded showing the §2 gate cleared (or, if it
|
||||
didn't, this ticket parks with the bench result documented and
|
||||
`hybrid` left non-default).
|
||||
4. Tests: hybrid-merge unit tests (RRF math; vec-absent fallback;
|
||||
filter chain runs on vec hits); a bench-row in the QA-modes
|
||||
journal.
|
||||
1. `arborist/qa/query.py` merges a fifth (vec) route via RRF (with
|
||||
route provenance on each hit), behind a `--retrieval-backend
|
||||
{fts5,hybrid}` knob; FTS5-only behavior byte-unchanged when the
|
||||
knob says `fts5` or `chunk_vecs` is absent (mixed-fleet fallback
|
||||
emits the `vec_route_unavailable` audit note).
|
||||
2. Accept-path-5 lands in `_filter_by_title_relevance`: a low-title-
|
||||
overlap vec hit survives only via a stronger span-level warrant
|
||||
(anchor / pointer-warrant / entity-predicate-date-quote /
|
||||
concept-synonym / explicitly-`UNGROUNDED`), never via similarity
|
||||
score alone.
|
||||
3. The **six** vec config fields fold into `governance_policy_hash`,
|
||||
summarized by `VEC_BACKEND_VERSION` (recipe-named quant, e.g.
|
||||
`int8sym`); **and** a cache-write guard blocks `providence_cache`
|
||||
persistence for `vec`/`hybrid` runs until that wiring is live
|
||||
(status `vec_experimental_no_cache`).
|
||||
4. A hybrid run's run-DAG records the vec retrieval stage (backend
|
||||
version, six fields, `top_k`, query-embedding hash, candidate
|
||||
chunk_ids + quantized distances / `candidate_root`).
|
||||
5. Bench data recorded for all **four conditions** (A FTS5-only / B
|
||||
vec-only / C RRF hybrid / D candidate-union-no-RRF) on the
|
||||
semantic-allusion + curated + adversarial-neighbor fixtures,
|
||||
showing the §2 gate cleared — including C-beats-D — (or, if it
|
||||
didn't, this ticket parks with the result documented and `hybrid`
|
||||
left non-default; if C ≈ D, ship the union, drop RRF).
|
||||
6. Tests: RRF math + route provenance; vec-absent fallback + the
|
||||
audit note; accept-path-5 (a known semantic-allusion hit survives
|
||||
low title overlap *only* with a warrant, a bare-similarity hit
|
||||
does not); cache-write guard fires for un-versioned hybrid;
|
||||
filter chain + Rule-8 still reject adversarial semantic-neighbor
|
||||
hits; a bench-row in the QA-modes journal.
|
||||
|
||||
## 6. References
|
||||
|
||||
- #000039 — the parent (Phase 0 + Phase 1, closed 2026-05-12);
|
||||
§4 (additive not replacement), §4.2 (RRF design), §8 (the gate),
|
||||
§14 (ingest integration), §14.6 (embed throughput).
|
||||
- **#000051 — federated vecpack distribution.** Supplies prereq #1
|
||||
(a corpus backfill) as a *distributable artifact* — backfill once
|
||||
on any CPU box, gossip the `chunk_vecs` over mesh — so the laptop
|
||||
never embeds. Read it for why vec data is safe to gossip with a
|
||||
cheap structural gate (soft data, never proof path).
|
||||
- `docs/bench-maxing.md` — the 5pp signal floor + bench discipline.
|
||||
- `docs/benchmarks.md` — bench harness orientation.
|
||||
- #000037 §3.1 — the Prometheus-Σ unconscious-sweep task that's the
|
||||
natural home for the lazy embed pass that backfills `chunk_vecs`.
|
||||
- CLAUDE.md "Retrieval pipeline" — the four FTS5 routes vec joins.
|
||||
natural home for the embed backfill *and* the vecpack producer.
|
||||
- CLAUDE.md "Retrieval pipeline" — the four FTS5 routes vec joins;
|
||||
"soft hash vs hard hash" — why vec stays out of the proof path.
|
||||
|
|
|
|||
216
docs/tickets/ticket-000051-federated-vecpack-distribution.md
Normal file
216
docs/tickets/ticket-000051-federated-vecpack-distribution.md
Normal file
|
|
@ -0,0 +1,216 @@
|
|||
# Ticket #000051 — Federated vecpack distribution (gossip the embedding backfill)
|
||||
|
||||
**Status:** open · awaiting go/no-go — doc-only scaffold. Removes the
|
||||
"who runs the days-long embed job" friction from #000050's gate by
|
||||
making `chunk_vecs` a *distributable artifact*: backfill once on any
|
||||
box with CPU (a cloud node, a Prometheus-Σ sweep box — #000037 §3.1),
|
||||
publish a **vecpack** over the existing mesh wire layer, every peer
|
||||
pulls + bulk-loads it. The transformer never runs on a laptop.
|
||||
**Opened:** 2026-05-12
|
||||
**Scope:** A serialize/verify/load path for per-shard embedding sets,
|
||||
plus a mesh payload type carrying them. A vecpack is
|
||||
`(shard_root, vec_backend_version, [(leaf_hash, embedding_blob), …])`
|
||||
— content-addressed by `leaf_hash` (the chunk's hard hash), keyed in
|
||||
transit on `leaf_hash` not `chunk_id` (`chunk_id` is a shard-local
|
||||
rowid; `leaf_hash` is portable). A receiving node maps
|
||||
`leaf_hash → local chunk_id` against its own `chunks` table and
|
||||
bulk-INSERTs into `chunk_vecs` — sub-millisecond per chunk, the cheap
|
||||
side of the #000039 §14.6 asymmetry. Storage cost is real (~384 B/chunk
|
||||
at int8sym, ~2.4 GB for the 6.24M-chunk wiki set, ~6% over the shards)
|
||||
but it is a *download + disk* cost, never a *compute* cost, on the
|
||||
receiver.
|
||||
**Audience:** fox + future blackops shifts + anyone running a public
|
||||
arborist node that wants the semantic layer without an accelerator.
|
||||
**Hard constraint:** vecpack payloads are **soft data** — embeddings
|
||||
are `audit_mode=UNGROUNDED`, never in the proof path (#000039 §4,
|
||||
CLAUDE.md "soft hash vs hard hash"). A wrong or adversarial gossiped
|
||||
embedding can only nudge retrieval recall on the receiver; it cannot
|
||||
corrupt a `cache_key`, a `document_root`, or an `audit_event_hash`.
|
||||
**So vecpacks do not need Merkle-proof-grade verification** — a cheap
|
||||
structural sanity gate is sufficient and correct (see §3). The `cache_key`
|
||||
8-dim invariant is untouched; the vec config still folds into
|
||||
`governance_policy_hash` per #000050 (this ticket distributes the
|
||||
vectors, #000050 makes a QA answer that *consulted* them
|
||||
policy-hash-honest).
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this is a separate ticket
|
||||
|
||||
#000050's prerequisite #1 — "a corpus backfill exists" — was written
|
||||
as a batch job *somebody runs*, with fox's laptop implicitly the
|
||||
somebody. It is not: the embed pass is ~10–100× heavier than the rest
|
||||
of ingest (#000039 §14.6) and a full-corpus backfill is hours-on-idle
|
||||
/ days-on-contended. fox does not have an accelerator and should never
|
||||
need one. The fix is distribution, not faster local embedding:
|
||||
|
||||
```
|
||||
backfill once, anywhere with CPU
|
||||
→ publish a vecpack per shard over mesh
|
||||
→ every peer pulls + bulk-loads it
|
||||
→ laptop pays download + sqlite INSERTs only
|
||||
```
|
||||
|
||||
That is a distinct mechanism from #000050's RRF fusion (which is about
|
||||
*using* `chunk_vecs` once present) and from #000039's backend (which
|
||||
is about *building* it locally), so it gets its own ticket per repo
|
||||
convention rather than bloating either.
|
||||
|
||||
This is also the concrete answer to the whitepaper §1 claim that "the
|
||||
embedding pass runs off the device" — gossip is *how* it reaches the
|
||||
device. The two strengthen each other.
|
||||
|
||||
## 2. Why vec data gossips well (the trust-model argument)
|
||||
|
||||
An embedding is a pure function of `(chunk content bytes, embedder
|
||||
config)`. Two consequences:
|
||||
|
||||
1. **Content-addressable.** `embedding = f(leaf_content,
|
||||
vec_backend_version)`. A vecpack keyed on `(leaf_hash,
|
||||
vec_backend_version)` is the natural unit — a peer either has the
|
||||
chunk (knows its `leaf_hash`) or doesn't; if it does, the embedding
|
||||
slots straight in. No `chunk_id` alignment problem (those are
|
||||
shard-local rowids and must NOT be trusted across DBs even when two
|
||||
shards were built from the same dump).
|
||||
2. **Soft by construction → cheap to verify.** Embeddings never enter
|
||||
a proof. The worst a bad vecpack does is degrade *recall* on the
|
||||
receiver — and even that is bounded, because the post-retrieval
|
||||
filter chain (title-relevance, rivalry exclusion, Rule-8
|
||||
title-mismatch, and the verifier itself) still runs on every
|
||||
candidate vec surfaces. So a vecpack does **not** need a Merkle
|
||||
inclusion proof or a signed-by-the-corpus-author guarantee. It
|
||||
needs: (a) the chunks it references actually exist locally with
|
||||
matching `leaf_hash`, (b) each blob is the right length for the
|
||||
declared `(dim, quant)`, (c) each decoded vector has a finite,
|
||||
sane norm, (d) `vec_backend_version` matches the receiver's
|
||||
`vec_meta` (or the receiver is creating `vec_meta` fresh from this
|
||||
pack). That is a cheap structural gate, not cryptographic
|
||||
verification. (Contrast: cache records gossiped over mesh *are*
|
||||
proof-bearing and *do* carry the full audit-chain machinery —
|
||||
vecpacks deliberately don't, and that's the right call.)
|
||||
|
||||
This is the same hard/soft discipline as #000035 (φ_PRG bytes pinned)
|
||||
and #000036 (soft channels bounded, never confused with hard proof):
|
||||
the layer that's safe to distribute loosely is exactly the layer that
|
||||
can't hurt you if it's wrong.
|
||||
|
||||
## 3. Vecpack format (sketch)
|
||||
|
||||
```
|
||||
vecpack-v1:
|
||||
shard_root # the shard this pack embeds (sanity-bind, not a proof)
|
||||
vec_backend_version # e.g. vec-v1-bge-small-en-v1.5-384int8sym-l2norm-flat
|
||||
dim # 384
|
||||
quant # "int8sym" | "float32"
|
||||
metric # "l2norm" (cosine-equivalent for unit vectors)
|
||||
count # number of (leaf_hash, blob) entries
|
||||
entries: [ (leaf_hash: 32 bytes, embedding_blob: dim*bytes_per_elem) , … ]
|
||||
```
|
||||
|
||||
Wire it as a mesh payload (`arborist/mesh/wire.py` `WireEnvelope`
|
||||
already does signed/AEAD-wrapped bodies — a vecpack is just another
|
||||
body type; the signature authenticates the *sender*, not the
|
||||
*correctness* of the embeddings, which is fine because they're soft).
|
||||
Large packs chunk into multiple envelopes keyed by `(shard_root,
|
||||
vec_backend_version, part_i, n_parts)`.
|
||||
|
||||
Receiver algorithm:
|
||||
|
||||
```
|
||||
on vecpack(shard_root, vec_backend_version, dim, quant, metric, entries):
|
||||
if local shard_root != shard_root: -> reject (wrong shard)
|
||||
if chunk_vecs exists and vec_meta.backend_version != vec_backend_version:
|
||||
-> reject (rebuild needed) or stage separately
|
||||
if not chunk_vecs: ensure_chunk_vecs_table(quant=quant) # creates vec_meta from this pack
|
||||
for (leaf_hash, blob) in entries:
|
||||
cid = SELECT chunk_id FROM chunks WHERE leaf_hash = ? # local rowid
|
||||
if cid is None: continue # don't have this chunk; skip
|
||||
if len(blob) != dim * bytes_per_elem(quant): skip + count_bad
|
||||
if not finite_sane_norm(decode(blob, quant)): skip + count_bad
|
||||
stage (cid, wrap(blob, quant)) # vec_f32(?) / vec_int8(?)
|
||||
bulk delete-then-insert staged rows into chunk_vecs
|
||||
emit status: {received, loaded, skipped_missing_chunk, skipped_bad, backend_version}
|
||||
```
|
||||
|
||||
Idempotent: re-loading the same vecpack is a no-op (delete-then-insert
|
||||
on the same `chunk_id` with the same blob). Incremental: a partial
|
||||
vecpack (some shards, some chunks) loads what it can; the rest stay
|
||||
FTS5-only until another pack arrives — the mixed-fleet case #000050 §3
|
||||
already has to handle.
|
||||
|
||||
## 4. Producer side
|
||||
|
||||
```
|
||||
arborist vecpack export --shard <db> [--out path] [--max-bytes N (split)]
|
||||
# SELECT leaf_hash, embedding FROM chunks JOIN chunk_vecs USING(chunk_id)
|
||||
# → vecpack-v1 file(s); records vec_backend_version + dim + quant from vec_meta
|
||||
arborist vecpack import --shard <db> --in path
|
||||
# the receiver algorithm above; prints the status dict
|
||||
```
|
||||
|
||||
Mesh-native variant (when mesh is enabled): a node advertises which
|
||||
`(shard_root, vec_backend_version)` packs it serves; a peer requests
|
||||
the ones it lacks; transfer rides the existing wire layer. The
|
||||
Prometheus-Σ unconscious-sweep task (#000037 §3.1) is the natural
|
||||
producer — it backfills `chunk_vecs` off-peak *and* exports the pack
|
||||
*and* answers pull requests, all as one low-priority background role.
|
||||
|
||||
## 5. Out of scope
|
||||
|
||||
- The local embed backend + `arborist embed` itself (#000039 — done).
|
||||
- RRF hybrid fusion in `query.py` (#000050).
|
||||
- A GPU embedder (orthogonal — #000039 §14.6; a faster producer, same
|
||||
vecpack output).
|
||||
- Strong/cryptographic verification of embedding *correctness* (a peer
|
||||
can re-embed a sample and spot-check if it wants higher assurance,
|
||||
but that's an optional belt-and-suspenders, not part of v1 — the
|
||||
soft-data argument in §2 is why it isn't required).
|
||||
- A content-distribution network / torrent-style fan-out for very
|
||||
large packs (if the 2.4 GB wiki pack ever needs it; v1 is
|
||||
point-to-point over mesh + a plain file export).
|
||||
|
||||
## 6. Acceptance criteria
|
||||
|
||||
1. `vecpack-v1` serialize/parse round-trips (`leaf_hash`-keyed,
|
||||
`vec_backend_version`/`dim`/`quant`/`metric` in the header).
|
||||
2. `arborist vecpack export` / `import` work file-to-file across two
|
||||
independently-built copies of the same shard (chunk_ids differ,
|
||||
leaf_hashes match → embeddings land correctly).
|
||||
3. The receiver's structural sanity gate rejects: wrong `shard_root`,
|
||||
`backend_version` mismatch against existing `vec_meta`, wrong blob
|
||||
length, non-finite vectors — each with a distinct status field, no
|
||||
silent acceptance.
|
||||
4. Import is idempotent and incremental (partial packs load what they
|
||||
can; missing chunks counted, not errored).
|
||||
5. Mesh payload type wired into `arborist/mesh/wire.py` (or a thin
|
||||
sibling), signed-by-sender, AEAD-wrapped like other bodies; large
|
||||
packs split/reassemble by part index.
|
||||
6. Tests: round-trip; cross-shard leaf_hash mapping; each sanity-gate
|
||||
rejection path; idempotent re-import; mesh envelope carry; the
|
||||
soft-data boundary (a deliberately-corrupted blob is rejected by
|
||||
the gate, and even if it weren't, it's `UNGROUNDED` and the
|
||||
post-retrieval filters still run — i.e. a unit test asserting a bad
|
||||
vec hit can't reach the proof path).
|
||||
7. Docs: a short section in `docs/mesh.md` (vecpack as a payload type)
|
||||
+ a pointer from #000050 §2 prereq #1 ("backfill once, distribute
|
||||
via #000051").
|
||||
|
||||
## 7. References
|
||||
|
||||
- #000039 — the local vec backend (Phase 0 + Phase 1, closed
|
||||
2026-05-12); §14.6 (embed throughput — the asymmetry this ticket
|
||||
routes around), §4 (additive, never proof path).
|
||||
- #000050 — RRF hybrid fusion; this ticket supplies prereq #1 (a
|
||||
corpus backfill) as a *distributable artifact* rather than a job
|
||||
fox runs locally.
|
||||
- #000037 §3.1 — the Prometheus-Σ unconscious-sweep task: the natural
|
||||
vecpack *producer* (backfill off-peak, export, serve pulls).
|
||||
- `arborist/mesh/wire.py` — the signed/AEAD wire envelope a vecpack
|
||||
payload rides on. `docs/mesh.md` — wire format + group-key state.
|
||||
- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst`
|
||||
§1 — "the embedding pass runs off the device"; vecpack distribution
|
||||
is the mechanism behind that sentence.
|
||||
- CLAUDE.md "soft hash vs hard hash" — why soft data (embeddings) is
|
||||
safe to gossip with a cheap structural gate instead of proof-grade
|
||||
verification. #000035 / #000036 — the same discipline applied to
|
||||
PRG bytes / soft channels.
|
||||
Loading…
Add table
Add a link
Reference in a new issue