From 2ad2dceada940a024ed20d6eb20f124134f03b25 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Tue, 12 May 2026 07:14:12 -0400 Subject: [PATCH] =?UTF-8?q?ticket=20#000039:=20record=20embed-throughput?= =?UTF-8?q?=20measurement=20+=20ingest-stays-fast=20rationale=20(=C2=A714.?= =?UTF-8?q?6)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New §14.6 captures the 2026-05-11 findings: - Measured embed rate: ~4.3 chunks/s on the contended dev box (load ~11 on 8 cores; the embed process got ~28% of one core). One wiki shard (~1.56M chunks) at that rate ≈ 100h ≈ 4+ days; all four ≈ ~16 days. The full int8 backfill was abandoned as not feasible to brute-force there. - The 54K-chunk partial on 000.db confirmed ~409 B/chunk apparent → 384 B amortized → ~2.4 GB for the full 6.24M-chunk corpus at int8 — the deterministic number a full backfill would only re-confirm, so finishing it bought nothing. - On an idle healthy box (batching + all cores) bge-small does ~50-200 chunks/s → full corpus ≈ ~9-35h (the ticket's earlier "~17h" is the optimistic end). - Per-chunk cost breakdown: bge-small ONNX inference ~5-30 ms/chunk dominates; the existing arborist ingest steps (chunker + SHA-256 leaf + Merkle commit + sqlite INSERTs + zstd + FTS5) are well under 1 ms total. So adding vec multiplies ingest by ~10-100×, entirely in the ONNX matmuls — the non-vec ingest path is unchanged and still runs at hundreds of chunks/s. - Implication for §14.2: this is *why* lazy-out-of-band is the default and `arborist ingest --embed` is the opt-in. Production guidance: a corpus-wide backfill is a one-time batch job (hours on idle / days on contended), best run off-peak or on a dedicated box; it does not slow ongoing ingest (which never embeds unless --embed is passed); a GPU/accelerated embedder is a drop-in via the pluggable Embedder callable if backfill latency matters. Status line updated to point at §14.6. Doc-only. --- ...cket-000039-sqlite-vec-optional-backend.md | 54 ++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-) diff --git a/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md b/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md index fd45090..9a85938 100644 --- a/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md +++ b/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md @@ -1,6 +1,6 @@ # Ticket #000039 — Optional `sqlite-vec` retrieval backend (A/B vs FTS5, hybrid not replacement) -**Status:** in progress · Phase 0 (doc) + **Phase 1 landed 2026-05-11**. Phase 1: `arborist/search/vec.py` — `VecBackend(SearchBackend)` (UNGROUNDED hits, never in proof path), `chunk_vecs` vec0 virtual table + `vec_meta` (sibling tables — don't touch chunks/documents/audit chain), `embed_documents()` ingest (delete-then-insert idempotent; vec0 doesn't honor INSERT-OR-REPLACE), pluggable `Embedder` callable with a fastembed `bge-small-en-v1.5` default. CLI: `arborist embed [--limit] [--batch-size]` + `arborist search --backend vec`. `[vec]` optional extra (sqlite-vec + fastembed). **Obvious v1 tuning** (`VEC_BACKEND_VERSION = vec-v1-bge-small-en-v1.5-384float32-cosine-flat`): embedder `BAAI/bge-small-en-v1.5`, dim 384, quant float32 (int8/binary = the production storage knob per §3.1, not wired in v1), metric cosine (bge outputs L2-normalized so cosine ≡ L2 ranking), ANN flat (vec0 default), top_k 20. 7 tests (`tests/test_search_vec.py`, stub embedder — plumbing only; semantic quality demonstrated on a real shard). **Demonstrated on `crawl_appliedcombinatorics_org.db`** (168 chunks embedded in ~37s incl. model load; semantic queries return topically-correct hits — "how many ways to choose k things from n" → top hit "AC Combinations"; chain-check on that shard reports 0 after embedding). The 5 hyperparams fold into `governance_policy_hash` in a later phase (§6 — not wired yet). **Ingest integration landed 2026-05-11** (see §14): `embed_documents()` is now **incremental by default** (embeds only `chunk_id`s not already in `chunk_vecs` — re-runs are cheap no-ops); `arborist embed --rebuild` does the DROP+recreate+full-re-embed for a `VEC_BACKEND_VERSION` bump; `arborist ingest --embed` is the **eager opt-in** (embed this run's new chunks after the chunk+Merkle-commit pass; default ingest does NOT embed — the lazy `arborist embed` pass / cron / Prometheus-Σ unconscious sweep is the usual path). **`--quant {float32,int8}` landed 2026-05-11** (`arborist embed --quant int8 [--rebuild]`; the `chunk_vecs` vec0 column is `int8[384]` vs `float[384]` per quant; int8 blobs are `vec_int8(?)`-wrapped — sqlite-vec v0.1.9 treats a bare blob as float32; quant change on an existing table requires `--rebuild` since the vec0 element type can't be altered in place; the quant folds into `VEC_BACKEND_VERSION` → `...-384int8-...`, recorded in `vec_meta`). **int8 head-to-head on `crawl_appliedcombinatorics_org.db`**: storage 1.60 MB → **0.42 MB (3.8× smaller; ~4× at corpus scale where blocks fill)**; recall ≈ float32 — Q2 "how many ways to choose k things from n" identical top-5, Q1 "pigeonhole principle counting" identical top-2 with a sub-noise rank-3/4 swap (Δdistance 0.002). So int8 is the obvious production config (§3.1's +6%-tax recommendation confirmed) — v1 default stays float32 for max fidelity; switching the default to int8 is a fox call. 16 vec tests. **Phase 2** (RRF hybrid fusion in `query.py`) gated on a ≥5pp recall-lift measurement on bench fixtures with no STRICT-rate regression (§8). +**Status:** in progress · Phase 0 (doc) + **Phase 1 landed 2026-05-11**. Phase 1: `arborist/search/vec.py` — `VecBackend(SearchBackend)` (UNGROUNDED hits, never in proof path), `chunk_vecs` vec0 virtual table + `vec_meta` (sibling tables — don't touch chunks/documents/audit chain), `embed_documents()` ingest (delete-then-insert idempotent; vec0 doesn't honor INSERT-OR-REPLACE), pluggable `Embedder` callable with a fastembed `bge-small-en-v1.5` default. CLI: `arborist embed [--limit] [--batch-size]` + `arborist search --backend vec`. `[vec]` optional extra (sqlite-vec + fastembed). **Obvious v1 tuning** (`VEC_BACKEND_VERSION = vec-v1-bge-small-en-v1.5-384float32-cosine-flat`): embedder `BAAI/bge-small-en-v1.5`, dim 384, quant float32 (int8/binary = the production storage knob per §3.1, not wired in v1), metric cosine (bge outputs L2-normalized so cosine ≡ L2 ranking), ANN flat (vec0 default), top_k 20. 7 tests (`tests/test_search_vec.py`, stub embedder — plumbing only; semantic quality demonstrated on a real shard). **Demonstrated on `crawl_appliedcombinatorics_org.db`** (168 chunks embedded in ~37s incl. model load; semantic queries return topically-correct hits — "how many ways to choose k things from n" → top hit "AC Combinations"; chain-check on that shard reports 0 after embedding). The 5 hyperparams fold into `governance_policy_hash` in a later phase (§6 — not wired yet). **Ingest integration landed 2026-05-11** (see §14): `embed_documents()` is now **incremental by default** (embeds only `chunk_id`s not already in `chunk_vecs` — re-runs are cheap no-ops); `arborist embed --rebuild` does the DROP+recreate+full-re-embed for a `VEC_BACKEND_VERSION` bump; `arborist ingest --embed` is the **eager opt-in** (embed this run's new chunks after the chunk+Merkle-commit pass; default ingest does NOT embed — the lazy `arborist embed` pass / cron / Prometheus-Σ unconscious sweep is the usual path). **`--quant {float32,int8}` landed 2026-05-11** (`arborist embed --quant int8 [--rebuild]`; the `chunk_vecs` vec0 column is `int8[384]` vs `float[384]` per quant; int8 blobs are `vec_int8(?)`-wrapped — sqlite-vec v0.1.9 treats a bare blob as float32; quant change on an existing table requires `--rebuild` since the vec0 element type can't be altered in place; the quant folds into `VEC_BACKEND_VERSION` → `...-384int8-...`, recorded in `vec_meta`). **int8 head-to-head on `crawl_appliedcombinatorics_org.db`**: storage 1.60 MB → **0.42 MB (3.8× smaller; ~4× at corpus scale where blocks fill)**; recall ≈ float32 — Q2 "how many ways to choose k things from n" identical top-5, Q1 "pigeonhole principle counting" identical top-2 with a sub-noise rank-3/4 swap (Δdistance 0.002). So int8 is the obvious production config (§3.1's +6%-tax recommendation confirmed) — v1 default stays float32 for max fidelity; switching the default to int8 is a fox call. 16 vec tests. **Embed throughput measured 2026-05-11** (§14.6): ~4.3 chunks/s on the contended dev box (~50-200/s idle); a 54K-chunk wiki backfill confirmed ~409 B/chunk → ~2.4 GB full-corpus at int8 (the deterministic number), then was **abandoned** — full corpus is hours-on-idle / days-on-contended, a one-time off-peak/dedicated-box batch job, not something to brute-force inline. The non-vec ingest path is unchanged (hundreds of chunks/s); adding vec multiplies ingest by ~10-100× (all in the ONNX matmuls) — which is exactly why lazy-out-of-band is the default and `ingest --embed` is the opt-in. **Phase 2** (RRF hybrid fusion in `query.py`) gated on a ≥5pp recall-lift measurement on bench fixtures with no STRICT-rate regression (§8). **Opened:** 2026-05-09 **Scope:** Spec an optional `sqlite-vec` backend that runs **alongside** the existing FTS5 retrieval pipeline (never replacing it), with phased gates @@ -888,3 +888,55 @@ the per-shard discriminator. The 5 vec hyperparams fold into 8-dim `cache_key` invariant is untouched (vec config goes into the existing `governance_policy_hash` dim, not a new one). Same versioned-default discipline as `tok-512-v1` / `norm-v1` etc. + +### 14.6 Embed throughput — measured, and why ingest stays fast (2026-05-11) + +Embedding is the **only** expensive part of the vec layer, and it's +expensive because it's CPU-bound ONNX inference — not because of +anything in arborist's ingest path. + +**Measured.** On the dev box (8 cores, but load average ~11 — parallel +agents saturating CPU), `arborist embed --quant int8` on wiki shard +`000.db` ran at **~4.3 chunks/s** (54,016 chunks embedded in 3 h 30 m; +the embed process got ~28 % of one core). At that rate one shard +(~1.56 M chunks) ≈ 100 h ≈ 4 + days, all four ≈ ~16 days. The full +backfill was **abandoned** as not feasible to brute-force here; the +54 K-chunk partial on `000.db` confirmed the per-chunk storage +(~409 B/chunk apparent → 384 B amortized → ~2.4 GB full corpus at +int8) — which is the deterministic number a backfill would only +re-confirm, so finishing it bought nothing. (On an *idle* healthy +box with batching + all cores, bge-small does ~50-200 chunks/s → +the full 6.24 M corpus ≈ ~9-35 h; the ticket's earlier "~17 h" +estimate is the optimistic end of that.) + +**Per-chunk cost breakdown:** + +| step | per-chunk cost (healthy box) | +|---|---| +| bge-small-en-v1.5 ONNX inference | ~5-30 ms (≫ everything else; chunk-length-dependent, 512-token cap) | +| chunker + SHA-256(leaf) + Merkle commit + sqlite INSERTs + zstd + FTS5 update | well under 1 ms total — the existing arborist ingest steps | + +So adding vec **multiplies ingest time by roughly 10-100×** — entirely +in the ONNX matmuls. The non-vec ingest path is unchanged and still +runs at hundreds of chunks/s. + +**Implication for the integration design (§14.2):** this is *why* +lazy-out-of-band is the default and `arborist ingest --embed` is the +opt-in: + +- `arborist ingest` (default) — no vec, fast, same as before. +- `arborist ingest --embed` — eager; right for incremental ingests + of a handful of docs, **wrong for a bulk wiki backfill** (the embed + cost dominates the run). +- `arborist embed` (separate command) / a cron / a Prometheus-Σ + unconscious-sweep task — the one-time multi-hour-to-multi-day pass, + best run off-peak or on a dedicated box, never blocking a + user-facing ingest. + +**Production guidance:** a real corpus-wide backfill is a one-time +batch job sized in hours-on-idle / days-on-contended. It does **not** +slow ongoing ingest (which never embeds unless you pass `--embed`). +If the backfill latency matters, options are: run it off-peak; use a +dedicated box; or wire GPU/accelerated embedding (out of scope for +this ticket — `default_embedder()` is a pluggable `Embedder` callable, +so a GPU/onnxruntime-gpu/external-endpoint variant is a drop-in).