ticket #000039: record embed-throughput measurement + ingest-stays-fast rationale (§14.6)
New §14.6 captures the 2026-05-11 findings: - Measured embed rate: ~4.3 chunks/s on the contended dev box (load ~11 on 8 cores; the embed process got ~28% of one core). One wiki shard (~1.56M chunks) at that rate ≈ 100h ≈ 4+ days; all four ≈ ~16 days. The full int8 backfill was abandoned as not feasible to brute-force there. - The 54K-chunk partial on 000.db confirmed ~409 B/chunk apparent → 384 B amortized → ~2.4 GB for the full 6.24M-chunk corpus at int8 — the deterministic number a full backfill would only re-confirm, so finishing it bought nothing. - On an idle healthy box (batching + all cores) bge-small does ~50-200 chunks/s → full corpus ≈ ~9-35h (the ticket's earlier "~17h" is the optimistic end). - Per-chunk cost breakdown: bge-small ONNX inference ~5-30 ms/chunk dominates; the existing arborist ingest steps (chunker + SHA-256 leaf + Merkle commit + sqlite INSERTs + zstd + FTS5) are well under 1 ms total. So adding vec multiplies ingest by ~10-100×, entirely in the ONNX matmuls — the non-vec ingest path is unchanged and still runs at hundreds of chunks/s. - Implication for §14.2: this is *why* lazy-out-of-band is the default and `arborist ingest --embed` is the opt-in. Production guidance: a corpus-wide backfill is a one-time batch job (hours on idle / days on contended), best run off-peak or on a dedicated box; it does not slow ongoing ingest (which never embeds unless --embed is passed); a GPU/accelerated embedder is a drop-in via the pluggable Embedder callable if backfill latency matters. Status line updated to point at §14.6. Doc-only.
This commit is contained in:
parent
5428b3b1d6
commit
2ad2dceada
1 changed files with 53 additions and 1 deletions
|
|
@ -1,6 +1,6 @@
|
|||
# Ticket #000039 — Optional `sqlite-vec` retrieval backend (A/B vs FTS5, hybrid not replacement)
|
||||
|
||||
**Status:** in progress · Phase 0 (doc) + **Phase 1 landed 2026-05-11**. Phase 1: `arborist/search/vec.py` — `VecBackend(SearchBackend)` (UNGROUNDED hits, never in proof path), `chunk_vecs` vec0 virtual table + `vec_meta` (sibling tables — don't touch chunks/documents/audit chain), `embed_documents()` ingest (delete-then-insert idempotent; vec0 doesn't honor INSERT-OR-REPLACE), pluggable `Embedder` callable with a fastembed `bge-small-en-v1.5` default. CLI: `arborist embed [--limit] [--batch-size]` + `arborist search --backend vec`. `[vec]` optional extra (sqlite-vec + fastembed). **Obvious v1 tuning** (`VEC_BACKEND_VERSION = vec-v1-bge-small-en-v1.5-384float32-cosine-flat`): embedder `BAAI/bge-small-en-v1.5`, dim 384, quant float32 (int8/binary = the production storage knob per §3.1, not wired in v1), metric cosine (bge outputs L2-normalized so cosine ≡ L2 ranking), ANN flat (vec0 default), top_k 20. 7 tests (`tests/test_search_vec.py`, stub embedder — plumbing only; semantic quality demonstrated on a real shard). **Demonstrated on `crawl_appliedcombinatorics_org.db`** (168 chunks embedded in ~37s incl. model load; semantic queries return topically-correct hits — "how many ways to choose k things from n" → top hit "AC Combinations"; chain-check on that shard reports 0 after embedding). The 5 hyperparams fold into `governance_policy_hash` in a later phase (§6 — not wired yet). **Ingest integration landed 2026-05-11** (see §14): `embed_documents()` is now **incremental by default** (embeds only `chunk_id`s not already in `chunk_vecs` — re-runs are cheap no-ops); `arborist embed --rebuild` does the DROP+recreate+full-re-embed for a `VEC_BACKEND_VERSION` bump; `arborist ingest --embed` is the **eager opt-in** (embed this run's new chunks after the chunk+Merkle-commit pass; default ingest does NOT embed — the lazy `arborist embed` pass / cron / Prometheus-Σ unconscious sweep is the usual path). **`--quant {float32,int8}` landed 2026-05-11** (`arborist embed --quant int8 [--rebuild]`; the `chunk_vecs` vec0 column is `int8[384]` vs `float[384]` per quant; int8 blobs are `vec_int8(?)`-wrapped — sqlite-vec v0.1.9 treats a bare blob as float32; quant change on an existing table requires `--rebuild` since the vec0 element type can't be altered in place; the quant folds into `VEC_BACKEND_VERSION` → `...-384int8-...`, recorded in `vec_meta`). **int8 head-to-head on `crawl_appliedcombinatorics_org.db`**: storage 1.60 MB → **0.42 MB (3.8× smaller; ~4× at corpus scale where blocks fill)**; recall ≈ float32 — Q2 "how many ways to choose k things from n" identical top-5, Q1 "pigeonhole principle counting" identical top-2 with a sub-noise rank-3/4 swap (Δdistance 0.002). So int8 is the obvious production config (§3.1's +6%-tax recommendation confirmed) — v1 default stays float32 for max fidelity; switching the default to int8 is a fox call. 16 vec tests. **Phase 2** (RRF hybrid fusion in `query.py`) gated on a ≥5pp recall-lift measurement on bench fixtures with no STRICT-rate regression (§8).
|
||||
**Status:** in progress · Phase 0 (doc) + **Phase 1 landed 2026-05-11**. Phase 1: `arborist/search/vec.py` — `VecBackend(SearchBackend)` (UNGROUNDED hits, never in proof path), `chunk_vecs` vec0 virtual table + `vec_meta` (sibling tables — don't touch chunks/documents/audit chain), `embed_documents()` ingest (delete-then-insert idempotent; vec0 doesn't honor INSERT-OR-REPLACE), pluggable `Embedder` callable with a fastembed `bge-small-en-v1.5` default. CLI: `arborist embed [--limit] [--batch-size]` + `arborist search --backend vec`. `[vec]` optional extra (sqlite-vec + fastembed). **Obvious v1 tuning** (`VEC_BACKEND_VERSION = vec-v1-bge-small-en-v1.5-384float32-cosine-flat`): embedder `BAAI/bge-small-en-v1.5`, dim 384, quant float32 (int8/binary = the production storage knob per §3.1, not wired in v1), metric cosine (bge outputs L2-normalized so cosine ≡ L2 ranking), ANN flat (vec0 default), top_k 20. 7 tests (`tests/test_search_vec.py`, stub embedder — plumbing only; semantic quality demonstrated on a real shard). **Demonstrated on `crawl_appliedcombinatorics_org.db`** (168 chunks embedded in ~37s incl. model load; semantic queries return topically-correct hits — "how many ways to choose k things from n" → top hit "AC Combinations"; chain-check on that shard reports 0 after embedding). The 5 hyperparams fold into `governance_policy_hash` in a later phase (§6 — not wired yet). **Ingest integration landed 2026-05-11** (see §14): `embed_documents()` is now **incremental by default** (embeds only `chunk_id`s not already in `chunk_vecs` — re-runs are cheap no-ops); `arborist embed --rebuild` does the DROP+recreate+full-re-embed for a `VEC_BACKEND_VERSION` bump; `arborist ingest --embed` is the **eager opt-in** (embed this run's new chunks after the chunk+Merkle-commit pass; default ingest does NOT embed — the lazy `arborist embed` pass / cron / Prometheus-Σ unconscious sweep is the usual path). **`--quant {float32,int8}` landed 2026-05-11** (`arborist embed --quant int8 [--rebuild]`; the `chunk_vecs` vec0 column is `int8[384]` vs `float[384]` per quant; int8 blobs are `vec_int8(?)`-wrapped — sqlite-vec v0.1.9 treats a bare blob as float32; quant change on an existing table requires `--rebuild` since the vec0 element type can't be altered in place; the quant folds into `VEC_BACKEND_VERSION` → `...-384int8-...`, recorded in `vec_meta`). **int8 head-to-head on `crawl_appliedcombinatorics_org.db`**: storage 1.60 MB → **0.42 MB (3.8× smaller; ~4× at corpus scale where blocks fill)**; recall ≈ float32 — Q2 "how many ways to choose k things from n" identical top-5, Q1 "pigeonhole principle counting" identical top-2 with a sub-noise rank-3/4 swap (Δdistance 0.002). So int8 is the obvious production config (§3.1's +6%-tax recommendation confirmed) — v1 default stays float32 for max fidelity; switching the default to int8 is a fox call. 16 vec tests. **Embed throughput measured 2026-05-11** (§14.6): ~4.3 chunks/s on the contended dev box (~50-200/s idle); a 54K-chunk wiki backfill confirmed ~409 B/chunk → ~2.4 GB full-corpus at int8 (the deterministic number), then was **abandoned** — full corpus is hours-on-idle / days-on-contended, a one-time off-peak/dedicated-box batch job, not something to brute-force inline. The non-vec ingest path is unchanged (hundreds of chunks/s); adding vec multiplies ingest by ~10-100× (all in the ONNX matmuls) — which is exactly why lazy-out-of-band is the default and `ingest --embed` is the opt-in. **Phase 2** (RRF hybrid fusion in `query.py`) gated on a ≥5pp recall-lift measurement on bench fixtures with no STRICT-rate regression (§8).
|
||||
**Opened:** 2026-05-09
|
||||
**Scope:** Spec an optional `sqlite-vec` backend that runs **alongside** the
|
||||
existing FTS5 retrieval pipeline (never replacing it), with phased gates
|
||||
|
|
@ -888,3 +888,55 @@ the per-shard discriminator. The 5 vec hyperparams fold into
|
|||
8-dim `cache_key` invariant is untouched (vec config goes into the
|
||||
existing `governance_policy_hash` dim, not a new one). Same
|
||||
versioned-default discipline as `tok-512-v1` / `norm-v1` etc.
|
||||
|
||||
### 14.6 Embed throughput — measured, and why ingest stays fast (2026-05-11)
|
||||
|
||||
Embedding is the **only** expensive part of the vec layer, and it's
|
||||
expensive because it's CPU-bound ONNX inference — not because of
|
||||
anything in arborist's ingest path.
|
||||
|
||||
**Measured.** On the dev box (8 cores, but load average ~11 — parallel
|
||||
agents saturating CPU), `arborist embed --quant int8` on wiki shard
|
||||
`000.db` ran at **~4.3 chunks/s** (54,016 chunks embedded in 3 h 30 m;
|
||||
the embed process got ~28 % of one core). At that rate one shard
|
||||
(~1.56 M chunks) ≈ 100 h ≈ 4 + days, all four ≈ ~16 days. The full
|
||||
backfill was **abandoned** as not feasible to brute-force here; the
|
||||
54 K-chunk partial on `000.db` confirmed the per-chunk storage
|
||||
(~409 B/chunk apparent → 384 B amortized → ~2.4 GB full corpus at
|
||||
int8) — which is the deterministic number a backfill would only
|
||||
re-confirm, so finishing it bought nothing. (On an *idle* healthy
|
||||
box with batching + all cores, bge-small does ~50-200 chunks/s →
|
||||
the full 6.24 M corpus ≈ ~9-35 h; the ticket's earlier "~17 h"
|
||||
estimate is the optimistic end of that.)
|
||||
|
||||
**Per-chunk cost breakdown:**
|
||||
|
||||
| step | per-chunk cost (healthy box) |
|
||||
|---|---|
|
||||
| bge-small-en-v1.5 ONNX inference | ~5-30 ms (≫ everything else; chunk-length-dependent, 512-token cap) |
|
||||
| chunker + SHA-256(leaf) + Merkle commit + sqlite INSERTs + zstd + FTS5 update | well under 1 ms total — the existing arborist ingest steps |
|
||||
|
||||
So adding vec **multiplies ingest time by roughly 10-100×** — entirely
|
||||
in the ONNX matmuls. The non-vec ingest path is unchanged and still
|
||||
runs at hundreds of chunks/s.
|
||||
|
||||
**Implication for the integration design (§14.2):** this is *why*
|
||||
lazy-out-of-band is the default and `arborist ingest --embed` is the
|
||||
opt-in:
|
||||
|
||||
- `arborist ingest` (default) — no vec, fast, same as before.
|
||||
- `arborist ingest --embed` — eager; right for incremental ingests
|
||||
of a handful of docs, **wrong for a bulk wiki backfill** (the embed
|
||||
cost dominates the run).
|
||||
- `arborist embed` (separate command) / a cron / a Prometheus-Σ
|
||||
unconscious-sweep task — the one-time multi-hour-to-multi-day pass,
|
||||
best run off-peak or on a dedicated box, never blocking a
|
||||
user-facing ingest.
|
||||
|
||||
**Production guidance:** a real corpus-wide backfill is a one-time
|
||||
batch job sized in hours-on-idle / days-on-contended. It does **not**
|
||||
slow ongoing ingest (which never embeds unless you pass `--embed`).
|
||||
If the backfill latency matters, options are: run it off-peak; use a
|
||||
dedicated box; or wire GPU/accelerated embedding (out of scope for
|
||||
this ticket — `default_embedder()` is a pluggable `Embedder` callable,
|
||||
so a GPU/onnxruntime-gpu/external-endpoint variant is a drop-in).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue