ticket #000039: --quant {float32,int8} + int8 head-to-head

Wire vector quantization (the §3.1 production knob): arborist embed
--quant int8 [--rebuild]. The chunk_vecs vec0 column becomes int8[384]
vs float[384] per quant; the quant folds into VEC_BACKEND_VERSION
(...-384int8-... / ...-384float32-...) and vec_meta records it per
shard. Switching quant on an existing chunk_vecs requires --rebuild
(the vec0 element type can't be altered in place — embed_documents
raises ValueError telling you to --rebuild).

int8 serialization: scale each bge component by 127 (theoretical
[-1,1] range), clamp to [-127,127], round, serialize_int8. The same
scaling on the query vector → distances comparable; cosine is
scale-invariant so the uniform x127 cancels in the ranking.

sqlite-vec v0.1.9 quirk worked around: a bare blob inserted into a
vec0 column is interpreted as float32 regardless of the column's
declared type — int8 vectors MUST be wrapped in vec_int8(...). So
the INSERT and the MATCH now wrap the blob in vec_f32(?) (float32)
or vec_int8(?) (int8) — constructor name from a fixed dict, no
injection surface. (Discovered the hard way: a bare int8 blob into
an int8[384] column → "expected int8, but a float32 vector was
provided".)

VecBackend reads the quant from the existing chunk_vecs schema (or
defaults to float32) so search uses the matching wrapper. New module
exports: QUANTS, EMBED_QUANT, vec_backend_version(quant), existing_quant.

int8 head-to-head on crawl_appliedcombinatorics_org.db (168 chunks):
- storage: float32 1,597,440 B -> int8 417,792 B = 3.8x smaller
  (~4x at corpus scale where the 1024-vector blocks fill; the
  ~28 KB of vec0 metadata doesn't quarter, hence 3.8 not 4.0).
- recall vs the float32 baseline:
    Q "how many ways to choose k things from n":
      identical top-5 (Combinations, Permutations, Exercises,
      Derangements, Graph Coloring).
    Q "pigeonhole principle counting":
      identical top-2 (Graph Coloring, Exercises); ranks 3-4 swap
      Derangements <-> Permutations at Δdistance 0.002 — sub-noise.
- embed speed unchanged (~3.9 chunks/s — model-load-dominated).
Conclusion: int8 is the obvious production config (§3.1's +6%-tax
recommendation confirmed empirically). v1 default stays float32 for
max fidelity; flipping the default to int8 is a fox call.

CLI (arborist/cli.py): arborist embed --quant {float32,int8}; output
JSON gains "quant"; embed_documents ValueError → exit 2 with the
"--rebuild" hint.

tests/test_search_vec.py (9 -> 16): test_int8_quant_roundtrips
(int8[384] schema, vec_meta version, search round-trip, quant
inferred by VecBackend), test_quant_mismatch_requires_rebuild,
test_invalid_quant_rejected.

#000039 status updated. Full suite: 2343 passed, 28 skipped.

(Unrelated parallel-clone work in the tree — Makefile, arborist/qa/
verify.py, bench/fixtures/5f/*, tests/test_bench_batteries.py,
tests/test_verify.py — is #000046's hard-fixture tier, not touched.)

docs/tickets/ticket-000039-sqlite-vec-optional-backend.md

@@ -1,6 +1,6 @@
 # Ticket #000039 — Optional `sqlite-vec` retrieval backend (A/B vs FTS5, hybrid not replacement)
 
-**Status:** in progress · Phase 0 (doc) + **Phase 1 landed 2026-05-11**. Phase 1: `arborist/search/vec.py` — `VecBackend(SearchBackend)` (UNGROUNDED hits, never in proof path), `chunk_vecs` vec0 virtual table + `vec_meta` (sibling tables — don't touch chunks/documents/audit chain), `embed_documents()` ingest (delete-then-insert idempotent; vec0 doesn't honor INSERT-OR-REPLACE), pluggable `Embedder` callable with a fastembed `bge-small-en-v1.5` default. CLI: `arborist embed [--limit] [--batch-size]` + `arborist search --backend vec`. `[vec]` optional extra (sqlite-vec + fastembed). **Obvious v1 tuning** (`VEC_BACKEND_VERSION = vec-v1-bge-small-en-v1.5-384float32-cosine-flat`): embedder `BAAI/bge-small-en-v1.5`, dim 384, quant float32 (int8/binary = the production storage knob per §3.1, not wired in v1), metric cosine (bge outputs L2-normalized so cosine ≡ L2 ranking), ANN flat (vec0 default), top_k 20. 7 tests (`tests/test_search_vec.py`, stub embedder — plumbing only; semantic quality demonstrated on a real shard). **Demonstrated on `crawl_appliedcombinatorics_org.db`** (168 chunks embedded in ~37s incl. model load; semantic queries return topically-correct hits — "how many ways to choose k things from n" → top hit "AC Combinations"; chain-check on that shard reports 0 after embedding). The 5 hyperparams fold into `governance_policy_hash` in a later phase (§6 — not wired yet). **Ingest integration landed 2026-05-11** (see §14): `embed_documents()` is now **incremental by default** (embeds only `chunk_id`s not already in `chunk_vecs` — re-runs are cheap no-ops); `arborist embed --rebuild` does the DROP+recreate+full-re-embed for a `VEC_BACKEND_VERSION` bump; `arborist ingest --embed` is the **eager opt-in** (embed this run's new chunks after the chunk+Merkle-commit pass; default ingest does NOT embed — the lazy `arborist embed` pass / cron / Prometheus-Σ unconscious sweep is the usual path). 9 vec tests. **Phase 2** (RRF hybrid fusion in `query.py`) gated on a ≥5pp recall-lift measurement on bench fixtures with no STRICT-rate regression (§8).
+**Status:** in progress · Phase 0 (doc) + **Phase 1 landed 2026-05-11**. Phase 1: `arborist/search/vec.py` — `VecBackend(SearchBackend)` (UNGROUNDED hits, never in proof path), `chunk_vecs` vec0 virtual table + `vec_meta` (sibling tables — don't touch chunks/documents/audit chain), `embed_documents()` ingest (delete-then-insert idempotent; vec0 doesn't honor INSERT-OR-REPLACE), pluggable `Embedder` callable with a fastembed `bge-small-en-v1.5` default. CLI: `arborist embed [--limit] [--batch-size]` + `arborist search --backend vec`. `[vec]` optional extra (sqlite-vec + fastembed). **Obvious v1 tuning** (`VEC_BACKEND_VERSION = vec-v1-bge-small-en-v1.5-384float32-cosine-flat`): embedder `BAAI/bge-small-en-v1.5`, dim 384, quant float32 (int8/binary = the production storage knob per §3.1, not wired in v1), metric cosine (bge outputs L2-normalized so cosine ≡ L2 ranking), ANN flat (vec0 default), top_k 20. 7 tests (`tests/test_search_vec.py`, stub embedder — plumbing only; semantic quality demonstrated on a real shard). **Demonstrated on `crawl_appliedcombinatorics_org.db`** (168 chunks embedded in ~37s incl. model load; semantic queries return topically-correct hits — "how many ways to choose k things from n" → top hit "AC Combinations"; chain-check on that shard reports 0 after embedding). The 5 hyperparams fold into `governance_policy_hash` in a later phase (§6 — not wired yet). **Ingest integration landed 2026-05-11** (see §14): `embed_documents()` is now **incremental by default** (embeds only `chunk_id`s not already in `chunk_vecs` — re-runs are cheap no-ops); `arborist embed --rebuild` does the DROP+recreate+full-re-embed for a `VEC_BACKEND_VERSION` bump; `arborist ingest --embed` is the **eager opt-in** (embed this run's new chunks after the chunk+Merkle-commit pass; default ingest does NOT embed — the lazy `arborist embed` pass / cron / Prometheus-Σ unconscious sweep is the usual path). **`--quant {float32,int8}` landed 2026-05-11** (`arborist embed --quant int8 [--rebuild]`; the `chunk_vecs` vec0 column is `int8[384]` vs `float[384]` per quant; int8 blobs are `vec_int8(?)`-wrapped — sqlite-vec v0.1.9 treats a bare blob as float32; quant change on an existing table requires `--rebuild` since the vec0 element type can't be altered in place; the quant folds into `VEC_BACKEND_VERSION` → `...-384int8-...`, recorded in `vec_meta`). **int8 head-to-head on `crawl_appliedcombinatorics_org.db`**: storage 1.60 MB → **0.42 MB (3.8× smaller; ~4× at corpus scale where blocks fill)**; recall ≈ float32 — Q2 "how many ways to choose k things from n" identical top-5, Q1 "pigeonhole principle counting" identical top-2 with a sub-noise rank-3/4 swap (Δdistance 0.002). So int8 is the obvious production config (§3.1's +6%-tax recommendation confirmed) — v1 default stays float32 for max fidelity; switching the default to int8 is a fox call. 16 vec tests. **Phase 2** (RRF hybrid fusion in `query.py`) gated on a ≥5pp recall-lift measurement on bench fixtures with no STRICT-rate regression (§8).
 **Opened:** 2026-05-09
 **Scope:** Spec an optional `sqlite-vec` backend that runs **alongside** the
 existing FTS5 retrieval pipeline (never replacing it), with phased gates