diff --git a/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md b/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md index 478c710..d2ad2dc 100644 --- a/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md +++ b/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md @@ -594,3 +594,195 @@ If storage budget tightens later `000.db`. - Upstream: (v0.1.9 at time of writing; expect breaking changes pre-v1). + +--- + +## 13. Phase 1 implementation plan (proposal, 2026-05-10) + +**Status:** awaiting fox go/no-go on the four §13.1 decisions +below. Phase 1 code does not start until they're settled. + +### 13.1 Phase-0 deliverables — proposed picks + +Four explicit decisions Phase 0 left open. Recommendations: + +1. **Embedder path** → **Path 1** (local sentence-transformer + bundled as optional dep). Preserves CLAUDE.md's "fresh checkout + needs only python3.12 + venv + sqlite3" rule; sidesteps the + Hermes 4-concurrent bottleneck (#000037 §11) entirely; no new + service to operate. Model: **`BAAI/bge-small-en-v1.5`** — + 384-dim, ~33 MB, MIT-licensed, top of MTEB-en/retrieval among + sub-100MB models, unit-normalized output (matches §6.1 + canonical config). +2. **Default quantization** → **int8 × 384 + flat** per the §9 + decision tree's primary leg. 6 % storage tax (~2.4 GB across + the 4-shard cluster), well within the 15 % budget. Speed/quality + default. Storage-conservative alternative `binary × 768 + flat` + (1.6 % tax) reachable via `--vec-quantization=binary --vec-dim=768`. +3. **1k-chunk smoke test** → §13.2 below. +4. **Bench protocol** → §13.3 below. + +### 13.2 Pre-flight smoke (gate before any Phase 1 code lands) + +```bash +make smoke-vec # new make target, ~2 min wall +``` + +What it does: + +- Pull 1k random `chunks.content` rows from `~/.arborist/shards/000.db` + into `/tmp/smoke-vec.db` (read-only on the source shard). +- Embed with `bge-small-en-v1.5` on CPU (single-threaded; ~10 ms/chunk + → ~10 s wall). +- INSERT into `chunk_vecs` virtual table under WAL + `synchronous=NORMAL` + (existing arborist convention). +- Kill -9 mid-insert (process group); reopen; check WAL + recovery + journal mode + row count consistency. +- Run 100 query embeddings × top-20 ANN. Measure p50/p95 query + latency. + +**Gate criteria:** insert throughput ≥ 100 chunk/s, p95 query +latency ≤ 50 ms, zero data loss across the kill -9 boundary. Fail +on any of those → Phase 1 blocked, surface to fox + upstream +issue. + +### 13.3 Bench protocol + +Fixtures (all already in `bench/`): + +- `bench/qa_questions_smoke.txt` — 5-question fast loop +- `bench/qa_questions_progressive_and.txt` — progressive-AND fallback fixture +- Existing `bench-emergent` random-word stress fixture +- Existing `qa-modes-bench` fixture for STRICT-rate guard + +Three conditions: + +| condition | retrieval routes | claim_lattice mode | +|---|---|---| +| Baseline | FTS5 only (current 4 routes) | claim_lattice (JSON) | +| Vec-only | Vec ANN top-20 only | claim_lattice (JSON) | +| Hybrid | FTS5 + Vec, RRF merge (k=60) | claim_lattice (JSON) | + +**Phase 1 success criteria** (from §8): +- Vec-only matches FTS5-only within ±5pp STRICT-rate (sanity) +- AND beats FTS5-only by ≥5pp on at least one semantic-allusion + bench-emergent fixture +- AND no UNGROUNDED-rate regression vs baseline + +**Phase 2 success criteria** (gates on §8): +- Hybrid lifts STRICT-rate by ≥5pp over baseline +- AND no UNGROUNDED-rate regression +- AND title-relevance hard check (Rule 8) catches mismatched vec + hits at the same rate it catches mismatched FTS5 hits + +### 13.4 Code structure + +``` +arborist/embed.py [NEW] + class Embedder: + - lazy-loads BAAI/bge-small-en-v1.5 once per process + - normalize=True (unit vectors, §6.1 canonical config) + - encode(texts: list[str]) -> np.ndarray[N, 384] dtype=int8 + - sentencetransformer dependency under [vec] extras + +arborist/search/vec.py [NEW, mirrors fts5.py] + class VecBackend(SearchBackend): + - audit_mode = AuditMode.UNGROUNDED # §1 hard constraint + - chunk_vecs virtual table creation (lazy; first ingest --embed) + - search(query, limit=20) -> list[Hit] via vec_distance_l2 + +arborist/store.py [PATCH] + - SCHEMA_SQL: NO new always-on tables (vec table is virtual, + created only when --embed runs) + - Add embedder_version, vec_quantization, vec_dim, vec_ann_index, + vec_distance_metric to governance_policy_hash inputs (NOT + cache_key — §6 hard constraint) + +arborist/qa/query.py [PATCH] + - if vec backend present: run RRF merge over (FTS5_routes_merged, + vec_route) with k=60. New code is ~6 lines per §7. + +arborist/cli.py [PATCH] + - ingest --embed flag (gated on _VEC_AVAILABLE import-check) + - query --retrieval={fts5|vec|hybrid} flag + - new subcommand: arborist vec rebuild [--db ... --batch N] + +Makefile [PATCH] + smoke-vec: # §13.2 pre-flight; gate-before-Phase-1 + rebuild-vec: # populate chunk_vecs for an existing shard + bench-vec: # run §13.3 three-condition bench + +pyproject.toml [PATCH] + [project.optional-dependencies] + vec = [ + "sqlite-vec>=0.1.9,<0.2", + "sentence-transformers>=2.7,<3", + "numpy", + ] +``` + +### 13.5 Test plan + +Unit tests: +- `tests/test_embed.py` — deterministic output for fixed input + (regression guard against silent model drift between revs); + unit-norm verification; int8 round-trip. +- `tests/test_search_vec.py` — `VecBackend.search` against a + 3-chunk in-memory shard with hand-crafted embeddings; verify + ranking matches expected; verify `audit_mode=UNGROUNDED`; + verify `_VEC_AVAILABLE=False` graceful degradation. +- `tests/test_governance_policy.py` — verify the 5 new vec-config + fields fold into `governance_policy_hash`; flipping any one + invalidates prior cached records. + +Integration tests (gated on `[vec]` extras installed; skipped +otherwise): +- `tests/test_vec_smoke_integration.py` — runs the §13.2 smoke on + a 100-chunk synthetic shard (smaller than 1k for CI speed); + asserts WAL recovery + row count parity. + +### 13.6 Out of scope for Phase 1 + +- ANN index variants (IVF, DiskANN, HNSW) — Phase 1 ships flat + only. ANN is opt-in via `vec_ann_index` policy field; flat works + at ≤10 M chunks per the upstream sqlite-vec docs. +- Re-embed cost on chunker bump — already a known-large cost + (§10.3); covered by `make rebuild-vec` not Phase 1. +- Edges compaction (§10.6) — different ticket entirely. + +### 13.7 Estimated size + +- New files: 2 (embed.py, search/vec.py) ~ 200 LOC. +- Patches: 4 (store.py, query.py, cli.py, Makefile) ~ 100 LOC. +- Tests: 4 files ~ 250 LOC + fixtures. +- Phase-1 doc append in `docs/`: this §13 + a journey-note bench + result file. +- pyproject.toml: 1 stanza. + +Total: ~550 LOC + ~250 test LOC. Single substantial commit if all +tests pass + smoke succeeds; otherwise broken into the natural +gates (smoke → embed.py → vec.py → integration tests → bench). + +### 13.8 Decisions fox needs to make to unblock Phase 1 + +| # | Decision | Recommendation | +|---|---|---| +| 1 | Embedder path 1 vs 2 vs 3 | Path 1 (local bundle) | +| 2 | Default model name | `BAAI/bge-small-en-v1.5` (MIT, 33 MB) | +| 3 | Default quantization × dim | `int8 × 384 + flat` | +| 4 | Approve `sentence-transformers` PyPI dep under `[vec]` | yes (~120 MB install footprint when extras pulled; default not pulled — `pip install '.[vec]'` only) | + +If fox approves all four → Phase 1 starts with §13.2 smoke as the +first gate. + +If fox rejects path 1 → Phase 1 blocked on standing up +`hermes-embed.ai.unturf.com` (path 2; outside this ticket). + +If fox rejects `int8 × 384` default → propose `binary × 768` as +default (1.6 % tax, ~10–20 % recall hit per §9). Either is +defensible. + +If fox rejects the dep footprint → fall back to the lighter +`onnxruntime` path with a `bge-small` ONNX model (still ~33 MB +weights but smaller runtime). Adds complexity but keeps the +no-server constraint.