ticket #000039: §13 Phase 1 implementation plan (proposal)

Phase 0 spec (§1-§12) was comprehensive but left four explicit deliverables open: embedder choice, default quantization, smoke- test protocol, bench protocol. §13 fills those four with concrete recommendations + a code structure / test plan / size estimate that fox can sign off on before Phase 1 code lands. Recommendations: - Embedder path 1 (local sentence-transformer bundled as optional dep); model BAAI/bge-small-en-v1.5 (MIT, 33 MB, 384-dim, unit-normalized, top of MTEB-en/retrieval among sub-100MB models) - Default quantization int8 × 384 + flat (6% storage tax, within the 15% budget per §3.1; binary × 768 reachable via --vec-quantization=binary) - Pre-Phase-1 smoke (§13.2): 1k chunks under WAL + synchronous=NORMAL, kill -9 mid-insert, recovery check; gate on insert ≥100 chunk/s, p95 query ≤50 ms, zero data loss - Bench protocol (§13.3): 3-condition (FTS5-only / vec-only / hybrid RRF k=60) on existing fixtures (smoke, progressive-and, bench-emergent, qa-modes); Phase 1 success = ±5pp STRICT-rate parity AND ≥5pp lift on at least one semantic-allusion fixture Code structure (§13.4): 2 new files (embed.py + search/vec.py) ~200 LOC, 4 patches (store.py + query.py + cli.py + Makefile) ~100 LOC, 4 test files ~250 LOC, pyproject.toml [vec] extras stanza. Single substantial commit when all gates pass. §13.8 lists the four go/no-go decisions fox needs to make to unblock Phase 1: embedder path, model name, default quantization, and approval of the sentence-transformers PyPI dep under [vec] extras (not pulled by default; only on pip install '.[vec]'). Fallback paths documented for each rejection. Phase 0 doc remains awaiting go/no-go; §13 doesn't change that gate, just provides the substance for fox's decision.
2026-05-10 10:40:07 -04:00 · 2026-05-10 10:40:07 -04:00 · 96e64b88e6
commit 96e64b88e6
parent 3b37c93810
1 changed files with 192 additions and 0 deletions
--- a/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md
+++ b/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md
@ -594,3 +594,195 @@ If storage budget tightens later
  `000.db`.
 - Upstream: <https://github.com/asg017/sqlite-vec> (v0.1.9 at time
  of writing; expect breaking changes pre-v1).
+
+---
+
+## 13. Phase 1 implementation plan (proposal, 2026-05-10)
+
+**Status:** awaiting fox go/no-go on the four §13.1 decisions
+below. Phase 1 code does not start until they're settled.
+
+### 13.1 Phase-0 deliverables — proposed picks
+
+Four explicit decisions Phase 0 left open. Recommendations:
+
+1. **Embedder path** → **Path 1** (local sentence-transformer
+   bundled as optional dep). Preserves CLAUDE.md's "fresh checkout
+   needs only python3.12 + venv + sqlite3" rule; sidesteps the
+   Hermes 4-concurrent bottleneck (#000037 §11) entirely; no new
+   service to operate. Model: **`BAAI/bge-small-en-v1.5`** —
+   384-dim, ~33 MB, MIT-licensed, top of MTEB-en/retrieval among
+   sub-100MB models, unit-normalized output (matches §6.1
+   canonical config).
+2. **Default quantization** → **int8 × 384 + flat** per the §9
+   decision tree's primary leg. 6 % storage tax (~2.4 GB across
+   the 4-shard cluster), well within the 15 % budget. Speed/quality
+   default. Storage-conservative alternative `binary × 768 + flat`
+   (1.6 % tax) reachable via `--vec-quantization=binary --vec-dim=768`.
+3. **1k-chunk smoke test** → §13.2 below.
+4. **Bench protocol** → §13.3 below.
+
+### 13.2 Pre-flight smoke (gate before any Phase 1 code lands)
+
+```bash
+make smoke-vec     # new make target, ~2 min wall
+```
+
+What it does:
+
+- Pull 1k random `chunks.content` rows from `~/.arborist/shards/000.db`
+  into `/tmp/smoke-vec.db` (read-only on the source shard).
+- Embed with `bge-small-en-v1.5` on CPU (single-threaded; ~10 ms/chunk
+  → ~10 s wall).
+- INSERT into `chunk_vecs` virtual table under WAL + `synchronous=NORMAL`
+  (existing arborist convention).
+- Kill -9 mid-insert (process group); reopen; check WAL
+  recovery + journal mode + row count consistency.
+- Run 100 query embeddings × top-20 ANN. Measure p50/p95 query
+  latency.
+
+**Gate criteria:** insert throughput ≥ 100 chunk/s, p95 query
+latency ≤ 50 ms, zero data loss across the kill -9 boundary. Fail
+on any of those → Phase 1 blocked, surface to fox + upstream
+issue.
+
+### 13.3 Bench protocol
+
+Fixtures (all already in `bench/`):
+
+- `bench/qa_questions_smoke.txt` — 5-question fast loop
+- `bench/qa_questions_progressive_and.txt` — progressive-AND fallback fixture
+- Existing `bench-emergent` random-word stress fixture
+- Existing `qa-modes-bench` fixture for STRICT-rate guard
+
+Three conditions:
+
+| condition | retrieval routes | claim_lattice mode |
+|---|---|---|
+| Baseline | FTS5 only (current 4 routes) | claim_lattice (JSON) |
+| Vec-only | Vec ANN top-20 only | claim_lattice (JSON) |
+| Hybrid | FTS5 + Vec, RRF merge (k=60) | claim_lattice (JSON) |
+
+**Phase 1 success criteria** (from §8):
+- Vec-only matches FTS5-only within ±5pp STRICT-rate (sanity)
+- AND beats FTS5-only by ≥5pp on at least one semantic-allusion
+  bench-emergent fixture
+- AND no UNGROUNDED-rate regression vs baseline
+
+**Phase 2 success criteria** (gates on §8):
+- Hybrid lifts STRICT-rate by ≥5pp over baseline
+- AND no UNGROUNDED-rate regression
+- AND title-relevance hard check (Rule 8) catches mismatched vec
+  hits at the same rate it catches mismatched FTS5 hits
+
+### 13.4 Code structure
+
+```
+arborist/embed.py                 [NEW]
+    class Embedder:
+        - lazy-loads BAAI/bge-small-en-v1.5 once per process
+        - normalize=True (unit vectors, §6.1 canonical config)
+        - encode(texts: list[str]) -> np.ndarray[N, 384] dtype=int8
+        - sentencetransformer dependency under [vec] extras
+
+arborist/search/vec.py            [NEW, mirrors fts5.py]
+    class VecBackend(SearchBackend):
+        - audit_mode = AuditMode.UNGROUNDED  # §1 hard constraint
+        - chunk_vecs virtual table creation (lazy; first ingest --embed)
+        - search(query, limit=20) -> list[Hit] via vec_distance_l2
+
+arborist/store.py                 [PATCH]
+    - SCHEMA_SQL: NO new always-on tables (vec table is virtual,
+      created only when --embed runs)
+    - Add embedder_version, vec_quantization, vec_dim, vec_ann_index,
+      vec_distance_metric to governance_policy_hash inputs (NOT
+      cache_key — §6 hard constraint)
+
+arborist/qa/query.py              [PATCH]
+    - if vec backend present: run RRF merge over (FTS5_routes_merged,
+      vec_route) with k=60. New code is ~6 lines per §7.
+
+arborist/cli.py                   [PATCH]
+    - ingest --embed flag (gated on _VEC_AVAILABLE import-check)
+    - query --retrieval={fts5|vec|hybrid} flag
+    - new subcommand: arborist vec rebuild [--db ... --batch N]
+
+Makefile                          [PATCH]
+    smoke-vec:        # §13.2 pre-flight; gate-before-Phase-1
+    rebuild-vec:      # populate chunk_vecs for an existing shard
+    bench-vec:        # run §13.3 three-condition bench
+
+pyproject.toml                    [PATCH]
+    [project.optional-dependencies]
+    vec = [
+        "sqlite-vec>=0.1.9,<0.2",
+        "sentence-transformers>=2.7,<3",
+        "numpy",
+    ]
+```
+
+### 13.5 Test plan
+
+Unit tests:
+- `tests/test_embed.py` — deterministic output for fixed input
+  (regression guard against silent model drift between revs);
+  unit-norm verification; int8 round-trip.
+- `tests/test_search_vec.py` — `VecBackend.search` against a
+  3-chunk in-memory shard with hand-crafted embeddings; verify
+  ranking matches expected; verify `audit_mode=UNGROUNDED`;
+  verify `_VEC_AVAILABLE=False` graceful degradation.
+- `tests/test_governance_policy.py` — verify the 5 new vec-config
+  fields fold into `governance_policy_hash`; flipping any one
+  invalidates prior cached records.
+
+Integration tests (gated on `[vec]` extras installed; skipped
+otherwise):
+- `tests/test_vec_smoke_integration.py` — runs the §13.2 smoke on
+  a 100-chunk synthetic shard (smaller than 1k for CI speed);
+  asserts WAL recovery + row count parity.
+
+### 13.6 Out of scope for Phase 1
+
+- ANN index variants (IVF, DiskANN, HNSW) — Phase 1 ships flat
+  only. ANN is opt-in via `vec_ann_index` policy field; flat works
+  at ≤10 M chunks per the upstream sqlite-vec docs.
+- Re-embed cost on chunker bump — already a known-large cost
+  (§10.3); covered by `make rebuild-vec` not Phase 1.
+- Edges compaction (§10.6) — different ticket entirely.
+
+### 13.7 Estimated size
+
+- New files: 2 (embed.py, search/vec.py) ~ 200 LOC.
+- Patches: 4 (store.py, query.py, cli.py, Makefile) ~ 100 LOC.
+- Tests: 4 files ~ 250 LOC + fixtures.
+- Phase-1 doc append in `docs/`: this §13 + a journey-note bench
+  result file.
+- pyproject.toml: 1 stanza.
+
+Total: ~550 LOC + ~250 test LOC. Single substantial commit if all
+tests pass + smoke succeeds; otherwise broken into the natural
+gates (smoke → embed.py → vec.py → integration tests → bench).
+
+### 13.8 Decisions fox needs to make to unblock Phase 1
+
+| # | Decision | Recommendation |
+|---|---|---|
+| 1 | Embedder path 1 vs 2 vs 3 | Path 1 (local bundle) |
+| 2 | Default model name | `BAAI/bge-small-en-v1.5` (MIT, 33 MB) |
+| 3 | Default quantization × dim | `int8 × 384 + flat` |
+| 4 | Approve `sentence-transformers` PyPI dep under `[vec]` | yes (~120 MB install footprint when extras pulled; default not pulled — `pip install '.[vec]'` only) |
+
+If fox approves all four → Phase 1 starts with §13.2 smoke as the
+first gate.
+
+If fox rejects path 1 → Phase 1 blocked on standing up
+`hermes-embed.ai.unturf.com` (path 2; outside this ticket).
+
+If fox rejects `int8 × 384` default → propose `binary × 768` as
+default (1.6 % tax, ~10–20 % recall hit per §9). Either is
+defensible.
+
+If fox rejects the dep footprint → fall back to the lighter
+`onnxruntime` path with a `bge-small` ONNX model (still ~33 MB
+weights but smaller runtime). Adds complexity but keeps the
+no-server constraint.