# Ticket #000039 — Optional `sqlite-vec` retrieval backend (A/B vs FTS5, hybrid not replacement)

**Status:** **closed · 2026-05-12** — Phase 0 (doc) + Phase 1 (the optional vec backend + ingest path + `--quant`) landed; **Phase 2 (RRF hybrid fusion in `query.py`) split to #000050**, gated on a corpus backfill + a ≥5pp recall bench (the corpus-wide `arborist embed` is a one-time hours-on-idle / days-on-contended batch job per §14.6 — not done here). The vec layer ships demonstrated on `crawl_appliedcombinatorics_org.db` (semantic hits topically correct) + a 54K-chunk partial on wiki shard 000 (confirming ~409 B/chunk → ~2.4 GB full-corpus at int8). 16 vec tests; full suite green. Detail below ↓. Phase 0 (doc) + **Phase 1 landed 2026-05-11**. Phase 1: `arborist/search/vec.py` — `VecBackend(SearchBackend)` (UNGROUNDED hits, never in proof path), `chunk_vecs` vec0 virtual table + `vec_meta` (sibling tables — don't touch chunks/documents/audit chain), `embed_documents()` ingest (delete-then-insert idempotent; vec0 doesn't honor INSERT-OR-REPLACE), pluggable `Embedder` callable with a fastembed `bge-small-en-v1.5` default. CLI: `arborist embed [--limit] [--batch-size]` + `arborist search --backend vec`. `[vec]` optional extra (sqlite-vec + fastembed). **Obvious v1 tuning** (`VEC_BACKEND_VERSION = vec-v1-bge-small-en-v1.5-384float32-cosine-flat`): embedder `BAAI/bge-small-en-v1.5`, dim 384, quant float32 (int8/binary = the production storage knob per §3.1, not wired in v1), metric cosine (bge outputs L2-normalized so cosine ≡ L2 ranking), ANN flat (vec0 default), top_k 20. 7 tests (`tests/test_search_vec.py`, stub embedder — plumbing only; semantic quality demonstrated on a real shard). **Demonstrated on `crawl_appliedcombinatorics_org.db`** (168 chunks embedded in ~37s incl. model load; semantic queries return topically-correct hits — "how many ways to choose k things from n" → top hit "AC Combinations"; chain-check on that shard reports 0 after embedding). The 5 hyperparams fold into `governance_policy_hash` in a later phase (§6 — not wired yet). **Ingest integration landed 2026-05-11** (see §14): `embed_documents()` is now **incremental by default** (embeds only `chunk_id`s not already in `chunk_vecs` — re-runs are cheap no-ops); `arborist embed --rebuild` does the DROP+recreate+full-re-embed for a `VEC_BACKEND_VERSION` bump; `arborist ingest --embed` is the **eager opt-in** (embed this run's new chunks after the chunk+Merkle-commit pass; default ingest does NOT embed — the lazy `arborist embed` pass / cron / Prometheus-Σ unconscious sweep is the usual path). **`--quant {float32,int8}` landed 2026-05-11** (`arborist embed --quant int8 [--rebuild]`; the `chunk_vecs` vec0 column is `int8[384]` vs `float[384]` per quant; int8 blobs are `vec_int8(?)`-wrapped — sqlite-vec v0.1.9 treats a bare blob as float32; quant change on an existing table requires `--rebuild` since the vec0 element type can't be altered in place; the quant folds into `VEC_BACKEND_VERSION` → `...-384int8-...`, recorded in `vec_meta`). **int8 head-to-head on `crawl_appliedcombinatorics_org.db`**: storage 1.60 MB → **0.42 MB (3.8× smaller; ~4× at corpus scale where blocks fill)**; recall ≈ float32 — Q2 "how many ways to choose k things from n" identical top-5, Q1 "pigeonhole principle counting" identical top-2 with a sub-noise rank-3/4 swap (Δdistance 0.002). So int8 is the obvious production config (§3.1's +6%-tax recommendation confirmed) — v1 default stays float32 for max fidelity; switching the default to int8 is a fox call. 16 vec tests. **Embed throughput measured 2026-05-11** (§14.6): ~4.3 chunks/s on the contended dev box (~50-200/s idle); a 54K-chunk wiki backfill confirmed ~409 B/chunk → ~2.4 GB full-corpus at int8 (the deterministic number), then was **abandoned** — full corpus is hours-on-idle / days-on-contended, a one-time off-peak/dedicated-box batch job, not something to brute-force inline. The non-vec ingest path is unchanged (hundreds of chunks/s); adding vec multiplies ingest by ~10-100× (all in the ONNX matmuls) — which is exactly why lazy-out-of-band is the default and `ingest --embed` is the opt-in. **Phase 2** (RRF hybrid fusion in `query.py`) gated on a ≥5pp recall-lift measurement on bench fixtures with no STRICT-rate regression (§8).
**Opened:** 2026-05-09
**Scope:** Spec an optional `sqlite-vec` backend that runs **alongside** the
existing FTS5 retrieval pipeline (never replacing it), with phased gates
on storage cost, embedder choice, and measured recall lift. Phase 0 is
this doc — no code, no schema bump. Phase 1 (ingest-side embedding +
`VecBackend` class) opens only when the storage projection in §3 lands
within budget on a real shard. Phase 2 (hybrid fusion in `query.py`)
opens only when Phase 1 measures ≥5pp recall lift on the bench fixtures
with no STRICT-rate regression.
**Audience:** fox + future blackops shifts + downstream Arborist
clients deciding whether to ship a vec layer.
**Hard constraint:** vec retrieval **never enters proof path** — hits
land `audit_mode=UNGROUNDED`, same as FTS5 today (per the soft-hash /
hard-hash separation in CLAUDE.md). No new `cache_key` dimension;
embedder identity + dimensionality + quantization fold into
`governance_policy_hash` like every other versioned default. Vec
storage is **additional** to FTS5, not a replacement (see §4 for the
architectural reasoning). Optional dep — `pip install '.[vec]'` —
graceful fallback when extension is not loadable. Pre-v1 dep
discipline: pin `sqlite-vec==X.Y.Z` and document the upgrade rule.

---

## 1. Problem statement

Arborist's retrieval today is four parallel FTS5 routes per shard
merged then reranked (CLAUDE.md "Retrieval pipeline" §1):

```text
body BM25  +  title LIKE  +  core-keyword (TF-IDF)  +  phrase-pattern
```

Each route closes a specific gap. Phrase-pattern (route 4) closes the
**lexical** allusion gap — "always been at war" verbatim-matches the
1984 article whose title shares zero tokens with the query. There is
no symmetric route that closes the **semantic** allusion gap — the
case where the query and target chunk share zero stems but mean the
same thing. Examples that today fall through every route:

```text
query:  "the document store that proves what it answered"
target: an Arborist documentation chunk that uses "content-addressed",
        "Merkle-committed", "audit chain", "providence cache" — zero
        stem overlap with the query, no verbatim 5-gram phrase, no
        title-LIKE candidate, no TF-IDF core overlap.

query:  "what did Orwell call the country at war with Oceania?"
target: 1984 article using "Eastasia" / "Eurasia" — phrase route
        catches "always been at war"-style verbatim quotes but not
        synonymic re-phrasings of the same fact.
```

A vector embedding backend would close that gap by mapping query and
chunk into a shared semantic space and returning top-K nearest. The
question is whether the storage and compute cost pencils out, and
whether vec retrieval **adds** to FTS5 or **replaces** it.

This ticket answers both questions with measured numbers, then
specifies the integration surface so a Phase 1 implementation can
proceed under a known budget.

---

## 2. Why `sqlite-vec` specifically (vs alternatives)

Arborist is SQLite-native by design. CLAUDE.md "Python only" rule
forbids Rust / C / non-Python sibling indexes inside the repo;
extension modules loaded by SQLite itself are fine because they
appear to Python as ordinary `sqlite3` rows.

| Candidate | Verdict | Why |
|---|---|---|
| `sqlite-vec` | **chosen** | Pure C extension dlopen-able into stock `sqlite3`. Apache-2.0 / MIT dual-license. Same DB file as FTS5 — no second backing store. Matches the `selectolax` optional-dep precedent. |
| FAISS (sibling file) | rejected | Adds a second on-disk format outside the audit-chain. Two-file consistency is a new failure mode (cache_key talks about a single shard, not a shard + a vector blob). |
| `pgvector` (Postgres) | rejected | New runtime dependency. CLAUDE.md "fresh checkout needs only python3.12 + venv + sqlite3" rule. |
| Hand-rolled NumPy + flat scan | rejected | Works at small scale; doesn't survive 6.24M chunks (§3). No ANN escape hatch when corpus grows. |
| LanceDB / Chroma / Weaviate | rejected | Server runtimes. Fresh-checkout rule again. |

Caveats on `sqlite-vec`:

- Pre-v1 (latest published v0.1.9). Repo explicitly warns "expect
  breaking changes." Pin tightly; treat upgrades as schema-bump
  events that re-embed every chunk.
- No formal max-dimension or transactional-write guarantees in
  upstream docs. Phase 0 deliverable: a 1k-chunk smoke test that
  measures (a) insert throughput, (b) WAL behavior, (c) crash
  recovery semantics. If any of those misbehave under WAL +
  `synchronous=NORMAL` (the existing arborist convention), Phase 1
  is gated on a fix or a different config.

---

## 3. Storage hypothesis — the real 6 GB → 38 GB number

Fox's hypothesis: **6 GB compressed wiki blows up to ~35 GB after
Merkle tree + FTS5 index.** Measured today on `~/.arborist/shards/`:

```text
shard 000.db   9.6 GB    867 K docs   1.56 M chunks
shard 001.db   9.5 GB    868 K docs   1.56 M chunks
shard 002.db   9.6 GB    867 K docs   1.56 M chunks
shard 003.db   9.6 GB    867 K docs   1.56 M chunks
TOTAL         38.3 GB    3.47 M docs  6.24 M chunks
```

Within one shard the breakdown (`SELECT name, SUM(pgsize) FROM dbstat
GROUP BY name`) is:

| Component | MB | % of shard | Note |
|---|---|---|---|
| `edges` | 3 670 | 38 % | Lossless `supersedes` / `derives_from` / related-doc links. **90.6 M total rows across the corpus — ~26 edges per document.** Biggest single cost, bigger than chunks themselves. |
| `chunks` (compressed content) | 2 649 | 28 % | Already zstd-compressed at rest. |
| `chunks_fts_data` (FTS5 inverted index) | 1 501 | 16 % | The full lexical index. |
| `idx_edges_dst_root` | 589 | 6 % | Edges secondary index. |
| `audit_events` | 377 | 4 % | Hard-hash chain, one row per state-change. |
| `documents` | 170 | 2 % | Document metadata. |
| `chunks` autoindex + leaf index | 251 | 3 % | |
| `merkle_nodes` | 114 | 1 % | Internal nodes only — leaves are folded into `chunks.leaf_hash`. |
| `concept_relations` | 10 | 0.1 % | Per-shard rivalry/synonym graph (#000018-area). |
| Everything else | ~ 270 | 3 % | |

So the 6 GB → 38 GB blowup is **not** "FTS5 + Merkle tree." It is:

```text
edges          ~38 %        relationship graph
chunks         ~28 %        compressed content (decompressed at query)
fts5           ~16 %        the actual inverted index
indexes        ~10 %        secondary indexes on the above
audit chain    ~4 %         hard-hash provenance receipts
merkle interior ~1 %        log-N internal nodes
```

The FTS5 index itself is **only ~16 %** of the shard. Most of the
blowup is **lossless provenance** (edges + chunks + audit). Anyone
proposing a vec backend should measure against that baseline, not
against the FTS5 fraction.

### 3.1 Vec storage projection at 6.24 M chunks

`sqlite-vec` supports float32 / int8 / binary across configurable
dimension. ANN index choice multiplies that. Using 6.24 M chunks
(today's full corpus) and the bytes-per-chunk math:

| Quantization | Dim | Bytes / chunk | Vec data total | + Flat index | + IVF (~1.1 ×) | + DiskANN/HNSW (~1.7 ×) |
|---|---|---|---|---|---|---|
| float32 | 384 | 1 536 | 9.6 GB | 9.6 GB | 10.6 GB | 16.3 GB |
| float32 | 768 | 3 072 | 19.2 GB | 19.2 GB | 21.1 GB | 32.6 GB |
| int8 | 384 | 384 | 2.4 GB | 2.4 GB | 2.6 GB | 4.1 GB |
| int8 | 768 | 768 | 4.8 GB | 4.8 GB | 5.3 GB | 8.2 GB |
| binary | 384 | 48 | 300 MB | 300 MB | 330 MB | 510 MB |
| binary | 768 | 96 | 600 MB | 600 MB | 660 MB | 1.0 GB |
| binary | 1024 | 128 | 800 MB | 800 MB | 880 MB | 1.4 GB |

Mapped onto the existing 38 GB shard footprint:

| Config | % tax over 38 GB | Comparable to |
|---|---|---|
| float32 × 384 + flat | + 25 % | Bigger than the entire FTS5 index. Reject. |
| float32 × 768 + DiskANN | + 85 % | Doubles disk. Reject outright. |
| **int8 × 384 + flat** | **+ 6 %** | **Comparable to a single secondary index. Acceptable.** |
| int8 × 768 + flat | + 13 % | Comparable to FTS5. Acceptable if recall justifies it. |
| **binary × 384 + flat** | **+ 0.8 %** | **Same scale as `concept_relations` (1.6 % tax tolerated).** |
| binary × 768 + flat | + 1.6 % | Tied with `concept_relations`. Acceptable. |
| binary × 1024 + flat | + 2.1 % | Acceptable. |

**Phase 0 storage budget:** ≤ 15 % tax over today's 38 GB total
(i.e. ≤ 5.7 GB of vec data across all shards). That admits
everything from binary × 1024 up through int8 × 768 + flat. It
excludes anything float32 and anything with DiskANN graph overhead
above ~int8 × 384.

### 3.2 What this looks like per-shard

At the recommended Phase-1 config (**int8 × 384 + flat** as the
default, configurable):

```text
per shard:    1.56 M chunks × 384 bytes = 600 MB
all 4 shards: 2.4 GB total
% of shard:   ~6 %  (sits between idx_edges_dst_root and audit_events)
```

At the storage-conservative config (**binary × 768 + flat**):

```text
per shard:    1.56 M chunks × 96 bytes = 150 MB
all 4 shards: 600 MB total
% of shard:   ~1.6 %  (same scale as concept_relations)
```

The Phase 0 deliverable picks one of these two as the default and
makes the other reachable via flag.

---

## 4. Architectural answer — additional, not same

> **Q: would vector search be the same as FTS, or additional?**
>
> A: **Additional. Always.** Vec is a fifth retrieval route,
> not a replacement for any of the four FTS5 routes.

### 4.1 Why never replacement

The four FTS5 routes each close a gap that vec cannot:

| Route | What it catches | Why vec misses it |
|---|---|---|
| Body BM25 | Term-frequency-weighted topical relevance | Embeddings smooth over rare terms; specialized vocab gets averaged into the centroid |
| Title LIKE | Exact title-prefix matches | Embeddings of short titles are dominated by stopword centroids — "the X of Y" embeds nearly identically for many X, Y |
| Core-keyword (TF-IDF) | Distilled-doc anchors | Cores are a different artifact (distillation output), not a chunk-level signal |
| Phrase-pattern (n=5/n=6) | Verbatim quote allusion ("always been at war") | Embeddings are bag-of-information; phrase order is lost. Verbatim recall is FTS5's job. |

Vec adds:

| New route | What it catches |
|---|---|
| Vec ANN top-K | Semantic paraphrase ("country at war with Oceania" → "Eastasia"); cross-vocabulary synonymy when the corpus uses unfamiliar terminology; concept-level matching when the user's words and the article's words share no stems |

So the architectural choice is fixed: **vec is a fifth parallel
route**, merged into the existing rerank pipeline.

### 4.2 Hybrid fusion design

Two standard merge strategies, both feasible with the existing
pipeline:

**Option A — Reciprocal Rank Fusion (RRF).** Each route emits
ranked hits; final score is `Σ_routes 1 / (k + rank_route)` with
`k=60` per the original RRF paper. Robust to score-scale
mismatch (BM25 scores are not comparable to cosine similarity).
Recommended.

**Option B — Score-level merge with normalization.** Convert each
route's score to a `[0,1]` percentile within its own distribution,
then weighted-sum. More tunable but introduces a `weights` config
that becomes a `governance_policy_hash` input. More moving parts
than RRF.

Phase 0 picks **Option A (RRF)** as the default. Option B is a
follow-up if RRF leaves measurable lift on the table.

### 4.3 Rivalry / title-relevance / phrase routes still apply

The post-merge filter chain (rivalry exclusion, four-accept-paths
title-relevance, stem-aware token matching, per-source context cap)
runs **after** the merge. Vec hits go through the same filter as
FTS5 hits. A vec hit that passes ANN top-K but fails the
title-relevance filter is dropped exactly like an FTS5 BM25 hit
that fails the same filter — so the existing retrieval-driven
hallucination guards (e.g. the spin-glass / QCD case from
CLAUDE.md "Title-relevance hard check (Rule 8)") still hold.

---

## 5. The embedder is the gating item

Arborist's only LLM endpoint today is Hermes-3 Llama-3.1-8B at
`hermes.ai.unturf.com`. Hermes is a chat model, **not** an
embedding model. Without an embedder there is nothing to put into
the vec table.

Phase 0 must pick one of these paths, in declining order of
preference:

1. **Local embedder bundled as optional dep.** A small
   sentence-transformer (e.g. `bge-small-en-v1.5`, 384-dim;
   `all-MiniLM-L6-v2`, 384-dim) shipped via `pip install '.[vec]'`.
   Runs on CPU in arborist's process; no network call; no
   concurrency interaction with Hermes. ~100 MB model file. **Strong
   default candidate** because it preserves the "fresh checkout
   needs only python3.12 + venv + sqlite3" rule (the embed model
   is an optional extra, not a runtime dep).
2. **Dedicated embedding endpoint.** Stand up a sibling endpoint
   alongside Hermes (`hermes-embed.ai.unturf.com`?) running
   `bge-large` or similar. Frees arborist from CPU embedding
   work. Costs an extra service to operate.
3. **Hermes itself for embeddings.** Hijack a generative model for
   embeddings via prompt + last-hidden-state pooling. Possible but
   research-grade — bench quality unknown, breaks the ~4-concurrent
   bottleneck (#000037 §11) at every ingest. **Not recommended.**

Phase 0 deliverable: pick path 1 or 2 with a named model. Phase 1
implementation depends on which.

### 5.1 Why embed at ingest, never at query

Embedding cost is amortized:

```text
ingest-time:  embed each chunk once,  store in vec table
query-time:   embed query string once (~10 ms on CPU),  ANN top-K
```

If the embedder lives in arborist's process (path 1), the only
network cost at query time is the ANN scan inside SQLite — no LLM
call. This sidesteps the Hermes ~4-concurrent bottleneck for
queries entirely. Ingest pays the embedding cost up front, once
per chunk, on the chunker-version-pinned content.

---

## 6. Versioning and `governance_policy_hash` integration

Embedder choice is a versioned default. Per CLAUDE.md "Versioned
defaults" rule (`tok-512-v1`, `norm-v1`, `wikitext-base-v1`,
`v9.8.0`), the vec layer adds:

```text
embedder-name      e.g. "bge-small-en-v1.5"
embedder-version   e.g. "@hf-rev-deadbeef"
quantization       e.g. "int8" / "binary" / "float32"
dimension          e.g. 384 / 768 / 1024
ann-index          e.g. "flat" / "ivf" / "diskann"
distance-metric    e.g. "cosine" / "l2_normalized" / "dot" / "hamming"
```

These **six** fields fold into `governance_policy_hash` (NOT into
a new cache_key dimension — keep the 8-dim invariant). Bumping any
one stales every prior cached answer that consulted the vec
backend, exactly as bumping the chunker stales every prior FTS5
hit. The vec table itself stays in place but is treated as
"cold": query-time lookups filter on `embedder_version =
current` and ignore older rows; a `make rebuild-vec` target
re-embeds.

### 6.1 Distance metric — the silent-correctness field

Switching distance metric without re-embedding silently changes
ranking. A query that returned chunk A as top-1 under cosine may
return chunk B under L2 if the embeddings are not unit-normalized.
Because nothing else in the pipeline observes the change (the vec
backend just emits ranked Hits; the verifier and cache_key are
ranking-blind), a metric-only flip would be **invisible drift** —
exactly the failure mode CLAUDE.md "Versioned defaults" exists to
prevent. Hence its inclusion in the policy hash.

The recommended canonical config:

```text
1. embedder produces unit-normalized vectors    (||v|| = 1)
2. ingest stores vectors as-is                  (no further normalize)
3. query stores  metric = "l2_normalized"        (in policy hash)
4. SQL uses      vec_distance_l2()              (== cosine ranking,
                                                 no per-query norm,
                                                 SIMD-friendly)
```

`metric = "cosine"` (i.e. `vec_distance_cosine()` at query time)
is the alternative when the embedder output is **not** guaranteed
unit-normalized; pays a per-row norm computation in exchange for
correctness regardless of input vector magnitude. Slower, never
needed if §1 holds.

`metric = "dot"` is for pre-scaled int8 / float16 where unit
normalization is impossible — the embedder writer is expected to
have rescaled the vectors so dot-product **ranks** identically to
cosine on the original float32 normalized vectors. Documented in
the embedder version note; bump `embedder-version` if the rescale
recipe changes.

`metric = "hamming"` is the only correct option for binary
quantization — cosine on bit-vectors is undefined. Switching from
binary + Hamming to int8 + L2 is a **two-field policy bump**
(quantization AND distance-metric), and re-embeds nothing because
the underlying float vectors weren't stored — `make rebuild-vec`
must re-embed from source.

### 6.2 Quantization × metric compatibility matrix

| Quantization | Valid metrics | Default |
|---|---|---|
| float32 (normalized) | `l2_normalized`, `cosine` | `l2_normalized` |
| float32 (unnormalized) | `cosine` | `cosine` |
| int8 (pre-scaled) | `dot`, `l2_normalized` | `dot` |
| binary | `hamming` | `hamming` |

Phase 1 enforces the matrix at ingest: a config that pairs `binary
+ cosine` errors out before any chunk is embedded. The error
message names the matrix entry that would resolve it.

---

## 7. SearchBackend integration — where the code lands

Existing surface (`arborist/search/base.py`):

```python
class SearchBackend(ABC):
    name: str
    audit_mode: AuditMode

    def search(self, query: str, limit: int = 20) -> list[Hit]: ...
```

Phase 1 adds `arborist/search/vec.py` mirroring `fts5.py`:

```python
class VecBackend(SearchBackend):
    name = "vec"
    audit_mode = AuditMode.UNGROUNDED   # same as FTS5 — soft signal

    def __init__(self, conn, embedder):
        super().__init__(conn)
        self.embedder = embedder    # callable: str -> np.ndarray

    def search(self, query, limit=20):
        qv = self.embedder(query)
        rows = self.conn.execute(
            "SELECT chunk_id, distance FROM chunk_vecs "
            "WHERE embedding MATCH ? AND k = ? "
            "ORDER BY distance",
            (qv.tobytes(), limit),
        ).fetchall()
        # JOIN back to chunks + documents, build Hits with the same
        # _build_snippet helper FTS5 already uses.
        ...
```

The merge with FTS5 happens in `arborist/qa/query.py` where the
four FTS5 routes already merge today. RRF is a six-line addition
once both rankings exist.

Optional-dep guard at import time (mirrors `[html]`):

```python
try:
    import sqlite_vec
    _VEC_AVAILABLE = True
except ImportError:
    _VEC_AVAILABLE = False
```

CLI surface only exposes `--retrieval=vec` or `--retrieval=hybrid`
when `_VEC_AVAILABLE`, otherwise the flag errors with an install
hint. Same pattern as `selectolax` for the HTML source.

---

## 8. Phased plan with measurement gates

### Phase 0 — this ticket (doc)

- Decide embedder (path 1 or path 2, named model).
- Decide default quantization + dimension (recommendation:
  **int8 × 384 + flat** as the speed/quality default; **binary ×
  768 + flat** as the storage-conservative alternative).
- Run the 1k-chunk smoke test under WAL + `synchronous=NORMAL`.
- Write the bench protocol: which fixtures, what signal floor.
  Lift target: ≥ 5pp recall@20 on bench-emergent random-word
  fixtures (bench-maxing.md signal floor).

### Phase 1 — code (gated on Phase 0 sign-off)

- `arborist/search/vec.py` with optional-dep guard.
- `chunk_vecs` virtual table created lazily on first ingest with
  `--embed`.
- `arborist ingest --embed` ingest-side flag (default off).
- `make rebuild-vec` Makefile target.
- Rebuilds populate `chunk_vecs` for all current `chunks` rows.
- Bench: vec-only retrieval vs FTS5-only on a single shard.
  **Phase 1 succeeds if** vec-only matches FTS5-only within ±5pp
  on STRICT-rate AND beats FTS5 by ≥ 5pp on at least one
  bench-emergent semantic-allusion fixture.

### Phase 2 — hybrid fusion (gated on Phase 1 success)

- RRF merge in `arborist/qa/query.py` between FTS5 routes and
  vec route.
- Bench: hybrid vs FTS5-only.
- **Phase 2 succeeds if** hybrid lifts STRICT-rate by ≥ 5pp over
  FTS5-only with no UNGROUNDED-rate regression.
- If Phase 2 fails by < 5pp lift: park the vec layer as opt-in
  per-call (`--retrieval=vec`) but do not make hybrid the default.
  The 2.4 GB tax is not paid for a sub-signal-floor lift.

### Phase 3 — `governance_policy_hash` wiring (mechanical)

- Fold the five vec-version fields (§6) into the policy hash.
- Document in CLAUDE.md "Versioned defaults" and "Schema
  invariants" sections.
- Update `make chain-check-shards` semantics if needed (it should
  not need changes — vec data is sibling, like `concept_relations`).

---

## 9. Decision tree on quantization

```text
Phase 0 budget: ≤ 15 % storage tax (≤ 5.7 GB across all shards).

If Phase 1 bench shows int8 × 384 hits the recall target
   → ship int8 × 384 + flat as default.
   → 6 % storage tax. Best speed/quality for the budget.

If Phase 1 bench shows int8 × 384 misses the target
   AND int8 × 768 hits it
   → ship int8 × 768 + flat.
   → 13 % storage tax. Borderline acceptable.

If Phase 1 bench shows int8 × 768 also misses
   → DO NOT ship float32 anywhere — too expensive.
   → Park the ticket. Reopen when a smaller, sharper embedder lands
     (research-bench problem, not an arborist problem).

If storage budget tightens later
   → fall back to binary × 768 + flat. < 2 % tax,
     ~10–20 % recall hit, still wins on the semantic-allusion
     fixtures the lexical pipeline misses entirely.
```

---

## 10. Risks and open questions

1. **`sqlite-vec` pre-v1 instability.** Mitigation: pin tightly,
   treat upgrades as schema-bump events, run the 1k-chunk smoke
   test after every dep bump. Phase 0 deliverable.
2. **Embedder model drift.** A new HuggingFace revision of
   `bge-small` invalidates every prior embedding. Mitigation: pin
   model+revision via `embedder_version` in §6, track upgrades
   under `make rebuild-vec`.
3. **Re-embed cost on chunker bump.** `tok-512-v1 → v2` already
   stales every FTS5 chunk. Adds re-embed work proportional to
   chunk count. At 6.24 M chunks × ~10 ms/chunk on a CPU
   bge-small ≈ 17 hours single-threaded; parallelizes trivially.
   Already a known-large cost; vec adds ~10–20 % to it.
4. **Verifier semantics unchanged.** Vec lifts retrieval recall;
   the lexical verifier (quote / span / entity / paraphrase)
   stays binary. A vec-retrieved chunk that the verifier cannot
   ground still lands UNGROUNDED. Worth confirming on the bench:
   does vec recall translate to STRICT-rate lift, or just to more
   UNGROUNDED hits being dropped at verification time?
5. **Title-relevance hard check (Rule 8) interaction.** Vec is
   precisely the retrieval mode most likely to surface
   structurally-unrelated cited chunks (the spin-glass / QCD
   case). Phase 1 bench must include the existing title-mismatch
   fixtures and confirm vec hits get caught by Rule 8 like any
   other retrieval-driven hallucination candidate. If vec hits
   bypass Rule 8 somehow, that is a Phase 1 blocker.
6. **`edges` is the dominant cost (38 % per §3), not FTS5.**
   Confirmed with the corpus-wide row count: **90.6 M edges
   across 3.47 M documents = ~26 edges/document.** If storage
   matters more than retrieval quality, compacting `edges` is a
   higher-leverage optimization than anything vec-related (a
   single int8 × 384 vec layer adds 6 % tax; trimming edges by
   10 % saves 3.8 % at zero quality cost). Out of scope for this
   ticket but worth a reminder when we benchmark.
7. **Concurrent-Hermes bottleneck (#000037 §11) is irrelevant
   here** as long as we use a local embedder (path 1 in §5). If
   we go path 2 or 3, vec ingest competes with inference traffic
   and the Prometheus-Σ controller (#000037) would need to
   throttle ingest under load.

---

## 11. What this ticket does NOT do

- Does not commit to shipping a vec backend. Phase 0 is doc-only;
  every later phase has a measurable gate.
- Does not bump `cache_key` dimensions. Vec config folds into
  `governance_policy_hash`, same pattern as every prior versioned
  default.
- Does not change the verifier or the audit chain. Vec is a soft
  signal at the retrieval layer only; proof-path stays identical.
- Does not replace any FTS5 route. §4 settles this: vec is a
  fifth route, never a substitute.
- Does not implement the unconscious falsification sweep
  (#000037 Phase 1). That is a separate ticket; vec just makes
  semantic neighbors easier to surface, which is one capability
  the sweep can use later.

---

## 12. References

- This ticket: doc-only Phase 0.
- `arborist/search/base.py` — `SearchBackend` ABC, the integration
  surface.
- `arborist/search/fts5.py` — the contract the vec backend mirrors.
- `arborist/qa/query.py` — where RRF merge would land.
- CLAUDE.md "Retrieval pipeline" — the four-route baseline.
- CLAUDE.md "Soft hash vs hard hash" — the rule that licenses vec
  retrieval as long as it never enters the proof path.
- CLAUDE.md "Versioned defaults" — the pattern §6 follows.
- #000018 (concept_relations) — the 1.6 % storage-tax precedent.
- #000037 §11 — the Hermes ~4-concurrent constraint that drives
  the "embed at ingest, not at query" rule.
- `~/.arborist/shards/` — 4 × 9.6 GB shards measured 2026-05-09;
  the per-table breakdown in §3 is `dbstat`-derived from
  `000.db`.
- Upstream: <https://github.com/asg017/sqlite-vec> (v0.1.9 at time
  of writing; expect breaking changes pre-v1).

---

## 13. Phase 1 implementation plan (proposal, 2026-05-10)

**Status:** awaiting fox go/no-go on the four §13.1 decisions
below. Phase 1 code does not start until they're settled.

### 13.1 Phase-0 deliverables — proposed picks

Four explicit decisions Phase 0 left open. Recommendations:

1. **Embedder path** → **Path 1** (local sentence-transformer
   bundled as optional dep). Preserves CLAUDE.md's "fresh checkout
   needs only python3.12 + venv + sqlite3" rule; sidesteps the
   Hermes 4-concurrent bottleneck (#000037 §11) entirely; no new
   service to operate. Model: **`BAAI/bge-small-en-v1.5`** —
   384-dim, ~33 MB, MIT-licensed, top of MTEB-en/retrieval among
   sub-100MB models, unit-normalized output (matches §6.1
   canonical config).
2. **Default quantization** → **int8 × 384 + flat** per the §9
   decision tree's primary leg. 6 % storage tax (~2.4 GB across
   the 4-shard cluster), well within the 15 % budget. Speed/quality
   default. Storage-conservative alternative `binary × 768 + flat`
   (1.6 % tax) reachable via `--vec-quantization=binary --vec-dim=768`.
3. **1k-chunk smoke test** → §13.2 below.
4. **Bench protocol** → §13.3 below.

### 13.2 Pre-flight smoke (gate before any Phase 1 code lands)

```bash
make smoke-vec     # new make target, ~2 min wall
```

What it does:

- Pull 1k random `chunks.content` rows from `~/.arborist/shards/000.db`
  into `/tmp/smoke-vec.db` (read-only on the source shard).
- Embed with `bge-small-en-v1.5` on CPU (single-threaded; ~10 ms/chunk
  → ~10 s wall).
- INSERT into `chunk_vecs` virtual table under WAL + `synchronous=NORMAL`
  (existing arborist convention).
- Kill -9 mid-insert (process group); reopen; check WAL
  recovery + journal mode + row count consistency.
- Run 100 query embeddings × top-20 ANN. Measure p50/p95 query
  latency.

**Gate criteria:** insert throughput ≥ 100 chunk/s, p95 query
latency ≤ 50 ms, zero data loss across the kill -9 boundary. Fail
on any of those → Phase 1 blocked, surface to fox + upstream
issue.

### 13.3 Bench protocol

Fixtures (all already in `bench/`):

- `bench/qa_questions_smoke.txt` — 5-question fast loop
- `bench/qa_questions_progressive_and.txt` — progressive-AND fallback fixture
- Existing `bench-emergent` random-word stress fixture
- Existing `qa-modes-bench` fixture for STRICT-rate guard

Three conditions:

| condition | retrieval routes | claim_lattice mode |
|---|---|---|
| Baseline | FTS5 only (current 4 routes) | claim_lattice (JSON) |
| Vec-only | Vec ANN top-20 only | claim_lattice (JSON) |
| Hybrid | FTS5 + Vec, RRF merge (k=60) | claim_lattice (JSON) |

**Phase 1 success criteria** (from §8):
- Vec-only matches FTS5-only within ±5pp STRICT-rate (sanity)
- AND beats FTS5-only by ≥5pp on at least one semantic-allusion
  bench-emergent fixture
- AND no UNGROUNDED-rate regression vs baseline

**Phase 2 success criteria** (gates on §8):
- Hybrid lifts STRICT-rate by ≥5pp over baseline
- AND no UNGROUNDED-rate regression
- AND title-relevance hard check (Rule 8) catches mismatched vec
  hits at the same rate it catches mismatched FTS5 hits

### 13.4 Code structure

```
arborist/embed.py                 [NEW]
    class Embedder:
        - lazy-loads BAAI/bge-small-en-v1.5 once per process
        - normalize=True (unit vectors, §6.1 canonical config)
        - encode(texts: list[str]) -> np.ndarray[N, 384] dtype=int8
        - sentencetransformer dependency under [vec] extras

arborist/search/vec.py            [NEW, mirrors fts5.py]
    class VecBackend(SearchBackend):
        - audit_mode = AuditMode.UNGROUNDED  # §1 hard constraint
        - chunk_vecs virtual table creation (lazy; first ingest --embed)
        - search(query, limit=20) -> list[Hit] via vec_distance_l2

arborist/store.py                 [PATCH]
    - SCHEMA_SQL: NO new always-on tables (vec table is virtual,
      created only when --embed runs)
    - Add embedder_version, vec_quantization, vec_dim, vec_ann_index,
      vec_distance_metric to governance_policy_hash inputs (NOT
      cache_key — §6 hard constraint)

arborist/qa/query.py              [PATCH]
    - if vec backend present: run RRF merge over (FTS5_routes_merged,
      vec_route) with k=60. New code is ~6 lines per §7.

arborist/cli.py                   [PATCH]
    - ingest --embed flag (gated on _VEC_AVAILABLE import-check)
    - query --retrieval={fts5|vec|hybrid} flag
    - new subcommand: arborist vec rebuild [--db ... --batch N]

Makefile                          [PATCH]
    smoke-vec:        # §13.2 pre-flight; gate-before-Phase-1
    rebuild-vec:      # populate chunk_vecs for an existing shard
    bench-vec:        # run §13.3 three-condition bench

pyproject.toml                    [PATCH]
    [project.optional-dependencies]
    vec = [
        "sqlite-vec>=0.1.9,<0.2",
        "sentence-transformers>=2.7,<3",
        "numpy",
    ]
```

### 13.5 Test plan

Unit tests:
- `tests/test_embed.py` — deterministic output for fixed input
  (regression guard against silent model drift between revs);
  unit-norm verification; int8 round-trip.
- `tests/test_search_vec.py` — `VecBackend.search` against a
  3-chunk in-memory shard with hand-crafted embeddings; verify
  ranking matches expected; verify `audit_mode=UNGROUNDED`;
  verify `_VEC_AVAILABLE=False` graceful degradation.
- `tests/test_governance_policy.py` — verify the 5 new vec-config
  fields fold into `governance_policy_hash`; flipping any one
  invalidates prior cached records.

CLI surface tests (per
``docs/calculator-test-patterns.md`` §6 — subprocess invocation
catches argparse + main() drift the import-only tests miss; this
hazard surfaced three times during the 2026-05-10 substrate
rename refactor in yesterday's commit chain, so we pin the
pattern):

- `tests/test_cli_vec_rebuild.py` — subprocess invocation of
  ``arborist vec rebuild --db <tmp_path/test.db> --batch 4``
  against a 12-chunk synthetic shard; assert exit 0, JSON
  status output if any, row-count parity in
  ``chunk_vecs``. Skipped via ``pytest.importorskip`` when
  ``[vec]`` extras absent.
- `tests/test_cli_ingest_embed_flag.py` — subprocess invocation
  of ``arborist ingest --source html --embed --author X``
  against a tmp shard; assert ``--embed`` flag wires through
  to the embedder; row count after ingest matches; vec rows
  match chunk rows 1:1.
- `tests/test_cli_query_retrieval_flag.py` — subprocess
  invocation of ``arborist query --retrieval=vec`` and
  ``--retrieval=hybrid`` and ``--retrieval=fts5`` against a
  pre-embedded shard fixture; assert all three exit 0 and
  produce structurally-distinct retrieval candidate sets
  (vec ≠ fts5 on at least one query).

Integration tests (gated on `[vec]` extras installed; skipped
otherwise):
- `tests/test_vec_smoke_integration.py` — runs the §13.2 smoke on
  a 100-chunk synthetic shard (smaller than 1k for CI speed);
  asserts WAL recovery + row count parity.

### 13.6 Out of scope for Phase 1

- ANN index variants (IVF, DiskANN, HNSW) — Phase 1 ships flat
  only. ANN is opt-in via `vec_ann_index` policy field; flat works
  at ≤10 M chunks per the upstream sqlite-vec docs.
- Re-embed cost on chunker bump — already a known-large cost
  (§10.3); covered by `make rebuild-vec` not Phase 1.
- Edges compaction (§10.6) — different ticket entirely.

### 13.7 Estimated size

- New files: 2 (embed.py, search/vec.py) ~ 200 LOC.
- Patches: 4 (store.py, query.py, cli.py, Makefile) ~ 100 LOC.
- Tests: 7 files ~ 400 LOC + fixtures (4 unit + 3 CLI subprocess
  per the §13.5 amendment 2026-05-10 + 1 integration). The CLI
  subprocess tests are ~30-40 LOC each — boilerplate + tmp_path
  + subprocess.run + JSON parse.
- Phase-1 doc append in `docs/`: this §13 + a journey-note bench
  result file.
- pyproject.toml: 1 stanza.

Total: ~550 LOC + ~400 test LOC. Single substantial commit if all
tests pass + smoke succeeds; otherwise broken into the natural
gates (smoke → embed.py → vec.py → CLI subprocess tests →
integration tests → bench).

### 13.8 Decisions fox needs to make to unblock Phase 1

| # | Decision | Recommendation |
|---|---|---|
| 1 | Embedder path 1 vs 2 vs 3 | Path 1 (local bundle) |
| 2 | Default model name | `BAAI/bge-small-en-v1.5` (MIT, 33 MB) |
| 3 | Default quantization × dim | `int8 × 384 + flat` |
| 4 | Approve `sentence-transformers` PyPI dep under `[vec]` | yes (~120 MB install footprint when extras pulled; default not pulled — `pip install '.[vec]'` only) |

If fox approves all four → Phase 1 starts with §13.2 smoke as the
first gate.

If fox rejects path 1 → Phase 1 blocked on standing up
`hermes-embed.ai.unturf.com` (path 2; outside this ticket).

If fox rejects `int8 × 384` default → propose `binary × 768` as
default (1.6 % tax, ~10–20 % recall hit per §9). Either is
defensible.

If fox rejects the dep footprint → fall back to the lighter
`onnxruntime` path with a `bge-small` ONNX model (still ~33 MB
weights but smaller runtime). Adds complexity but keeps the
no-server constraint.

---

## 14. Ingest integration & idempotency (landed 2026-05-11)

The load-bearing fact: **a `chunk_id`'s content is immutable in
arborist.** A chunk is content-addressed-ish (`leaf_hash`) — same
content → same `chunk_id`; different content → a *new* `chunk_id`
(re-ingest with changed content makes a new `document_root` + new
chunk_ids, linked by `supersedes`; a chunker-version bump re-chunks
→ new chunk_ids). So **a chunk, once embedded, never needs
re-embedding** — the only re-embed trigger is the *embedder*
changing (`VEC_BACKEND_VERSION` bump).

### 14.1 Idempotency table

| Event | `chunk_vecs` effect |
|---|---|
| Re-ingest same content | doc_root unchanged, chunks no-op-insert → already embedded → no-op |
| Re-ingest changed content (new doc_root, new chunk_ids) | old chunks + old vecs stay (lossless history); new chunk_ids embedded on next pass |
| Chunker bump (`tok-512-v1 → v2`) | re-chunk → new chunk_ids → embedded next pass; old chunk_ids' vecs are orphan-ish but harmless (old chunks still exist) |
| Cold eviction (`content = NULL`) | vec row persists — still valid; on rehydrate the same `chunk_id` returns with the same content |
| Embedder bump (`VEC_BACKEND_VERSION` changes) | the *only* full-re-embed case — `arborist embed --rebuild` (DROP+recreate+full pass); the `vec_meta.backend_version` row records the populating version |
| Superseded docs | old chunks stay embedded → searchable via vec — consistent with `chunks_fts` keeping superseded docs; the retrieval `supersedes`-filter applies equally |

### 14.2 Two integration models

- **Lazy / out-of-band (default).** `arborist ingest` does *not*
  embed. `arborist embed` (incremental — embeds only `chunk_id`s
  not in `chunk_vecs`) populates on demand; the natural ops home is
  a cron after the nightly ingest, or — better — a **Prometheus-Σ
  unconscious-sweep task** (#000037 §3.1 already walks "ingested-
  but-not-yet-probed" documents; embed-the-new-chunks is exactly
  that shape, so vec needn't carry its own cron). Ingest stays fast;
  embedding is a separable, resumable, optional layer; `chunk_vecs`
  trails ingest by ≤ one pass.
- **Eager opt-in.** `arborist ingest --embed` — after the
  chunk+Merkle-commit pass, embed this run's new chunks (incremental,
  so it's just the delta). `chunk_vecs` stays in sync. Pays
  ~10-20 ms/chunk CPU per ingest — fine for incremental ingests,
  not for a bulk wiki backfill.

(#000039 §5.1's "embed at ingest, never at query" is about *when not*
to embed the query corpus — it doesn't mandate embedding inside the
ingest call.)

### 14.3 Commands

| Command | Behavior |
|---|---|
| `arborist --db D embed` | incremental — embed `chunk_id`s not yet in `chunk_vecs` (cheap no-op once everything's embedded) |
| `arborist --db D embed --rebuild` | DROP + recreate `chunk_vecs`, then full re-embed — the clean `VEC_BACKEND_VERSION`-bump path (a search mid-rebuild never mixes old/new-model embeddings: the recreated table starts empty and grows new-model) |
| `arborist --db D embed --limit N` | cap chunks processed — smoke-test on a real shard |
| `arborist ... ingest --source ... --embed` | eager: after ingest, incremental-embed this run's new chunks |
| `arborist --db D search Q --backend vec` | semantic ANN; errors with an install/embed hint if `[vec]` missing or `chunk_vecs` empty |

### 14.4 Concurrency

WAL serializes writers → `arborist embed` and `arborist ingest` on
the *same* shard take turns. In the `--shards-dir` layout each shard
is its own file → embed-shard-0 and ingest-shard-1 don't contend.
Embed-during-ingest is safe under the lazy model: embed sees a
consistent per-transaction snapshot; ingest's new chunks get picked
up on the next incremental embed pass.

### 14.5 Versioning

`vec_meta.backend_version` (written by `ensure_chunk_vecs_table`) is
the per-shard discriminator. The 5 vec hyperparams fold into
`governance_policy_hash` in a later phase (§6 — not wired yet); the
8-dim `cache_key` invariant is untouched (vec config goes into the
existing `governance_policy_hash` dim, not a new one). Same
versioned-default discipline as `tok-512-v1` / `norm-v1` etc.

### 14.6 Embed throughput — measured, and why ingest stays fast (2026-05-11)

Embedding is the **only** expensive part of the vec layer, and it's
expensive because it's CPU-bound ONNX inference — not because of
anything in arborist's ingest path.

**Measured.** On the dev box (8 cores, but load average ~11 — parallel
agents saturating CPU), `arborist embed --quant int8` on wiki shard
`000.db` ran at **~4.3 chunks/s** (54,016 chunks embedded in 3 h 30 m;
the embed process got ~28 % of one core). At that rate one shard
(~1.56 M chunks) ≈ 100 h ≈ 4 + days, all four ≈ ~16 days. The full
backfill was **abandoned** as not feasible to brute-force here; the
54 K-chunk partial on `000.db` confirmed the per-chunk storage
(~409 B/chunk apparent → 384 B amortized → ~2.4 GB full corpus at
int8) — which is the deterministic number a backfill would only
re-confirm, so finishing it bought nothing. (On an *idle* healthy
box with batching + all cores, bge-small does ~50-200 chunks/s →
the full 6.24 M corpus ≈ ~9-35 h; the ticket's earlier "~17 h"
estimate is the optimistic end of that.)

**Per-chunk cost breakdown:**

| step | per-chunk cost (healthy box) |
|---|---|
| bge-small-en-v1.5 ONNX inference | ~5-30 ms (≫ everything else; chunk-length-dependent, 512-token cap) |
| chunker + SHA-256(leaf) + Merkle commit + sqlite INSERTs + zstd + FTS5 update | well under 1 ms total — the existing arborist ingest steps |

So adding vec **multiplies ingest time by roughly 10-100×** — entirely
in the ONNX matmuls. The non-vec ingest path is unchanged and still
runs at hundreds of chunks/s.

**Implication for the integration design (§14.2):** this is *why*
lazy-out-of-band is the default and `arborist ingest --embed` is the
opt-in:

- `arborist ingest` (default) — no vec, fast, same as before.
- `arborist ingest --embed` — eager; right for incremental ingests
  of a handful of docs, **wrong for a bulk wiki backfill** (the embed
  cost dominates the run).
- `arborist embed` (separate command) / a cron / a Prometheus-Σ
  unconscious-sweep task — the one-time multi-hour-to-multi-day pass,
  best run off-peak or on a dedicated box, never blocking a
  user-facing ingest.

**Production guidance:** a real corpus-wide backfill is a one-time
batch job sized in hours-on-idle / days-on-contended. It does **not**
slow ongoing ingest (which never embeds unless you pass `--embed`).
If the backfill latency matters, options are: run it off-peak; use a
dedicated box; or wire GPU/accelerated embedding (out of scope for
this ticket — `default_embedder()` is a pluggable `Embedder` callable,
so a GPU/onnxruntime-gpu/external-endpoint variant is a drop-in).

**Strategic framing:** this ~10-100× ingest-cost asymmetry — and the
matching query-side asymmetry (a vec query embeds the query string
first; an FTS5 query is B-tree lookups) — is *why* arborist is
lexical-first by default and dense-vector is an opt-in additive layer:
the cheap, proof-bearing, mobile-viable path is what ships; the
expensive semantic layer is there when you want it and can afford it.
See `docs/lexical-first-rationale.md`.
