arborist/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md

Ticket #000039 — Optional sqlite-vec retrieval backend (A/B vs FTS5, hybrid not replacement)

Status: closed · 2026-05-12 — Phase 0 (doc) + Phase 1 (the optional vec backend + ingest path + --quant) landed; Phase 2 (RRF hybrid fusion in query.py) split to #000050, gated on a corpus backfill + a ≥5pp recall bench (the corpus-wide arborist embed is a one-time hours-on-idle / days-on-contended batch job per §14.6 — not done here). The vec layer ships demonstrated on crawl_appliedcombinatorics_org.db (semantic hits topically correct) + a 54K-chunk partial on wiki shard 000 (confirming ~409 B/chunk → ~2.4 GB full-corpus at int8). 16 vec tests; full suite green. Detail below ↓. Phase 0 (doc) + Phase 1 landed 2026-05-11. Phase 1: arborist/search/vec.py — VecBackend(SearchBackend) (UNGROUNDED hits, never in proof path), chunk_vecs vec0 virtual table + vec_meta (sibling tables — don't touch chunks/documents/audit chain), embed_documents() ingest (delete-then-insert idempotent; vec0 doesn't honor INSERT-OR-REPLACE), pluggable Embedder callable with a fastembed bge-small-en-v1.5 default. CLI: arborist embed [--limit] [--batch-size] + arborist search --backend vec. [vec] optional extra (sqlite-vec + fastembed). Obvious v1 tuning (VEC_BACKEND_VERSION = vec-v1-bge-small-en-v1.5-384float32-cosine-flat): embedder BAAI/bge-small-en-v1.5, dim 384, quant float32 (int8/binary = the production storage knob per §3.1, not wired in v1), metric cosine (bge outputs L2-normalized so cosine ≡ L2 ranking), ANN flat (vec0 default), top_k 20. 7 tests (tests/test_search_vec.py, stub embedder — plumbing only; semantic quality demonstrated on a real shard). Demonstrated on crawl_appliedcombinatorics_org.db (168 chunks embedded in ~37s incl. model load; semantic queries return topically-correct hits — "how many ways to choose k things from n" → top hit "AC Combinations"; chain-check on that shard reports 0 after embedding). The 5 hyperparams fold into governance_policy_hash in a later phase (§6 — not wired yet). Ingest integration landed 2026-05-11 (see §14): embed_documents() is now incremental by default (embeds only chunk_ids not already in chunk_vecs — re-runs are cheap no-ops); arborist embed --rebuild does the DROP+recreate+full-re-embed for a VEC_BACKEND_VERSION bump; arborist ingest --embed is the eager opt-in (embed this run's new chunks after the chunk+Merkle-commit pass; default ingest does NOT embed — the lazy arborist embed pass / cron / Prometheus-Σ unconscious sweep is the usual path). --quant {float32,int8} landed 2026-05-11 (arborist embed --quant int8 [--rebuild]; the chunk_vecs vec0 column is int8[384] vs float[384] per quant; int8 blobs are vec_int8(?)-wrapped — sqlite-vec v0.1.9 treats a bare blob as float32; quant change on an existing table requires --rebuild since the vec0 element type can't be altered in place; the quant folds into VEC_BACKEND_VERSION → ...-384int8-..., recorded in vec_meta). int8 head-to-head on crawl_appliedcombinatorics_org.db: storage 1.60 MB → 0.42 MB (3.8× smaller; ~4× at corpus scale where blocks fill); recall ≈ float32 — Q2 "how many ways to choose k things from n" identical top-5, Q1 "pigeonhole principle counting" identical top-2 with a sub-noise rank-3/4 swap (Δdistance 0.002). So int8 is the obvious production config (§3.1's +6%-tax recommendation confirmed) — v1 default stays float32 for max fidelity; switching the default to int8 is a fox call. 16 vec tests. Embed throughput measured 2026-05-11 (§14.6): ~4.3 chunks/s on the contended dev box (~50-200/s idle); a 54K-chunk wiki backfill confirmed ~409 B/chunk → ~2.4 GB full-corpus at int8 (the deterministic number), then was abandoned — full corpus is hours-on-idle / days-on-contended, a one-time off-peak/dedicated-box batch job, not something to brute-force inline. The non-vec ingest path is unchanged (hundreds of chunks/s); adding vec multiplies ingest by ~10-100× (all in the ONNX matmuls) — which is exactly why lazy-out-of-band is the default and ingest --embed is the opt-in. Phase 2 (RRF hybrid fusion in query.py) gated on a ≥5pp recall-lift measurement on bench fixtures with no STRICT-rate regression (§8). Opened: 2026-05-09 Scope: Spec an optional sqlite-vec backend that runs alongside the existing FTS5 retrieval pipeline (never replacing it), with phased gates on storage cost, embedder choice, and measured recall lift. Phase 0 is this doc — no code, no schema bump. Phase 1 (ingest-side embedding + VecBackend class) opens only when the storage projection in §3 lands within budget on a real shard. Phase 2 (hybrid fusion in query.py) opens only when Phase 1 measures ≥5pp recall lift on the bench fixtures with no STRICT-rate regression. Audience: fox + future blackops shifts + downstream Arborist clients deciding whether to ship a vec layer. Hard constraint: vec retrieval never enters proof path — hits land audit_mode=UNGROUNDED, same as FTS5 today (per the soft-hash / hard-hash separation in CLAUDE.md). No new cache_key dimension; embedder identity + dimensionality + quantization fold into governance_policy_hash like every other versioned default. Vec storage is additional to FTS5, not a replacement (see §4 for the architectural reasoning). Optional dep — pip install '.[vec]' — graceful fallback when extension is not loadable. Pre-v1 dep discipline: pin sqlite-vec==X.Y.Z and document the upgrade rule.

---

1. Problem statement

Arborist's retrieval today is four parallel FTS5 routes per shard merged then reranked (CLAUDE.md "Retrieval pipeline" §1):

body BM25  +  title LIKE  +  core-keyword (TF-IDF)  +  phrase-pattern

Each route closes a specific gap. Phrase-pattern (route 4) closes the lexical allusion gap — "always been at war" verbatim-matches the 1984 article whose title shares zero tokens with the query. There is no symmetric route that closes the semantic allusion gap — the case where the query and target chunk share zero stems but mean the same thing. Examples that today fall through every route:

query:  "the document store that proves what it answered"
target: an Arborist documentation chunk that uses "content-addressed",
        "Merkle-committed", "audit chain", "providence cache" — zero
        stem overlap with the query, no verbatim 5-gram phrase, no
        title-LIKE candidate, no TF-IDF core overlap.

query:  "what did Orwell call the country at war with Oceania?"
target: 1984 article using "Eastasia" / "Eurasia" — phrase route
        catches "always been at war"-style verbatim quotes but not
        synonymic re-phrasings of the same fact.

A vector embedding backend would close that gap by mapping query and chunk into a shared semantic space and returning top-K nearest. The question is whether the storage and compute cost pencils out, and whether vec retrieval adds to FTS5 or replaces it.

This ticket answers both questions with measured numbers, then specifies the integration surface so a Phase 1 implementation can proceed under a known budget.

---

2. Why sqlite-vec specifically (vs alternatives)

Arborist is SQLite-native by design. CLAUDE.md "Python only" rule forbids Rust / C / non-Python sibling indexes inside the repo; extension modules loaded by SQLite itself are fine because they appear to Python as ordinary sqlite3 rows.

Candidate	Verdict	Why
sqlite-vec	chosen	Pure C extension dlopen-able into stock sqlite3. Apache-2.0 / MIT dual-license. Same DB file as FTS5 — no second backing store. Matches the selectolax optional-dep precedent.
FAISS (sibling file)	rejected	Adds a second on-disk format outside the audit-chain. Two-file consistency is a new failure mode (cache_key talks about a single shard, not a shard + a vector blob).
pgvector (Postgres)	rejected	New runtime dependency. CLAUDE.md "fresh checkout needs only python3.12 + venv + sqlite3" rule.
Hand-rolled NumPy + flat scan	rejected	Works at small scale; doesn't survive 6.24M chunks (§3). No ANN escape hatch when corpus grows.
LanceDB / Chroma / Weaviate	rejected	Server runtimes. Fresh-checkout rule again.

Caveats on sqlite-vec:

• Pre-v1 (latest published v0.1.9). Repo explicitly warns "expect breaking changes." Pin tightly; treat upgrades as schema-bump events that re-embed every chunk.
• No formal max-dimension or transactional-write guarantees in upstream docs. Phase 0 deliverable: a 1k-chunk smoke test that measures (a) insert throughput, (b) WAL behavior, (c) crash recovery semantics. If any of those misbehave under WAL + synchronous=NORMAL (the existing arborist convention), Phase 1 is gated on a fix or a different config.

---

3. Storage hypothesis — the real 6 GB → 38 GB number

Fox's hypothesis: 6 GB compressed wiki blows up to ~35 GB after Merkle tree + FTS5 index. Measured today on ~/.arborist/shards/:

shard 000.db   9.6 GB    867 K docs   1.56 M chunks
shard 001.db   9.5 GB    868 K docs   1.56 M chunks
shard 002.db   9.6 GB    867 K docs   1.56 M chunks
shard 003.db   9.6 GB    867 K docs   1.56 M chunks
TOTAL         38.3 GB    3.47 M docs  6.24 M chunks

Within one shard the breakdown (SELECT name, SUM(pgsize) FROM dbstat GROUP BY name) is:

Component	MB	% of shard	Note
edges	3 670	38 %	Lossless supersedes / derives_from / related-doc links. 90.6 M total rows across the corpus — ~26 edges per document. Biggest single cost, bigger than chunks themselves.
chunks (compressed content)	2 649	28 %	Already zstd-compressed at rest.
chunks_fts_data (FTS5 inverted index)	1 501	16 %	The full lexical index.
idx_edges_dst_root	589	6 %	Edges secondary index.
audit_events	377	4 %	Hard-hash chain, one row per state-change.
documents	170	2 %	Document metadata.
chunks autoindex + leaf index	251	3 %
merkle_nodes	114	1 %	Internal nodes only — leaves are folded into chunks.leaf_hash.
concept_relations	10	0.1 %	Per-shard rivalry/synonym graph (#000018-area).
Everything else	~ 270	3 %

So the 6 GB → 38 GB blowup is not "FTS5 + Merkle tree." It is:

edges          ~38 %        relationship graph
chunks         ~28 %        compressed content (decompressed at query)
fts5           ~16 %        the actual inverted index
indexes        ~10 %        secondary indexes on the above
audit chain    ~4 %         hard-hash provenance receipts
merkle interior ~1 %        log-N internal nodes

The FTS5 index itself is only ~16 % of the shard. Most of the blowup is lossless provenance (edges + chunks + audit). Anyone proposing a vec backend should measure against that baseline, not against the FTS5 fraction.

3.1 Vec storage projection at 6.24 M chunks

sqlite-vec supports float32 / int8 / binary across configurable dimension. ANN index choice multiplies that. Using 6.24 M chunks (today's full corpus) and the bytes-per-chunk math:

Quantization	Dim	Bytes / chunk	Vec data total	+ Flat index	+ IVF (~1.1 ×)	+ DiskANN/HNSW (~1.7 ×)
float32	384	1 536	9.6 GB	9.6 GB	10.6 GB	16.3 GB
float32	768	3 072	19.2 GB	19.2 GB	21.1 GB	32.6 GB
int8	384	384	2.4 GB	2.4 GB	2.6 GB	4.1 GB
int8	768	768	4.8 GB	4.8 GB	5.3 GB	8.2 GB
binary	384	48	300 MB	300 MB	330 MB	510 MB
binary	768	96	600 MB	600 MB	660 MB	1.0 GB
binary	1024	128	800 MB	800 MB	880 MB	1.4 GB

Mapped onto the existing 38 GB shard footprint:

Config	% tax over 38 GB	Comparable to
float32 × 384 + flat	+ 25 %	Bigger than the entire FTS5 index. Reject.
float32 × 768 + DiskANN	+ 85 %	Doubles disk. Reject outright.
int8 × 384 + flat	+ 6 %	Comparable to a single secondary index. Acceptable.
int8 × 768 + flat	+ 13 %	Comparable to FTS5. Acceptable if recall justifies it.
binary × 384 + flat	+ 0.8 %	Same scale as concept_relations (1.6 % tax tolerated).
binary × 768 + flat	+ 1.6 %	Tied with concept_relations. Acceptable.
binary × 1024 + flat	+ 2.1 %	Acceptable.

Phase 0 storage budget: ≤ 15 % tax over today's 38 GB total (i.e. ≤ 5.7 GB of vec data across all shards). That admits everything from binary × 1024 up through int8 × 768 + flat. It excludes anything float32 and anything with DiskANN graph overhead above ~int8 × 384.

3.2 What this looks like per-shard

At the recommended Phase-1 config (int8 × 384 + flat as the default, configurable):

per shard:    1.56 M chunks × 384 bytes = 600 MB
all 4 shards: 2.4 GB total
% of shard:   ~6 %  (sits between idx_edges_dst_root and audit_events)

At the storage-conservative config (binary × 768 + flat):

per shard:    1.56 M chunks × 96 bytes = 150 MB
all 4 shards: 600 MB total
% of shard:   ~1.6 %  (same scale as concept_relations)

The Phase 0 deliverable picks one of these two as the default and makes the other reachable via flag.

---

4. Architectural answer — additional, not same

Q: would vector search be the same as FTS, or additional?

A: Additional. Always. Vec is a fifth retrieval route, not a replacement for any of the four FTS5 routes.

4.1 Why never replacement

The four FTS5 routes each close a gap that vec cannot:

Route	What it catches	Why vec misses it
Body BM25	Term-frequency-weighted topical relevance	Embeddings smooth over rare terms; specialized vocab gets averaged into the centroid
Title LIKE	Exact title-prefix matches	Embeddings of short titles are dominated by stopword centroids — "the X of Y" embeds nearly identically for many X, Y
Core-keyword (TF-IDF)	Distilled-doc anchors	Cores are a different artifact (distillation output), not a chunk-level signal
Phrase-pattern (n=5/n=6)	Verbatim quote allusion ("always been at war")	Embeddings are bag-of-information; phrase order is lost. Verbatim recall is FTS5's job.

Vec adds:

New route	What it catches
Vec ANN top-K	Semantic paraphrase ("country at war with Oceania" → "Eastasia"); cross-vocabulary synonymy when the corpus uses unfamiliar terminology; concept-level matching when the user's words and the article's words share no stems

So the architectural choice is fixed: vec is a fifth parallel route, merged into the existing rerank pipeline.

4.2 Hybrid fusion design

Two standard merge strategies, both feasible with the existing pipeline:

Option A — Reciprocal Rank Fusion (RRF). Each route emits ranked hits; final score is Σ_routes 1 / (k + rank_route) with k=60 per the original RRF paper. Robust to score-scale mismatch (BM25 scores are not comparable to cosine similarity). Recommended.

Option B — Score-level merge with normalization. Convert each route's score to a [0,1] percentile within its own distribution, then weighted-sum. More tunable but introduces a weights config that becomes a governance_policy_hash input. More moving parts than RRF.

Phase 0 picks Option A (RRF) as the default. Option B is a follow-up if RRF leaves measurable lift on the table.

4.3 Rivalry / title-relevance / phrase routes still apply

The post-merge filter chain (rivalry exclusion, four-accept-paths title-relevance, stem-aware token matching, per-source context cap) runs after the merge. Vec hits go through the same filter as FTS5 hits. A vec hit that passes ANN top-K but fails the title-relevance filter is dropped exactly like an FTS5 BM25 hit that fails the same filter — so the existing retrieval-driven hallucination guards (e.g. the spin-glass / QCD case from CLAUDE.md "Title-relevance hard check (Rule 8)") still hold.

---

5. The embedder is the gating item

Arborist's only LLM endpoint today is Hermes-3 Llama-3.1-8B at hermes.ai.unturf.com. Hermes is a chat model, not an embedding model. Without an embedder there is nothing to put into the vec table.

Phase 0 must pick one of these paths, in declining order of preference:

1. Local embedder bundled as optional dep. A small sentence-transformer (e.g. bge-small-en-v1.5, 384-dim; all-MiniLM-L6-v2, 384-dim) shipped via pip install '.[vec]'. Runs on CPU in arborist's process; no network call; no concurrency interaction with Hermes. ~100 MB model file. Strong default candidate because it preserves the "fresh checkout needs only python3.12 + venv + sqlite3" rule (the embed model is an optional extra, not a runtime dep).
2. Dedicated embedding endpoint. Stand up a sibling endpoint alongside Hermes (hermes-embed.ai.unturf.com?) running bge-large or similar. Frees arborist from CPU embedding work. Costs an extra service to operate.
3. Hermes itself for embeddings. Hijack a generative model for embeddings via prompt + last-hidden-state pooling. Possible but research-grade — bench quality unknown, breaks the ~4-concurrent bottleneck (#000037 §11) at every ingest. Not recommended.

Phase 0 deliverable: pick path 1 or 2 with a named model. Phase 1 implementation depends on which.

5.1 Why embed at ingest, never at query

Embedding cost is amortized:

ingest-time:  embed each chunk once,  store in vec table
query-time:   embed query string once (~10 ms on CPU),  ANN top-K

If the embedder lives in arborist's process (path 1), the only network cost at query time is the ANN scan inside SQLite — no LLM call. This sidesteps the Hermes ~4-concurrent bottleneck for queries entirely. Ingest pays the embedding cost up front, once per chunk, on the chunker-version-pinned content.

---

6. Versioning and governance_policy_hash integration

Embedder choice is a versioned default. Per CLAUDE.md "Versioned defaults" rule (tok-512-v1, norm-v1, wikitext-base-v1, v9.8.0), the vec layer adds:

embedder-name      e.g. "bge-small-en-v1.5"
embedder-version   e.g. "@hf-rev-deadbeef"
quantization       e.g. "int8" / "binary" / "float32"
dimension          e.g. 384 / 768 / 1024
ann-index          e.g. "flat" / "ivf" / "diskann"
distance-metric    e.g. "cosine" / "l2_normalized" / "dot" / "hamming"

These six fields fold into governance_policy_hash (NOT into a new cache_key dimension — keep the 8-dim invariant). Bumping any one stales every prior cached answer that consulted the vec backend, exactly as bumping the chunker stales every prior FTS5 hit. The vec table itself stays in place but is treated as "cold": query-time lookups filter on embedder_version = current and ignore older rows; a make rebuild-vec target re-embeds.

6.1 Distance metric — the silent-correctness field

Switching distance metric without re-embedding silently changes ranking. A query that returned chunk A as top-1 under cosine may return chunk B under L2 if the embeddings are not unit-normalized. Because nothing else in the pipeline observes the change (the vec backend just emits ranked Hits; the verifier and cache_key are ranking-blind), a metric-only flip would be invisible drift — exactly the failure mode CLAUDE.md "Versioned defaults" exists to prevent. Hence its inclusion in the policy hash.

The recommended canonical config:

1. embedder produces unit-normalized vectors    (||v|| = 1)
2. ingest stores vectors as-is                  (no further normalize)
3. query stores  metric = "l2_normalized"        (in policy hash)
4. SQL uses      vec_distance_l2()              (== cosine ranking,
                                                 no per-query norm,
                                                 SIMD-friendly)

metric = "cosine" (i.e. vec_distance_cosine() at query time) is the alternative when the embedder output is not guaranteed unit-normalized; pays a per-row norm computation in exchange for correctness regardless of input vector magnitude. Slower, never needed if §1 holds.

metric = "dot" is for pre-scaled int8 / float16 where unit normalization is impossible — the embedder writer is expected to have rescaled the vectors so dot-product ranks identically to cosine on the original float32 normalized vectors. Documented in the embedder version note; bump embedder-version if the rescale recipe changes.

metric = "hamming" is the only correct option for binary quantization — cosine on bit-vectors is undefined. Switching from binary + Hamming to int8 + L2 is a two-field policy bump (quantization AND distance-metric), and re-embeds nothing because the underlying float vectors weren't stored — make rebuild-vec must re-embed from source.

6.2 Quantization × metric compatibility matrix

Quantization	Valid metrics	Default
float32 (normalized)	l2_normalized, cosine	l2_normalized
float32 (unnormalized)	cosine	cosine
int8 (pre-scaled)	dot, l2_normalized	dot
binary	hamming	hamming

Phase 1 enforces the matrix at ingest: a config that pairs `binary

• cosine` errors out before any chunk is embedded. The error message names the matrix entry that would resolve it.

---

7. SearchBackend integration — where the code lands

Existing surface (arborist/search/base.py):

class SearchBackend(ABC):
    name: str
    audit_mode: AuditMode

    def search(self, query: str, limit: int = 20) -> list[Hit]: ...

Phase 1 adds arborist/search/vec.py mirroring fts5.py:

class VecBackend(SearchBackend):
    name = "vec"
    audit_mode = AuditMode.UNGROUNDED   # same as FTS5 — soft signal

    def __init__(self, conn, embedder):
        super().__init__(conn)
        self.embedder = embedder    # callable: str -> np.ndarray

    def search(self, query, limit=20):
        qv = self.embedder(query)
        rows = self.conn.execute(
            "SELECT chunk_id, distance FROM chunk_vecs "
            "WHERE embedding MATCH ? AND k = ? "
            "ORDER BY distance",
            (qv.tobytes(), limit),
        ).fetchall()
        # JOIN back to chunks + documents, build Hits with the same
        # _build_snippet helper FTS5 already uses.
        ...

The merge with FTS5 happens in arborist/qa/query.py where the four FTS5 routes already merge today. RRF is a six-line addition once both rankings exist.

Optional-dep guard at import time (mirrors [html]):

try:
    import sqlite_vec
    _VEC_AVAILABLE = True
except ImportError:
    _VEC_AVAILABLE = False

CLI surface only exposes --retrieval=vec or --retrieval=hybrid when _VEC_AVAILABLE, otherwise the flag errors with an install hint. Same pattern as selectolax for the HTML source.

---

8. Phased plan with measurement gates

Phase 0 — this ticket (doc)

• Decide embedder (path 1 or path 2, named model).
• Decide default quantization + dimension (recommendation: int8 × 384 + flat as the speed/quality default; binary × 768 + flat as the storage-conservative alternative).
• Run the 1k-chunk smoke test under WAL + synchronous=NORMAL.
• Write the bench protocol: which fixtures, what signal floor. Lift target: ≥ 5pp recall@20 on bench-emergent random-word fixtures (bench-maxing.md signal floor).

Phase 1 — code (gated on Phase 0 sign-off)

• arborist/search/vec.py with optional-dep guard.
• chunk_vecs virtual table created lazily on first ingest with --embed.
• arborist ingest --embed ingest-side flag (default off).
• make rebuild-vec Makefile target.
• Rebuilds populate chunk_vecs for all current chunks rows.
• Bench: vec-only retrieval vs FTS5-only on a single shard. Phase 1 succeeds if vec-only matches FTS5-only within ±5pp on STRICT-rate AND beats FTS5 by ≥ 5pp on at least one bench-emergent semantic-allusion fixture.

Phase 2 — hybrid fusion (gated on Phase 1 success)

• RRF merge in arborist/qa/query.py between FTS5 routes and vec route.
• Bench: hybrid vs FTS5-only.
• Phase 2 succeeds if hybrid lifts STRICT-rate by ≥ 5pp over FTS5-only with no UNGROUNDED-rate regression.
• If Phase 2 fails by < 5pp lift: park the vec layer as opt-in per-call (--retrieval=vec) but do not make hybrid the default. The 2.4 GB tax is not paid for a sub-signal-floor lift.

Phase 3 — governance_policy_hash wiring (mechanical)

• Fold the five vec-version fields (§6) into the policy hash.
• Document in CLAUDE.md "Versioned defaults" and "Schema invariants" sections.
• Update make chain-check-shards semantics if needed (it should not need changes — vec data is sibling, like concept_relations).

---

9. Decision tree on quantization

Phase 0 budget: ≤ 15 % storage tax (≤ 5.7 GB across all shards).

If Phase 1 bench shows int8 × 384 hits the recall target
   → ship int8 × 384 + flat as default.
   → 6 % storage tax. Best speed/quality for the budget.

If Phase 1 bench shows int8 × 384 misses the target
   AND int8 × 768 hits it
   → ship int8 × 768 + flat.
   → 13 % storage tax. Borderline acceptable.

If Phase 1 bench shows int8 × 768 also misses
   → DO NOT ship float32 anywhere — too expensive.
   → Park the ticket. Reopen when a smaller, sharper embedder lands
     (research-bench problem, not an arborist problem).

If storage budget tightens later
   → fall back to binary × 768 + flat. < 2 % tax,
     ~10–20 % recall hit, still wins on the semantic-allusion
     fixtures the lexical pipeline misses entirely.

---

10. Risks and open questions

1. sqlite-vec pre-v1 instability. Mitigation: pin tightly, treat upgrades as schema-bump events, run the 1k-chunk smoke test after every dep bump. Phase 0 deliverable.
2. Embedder model drift. A new HuggingFace revision of bge-small invalidates every prior embedding. Mitigation: pin model+revision via embedder_version in §6, track upgrades under make rebuild-vec.
3. Re-embed cost on chunker bump. tok-512-v1 → v2 already stales every FTS5 chunk. Adds re-embed work proportional to chunk count. At 6.24 M chunks × ~10 ms/chunk on a CPU bge-small ≈ 17 hours single-threaded; parallelizes trivially. Already a known-large cost; vec adds ~10–20 % to it.
4. Verifier semantics unchanged. Vec lifts retrieval recall; the lexical verifier (quote / span / entity / paraphrase) stays binary. A vec-retrieved chunk that the verifier cannot ground still lands UNGROUNDED. Worth confirming on the bench: does vec recall translate to STRICT-rate lift, or just to more UNGROUNDED hits being dropped at verification time?
5. Title-relevance hard check (Rule 8) interaction. Vec is precisely the retrieval mode most likely to surface structurally-unrelated cited chunks (the spin-glass / QCD case). Phase 1 bench must include the existing title-mismatch fixtures and confirm vec hits get caught by Rule 8 like any other retrieval-driven hallucination candidate. If vec hits bypass Rule 8 somehow, that is a Phase 1 blocker.
6. edges is the dominant cost (38 % per §3), not FTS5. Confirmed with the corpus-wide row count: 90.6 M edges across 3.47 M documents = ~26 edges/document. If storage matters more than retrieval quality, compacting edges is a higher-leverage optimization than anything vec-related (a single int8 × 384 vec layer adds 6 % tax; trimming edges by 10 % saves 3.8 % at zero quality cost). Out of scope for this ticket but worth a reminder when we benchmark.
7. Concurrent-Hermes bottleneck (#000037 §11) is irrelevant here as long as we use a local embedder (path 1 in §5). If we go path 2 or 3, vec ingest competes with inference traffic and the Prometheus-Σ controller (#000037) would need to throttle ingest under load.

---

11. What this ticket does NOT do

• Does not commit to shipping a vec backend. Phase 0 is doc-only; every later phase has a measurable gate.
• Does not bump cache_key dimensions. Vec config folds into governance_policy_hash, same pattern as every prior versioned default.
• Does not change the verifier or the audit chain. Vec is a soft signal at the retrieval layer only; proof-path stays identical.
• Does not replace any FTS5 route. §4 settles this: vec is a fifth route, never a substitute.
• Does not implement the unconscious falsification sweep (#000037 Phase 1). That is a separate ticket; vec just makes semantic neighbors easier to surface, which is one capability the sweep can use later.

---

12. References

• This ticket: doc-only Phase 0.
• arborist/search/base.py — SearchBackend ABC, the integration surface.
• arborist/search/fts5.py — the contract the vec backend mirrors.
• arborist/qa/query.py — where RRF merge would land.
• CLAUDE.md "Retrieval pipeline" — the four-route baseline.
• CLAUDE.md "Soft hash vs hard hash" — the rule that licenses vec retrieval as long as it never enters the proof path.
• CLAUDE.md "Versioned defaults" — the pattern §6 follows.
• #000018 (concept_relations) — the 1.6 % storage-tax precedent.
• #000037 §11 — the Hermes ~4-concurrent constraint that drives the "embed at ingest, not at query" rule.
• ~/.arborist/shards/ — 4 × 9.6 GB shards measured 2026-05-09; the per-table breakdown in §3 is dbstat-derived from 000.db.
• Upstream: https://github.com/asg017/sqlite-vec (v0.1.9 at time of writing; expect breaking changes pre-v1).

---

13. Phase 1 implementation plan (proposal, 2026-05-10)

Status: awaiting fox go/no-go on the four §13.1 decisions below. Phase 1 code does not start until they're settled.

13.1 Phase-0 deliverables — proposed picks

Four explicit decisions Phase 0 left open. Recommendations:

1. Embedder path → Path 1 (local sentence-transformer bundled as optional dep). Preserves CLAUDE.md's "fresh checkout needs only python3.12 + venv + sqlite3" rule; sidesteps the Hermes 4-concurrent bottleneck (#000037 §11) entirely; no new service to operate. Model: BAAI/bge-small-en-v1.5 — 384-dim, ~33 MB, MIT-licensed, top of MTEB-en/retrieval among sub-100MB models, unit-normalized output (matches §6.1 canonical config).
2. Default quantization → int8 × 384 + flat per the §9 decision tree's primary leg. 6 % storage tax (~2.4 GB across the 4-shard cluster), well within the 15 % budget. Speed/quality default. Storage-conservative alternative binary × 768 + flat (1.6 % tax) reachable via --vec-quantization=binary --vec-dim=768.
3. 1k-chunk smoke test → §13.2 below.
4. Bench protocol → §13.3 below.

13.2 Pre-flight smoke (gate before any Phase 1 code lands)

make smoke-vec     # new make target, ~2 min wall

What it does:

• Pull 1k random chunks.content rows from ~/.arborist/shards/000.db into /tmp/smoke-vec.db (read-only on the source shard).
• Embed with bge-small-en-v1.5 on CPU (single-threaded; ~10 ms/chunk → ~10 s wall).
• INSERT into chunk_vecs virtual table under WAL + synchronous=NORMAL (existing arborist convention).
• Kill -9 mid-insert (process group); reopen; check WAL recovery + journal mode + row count consistency.
• Run 100 query embeddings × top-20 ANN. Measure p50/p95 query latency.

Gate criteria: insert throughput ≥ 100 chunk/s, p95 query latency ≤ 50 ms, zero data loss across the kill -9 boundary. Fail on any of those → Phase 1 blocked, surface to fox + upstream issue.

13.3 Bench protocol

Fixtures (all already in bench/):

• bench/qa_questions_smoke.txt — 5-question fast loop
• bench/qa_questions_progressive_and.txt — progressive-AND fallback fixture
• Existing bench-emergent random-word stress fixture
• Existing qa-modes-bench fixture for STRICT-rate guard

Three conditions:

condition	retrieval routes	claim_lattice mode
Baseline	FTS5 only (current 4 routes)	claim_lattice (JSON)
Vec-only	Vec ANN top-20 only	claim_lattice (JSON)
Hybrid	FTS5 + Vec, RRF merge (k=60)	claim_lattice (JSON)

Phase 1 success criteria (from §8):

• Vec-only matches FTS5-only within ±5pp STRICT-rate (sanity)
• AND beats FTS5-only by ≥5pp on at least one semantic-allusion bench-emergent fixture
• AND no UNGROUNDED-rate regression vs baseline

Phase 2 success criteria (gates on §8):

• Hybrid lifts STRICT-rate by ≥5pp over baseline
• AND no UNGROUNDED-rate regression
• AND title-relevance hard check (Rule 8) catches mismatched vec hits at the same rate it catches mismatched FTS5 hits

13.4 Code structure

arborist/embed.py                 [NEW]
    class Embedder:
        - lazy-loads BAAI/bge-small-en-v1.5 once per process
        - normalize=True (unit vectors, §6.1 canonical config)
        - encode(texts: list[str]) -> np.ndarray[N, 384] dtype=int8
        - sentencetransformer dependency under [vec] extras

arborist/search/vec.py            [NEW, mirrors fts5.py]
    class VecBackend(SearchBackend):
        - audit_mode = AuditMode.UNGROUNDED  # §1 hard constraint
        - chunk_vecs virtual table creation (lazy; first ingest --embed)
        - search(query, limit=20) -> list[Hit] via vec_distance_l2

arborist/store.py                 [PATCH]
    - SCHEMA_SQL: NO new always-on tables (vec table is virtual,
      created only when --embed runs)
    - Add embedder_version, vec_quantization, vec_dim, vec_ann_index,
      vec_distance_metric to governance_policy_hash inputs (NOT
      cache_key — §6 hard constraint)

arborist/qa/query.py              [PATCH]
    - if vec backend present: run RRF merge over (FTS5_routes_merged,
      vec_route) with k=60. New code is ~6 lines per §7.

arborist/cli.py                   [PATCH]
    - ingest --embed flag (gated on _VEC_AVAILABLE import-check)
    - query --retrieval={fts5|vec|hybrid} flag
    - new subcommand: arborist vec rebuild [--db ... --batch N]

Makefile                          [PATCH]
    smoke-vec:        # §13.2 pre-flight; gate-before-Phase-1
    rebuild-vec:      # populate chunk_vecs for an existing shard
    bench-vec:        # run §13.3 three-condition bench

pyproject.toml                    [PATCH]
    [project.optional-dependencies]
    vec = [
        "sqlite-vec>=0.1.9,<0.2",
        "sentence-transformers>=2.7,<3",
        "numpy",
    ]

13.5 Test plan

Unit tests:

• tests/test_embed.py — deterministic output for fixed input (regression guard against silent model drift between revs); unit-norm verification; int8 round-trip.
• tests/test_search_vec.py — VecBackend.search against a 3-chunk in-memory shard with hand-crafted embeddings; verify ranking matches expected; verify audit_mode=UNGROUNDED; verify _VEC_AVAILABLE=False graceful degradation.
• tests/test_governance_policy.py — verify the 5 new vec-config fields fold into governance_policy_hash; flipping any one invalidates prior cached records.

CLI surface tests (per docs/calculator-test-patterns.md §6 — subprocess invocation catches argparse + main() drift the import-only tests miss; this hazard surfaced three times during the 2026-05-10 substrate rename refactor in yesterday's commit chain, so we pin the pattern):

• tests/test_cli_vec_rebuild.py — subprocess invocation of arborist vec rebuild --db <tmp_path/test.db> --batch 4 against a 12-chunk synthetic shard; assert exit 0, JSON status output if any, row-count parity in chunk_vecs. Skipped via pytest.importorskip when [vec] extras absent.
• tests/test_cli_ingest_embed_flag.py — subprocess invocation of arborist ingest --source html --embed --author X against a tmp shard; assert --embed flag wires through to the embedder; row count after ingest matches; vec rows match chunk rows 1:1.
• tests/test_cli_query_retrieval_flag.py — subprocess invocation of arborist query --retrieval=vec and --retrieval=hybrid and --retrieval=fts5 against a pre-embedded shard fixture; assert all three exit 0 and produce structurally-distinct retrieval candidate sets (vec ≠ fts5 on at least one query).

Integration tests (gated on [vec] extras installed; skipped otherwise):

• tests/test_vec_smoke_integration.py — runs the §13.2 smoke on a 100-chunk synthetic shard (smaller than 1k for CI speed); asserts WAL recovery + row count parity.

13.6 Out of scope for Phase 1

• ANN index variants (IVF, DiskANN, HNSW) — Phase 1 ships flat only. ANN is opt-in via vec_ann_index policy field; flat works at ≤10 M chunks per the upstream sqlite-vec docs.
• Re-embed cost on chunker bump — already a known-large cost (§10.3); covered by make rebuild-vec not Phase 1.
• Edges compaction (§10.6) — different ticket entirely.

13.7 Estimated size

• New files: 2 (embed.py, search/vec.py) ~ 200 LOC.
• Patches: 4 (store.py, query.py, cli.py, Makefile) ~ 100 LOC.
• Tests: 7 files ~ 400 LOC + fixtures (4 unit + 3 CLI subprocess per the §13.5 amendment 2026-05-10 + 1 integration). The CLI subprocess tests are ~30-40 LOC each — boilerplate + tmp_path
  • subprocess.run + JSON parse.
• Phase-1 doc append in docs/: this §13 + a journey-note bench result file.
• pyproject.toml: 1 stanza.

Total: ~550 LOC + ~400 test LOC. Single substantial commit if all tests pass + smoke succeeds; otherwise broken into the natural gates (smoke → embed.py → vec.py → CLI subprocess tests → integration tests → bench).

13.8 Decisions fox needs to make to unblock Phase 1

#	Decision	Recommendation
1	Embedder path 1 vs 2 vs 3	Path 1 (local bundle)
2	Default model name	BAAI/bge-small-en-v1.5 (MIT, 33 MB)
3	Default quantization × dim	int8 × 384 + flat
4	Approve sentence-transformers PyPI dep under [vec]	yes (~120 MB install footprint when extras pulled; default not pulled — pip install '.[vec]' only)

If fox approves all four → Phase 1 starts with §13.2 smoke as the first gate.

If fox rejects path 1 → Phase 1 blocked on standing up hermes-embed.ai.unturf.com (path 2; outside this ticket).

If fox rejects int8 × 384 default → propose binary × 768 as default (1.6 % tax, ~10–20 % recall hit per §9). Either is defensible.

If fox rejects the dep footprint → fall back to the lighter onnxruntime path with a bge-small ONNX model (still ~33 MB weights but smaller runtime). Adds complexity but keeps the no-server constraint.

---

14. Ingest integration & idempotency (landed 2026-05-11)

The load-bearing fact: a chunk_id's content is immutable in arborist. A chunk is content-addressed-ish (leaf_hash) — same content → same chunk_id; different content → a new chunk_id (re-ingest with changed content makes a new document_root + new chunk_ids, linked by supersedes; a chunker-version bump re-chunks → new chunk_ids). So a chunk, once embedded, never needs re-embedding — the only re-embed trigger is the embedder changing (VEC_BACKEND_VERSION bump).

14.1 Idempotency table

Event	chunk_vecs effect
Re-ingest same content	doc_root unchanged, chunks no-op-insert → already embedded → no-op
Re-ingest changed content (new doc_root, new chunk_ids)	old chunks + old vecs stay (lossless history); new chunk_ids embedded on next pass
Chunker bump (tok-512-v1 → v2)	re-chunk → new chunk_ids → embedded next pass; old chunk_ids' vecs are orphan-ish but harmless (old chunks still exist)
Cold eviction (content = NULL)	vec row persists — still valid; on rehydrate the same chunk_id returns with the same content
Embedder bump (VEC_BACKEND_VERSION changes)	the only full-re-embed case — arborist embed --rebuild (DROP+recreate+full pass); the vec_meta.backend_version row records the populating version
Superseded docs	old chunks stay embedded → searchable via vec — consistent with chunks_fts keeping superseded docs; the retrieval supersedes-filter applies equally

14.2 Two integration models

• Lazy / out-of-band (default). arborist ingest does not embed. arborist embed (incremental — embeds only chunk_ids not in chunk_vecs) populates on demand; the natural ops home is a cron after the nightly ingest, or — better — a Prometheus-Σ unconscious-sweep task (#000037 §3.1 already walks "ingested- but-not-yet-probed" documents; embed-the-new-chunks is exactly that shape, so vec needn't carry its own cron). Ingest stays fast; embedding is a separable, resumable, optional layer; chunk_vecs trails ingest by ≤ one pass.
• Eager opt-in. arborist ingest --embed — after the chunk+Merkle-commit pass, embed this run's new chunks (incremental, so it's just the delta). chunk_vecs stays in sync. Pays ~10-20 ms/chunk CPU per ingest — fine for incremental ingests, not for a bulk wiki backfill.

(#000039 §5.1's "embed at ingest, never at query" is about when not to embed the query corpus — it doesn't mandate embedding inside the ingest call.)

14.3 Commands

Command	Behavior
arborist --db D embed	incremental — embed chunk_ids not yet in chunk_vecs (cheap no-op once everything's embedded)
arborist --db D embed --rebuild	DROP + recreate chunk_vecs, then full re-embed — the clean VEC_BACKEND_VERSION-bump path (a search mid-rebuild never mixes old/new-model embeddings: the recreated table starts empty and grows new-model)
arborist --db D embed --limit N	cap chunks processed — smoke-test on a real shard
arborist ... ingest --source ... --embed	eager: after ingest, incremental-embed this run's new chunks
arborist --db D search Q --backend vec	semantic ANN; errors with an install/embed hint if [vec] missing or chunk_vecs empty

14.4 Concurrency

WAL serializes writers → arborist embed and arborist ingest on the same shard take turns. In the --shards-dir layout each shard is its own file → embed-shard-0 and ingest-shard-1 don't contend. Embed-during-ingest is safe under the lazy model: embed sees a consistent per-transaction snapshot; ingest's new chunks get picked up on the next incremental embed pass.

14.5 Versioning

vec_meta.backend_version (written by ensure_chunk_vecs_table) is the per-shard discriminator. The 5 vec hyperparams fold into governance_policy_hash in a later phase (§6 — not wired yet); the 8-dim cache_key invariant is untouched (vec config goes into the existing governance_policy_hash dim, not a new one). Same versioned-default discipline as tok-512-v1 / norm-v1 etc.

14.6 Embed throughput — measured, and why ingest stays fast (2026-05-11)

Embedding is the only expensive part of the vec layer, and it's expensive because it's CPU-bound ONNX inference — not because of anything in arborist's ingest path.

Measured. On the dev box (8 cores, but load average ~11 — parallel agents saturating CPU), arborist embed --quant int8 on wiki shard 000.db ran at ~4.3 chunks/s (54,016 chunks embedded in 3 h 30 m; the embed process got ~28 % of one core). At that rate one shard (~1.56 M chunks) ≈ 100 h ≈ 4 + days, all four ≈ ~16 days. The full backfill was abandoned as not feasible to brute-force here; the 54 K-chunk partial on 000.db confirmed the per-chunk storage (~409 B/chunk apparent → 384 B amortized → ~2.4 GB full corpus at int8) — which is the deterministic number a backfill would only re-confirm, so finishing it bought nothing. (On an idle healthy box with batching + all cores, bge-small does ~50-200 chunks/s → the full 6.24 M corpus ≈ ~9-35 h; the ticket's earlier "~17 h" estimate is the optimistic end of that.)

Per-chunk cost breakdown:

step	per-chunk cost (healthy box)
bge-small-en-v1.5 ONNX inference	~5-30 ms (≫ everything else; chunk-length-dependent, 512-token cap)
chunker + SHA-256(leaf) + Merkle commit + sqlite INSERTs + zstd + FTS5 update	well under 1 ms total — the existing arborist ingest steps

So adding vec multiplies ingest time by roughly 10-100× — entirely in the ONNX matmuls. The non-vec ingest path is unchanged and still runs at hundreds of chunks/s.

Implication for the integration design (§14.2): this is why lazy-out-of-band is the default and arborist ingest --embed is the opt-in:

• arborist ingest (default) — no vec, fast, same as before.
• arborist ingest --embed — eager; right for incremental ingests of a handful of docs, wrong for a bulk wiki backfill (the embed cost dominates the run).
• arborist embed (separate command) / a cron / a Prometheus-Σ unconscious-sweep task — the one-time multi-hour-to-multi-day pass, best run off-peak or on a dedicated box, never blocking a user-facing ingest.

Production guidance: a real corpus-wide backfill is a one-time batch job sized in hours-on-idle / days-on-contended. It does not slow ongoing ingest (which never embeds unless you pass --embed). If the backfill latency matters, options are: run it off-peak; use a dedicated box; or wire GPU/accelerated embedding (out of scope for this ticket — default_embedder() is a pluggable Embedder callable, so a GPU/onnxruntime-gpu/external-endpoint variant is a drop-in).

Strategic framing: this ~10-100× ingest-cost asymmetry — and the matching query-side asymmetry (a vec query embeds the query string first; an FTS5 query is B-tree lookups) — is why arborist is lexical-first by default and dense-vector is an opt-in additive layer: the cheap, proof-bearing, mobile-viable path is what ships; the expensive semantic layer is there when you want it and can afford it. See docs/lexical-first-rationale.md.