# Lexical-first retrieval — why the cheap path is the default

**What this is:** the rationale for arborist's retrieval philosophy —
lexical-first (FTS5 BM25 + content-addressed Merkle proofs) as the
default, with dense-vector semantic retrieval as an *optional,
additive* layer (#000039). Not a ticket; a positioning/architecture
reference. The headline: arborist's ingest is **10-100× cheaper per
document** than building a vector-DB representation, on the same
SQLite substrate — which is the difference between "ingest + search
runs on a phone" and "the NPU is now a hand-warmer."

---

## 1. The cost asymmetry, measured

| step | per-chunk cost (healthy CPU core) |
|---|---|
| **bge-small-en-v1.5 ONNX inference** (a vector-DB-style embed) | ~5-30 ms — chunk-length-dependent, 512-token cap; *worse* under load (~250 ms/chunk ≈ 4 chunks/s measured on a contended dev box at load 11/8 cores) |
| arborist's ingest pipeline: chunker (`tok-512-v1`) + SHA-256 leaf hash + Merkle commit + SQLite INSERTs (`documents` / `chunks` / `edges` / `audit_events`) + zstd content compression + FTS5 inverted-index update | **well under 1 ms total** |

So adding embeddings multiplies ingest time by **~10-100×**, entirely
in the ONNX matmuls. The non-vec ingest path runs at hundreds of
chunks/s; a corpus-wide embed of arborist's 6.24 M-chunk wiki shard
set is a one-time hours-on-idle / days-on-contended *batch job*
(#000039 §14.6) — not something you'd do on a phone, ever.

**And the query side is asymmetric too.** An FTS5 query: tokenize,
look up posting lists in the inverted-index B-tree per term, BM25-rank
— a handful of B-tree traversals, microseconds to low-milliseconds. A
vector query: tokenize, run the embedder (**one transformer forward
pass — the same ~5-30 ms as embedding a document chunk**), *then* the
ANN scan. On a phone, every search would pay a transformer forward
pass before it even touches the index.

## 2. Same SQLite file, different philosophy

Both approaches put a `.db` file on disk. The split is in *what's in
it* and *how retrieval works*:

| | lexical-first (arborist) | embedding-first (a vector DB) |
|---|---|---|
| index | FTS5 inverted index (term → posting list) | dense vectors + ANN index (IVF / HNSW / flat) |
| build cost | ~free (sort tokens into B-trees) | a transformer forward pass per chunk |
| query cost | B-tree lookups + BM25 | embed the query (transformer) + ANN scan |
| matching | exact term / phrase / stem | semantic similarity |
| proof-bearing? | **yes** — chunks are content-addressed (`leaf_hash`), Merkle-rooted, the retrieved span is inclusion-provable against `C(M)` | **no** — embeddings are lossy, non-canonical; they can't enter a proof path (CLAUDE.md "soft hash vs hard hash") |

That last row is the deep version of the point. arborist *is* the
Merkle Providence model — content-addressed chunks, Merkle inclusion
proofs, the audit chain — and that model is **cheap by construction**:
the proof-bearing layer (hard hashes, FTS5-retrievable chunks) costs
~nothing to build. Embeddings are a *soft* signal — they help recall,
they never enter a proof — and they're the expensive bolt-on. The
"soft hash vs hard hash" discipline (hard = SHA-256 commitments /
proofs / cache_key; soft = embeddings / TF-IDF / similarity, never in
the proof path) maps directly onto "cheap-and-mandatory vs
expensive-and-optional."

## 3. The consequence: edge / mobile viability

A phone (or a Raspberry Pi, or a permacomputer node) can:

- **Ingest** documents at arborist's full rate — sorting tokens into
  B-trees, hashing, writing SQLite rows is what mobile CPUs do well.
- **Query** the lexical index essentially for free — B-tree lookups,
  no neural net in the path.
- **Use** a precomputed int8 `chunk_vecs` someone else built
  off-device, if a semantic layer is wanted — querying it still costs
  a query-embedding forward pass per search, but the *build* cost
  (the days-long batch job) was paid elsewhere.

A vector-DB-first design can't do the first two cheaply: ingest pegs
the CPU/NPU per chunk, and even browsing pays a forward pass per
query. On battery, that's the difference between a usable offline
knowledge store and a hand-warmer that's flat by lunch.

## 4. The honest caveat

Lexical-first **trades** the *semantic allusion gap* — the case where
a query and the target chunk share zero stems but mean the same thing
("what did Orwell call the country at war with Oceania?" → an article
that says "Eastasia" / "Eurasia" but never the query's words). FTS5's
four routes (body BM25, title-LIKE, core-keyword TF-IDF, phrase-
pattern) close the *lexical* allusion gap, not the semantic one. Dense
vectors close the semantic gap.

That's exactly why arborist's answer isn't "we don't do vectors" —
it's: **vectors are an opt-in, additive 5th retrieval route, never the
default**, and the embed pass is lazy / out-of-band (a cron, or a
Prometheus-Σ unconscious-sweep task per #000037 §3.1) so the heavy
transformer work runs off-device / off-peak. The cheap, mobile-viable,
proof-bearing path is what ships by default; the semantic layer is
there when you want it and can afford it, and it doesn't compromise
the rest. (`int8` quantization keeps the storage tax at +6% over the
shards — vs +25% at float32 — and the embeddings still never enter the
proof path.)

## 5. References

- `docs/tickets/ticket-000039-sqlite-vec-optional-backend.md` — the
  optional vec backend; §3 (storage), §4 (additive not replacement),
  §14 (ingest integration + idempotency), §14.6 (the embed-throughput
  measurement these numbers come from).
- `docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md` — Phase 2:
  wiring vec as a 5th retrieval route (gated on a corpus backfill + a
  recall bench).
- CLAUDE.md "Retrieval pipeline" — the four FTS5 routes; "soft hash vs
  hard hash" — the discipline this rationale generalizes.
- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst`
  — the canonical Merkle Providence Reverse RAG paper (this doc is the
  arborist-side positioning note; folding the mobile-viability argument
  into the paper proper is a possible follow-up).
- #000037 §3.1 — the Prometheus-Σ unconscious-sweep task that's the
  natural home for the lazy embed pass.