arborist/docs/lexical-first-rationale.md

Lexical-first retrieval — why the cheap path is the default

What this is: the rationale for arborist's retrieval philosophy — lexical-first (FTS5 BM25 + content-addressed Merkle proofs) as the default, with dense-vector semantic retrieval as an optional, additive layer (#000039). Not a ticket; a positioning/architecture reference. The headline: arborist's ingest is 10-100× cheaper per document than building a vector-DB representation, on the same SQLite substrate — which is the difference between "ingest + search runs on a phone" and "the NPU is now a hand-warmer."

---

1. The cost asymmetry, measured

step	per-chunk cost (healthy CPU core)
bge-small-en-v1.5 ONNX inference (a vector-DB-style embed)	~5-30 ms — chunk-length-dependent, 512-token cap; worse under load (~250 ms/chunk ≈ 4 chunks/s measured on a contended dev box at load 11/8 cores)
arborist's ingest pipeline: chunker (tok-512-v1) + SHA-256 leaf hash + Merkle commit + SQLite INSERTs (documents / chunks / edges / audit_events) + zstd content compression + FTS5 inverted-index update	well under 1 ms total

So adding embeddings multiplies ingest time by ~10-100×, entirely in the ONNX matmuls. The non-vec ingest path runs at hundreds of chunks/s; a corpus-wide embed of arborist's 6.24 M-chunk wiki shard set is a one-time hours-on-idle / days-on-contended batch job (#000039 §14.6) — not something you'd do on a phone, ever.

And the query side is asymmetric too. An FTS5 query: tokenize, look up posting lists in the inverted-index B-tree per term, BM25-rank — a handful of B-tree traversals, microseconds to low-milliseconds. A vector query: tokenize, run the embedder (one transformer forward pass — the same ~5-30 ms as embedding a document chunk), then the ANN scan. On a phone, every search would pay a transformer forward pass before it even touches the index.

2. Same SQLite file, different philosophy

Both approaches put a .db file on disk. The split is in what's in it and how retrieval works:

	lexical-first (arborist)	embedding-first (a vector DB)
index	FTS5 inverted index (term → posting list)	dense vectors + ANN index (IVF / HNSW / flat)
build cost	~free (sort tokens into B-trees)	a transformer forward pass per chunk
query cost	B-tree lookups + BM25	embed the query (transformer) + ANN scan
matching	exact term / phrase / stem	semantic similarity
proof-bearing?	yes — chunks are content-addressed (leaf_hash), Merkle-rooted, the retrieved span is inclusion-provable against C(M)	no — embeddings are lossy, non-canonical; they can't enter a proof path (CLAUDE.md "soft hash vs hard hash")

That last row is the deep version of the point. arborist is the Merkle Providence model — content-addressed chunks, Merkle inclusion proofs, the audit chain — and that model is cheap by construction: the proof-bearing layer (hard hashes, FTS5-retrievable chunks) costs ~nothing to build. Embeddings are a soft signal — they help recall, they never enter a proof — and they're the expensive bolt-on. The "soft hash vs hard hash" discipline (hard = SHA-256 commitments / proofs / cache_key; soft = embeddings / TF-IDF / similarity, never in the proof path) maps directly onto "cheap-and-mandatory vs expensive-and-optional."

3. The consequence: edge / mobile viability

A phone (or a Raspberry Pi, or a permacomputer node) can:

• Ingest documents at arborist's full rate — sorting tokens into B-trees, hashing, writing SQLite rows is what mobile CPUs do well.
• Query the lexical index essentially for free — B-tree lookups, no neural net in the path.
• Use a precomputed int8 chunk_vecs someone else built off-device, if a semantic layer is wanted — querying it still costs a query-embedding forward pass per search, but the build cost (the days-long batch job) was paid elsewhere.

A vector-DB-first design can't do the first two cheaply: ingest pegs the CPU/NPU per chunk, and even browsing pays a forward pass per query. On battery, that's the difference between a usable offline knowledge store and a hand-warmer that's flat by lunch.

4. The honest caveat

Lexical-first trades the semantic allusion gap — the case where a query and the target chunk share zero stems but mean the same thing ("what did Orwell call the country at war with Oceania?" → an article that says "Eastasia" / "Eurasia" but never the query's words). FTS5's four routes (body BM25, title-LIKE, core-keyword TF-IDF, phrase- pattern) close the lexical allusion gap, not the semantic one. Dense vectors close the semantic gap.

That's exactly why arborist's answer isn't "we don't do vectors" — it's: vectors are an opt-in, additive 5th retrieval route, never the default, and the embed pass is lazy / out-of-band (a cron, or a Prometheus-Σ unconscious-sweep task per #000037 §3.1) so the heavy transformer work runs off-device / off-peak. The cheap, mobile-viable, proof-bearing path is what ships by default; the semantic layer is there when you want it and can afford it, and it doesn't compromise the rest. (int8 quantization keeps the storage tax at +6% over the shards — vs +25% at float32 — and the embeddings still never enter the proof path.)

5. References

• docs/tickets/ticket-000039-sqlite-vec-optional-backend.md — the optional vec backend; §3 (storage), §4 (additive not replacement), §14 (ingest integration + idempotency), §14.6 (the embed-throughput measurement these numbers come from).
• docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md — Phase 2: wiring vec as a 5th retrieval route (gated on a corpus backfill + a recall bench).
• CLAUDE.md "Retrieval pipeline" — the four FTS5 routes; "soft hash vs hard hash" — the discipline this rationale generalizes.
• ~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst — the canonical Merkle Providence Reverse RAG paper (this doc is the arborist-side positioning note; folding the mobile-viability argument into the paper proper is a possible follow-up).
• #000037 §3.1 — the Prometheus-Σ unconscious-sweep task that's the natural home for the lazy embed pass.