diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 47d2358..3507931 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -82,6 +82,15 @@ proposing change. Examples that stay un-numbered: exemplars (`pi_star_arithmetic` / `_logic` / `_code` / `_time_series`). Explicit "doesn't apply to verifier-style code" caveat.) +- `lexical-first-rationale.md` (positioning/architecture reference + for why lexical-first FTS5 + content-addressed Merkle proofs is + arborist's *default* retrieval and dense-vector semantic search is + an opt-in additive layer: arborist's per-document ingest is + ~10-100× cheaper than building a vector representation — same + SQLite substrate, different philosophy — which is the difference + between "ingest + search runs on a phone" and an NPU hand-warmer. + Generalizes the "soft hash vs hard hash" discipline; the numbers + come from #000039 §14.6. Written 2026-05-12.) If a doc proposes change AND awaits a decision AND has scoped implementation cost, it's a ticket. Otherwise it's reference. diff --git a/docs/lexical-first-rationale.md b/docs/lexical-first-rationale.md new file mode 100644 index 0000000..b42978c --- /dev/null +++ b/docs/lexical-first-rationale.md @@ -0,0 +1,114 @@ +# Lexical-first retrieval — why the cheap path is the default + +**What this is:** the rationale for arborist's retrieval philosophy — +lexical-first (FTS5 BM25 + content-addressed Merkle proofs) as the +default, with dense-vector semantic retrieval as an *optional, +additive* layer (#000039). Not a ticket; a positioning/architecture +reference. The headline: arborist's ingest is **10-100× cheaper per +document** than building a vector-DB representation, on the same +SQLite substrate — which is the difference between "ingest + search +runs on a phone" and "the NPU is now a hand-warmer." + +--- + +## 1. The cost asymmetry, measured + +| step | per-chunk cost (healthy CPU core) | +|---|---| +| **bge-small-en-v1.5 ONNX inference** (a vector-DB-style embed) | ~5-30 ms — chunk-length-dependent, 512-token cap; *worse* under load (~250 ms/chunk ≈ 4 chunks/s measured on a contended dev box at load 11/8 cores) | +| arborist's ingest pipeline: chunker (`tok-512-v1`) + SHA-256 leaf hash + Merkle commit + SQLite INSERTs (`documents` / `chunks` / `edges` / `audit_events`) + zstd content compression + FTS5 inverted-index update | **well under 1 ms total** | + +So adding embeddings multiplies ingest time by **~10-100×**, entirely +in the ONNX matmuls. The non-vec ingest path runs at hundreds of +chunks/s; a corpus-wide embed of arborist's 6.24 M-chunk wiki shard +set is a one-time hours-on-idle / days-on-contended *batch job* +(#000039 §14.6) — not something you'd do on a phone, ever. + +**And the query side is asymmetric too.** An FTS5 query: tokenize, +look up posting lists in the inverted-index B-tree per term, BM25-rank +— a handful of B-tree traversals, microseconds to low-milliseconds. A +vector query: tokenize, run the embedder (**one transformer forward +pass — the same ~5-30 ms as embedding a document chunk**), *then* the +ANN scan. On a phone, every search would pay a transformer forward +pass before it even touches the index. + +## 2. Same SQLite file, different philosophy + +Both approaches put a `.db` file on disk. The split is in *what's in +it* and *how retrieval works*: + +| | lexical-first (arborist) | embedding-first (a vector DB) | +|---|---|---| +| index | FTS5 inverted index (term → posting list) | dense vectors + ANN index (IVF / HNSW / flat) | +| build cost | ~free (sort tokens into B-trees) | a transformer forward pass per chunk | +| query cost | B-tree lookups + BM25 | embed the query (transformer) + ANN scan | +| matching | exact term / phrase / stem | semantic similarity | +| proof-bearing? | **yes** — chunks are content-addressed (`leaf_hash`), Merkle-rooted, the retrieved span is inclusion-provable against `C(M)` | **no** — embeddings are lossy, non-canonical; they can't enter a proof path (CLAUDE.md "soft hash vs hard hash") | + +That last row is the deep version of the point. arborist *is* the +Merkle Providence model — content-addressed chunks, Merkle inclusion +proofs, the audit chain — and that model is **cheap by construction**: +the proof-bearing layer (hard hashes, FTS5-retrievable chunks) costs +~nothing to build. Embeddings are a *soft* signal — they help recall, +they never enter a proof — and they're the expensive bolt-on. The +"soft hash vs hard hash" discipline (hard = SHA-256 commitments / +proofs / cache_key; soft = embeddings / TF-IDF / similarity, never in +the proof path) maps directly onto "cheap-and-mandatory vs +expensive-and-optional." + +## 3. The consequence: edge / mobile viability + +A phone (or a Raspberry Pi, or a permacomputer node) can: + +- **Ingest** documents at arborist's full rate — sorting tokens into + B-trees, hashing, writing SQLite rows is what mobile CPUs do well. +- **Query** the lexical index essentially for free — B-tree lookups, + no neural net in the path. +- **Use** a precomputed int8 `chunk_vecs` someone else built + off-device, if a semantic layer is wanted — querying it still costs + a query-embedding forward pass per search, but the *build* cost + (the days-long batch job) was paid elsewhere. + +A vector-DB-first design can't do the first two cheaply: ingest pegs +the CPU/NPU per chunk, and even browsing pays a forward pass per +query. On battery, that's the difference between a usable offline +knowledge store and a hand-warmer that's flat by lunch. + +## 4. The honest caveat + +Lexical-first **trades** the *semantic allusion gap* — the case where +a query and the target chunk share zero stems but mean the same thing +("what did Orwell call the country at war with Oceania?" → an article +that says "Eastasia" / "Eurasia" but never the query's words). FTS5's +four routes (body BM25, title-LIKE, core-keyword TF-IDF, phrase- +pattern) close the *lexical* allusion gap, not the semantic one. Dense +vectors close the semantic gap. + +That's exactly why arborist's answer isn't "we don't do vectors" — +it's: **vectors are an opt-in, additive 5th retrieval route, never the +default**, and the embed pass is lazy / out-of-band (a cron, or a +Prometheus-Σ unconscious-sweep task per #000037 §3.1) so the heavy +transformer work runs off-device / off-peak. The cheap, mobile-viable, +proof-bearing path is what ships by default; the semantic layer is +there when you want it and can afford it, and it doesn't compromise +the rest. (`int8` quantization keeps the storage tax at +6% over the +shards — vs +25% at float32 — and the embeddings still never enter the +proof path.) + +## 5. References + +- `docs/tickets/ticket-000039-sqlite-vec-optional-backend.md` — the + optional vec backend; §3 (storage), §4 (additive not replacement), + §14 (ingest integration + idempotency), §14.6 (the embed-throughput + measurement these numbers come from). +- `docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md` — Phase 2: + wiring vec as a 5th retrieval route (gated on a corpus backfill + a + recall bench). +- CLAUDE.md "Retrieval pipeline" — the four FTS5 routes; "soft hash vs + hard hash" — the discipline this rationale generalizes. +- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` + — the canonical Merkle Providence Reverse RAG paper (this doc is the + arborist-side positioning note; folding the mobile-viability argument + into the paper proper is a possible follow-up). +- #000037 §3.1 — the Prometheus-Σ unconscious-sweep task that's the + natural home for the lazy embed pass. diff --git a/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md b/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md index c833a92..ec2a3bc 100644 --- a/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md +++ b/docs/tickets/ticket-000039-sqlite-vec-optional-backend.md @@ -940,3 +940,11 @@ If the backfill latency matters, options are: run it off-peak; use a dedicated box; or wire GPU/accelerated embedding (out of scope for this ticket — `default_embedder()` is a pluggable `Embedder` callable, so a GPU/onnxruntime-gpu/external-endpoint variant is a drop-in). + +**Strategic framing:** this ~10-100× ingest-cost asymmetry — and the +matching query-side asymmetry (a vec query embeds the query string +first; an FTS5 query is B-tree lookups) — is *why* arborist is +lexical-first by default and dense-vector is an opt-in additive layer: +the cheap, proof-bearing, mobile-viable path is what ships; the +expensive semantic layer is there when you want it and can afford it. +See `docs/lexical-first-rationale.md`.