Capture the positioning fox articulated: arborist's per-document ingest
is ~10-100x cheaper than building a vector-DB representation — same
SQLite substrate, different retrieval philosophy — which is the
difference between "ingest + search runs on a phone" and "the NPU is
now a hand-warmer."
New docs/lexical-first-rationale.md (positioning/architecture
reference, not a ticket): the cost asymmetry with the measured numbers
(FTS5 + SHA-256 leaf + Merkle commit + sqlite + zstd pipeline << 1 ms/
chunk vs bge-small ONNX inference ~5-30 ms/chunk, worse contended; the
query side too — a vec query embeds the query string first, an FTS5
query is B-tree lookups); the same-SQLite-different-philosophy table
(inverted index vs dense vectors + ANN; build cost; query cost;
matching; proof-bearing); the deep version of the point — arborist IS
the Merkle Providence model and that model is cheap by construction,
embeddings are a soft signal (CLAUDE.md "soft hash vs hard hash") that
never enter a proof and are the expensive bolt-on; the edge/mobile
consequence; the honest caveat (lexical-first trades the semantic
allusion gap — which is why vec is opt-in/additive, never the default,
and the embed pass is lazy/out-of-band so the heavy transformer work
runs off-device/off-peak; int8 keeps the storage tax at +6%).
Wired in: TICKETS.md "Distinction from other docs" reference list gains
the doc; #000039 §14.6 gains a "Strategic framing" pointer to it.
(A possible follow-up: fold the mobile-viability argument into the
Merkle Providence Reverse RAG whitepaper proper — noted in the doc's
references; not done here, that's a deliberate cross-repo paper edit.)
Doc-only.
New §14.6 captures the 2026-05-11 findings:
- Measured embed rate: ~4.3 chunks/s on the contended dev box (load
~11 on 8 cores; the embed process got ~28% of one core). One wiki
shard (~1.56M chunks) at that rate ≈ 100h ≈ 4+ days; all four ≈
~16 days. The full int8 backfill was abandoned as not feasible to
brute-force there.
- The 54K-chunk partial on 000.db confirmed ~409 B/chunk apparent
→ 384 B amortized → ~2.4 GB for the full 6.24M-chunk corpus at
int8 — the deterministic number a full backfill would only
re-confirm, so finishing it bought nothing.
- On an idle healthy box (batching + all cores) bge-small does
~50-200 chunks/s → full corpus ≈ ~9-35h (the ticket's earlier
"~17h" is the optimistic end).
- Per-chunk cost breakdown: bge-small ONNX inference ~5-30 ms/chunk
dominates; the existing arborist ingest steps (chunker + SHA-256
leaf + Merkle commit + sqlite INSERTs + zstd + FTS5) are well
under 1 ms total. So adding vec multiplies ingest by ~10-100×,
entirely in the ONNX matmuls — the non-vec ingest path is
unchanged and still runs at hundreds of chunks/s.
- Implication for §14.2: this is *why* lazy-out-of-band is the
default and `arborist ingest --embed` is the opt-in. Production
guidance: a corpus-wide backfill is a one-time batch job (hours
on idle / days on contended), best run off-peak or on a dedicated
box; it does not slow ongoing ingest (which never embeds unless
--embed is passed); a GPU/accelerated embedder is a drop-in via
the pluggable Embedder callable if backfill latency matters.
Status line updated to point at §14.6. Doc-only.
Wire vector quantization (the §3.1 production knob): arborist embed
--quant int8 [--rebuild]. The chunk_vecs vec0 column becomes int8[384]
vs float[384] per quant; the quant folds into VEC_BACKEND_VERSION
(...-384int8-... / ...-384float32-...) and vec_meta records it per
shard. Switching quant on an existing chunk_vecs requires --rebuild
(the vec0 element type can't be altered in place — embed_documents
raises ValueError telling you to --rebuild).
int8 serialization: scale each bge component by 127 (theoretical
[-1,1] range), clamp to [-127,127], round, serialize_int8. The same
scaling on the query vector → distances comparable; cosine is
scale-invariant so the uniform x127 cancels in the ranking.
sqlite-vec v0.1.9 quirk worked around: a bare blob inserted into a
vec0 column is interpreted as float32 regardless of the column's
declared type — int8 vectors MUST be wrapped in vec_int8(...). So
the INSERT and the MATCH now wrap the blob in vec_f32(?) (float32)
or vec_int8(?) (int8) — constructor name from a fixed dict, no
injection surface. (Discovered the hard way: a bare int8 blob into
an int8[384] column → "expected int8, but a float32 vector was
provided".)
VecBackend reads the quant from the existing chunk_vecs schema (or
defaults to float32) so search uses the matching wrapper. New module
exports: QUANTS, EMBED_QUANT, vec_backend_version(quant), existing_quant.
int8 head-to-head on crawl_appliedcombinatorics_org.db (168 chunks):
- storage: float32 1,597,440 B -> int8 417,792 B = 3.8x smaller
(~4x at corpus scale where the 1024-vector blocks fill; the
~28 KB of vec0 metadata doesn't quarter, hence 3.8 not 4.0).
- recall vs the float32 baseline:
Q "how many ways to choose k things from n":
identical top-5 (Combinations, Permutations, Exercises,
Derangements, Graph Coloring).
Q "pigeonhole principle counting":
identical top-2 (Graph Coloring, Exercises); ranks 3-4 swap
Derangements <-> Permutations at Δdistance 0.002 — sub-noise.
- embed speed unchanged (~3.9 chunks/s — model-load-dominated).
Conclusion: int8 is the obvious production config (§3.1's +6%-tax
recommendation confirmed empirically). v1 default stays float32 for
max fidelity; flipping the default to int8 is a fox call.
CLI (arborist/cli.py): arborist embed --quant {float32,int8}; output
JSON gains "quant"; embed_documents ValueError → exit 2 with the
"--rebuild" hint.
tests/test_search_vec.py (9 -> 16): test_int8_quant_roundtrips
(int8[384] schema, vec_meta version, search round-trip, quant
inferred by VecBackend), test_quant_mismatch_requires_rebuild,
test_invalid_quant_rejected.
#000039 status updated. Full suite: 2343 passed, 28 skipped.
(Unrelated parallel-clone work in the tree — Makefile, arborist/qa/
verify.py, bench/fixtures/5f/*, tests/test_bench_batteries.py,
tests/test_verify.py — is #000046's hard-fixture tier, not touched.)
Fox: "i like both" — keep the lazy out-of-band pass as the default
AND add the eager opt-in. Plus the Phase-1 gap fix (incremental embed).
Key insight folded into the design (new ticket section 14): a chunk_id's
content is immutable in arborist — same content → same chunk_id;
different content → a NEW chunk_id (re-ingest makes a new doc_root +
new chunk_ids linked by supersedes; a chunker bump re-chunks → new
chunk_ids). So a chunk, once embedded, never needs re-embedding — the
ONLY re-embed trigger is the embedder changing (VEC_BACKEND_VERSION
bump). That makes the idempotency story clean.
arborist/search/vec.py — embed_documents() now:
- incremental=True (default): embed only chunk_ids NOT already in
chunk_vecs (chunk_id NOT IN (SELECT chunk_id FROM chunk_vecs)).
This is the after-ingest / cron / Prometheus-Sigma-sweep path —
it picks up exactly the newly-ingested chunks; re-running is a
cheap no-op once everything's embedded.
- incremental=False: re-embed every chunk with content (delete-then-
insert all) — the embedder-changed case.
- rebuild=True: DROP + recreate chunk_vecs first, then a full pass —
the clean VEC_BACKEND_VERSION-bump path (a search mid-rebuild never
mixes old- and new-model embeddings: the recreated table starts
empty and grows new-model as the pass runs). Implies non-incremental.
- Cold-evicted chunks (content NULL) still skipped; vec rows persist
and stay valid (content is identical on rehydrate).
CLI (arborist/cli.py):
- arborist embed --rebuild — the DROP+recreate+full-re-embed path
(default is incremental). Output JSON now reports "mode".
- arborist ingest --embed — eager opt-in: after the chunk+Merkle-
commit pass, incremental-embed this run's new chunks. Default
ingest does NOT embed. ingest output gains "chunks_embedded" when
--embed is set. Only surfaced when the [vec] extra is installed.
- Hoisted the _vec_ok check up to the top of build_parser so both
the ingest --embed flag and the search --backend / embed subcommand
can gate on it.
tests/test_search_vec.py (7 -> 9): test_embed_incremental_only_embeds
_new_chunks (second pass after a follow-up ingest embeds only the new
chunk; third pass is a no-op), test_embed_rebuild_re_embeds_all
(DROP+recreate+full pass; vec_meta still records the version).
Verified on crawl_appliedcombinatorics_org.db: incremental on an
already-embedded shard reports chunks_embedded=0 in ~1.8s; --rebuild
re-embeds all 168 in ~61s; semantic search after rebuild still returns
topically-correct hits ("pigeonhole principle counting" -> "AC Graph
Coloring" chunk containing "Generalized Pigeon Hole Principle").
Ticket section 14 added: the idempotency table (re-ingest / chunker
bump / cold eviction / embedder bump / superseded docs), the two
integration models (lazy default + eager opt-in; the lazy pass's
natural home is a Prometheus-Sigma unconscious-sweep task per #000037
section 3.1), the command matrix, concurrency notes, versioning.
Status line updated.
Full suite: 2339 passed, 28 skipped.
(Unrelated parallel-clone work in the working tree — Makefile,
arborist/qa/verify.py, bench/fixtures/5f/*, tests/test_bench_batteries.py,
tests/test_verify.py — is #000046's hard-fixture tier, not touched here.)
Implements the optional vec backend from the #000039 doc, with the
"obvious" v1 tuning, and demonstrates it on a real corpus shard.
arborist/search/vec.py (new):
- VecBackend(SearchBackend) — ANN over chunk_vecs, UNGROUNDED hits
(same as FTS5; vec changes recall, never warrant — embeddings are
soft signal, never in the proof path).
- chunk_vecs vec0 virtual table + vec_meta — sibling tables, additive,
don't touch chunks/documents/the audit chain.
- embed_documents() — batched ingest; delete-then-insert per chunk_id
(vec0 doesn't honor INSERT-OR-REPLACE — re-inserting an existing PK
is a hard UNIQUE error), so re-runs are idempotent and content-
changed → re-embed works. Skips cold-evicted chunks (content NULL).
- Pluggable Embedder callable; default = fastembed bge-small-en-v1.5
(~130 MB ONNX, downloads on first use). load_vec_extension(conn)
toggles enable_load_extension + sqlite_vec.load.
- v1 hyperparams (VEC_BACKEND_VERSION = vec-v1-bge-small-en-v1.5-
384float32-cosine-flat): model bge-small-en-v1.5, dim 384, quant
float32 (int8/binary = the production storage knob per §3.1, not
wired in v1), metric cosine (bge outputs L2-normalized, so cosine
ranking ≡ L2 ranking), ANN flat (vec0 default), top_k 20. These
five fold into governance_policy_hash in a later phase (§6).
CLI (arborist/cli.py):
- — populate chunk_vecs
for --db; prints progress + timing.
- — semantic ANN search (errors with
an install/embed hint if [vec] missing or chunk_vecs empty).
- Both surfaced only when sqlite_vec imports (mirrors the [html] /
selectolax pattern).
pyproject.toml: [vec] optional extra (sqlite-vec>=0.1.9, fastembed>=0.4);
added to [dev]. Note: sentence-transformers is the heavier "official"
embedder path §5 names; fastembed is the lightweight ONNX one.
tests/test_search_vec.py (7 tests, skip-if-no-[vec]): deterministic
stub embedder (hash → unit vector) so the suite exercises the
sqlite-vec plumbing — ext load, schema, ingest, KNN, JOIN, Hit shape,
limit, idempotent re-embed, --limit cap, empty/unpopulated — without
the heavy fastembed model. Semantic quality is demonstrated on a
shard, not unit-tested.
Demonstrated on ~/.arborist/shards/crawl_appliedcombinatorics_org.db:
168 chunks embedded in ~37 s (mostly model load); semantic queries
return topically-correct hits — "how many ways to choose k things
from n" → top hit "AC Combinations", "binomial coefficient counting"
→ "AC Introduction" (integer-solution counting) + "AC Combinatorial
Proofs". None of the query tokens need stem-match the chunk — the
semantic-allusion-gap closure the ticket promised. chain-check on
that shard reports 0 after embedding (chunk_vecs is a sibling table).
#000039 status flipped to "in progress · Phase 1 landed"; Phase 2
(RRF hybrid fusion in query.py) gated on a ≥5pp recall-lift
measurement with no STRICT-rate regression (§8).
(Unrelated: tests/test_weights.py::test_as_dict_returns_all_eleven_fields
fails in the working tree — that's a parallel-clone in-flight change
to arborist/substrate/weights.py + its test, not touched here.)
Phase 1 plan §13.5 named four test files (3 unit + 1 integration)
but did NOT name CLI subprocess tests for the three new CLI
surfaces §13.4 introduces:
- arborist ingest --embed (flag on existing subcommand)
- arborist query --retrieval={fts5|vec|hybrid} (new flag)
- arborist vec rebuild (new subcommand)
Per docs/calculator-test-patterns.md §6 (codified earlier today
in commit 0725eb4 from the three-module pattern bench): import-only
tests miss argparse + main() drift. Yesterday's substrate refactor
caught this hazard three times — fork_score.py import (85be5eb),
Makefile bench-fork-score target (209d670), .gitlab-ci.yml job
+ script (b320e27). Each fix was 1-3 lines, but each had been
shipped to main + would have surfaced as a noisy CI failure on
next pipeline run.
§13.5 now adds three CLI subprocess test files:
- tests/test_cli_vec_rebuild.py
- tests/test_cli_ingest_embed_flag.py
- tests/test_cli_query_retrieval_flag.py
Each gated via pytest.importorskip("sqlite_vec") so they skip
cleanly when [vec] extras absent. Pattern matches fox's
test_cli_baseline_runs_clean / test_cli_invalid_input_exits_2
in tests/test_t3_bound_calculator.py (the exemplar for
calculator-style CLI tests).
§13.7 size estimate revised: 4 test files → 7 test files (+3 CLI
subprocess), 250 → 400 test LOC. CLI subprocess tests are
~30-40 LOC each (boilerplate + tmp_path + subprocess.run +
JSON parse). Phase 1 total grows from ~550+250 → ~550+400 LOC.
Doc-only edit; doesn't unblock or block fox's §13.8 four
decisions — the test-plan addition is mechanical discipline,
not a scope change. Phase 1 still gates on the four §13.8
decisions before any code lands.
Cross-ref: docs/calculator-test-patterns.md §6 (CLI subprocess
pattern) + the three substrate-rename defect commits caught by
that pattern in retrospect (85be5eb / 209d670 / b320e27).
Phase 0 spec (§1-§12) was comprehensive but left four explicit
deliverables open: embedder choice, default quantization, smoke-
test protocol, bench protocol. §13 fills those four with concrete
recommendations + a code structure / test plan / size estimate
that fox can sign off on before Phase 1 code lands.
Recommendations:
- Embedder path 1 (local sentence-transformer bundled as
optional dep); model BAAI/bge-small-en-v1.5 (MIT, 33 MB,
384-dim, unit-normalized, top of MTEB-en/retrieval among
sub-100MB models)
- Default quantization int8 × 384 + flat (6% storage tax,
within the 15% budget per §3.1; binary × 768 reachable via
--vec-quantization=binary)
- Pre-Phase-1 smoke (§13.2): 1k chunks under WAL +
synchronous=NORMAL, kill -9 mid-insert, recovery check;
gate on insert ≥100 chunk/s, p95 query ≤50 ms, zero data
loss
- Bench protocol (§13.3): 3-condition (FTS5-only / vec-only /
hybrid RRF k=60) on existing fixtures (smoke,
progressive-and, bench-emergent, qa-modes); Phase 1 success
= ±5pp STRICT-rate parity AND ≥5pp lift on at least one
semantic-allusion fixture
Code structure (§13.4): 2 new files (embed.py + search/vec.py)
~200 LOC, 4 patches (store.py + query.py + cli.py + Makefile)
~100 LOC, 4 test files ~250 LOC, pyproject.toml [vec] extras
stanza. Single substantial commit when all gates pass.
§13.8 lists the four go/no-go decisions fox needs to make to
unblock Phase 1: embedder path, model name, default quantization,
and approval of the sentence-transformers PyPI dep under [vec]
extras (not pulled by default; only on pip install '.[vec]').
Fallback paths documented for each rejection.
Phase 0 doc remains awaiting go/no-go; §13 doesn't change that
gate, just provides the substance for fox's decision.
Parallel-shift session drafted #000039 earlier today and updated
TICKETS.md with the index row, but the ticket file itself sat
untracked in working tree (same situation #000037 had until that
ticket landed in commit 178cc42).
Committing the file as-drafted by the original author so the
design log entry is intact. No content changes from this
session; the file is exactly what the parallel shift produced.
Per the design log convention in TICKETS.md ("Do not delete
tickets; they are the design log") — every opened ticket file
ships with its index row.
