Wire vector quantization (the §3.1 production knob): arborist embed
--quant int8 [--rebuild]. The chunk_vecs vec0 column becomes int8[384]
vs float[384] per quant; the quant folds into VEC_BACKEND_VERSION
(...-384int8-... / ...-384float32-...) and vec_meta records it per
shard. Switching quant on an existing chunk_vecs requires --rebuild
(the vec0 element type can't be altered in place — embed_documents
raises ValueError telling you to --rebuild).
int8 serialization: scale each bge component by 127 (theoretical
[-1,1] range), clamp to [-127,127], round, serialize_int8. The same
scaling on the query vector → distances comparable; cosine is
scale-invariant so the uniform x127 cancels in the ranking.
sqlite-vec v0.1.9 quirk worked around: a bare blob inserted into a
vec0 column is interpreted as float32 regardless of the column's
declared type — int8 vectors MUST be wrapped in vec_int8(...). So
the INSERT and the MATCH now wrap the blob in vec_f32(?) (float32)
or vec_int8(?) (int8) — constructor name from a fixed dict, no
injection surface. (Discovered the hard way: a bare int8 blob into
an int8[384] column → "expected int8, but a float32 vector was
provided".)
VecBackend reads the quant from the existing chunk_vecs schema (or
defaults to float32) so search uses the matching wrapper. New module
exports: QUANTS, EMBED_QUANT, vec_backend_version(quant), existing_quant.
int8 head-to-head on crawl_appliedcombinatorics_org.db (168 chunks):
- storage: float32 1,597,440 B -> int8 417,792 B = 3.8x smaller
(~4x at corpus scale where the 1024-vector blocks fill; the
~28 KB of vec0 metadata doesn't quarter, hence 3.8 not 4.0).
- recall vs the float32 baseline:
Q "how many ways to choose k things from n":
identical top-5 (Combinations, Permutations, Exercises,
Derangements, Graph Coloring).
Q "pigeonhole principle counting":
identical top-2 (Graph Coloring, Exercises); ranks 3-4 swap
Derangements <-> Permutations at Δdistance 0.002 — sub-noise.
- embed speed unchanged (~3.9 chunks/s — model-load-dominated).
Conclusion: int8 is the obvious production config (§3.1's +6%-tax
recommendation confirmed empirically). v1 default stays float32 for
max fidelity; flipping the default to int8 is a fox call.
CLI (arborist/cli.py): arborist embed --quant {float32,int8}; output
JSON gains "quant"; embed_documents ValueError → exit 2 with the
"--rebuild" hint.
tests/test_search_vec.py (9 -> 16): test_int8_quant_roundtrips
(int8[384] schema, vec_meta version, search round-trip, quant
inferred by VecBackend), test_quant_mismatch_requires_rebuild,
test_invalid_quant_rejected.
#000039 status updated. Full suite: 2343 passed, 28 skipped.
(Unrelated parallel-clone work in the tree — Makefile, arborist/qa/
verify.py, bench/fixtures/5f/*, tests/test_bench_batteries.py,
tests/test_verify.py — is #000046's hard-fixture tier, not touched.)
Fox: "i like both" — keep the lazy out-of-band pass as the default
AND add the eager opt-in. Plus the Phase-1 gap fix (incremental embed).
Key insight folded into the design (new ticket section 14): a chunk_id's
content is immutable in arborist — same content → same chunk_id;
different content → a NEW chunk_id (re-ingest makes a new doc_root +
new chunk_ids linked by supersedes; a chunker bump re-chunks → new
chunk_ids). So a chunk, once embedded, never needs re-embedding — the
ONLY re-embed trigger is the embedder changing (VEC_BACKEND_VERSION
bump). That makes the idempotency story clean.
arborist/search/vec.py — embed_documents() now:
- incremental=True (default): embed only chunk_ids NOT already in
chunk_vecs (chunk_id NOT IN (SELECT chunk_id FROM chunk_vecs)).
This is the after-ingest / cron / Prometheus-Sigma-sweep path —
it picks up exactly the newly-ingested chunks; re-running is a
cheap no-op once everything's embedded.
- incremental=False: re-embed every chunk with content (delete-then-
insert all) — the embedder-changed case.
- rebuild=True: DROP + recreate chunk_vecs first, then a full pass —
the clean VEC_BACKEND_VERSION-bump path (a search mid-rebuild never
mixes old- and new-model embeddings: the recreated table starts
empty and grows new-model as the pass runs). Implies non-incremental.
- Cold-evicted chunks (content NULL) still skipped; vec rows persist
and stay valid (content is identical on rehydrate).
CLI (arborist/cli.py):
- arborist embed --rebuild — the DROP+recreate+full-re-embed path
(default is incremental). Output JSON now reports "mode".
- arborist ingest --embed — eager opt-in: after the chunk+Merkle-
commit pass, incremental-embed this run's new chunks. Default
ingest does NOT embed. ingest output gains "chunks_embedded" when
--embed is set. Only surfaced when the [vec] extra is installed.
- Hoisted the _vec_ok check up to the top of build_parser so both
the ingest --embed flag and the search --backend / embed subcommand
can gate on it.
tests/test_search_vec.py (7 -> 9): test_embed_incremental_only_embeds
_new_chunks (second pass after a follow-up ingest embeds only the new
chunk; third pass is a no-op), test_embed_rebuild_re_embeds_all
(DROP+recreate+full pass; vec_meta still records the version).
Verified on crawl_appliedcombinatorics_org.db: incremental on an
already-embedded shard reports chunks_embedded=0 in ~1.8s; --rebuild
re-embeds all 168 in ~61s; semantic search after rebuild still returns
topically-correct hits ("pigeonhole principle counting" -> "AC Graph
Coloring" chunk containing "Generalized Pigeon Hole Principle").
Ticket section 14 added: the idempotency table (re-ingest / chunker
bump / cold eviction / embedder bump / superseded docs), the two
integration models (lazy default + eager opt-in; the lazy pass's
natural home is a Prometheus-Sigma unconscious-sweep task per #000037
section 3.1), the command matrix, concurrency notes, versioning.
Status line updated.
Full suite: 2339 passed, 28 skipped.
(Unrelated parallel-clone work in the working tree — Makefile,
arborist/qa/verify.py, bench/fixtures/5f/*, tests/test_bench_batteries.py,
tests/test_verify.py — is #000046's hard-fixture tier, not touched here.)
Implements the optional vec backend from the #000039 doc, with the
"obvious" v1 tuning, and demonstrates it on a real corpus shard.
arborist/search/vec.py (new):
- VecBackend(SearchBackend) — ANN over chunk_vecs, UNGROUNDED hits
(same as FTS5; vec changes recall, never warrant — embeddings are
soft signal, never in the proof path).
- chunk_vecs vec0 virtual table + vec_meta — sibling tables, additive,
don't touch chunks/documents/the audit chain.
- embed_documents() — batched ingest; delete-then-insert per chunk_id
(vec0 doesn't honor INSERT-OR-REPLACE — re-inserting an existing PK
is a hard UNIQUE error), so re-runs are idempotent and content-
changed → re-embed works. Skips cold-evicted chunks (content NULL).
- Pluggable Embedder callable; default = fastembed bge-small-en-v1.5
(~130 MB ONNX, downloads on first use). load_vec_extension(conn)
toggles enable_load_extension + sqlite_vec.load.
- v1 hyperparams (VEC_BACKEND_VERSION = vec-v1-bge-small-en-v1.5-
384float32-cosine-flat): model bge-small-en-v1.5, dim 384, quant
float32 (int8/binary = the production storage knob per §3.1, not
wired in v1), metric cosine (bge outputs L2-normalized, so cosine
ranking ≡ L2 ranking), ANN flat (vec0 default), top_k 20. These
five fold into governance_policy_hash in a later phase (§6).
CLI (arborist/cli.py):
- — populate chunk_vecs
for --db; prints progress + timing.
- — semantic ANN search (errors with
an install/embed hint if [vec] missing or chunk_vecs empty).
- Both surfaced only when sqlite_vec imports (mirrors the [html] /
selectolax pattern).
pyproject.toml: [vec] optional extra (sqlite-vec>=0.1.9, fastembed>=0.4);
added to [dev]. Note: sentence-transformers is the heavier "official"
embedder path §5 names; fastembed is the lightweight ONNX one.
tests/test_search_vec.py (7 tests, skip-if-no-[vec]): deterministic
stub embedder (hash → unit vector) so the suite exercises the
sqlite-vec plumbing — ext load, schema, ingest, KNN, JOIN, Hit shape,
limit, idempotent re-embed, --limit cap, empty/unpopulated — without
the heavy fastembed model. Semantic quality is demonstrated on a
shard, not unit-tested.
Demonstrated on ~/.arborist/shards/crawl_appliedcombinatorics_org.db:
168 chunks embedded in ~37 s (mostly model load); semantic queries
return topically-correct hits — "how many ways to choose k things
from n" → top hit "AC Combinations", "binomial coefficient counting"
→ "AC Introduction" (integer-solution counting) + "AC Combinatorial
Proofs". None of the query tokens need stem-match the chunk — the
semantic-allusion-gap closure the ticket promised. chain-check on
that shard reports 0 after embedding (chunk_vecs is a sibling table).
#000039 status flipped to "in progress · Phase 1 landed"; Phase 2
(RRF hybrid fusion in query.py) gated on a ≥5pp recall-lift
measurement with no STRICT-rate regression (§8).
(Unrelated: tests/test_weights.py::test_as_dict_returns_all_eleven_fields
fails in the working tree — that's a parallel-clone in-flight change
to arborist/substrate/weights.py + its test, not touched here.)
