Fox: "i like both" — keep the lazy out-of-band pass as the default
AND add the eager opt-in. Plus the Phase-1 gap fix (incremental embed).
Key insight folded into the design (new ticket section 14): a chunk_id's
content is immutable in arborist — same content → same chunk_id;
different content → a NEW chunk_id (re-ingest makes a new doc_root +
new chunk_ids linked by supersedes; a chunker bump re-chunks → new
chunk_ids). So a chunk, once embedded, never needs re-embedding — the
ONLY re-embed trigger is the embedder changing (VEC_BACKEND_VERSION
bump). That makes the idempotency story clean.
arborist/search/vec.py — embed_documents() now:
- incremental=True (default): embed only chunk_ids NOT already in
chunk_vecs (chunk_id NOT IN (SELECT chunk_id FROM chunk_vecs)).
This is the after-ingest / cron / Prometheus-Sigma-sweep path —
it picks up exactly the newly-ingested chunks; re-running is a
cheap no-op once everything's embedded.
- incremental=False: re-embed every chunk with content (delete-then-
insert all) — the embedder-changed case.
- rebuild=True: DROP + recreate chunk_vecs first, then a full pass —
the clean VEC_BACKEND_VERSION-bump path (a search mid-rebuild never
mixes old- and new-model embeddings: the recreated table starts
empty and grows new-model as the pass runs). Implies non-incremental.
- Cold-evicted chunks (content NULL) still skipped; vec rows persist
and stay valid (content is identical on rehydrate).
CLI (arborist/cli.py):
- arborist embed --rebuild — the DROP+recreate+full-re-embed path
(default is incremental). Output JSON now reports "mode".
- arborist ingest --embed — eager opt-in: after the chunk+Merkle-
commit pass, incremental-embed this run's new chunks. Default
ingest does NOT embed. ingest output gains "chunks_embedded" when
--embed is set. Only surfaced when the [vec] extra is installed.
- Hoisted the _vec_ok check up to the top of build_parser so both
the ingest --embed flag and the search --backend / embed subcommand
can gate on it.
tests/test_search_vec.py (7 -> 9): test_embed_incremental_only_embeds
_new_chunks (second pass after a follow-up ingest embeds only the new
chunk; third pass is a no-op), test_embed_rebuild_re_embeds_all
(DROP+recreate+full pass; vec_meta still records the version).
Verified on crawl_appliedcombinatorics_org.db: incremental on an
already-embedded shard reports chunks_embedded=0 in ~1.8s; --rebuild
re-embeds all 168 in ~61s; semantic search after rebuild still returns
topically-correct hits ("pigeonhole principle counting" -> "AC Graph
Coloring" chunk containing "Generalized Pigeon Hole Principle").
Ticket section 14 added: the idempotency table (re-ingest / chunker
bump / cold eviction / embedder bump / superseded docs), the two
integration models (lazy default + eager opt-in; the lazy pass's
natural home is a Prometheus-Sigma unconscious-sweep task per #000037
section 3.1), the command matrix, concurrency notes, versioning.
Status line updated.
Full suite: 2339 passed, 28 skipped.
(Unrelated parallel-clone work in the working tree — Makefile,
arborist/qa/verify.py, bench/fixtures/5f/*, tests/test_bench_batteries.py,
tests/test_verify.py — is #000046's hard-fixture tier, not touched here.)
russell@unturf.com
86d47e6aee