ticket #000039: ingest integration + incremental embed (both models)

Fox: "i like both" — keep the lazy out-of-band pass as the default
AND add the eager opt-in. Plus the Phase-1 gap fix (incremental embed).

Key insight folded into the design (new ticket section 14): a chunk_id's
content is immutable in arborist — same content → same chunk_id;
different content → a NEW chunk_id (re-ingest makes a new doc_root +
new chunk_ids linked by supersedes; a chunker bump re-chunks → new
chunk_ids). So a chunk, once embedded, never needs re-embedding — the
ONLY re-embed trigger is the embedder changing (VEC_BACKEND_VERSION
bump). That makes the idempotency story clean.

arborist/search/vec.py — embed_documents() now:
- incremental=True (default): embed only chunk_ids NOT already in
  chunk_vecs (chunk_id NOT IN (SELECT chunk_id FROM chunk_vecs)).
  This is the after-ingest / cron / Prometheus-Sigma-sweep path —
  it picks up exactly the newly-ingested chunks; re-running is a
  cheap no-op once everything's embedded.
- incremental=False: re-embed every chunk with content (delete-then-
  insert all) — the embedder-changed case.
- rebuild=True: DROP + recreate chunk_vecs first, then a full pass —
  the clean VEC_BACKEND_VERSION-bump path (a search mid-rebuild never
  mixes old- and new-model embeddings: the recreated table starts
  empty and grows new-model as the pass runs). Implies non-incremental.
- Cold-evicted chunks (content NULL) still skipped; vec rows persist
  and stay valid (content is identical on rehydrate).

CLI (arborist/cli.py):
- arborist embed --rebuild — the DROP+recreate+full-re-embed path
  (default is incremental). Output JSON now reports "mode".
- arborist ingest --embed — eager opt-in: after the chunk+Merkle-
  commit pass, incremental-embed this run's new chunks. Default
  ingest does NOT embed. ingest output gains "chunks_embedded" when
  --embed is set. Only surfaced when the [vec] extra is installed.
- Hoisted the _vec_ok check up to the top of build_parser so both
  the ingest --embed flag and the search --backend / embed subcommand
  can gate on it.

tests/test_search_vec.py (7 -> 9): test_embed_incremental_only_embeds
  _new_chunks (second pass after a follow-up ingest embeds only the new
  chunk; third pass is a no-op), test_embed_rebuild_re_embeds_all
  (DROP+recreate+full pass; vec_meta still records the version).

Verified on crawl_appliedcombinatorics_org.db: incremental on an
already-embedded shard reports chunks_embedded=0 in ~1.8s; --rebuild
re-embeds all 168 in ~61s; semantic search after rebuild still returns
topically-correct hits ("pigeonhole principle counting" -> "AC Graph
Coloring" chunk containing "Generalized Pigeon Hole Principle").

Ticket section 14 added: the idempotency table (re-ingest / chunker
bump / cold eviction / embedder bump / superseded docs), the two
integration models (lazy default + eager opt-in; the lazy pass's
natural home is a Prometheus-Sigma unconscious-sweep task per #000037
section 3.1), the command matrix, concurrency notes, versioning.
Status line updated.

Full suite: 2339 passed, 28 skipped.

(Unrelated parallel-clone work in the working tree — Makefile,
arborist/qa/verify.py, bench/fixtures/5f/*, tests/test_bench_batteries.py,
tests/test_verify.py — is #000046's hard-fixture tier, not touched here.)

tests/test_search_vec.py

@@ -154,6 +154,52 @@ def test_embed_limit_caps_count(db):
 def test_embed_is_idempotent(db):
     conn, _ = db
     embed_documents(conn, embedder=_stub_embedder)
-    embed_documents(conn, embedder=_stub_embedder)  # re-run; INSERT OR REPLACE
+    embed_documents(conn, embedder=_stub_embedder)  # re-run
     cnt = conn.execute("SELECT COUNT(*) FROM chunk_vecs").fetchone()[0]
     assert cnt == 3
+
+
+def test_embed_incremental_only_embeds_new_chunks(db):
+    """Default (incremental=True): a second embed pass after ingesting a
+    new doc embeds only the new chunk — re-running is cheap (#000039)."""
+    conn, _ = db
+    first = embed_documents(conn, embedder=_stub_embedder)
+    assert first == 3
+    # Simulate a follow-up ingest: one new document + chunk.
+    new_root = "root_d" * 8
+    conn.execute(
+        "INSERT INTO documents("
+        "  document_root, document_uri, source_type, kind, title,"
+        "  chunking_version, canonicalization_version, schema_version, ingest_ts"
+        ") VALUES (?, 'uri://d', 'test', 'surface', 'Delta doc',"
+        "  'tok-512-v1', 'norm-v1', 'v9.8.0', 0)",
+        (new_root,),
+    )
+    conn.execute(
+        "INSERT INTO chunks(document_root, idx, leaf_hash, content, tier) "
+        "VALUES (?, 0, ?, 'delta content about derangements', 'hot')",
+        (new_root, hashlib.sha256(b"delta").hexdigest()),
+    )
+    second = embed_documents(conn, embedder=_stub_embedder)  # incremental
+    assert second == 1, "only the new chunk should embed on the second pass"
+    assert conn.execute("SELECT COUNT(*) FROM chunk_vecs").fetchone()[0] == 4
+    # A third pass with nothing new is a no-op.
+    third = embed_documents(conn, embedder=_stub_embedder)
+    assert third == 0
+
+
+def test_embed_rebuild_re_embeds_all(db):
+    """rebuild=True: DROP + recreate + full pass — every chunk re-embedded
+    (the clean VEC_BACKEND_VERSION-bump path)."""
+    conn, _ = db
+    embed_documents(conn, embedder=_stub_embedder)
+    assert conn.execute("SELECT COUNT(*) FROM chunk_vecs").fetchone()[0] == 3
+    n = embed_documents(conn, embedder=_stub_embedder, rebuild=True)
+    assert n == 3, "rebuild re-embeds all chunks"
+    assert conn.execute("SELECT COUNT(*) FROM chunk_vecs").fetchone()[0] == 3
+    # vec_meta still records the backend version after the recreate.
+    from arborist.search.vec import VEC_BACKEND_VERSION
+    row = conn.execute(
+        "SELECT value FROM vec_meta WHERE key='backend_version'"
+    ).fetchone()
+    assert row[0] == VEC_BACKEND_VERSION