arborist/docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md

Ticket #000050 — Vec RRF hybrid fusion (the #000039 Phase 2)

Status: open · awaiting go/no-go — gated on (a) a corpus backfill and (b) a bench measurement (see "Gate" below). Doc-only scaffold; the design already lives in docs/tickets/ticket-000039-sqlite-vec-optional-backend.md §4.2 (RRF) + §8 (the measurement gate). #000039 closed 2026-05-12 with Phase 0 (doc) + Phase 1 (arborist/search/vec.py — VecBackend, chunk_vecs, embed_documents, arborist embed / search --backend vec, [vec] extra, ingest integration, --quant {float32,int8}) landed; this ticket carries the remaining Phase 2. Opened: 2026-05-12 Scope: Wire VecBackend as a fifth retrieval route in arborist/qa/query.py, merged with the four existing FTS5 routes (body BM25, title-LIKE, core-keyword TF-IDF, phrase-pattern) via Reciprocal Rank Fusion (RRF — Σ_routes 1/(k+rank), k=60), so the assembled context for a Q&A run draws from both lexical and semantic candidates. Vec hits stay UNGROUNDED (soft signal, never proof path); the post-merge filter chain (rivalry exclusion, four-accept-paths title-relevance, stem-aware token matching, per-source context cap, Rule-8 title-mismatch) runs on vec hits exactly as on FTS5 hits. Not a replacement for any FTS5 route — vec is additive (#000039 §4). Audience: fox + future blackops shifts + whoever runs the recall bench. Hard constraint: vec retrieval never enters the proof path (hits land audit_mode=UNGROUNDED). The 8-dim cache_key invariant is untouched — the vec hyperparams fold into governance_policy_hash (#000039 §6), which Phase 2 must wire (Phase 1 left it noted-only). RRF (rank-based) is the merge, not raw-score addition — BM25 scores and cosine distances aren't comparable scales. No change to the four FTS5 routes' behavior; hybrid is opt-in / governance-gated until the bench clears it.

---

1. Why this is a separate ticket

#000039's phased plan: Phase 0 (doc) → Phase 1 (the backend + ingest path) → Phase 2 (hybrid fusion), which opens only when Phase 1 measures ≥ 5pp recall lift on the bench fixtures with no STRICT-rate regression (#000039 §8). Phase 1 shipped 2026-05-11 and #000039 closed 2026-05-12; Phase 2 is real, scoped, and remaining, so it gets its own ticket per the repo convention rather than leaving #000039 open indefinitely.

2. Prerequisites (the gate)

Phase 2 implementation does not open until both:

1. A corpus backfill exists — anywhere, not necessarily where the bench runs. arborist embed --quant int8 over the shard set. Per #000039 §14.6 this is a one-time batch job — hours on an idle box, days on a contended one (~4 chunks/s measured on the dev box; ~50-200/s idle), ~2.4 GB total at int8 (+6 % over the 38 GB shards). The embed pass is ~10–100× heavier than the rest of ingest, so it does not run on a laptop — it runs once on any box with CPU (a cloud node, a Prometheus-Σ unconscious-sweep box — #000037 §3.1) and the resulting chunk_vecs is distributed as a vecpack over mesh so every peer pulls + bulk-loads it (sub-ms per chunk on the receiver). That distribution mechanism is #000051 — this prereq is "a vecpack exists and has been imported on the bench box", not "fox embedded the corpus locally".
2. A recall bench clears the gate. Run four conditions on the bench fixtures (the §2a semantic-allusion fixture set + the bench-emergent shapes + the curated set + the adversarial semantic-neighbor pack from §3 below): (A) FTS5-only baseline, (B) vec-only, (C) FTS5
   • vec RRF hybrid, (D) FTS5 + vec candidate-union but the existing reranker only (no RRF). D is the control that distinguishes "RRF helped" from "more candidates helped" — if C ≈ D, RRF is not earning its complexity. Gate (5pp signal floor per docs/bench-maxing.md):
   • vec-only (B) beats FTS5-only (A) by ≥ 5pp recall@K on ≥ 1 semantic-allusion fixture and surfaces ≥ 1 correct candidate absent from A's top-K. (B is a diagnostic, not required to match A's overall STRICT-rate — vec is allowed to lose on exact title / rare-token shapes; punishing it for not being FTS5 is the wrong test.)
   • hybrid (C) lifts the proof-quality-label rate ≥ 5pp over A on the target fixtures (STRICT for quote/span/entity/paraphrase; EVIDENCE-WARRANTED for lattice modes — "the current proof-quality label family", not literally "STRICT") with no statistically meaningful regression on the lexical/title/false-neighbor fixtures and no UNGROUNDED-rate regression.
   • C must beat D (or be no worse) — else ship the cheaper union, not RRF. If hybrid lift is sub-floor: park here — vec stays opt-in per-call (--backend vec), hybrid does NOT become the default, and the ~2.4 GB tax isn't paid for nothing.

2a. Semantic-allusion fixture set (the cases the §2 bench must include)

The §2 gate's "semantic-allusion fixtures" aren't a vague category — this is the running list of concrete cases where the query's words share zero tokens with the target article's title, so the four FTS5 routes (body BM25, title-LIKE, title-token, phrase-pattern) structurally can't surface it, but a dense-vector route should. The vec-only bench (condition B) must beat FTS5-only (A) by ≥ 5pp recall@K on ≥ 1 of these and surface ≥ 1 correct candidate absent from A's top-K; the hybrid bench (C) must lift the proof-quality-label rate on the set. When a new field case of this shape shows up, add a row here.

query	target article (title shares 0 tokens with query)	why FTS5 misses it
"what did Orwell call the country at war with Oceania?"	Nineteen Eighty-Four (→ Eastasia / Eurasia)	query tokens {orwell, country, war, oceania}; the article's distinctive content ("Eastasia", "always been at war with") isn't in the title, and the title shares no token with the query — only the verbatim phrase route ("always been at war") rescues a piece of it today (CLAUDE.md retrieval-pipeline §1)
"what is a CPU?"	Central processing unit	query token "CPU" ≠ the title "Central processing unit" — FTS5 pulls the "CPU *" satellites (CPU design, CPU socket, CPU time, CPU cache, CPU-Z, CPU multiplier, CPU disambiguation) where "CPU" is in the title; the canonical article never surfaces. (Field case 2026-05-13, fox. Note: #000053 fixed the verifier's separate acronym blind spot — EVIDENCE-WARRANTED instead of a spurious TITLE_MISMATCH — but the answer is still grounded in "CPU design", not the real article; that's this gap.)
"what is a GPU?"	Graphics processing unit	identical shape to CPU — FTS5 returns GPU cluster / GPU (disambiguation) / Molecular modeling on GPU / Evergreen (GPU family) / etc.; observed 0/1 UNGROUNDED because the model recited the (modern) GPU-lead definition and no 2003-dump GPU-* satellite carried that text. (Field case 2026-05-13, fox.)

(The abbreviation→expansion subclass — CPU↔Central processing unit, GPU↔Graphics processing unit — is also addressable upstream by a concepts/ synonym edge, cheaper than vec; both fixes are complementary and the bench should record which one (or both) closes each row. The Orwell row is not an abbreviation case — it's genuine conceptual allusion, the kind only dense vectors reach — so it's the one that actually justifies the vec layer if the abbreviation cases turn out to be all concepts/ could-have-fixed.)

3. Implementation sketch (when the gate clears)

RRF merge. arborist/qa/query.py: after the four FTS5 routes produce ranked hit lists and before the rerank/filter chain, add a fifth list from VecBackend.search(query, limit=K). Merge all five by RRF: score(doc) = Σ_route 1/(60 + rank_route(doc)), before the accept-path filtering. Carry route provenance on each merged hit (routes: [...], route_ranks: {body_bm25: 14, vec: 2}, rrf_score) — needed for the four-condition bench and for later failure analysis. The merged ranking then feeds the existing body-coverage sqrt rerank, title-token boost, _filter_by_title_relevance, rivalry exclusion, stem-aware matching, per-source cap, wikitext-base prose — unchanged.

Accept-path-5 (semantic-warrant). This is the load-bearing addition, not an afterthought: the existing title-relevance gate may drop exactly the low-title-overlap semantic candidates vec exists to find (the "what did Orwell call the country at war with Oceania?" → Nineteen Eighty-Four / Eastasia case — right chunk, zero query/title token overlap), so without a vec-safe accept path the bench can show no lift even when vec retrieval works. Add a fifth accept path in _filter_by_title_relevance that lets a vec hit survive low title overlap only if it earns a stronger span-level signal — at least one of: (a) the cited span contains a deterministic anchor from the question or the generated claim; (b) a claim-lattice pointer resolves to a span whose anchor-class warrant matches the claim; (c) an entity/predicate/date/quote warrant passes on the span; (d) a concept_relations synonym edge bridges a query anchor to a span anchor; (e) the answer is explicitly exploratory / UNGROUNDED and does not reach an EVIDENCE-WARRANTED / STRICT label. In short: vec may bypass title overlap only by earning a stronger warrant, never by embedding proximity alone. Decide the exact predicate set at implementation time from the bench, but the principle — "vec hits clear the gate via warrant strength, not similarity score" — is non-negotiable.

Mixed fleet. Skip the vec route cleanly when chunk_vecs is absent/empty on a shard (some shards have an imported vecpack — #000051 — some don't) — falls back to FTS5-only for that shard, with an audit note vec_route_unavailable so a hybrid-labelled run never silently ran FTS5-only.

Governance — wire it before any cached hybrid run, not "later". Fold the six vec config fields — embedder_name, embedder_version (pinned revision/digest, separate from the name), quantization (named with its recipe: int8sym = symmetric-global-scale ×127/clamp/round, not bare int8), dimension, distance_metric, ann_index — into governance_policy_hash (#000039 §6), summarized by the compact VEC_BACKEND_VERSION token (e.g. vec-v1-bge-small-en-v1.5-384int8sym-l2norm-flat; the token is a derived summary, the six fields are the auditable payload). Until that wiring lands, a cache-write guard: a QA run whose retrieval backend is vec/hybrid may return an answer but must not persist a providence_cache record (status vec_experimental_no_cache) — because changing the embedder changes the candidates changes the context changes the answer, and an un-versioned cached hybrid answer is exactly the silent provenance drift the substrate exists to prevent. The 8-dim cache_key invariant is untouched (existing dim, not a new one). The hash must also cover the fusion policy — rrf_k, vec_top_k, the route set, accept_path_version, the fallback / coverage policy, the candidate-budget policy — not just the six vec fields; see §5a.1 (changing rrf_k shifts the answer as much as changing the embedder).

Run-DAG metadata. A hybrid run's per-run Merkle-DAG records the vec retrieval stage: route="vec", VEC_BACKEND_VERSION, the six config fields, top_k, a query_embedding_hash, and the candidate (chunk_id, quantized_distance) list (or a candidate_root over them). governance_policy_hash says which regime was allowed; the run-DAG says what actually happened this run — both must be visible for replay (same lesson as the --retrieval-keywords audit gap, #000001).

The knob. arborist query / the bench harness: a --retrieval-backend {fts5,hybrid} knob (default fts5 until the bench clears hybrid, then flipping the default is a fox call + a governance_policy_hash bump). No vec-only QA path except explicitly-experimental (vec-only is a retrieval diagnostic, not an answer regime).

Adversarial semantic-neighbor fixtures. Vec opens a new failure class: a chunk that is semantically near the query but factually wrong — same topic / wrong entity, same title family / wrong franchise installment, same era / wrong date, a paraphrase that retrieves the background source instead of the primary one (the "Mr. Burns" / "which Back to the Future" shape, now reachable through a semantic side door). Build a small adversarial pack for the §2 bench whose pass condition is: hybrid must not raise the rate of confidently-wrong warranted answers, and the title-relevance / Rule-8 hard checks must still catch the mismatched vec hits. This pack is part of the gate, not a nice-to-have.

4. Out of scope

• The corpus backfill itself (a batch op — #000039 §14.6) and its distribution (vecpack export/import + mesh payload — #000051). Prereq #1 above is "a vecpack exists and is imported on the bench box", which #000051 delivers.
• A GPU/accelerated embedder (drop-in via default_embedder()'s pluggable Embedder callable; orthogonal — #000039 §14.6).
• int8 vs float32 default (settled: int8 is the production config per #000039's head-to-head; switching the v1 default is a separate VEC_BACKEND_VERSION bump, not Phase 2's call).
• binary quantization + two-stage re-rank (#000039 §3.1's storage-emergency option; its own future ticket if corpus scale ever makes +0.8 % vs +6 % matter).

5. Acceptance criteria

1. arborist/qa/query.py merges a fifth (vec) route via RRF (with route provenance on each hit), behind a --retrieval-backend {fts5,hybrid} knob; FTS5-only behavior byte-unchanged when the knob says fts5 or chunk_vecs is absent (mixed-fleet fallback emits the vec_route_unavailable audit note).
2. Accept-path-5 lands in _filter_by_title_relevance: a low-title- overlap vec hit survives only via a stronger span-level warrant (anchor / pointer-warrant / entity-predicate-date-quote / concept-synonym / explicitly-UNGROUNDED), never via similarity score alone.
3. The six vec config fields fold into governance_policy_hash, summarized by VEC_BACKEND_VERSION (recipe-named quant, e.g. int8sym); and a cache-write guard blocks providence_cache persistence for vec/hybrid runs until that wiring is live (status vec_experimental_no_cache).
4. A hybrid run's run-DAG records the vec retrieval stage (backend version, six fields, top_k, query-embedding hash, candidate chunk_ids + quantized distances / candidate_root).
5. Bench data recorded for all four conditions (A FTS5-only / B vec-only / C RRF hybrid / D candidate-union-no-RRF) on the semantic-allusion + curated + adversarial-neighbor fixtures, showing the §2 gate cleared — including C-beats-D — (or, if it didn't, this ticket parks with the result documented and hybrid left non-default; if C ≈ D, ship the union, drop RRF).
6. Tests: RRF math + route provenance; vec-absent fallback + the audit note; accept-path-5 (a known semantic-allusion hit survives low title overlap only with a warrant, a bare-similarity hit does not); cache-write guard fires for un-versioned hybrid; filter chain + Rule-8 still reject adversarial semantic-neighbor hits; a bench-row in the QA-modes journal.

5a. Review amendments — 2026-05-13 (Dav1d de-novo review)

Fold into the Phase-2 spec before code. None change the architecture (vec additive, RRF rank-based, soft/UNGROUNDED, FTS5 default, 8-dim cache-key untouched) — they close measurement and provenance gaps.

1. governance_policy_hash must cover the fusion policy, not just the six vec-config fields. Hybrid answer context depends just as much on retrieval_backend (fts5 | hybrid), vec_top_k, rrf_k (default 60), the hybrid_route_set (which five routes), accept_path_version (the title-relevance / semantic-warrant gate version), the fallback_policy (what happens when chunk_vecs is absent), and the candidate_budget_policy (final top-K × per-source cap interaction). All of these + the six vec-config fields fold into governance_policy_hash. Still no ninth cache-key dimension — it all rides the existing governance_policy_hash dim. Serialize as canonical JSON (sorted keys, explicit scalar types, recipe-named quant, stable version token) → sha256 → the hash. (π* formalization as vec_config@v1 is later, not a blocker.)

2. RRF must survive the post-merge rerank — else the bench tests the old reranker, not RRF. If the existing body-coverage sqrt rerank + title-token boost fully re-sort the candidate list after RRF, the fused order is erased and "RRF failed" is unmeasurable. Implementation rule (Option A): the normalized RRF score becomes one feature in the existing reranker — final_score = norm_rrf_score + bounded_body_coverage_boost + bounded_title_boost − hard_penalties — with the boosts bounded so they adjust within the RRF prior, not overwrite it. Hard filters still drop candidates. Add test_rrf_signal_survives_post_merge_rerank. Deterministic tie-break: sort key = (−rrf_score, source_role_priority, source_root_hex, chunk_id) (exact key doesn't matter; stable + documented does) — test_rrf_tie_break_is_deterministic.

3. Split accept-path-5 into 5a vs 5b. 5a — semantic-warrant survival: a low-title-overlap vec hit that earns a stronger span-level warrant (span-contains-anchor / claim-pointer-warrant / entity-predicate-date-quote / concept-synonym-bridge) may enter normal context and later become evidence-warranted. 5b — exploratory-UNGROUNDED survival: a hit admitted only because the answer is explicitly exploratory may enter context only under the exploratory label and is hard-excluded from verified-evidence counting — it can never support a STRICT / EVIDENCE-WARRANTED output. Don't merge these into one path: an "exploratory" route must not become a back-door bridge into warranted answer context. Tests: test_accept_path_5_warranted_span_survives_low_title_overlap, test_accept_path_5_similarity_only_does_not_survive_low_title_overlap, test_exploratory_ungrounded_vec_candidate_cannot_increment_verified_count.

4. Mixed-fleet coverage is explicit, not silent. When a hybrid run spans shards where some have chunk_vecs and some don't, the run-DAG records vec_coverage = {shards_total, shards_with_vec, shards_without_vec, unavailable_shards:[…], coverage_mode: full|partial}. Bench-validity rule: a hybrid bench result is valid only at the declared coverage threshold — ideally full coverage on the benchmark shards. Production rule: a hybrid-labelled answer's audit metadata states full-vs-partial coverage. Else an operator thinks they ran hybrid while most shards silently fell back to FTS5. Tests: test_vec_absent_on_one_shard_records_partial_coverage, test_hybrid_label_does_not_silently_mean_fts5_only.

5. Bench needs exact denominators + a row schema. Per-query bench row: {query_id, condition: fts5|vec|rrf_hybrid|union_no_rrf, retrieval_backend_version, vec_coverage_mode, top_k, proof_quality_label, primary_answer_source_rank, used_sources, wrong_warrant: bool, ungrounded_output: bool, latency_ms, context_chars}. Derived: target_fixture_lift = rate(C good) − rate(A good); lexical_regression = rate(C good | lexical) − rate(A good | lexical); semantic_false_neighbor_rate = rate(confident-wrong-warranted | adversarial-neighbor); ungrounded_regression = rate(C UNGROUNDED out) − rate(A UNGROUNDED out). "Good label" by mode: quote/span/entity/paraphrase → STRICT or EVIDENCE-WARRANTED; lattice → EVIDENCE-WARRANTED (or the current top verified label); exploratory mode → excluded from proof-quality lift entirely.

6. Consume #000051 metadata in the run-DAG (audit/replay, never proof): {vecpack_id, vecpack_manifest_root, vecpack_trust_tier, vecpack_import_event}. If a hybrid answer rested on gossiped embeddings, replay needs to know which vecpack supplied them — and #000050's cache-policy decides which trust tiers (#000051 §6a.4) a cached hybrid answer may rest on (Tier 2+ once governance-hash wiring is live; Tier 0/1 → raw search / experimental only).

7. Implementation order — governance + cache guard before any RRF code can persist. (1) governance_policy_hash wiring for vec + fusion policy; (2) cache-write guard for vec/hybrid when the policy hash is incomplete; (3) mixed-fleet fallback + vec_coverage metadata; (4) run-DAG vec-stage metadata; (5) RRF merge as the fifth route behind --retrieval-backend hybrid; (6) accept-path-5a; (7) accept-path-5b (if still needed after 5a); (8) four-condition bench harness rows; (9) adversarial semantic-neighbor fixture pack; (10) the C-vs-A-vs-D decision (keep RRF / ship union / keep vec opt-in).

8. Test additions (append to §5.6): test_hybrid_governance_hash_includes_rrf_k_and_vec_top_k, test_hybrid_cache_write_blocked_without_vec_policy_hash, test_rrf_score_does_not_use_raw_bm25_or_raw_distance, test_rrf_tie_break_is_deterministic, test_rrf_signal_survives_post_merge_rerank, test_vec_absent_on_one_shard_records_partial_coverage, test_hybrid_label_does_not_silently_mean_fts5_only, test_accept_path_5_warranted_span_survives_low_title_overlap, test_accept_path_5_similarity_only_does_not_survive_low_title_overlap, test_exploratory_ungrounded_vec_candidate_cannot_increment_verified_count, test_rule8_rejects_wrong_franchise_semantic_neighbor, test_candidate_union_control_written_to_bench_journal.

6. References

• #000039 — the parent (Phase 0 + Phase 1, closed 2026-05-12); §4 (additive not replacement), §4.2 (RRF design), §8 (the gate), §14 (ingest integration), §14.6 (embed throughput).
• #000051 — federated vecpack distribution. Supplies prereq #1 (a corpus backfill) as a distributable artifact — backfill once on any CPU box, gossip the chunk_vecs over mesh — so the laptop never embeds. Read it for why vec data is safe to gossip with a cheap structural gate (soft data, never proof path).
• docs/bench-maxing.md — the 5pp signal floor + bench discipline.
• docs/benchmarks.md — bench harness orientation.
• #000037 §3.1 — the Prometheus-Σ unconscious-sweep task that's the natural home for the embed backfill and the vecpack producer.
• CLAUDE.md "Retrieval pipeline" — the four FTS5 routes vec joins; "soft hash vs hard hash" — why vec stays out of the proof path.