diff --git a/docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md b/docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md index a4d6abf..4e43aeb 100644 --- a/docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md +++ b/docs/tickets/ticket-000050-vec-rrf-hybrid-fusion.md @@ -145,7 +145,11 @@ because changing the embedder changes the candidates changes the context changes the answer, and an un-versioned cached hybrid answer is exactly the silent provenance drift the substrate exists to prevent. The 8-dim `cache_key` invariant is untouched (existing dim, not a new -one). +one). **The hash must also cover the *fusion policy* — `rrf_k`, +`vec_top_k`, the route set, `accept_path_version`, the fallback / +coverage policy, the candidate-budget policy — not just the six vec +fields; see §5a.1** (changing `rrf_k` shifts the answer as much as +changing the embedder). **Run-DAG metadata.** A hybrid run's per-run Merkle-DAG records the vec retrieval stage: `route="vec"`, `VEC_BACKEND_VERSION`, the six config @@ -223,6 +227,118 @@ a nice-to-have. filter chain + Rule-8 still reject adversarial semantic-neighbor hits; a bench-row in the QA-modes journal. +## 5a. Review amendments — 2026-05-13 (Dav1d de-novo review) + +Fold into the Phase-2 spec before code. None change the architecture +(vec additive, RRF rank-based, soft/`UNGROUNDED`, FTS5 default, +8-dim cache-key untouched) — they close measurement and provenance +gaps. + +1. **`governance_policy_hash` must cover the *fusion policy*, not just + the six vec-config fields.** Hybrid answer context depends just as + much on `retrieval_backend` (`fts5` | `hybrid`), `vec_top_k`, + `rrf_k` (default 60), the `hybrid_route_set` (which five routes), + `accept_path_version` (the title-relevance / semantic-warrant gate + version), the `fallback_policy` (what happens when `chunk_vecs` is + absent), and the `candidate_budget_policy` (final top-K × per-source + cap interaction). All of these + the six vec-config fields fold into + `governance_policy_hash`. Still **no ninth cache-key dimension** — + it all rides the existing `governance_policy_hash` dim. Serialize as + canonical JSON (sorted keys, explicit scalar types, recipe-named + quant, stable version token) → sha256 → the hash. (π* formalization + as `vec_config@v1` is later, not a blocker.) + +2. **RRF must survive the post-merge rerank — else the bench tests the + old reranker, not RRF.** If the existing body-coverage `sqrt` + rerank + title-token boost fully re-sort the candidate list after + RRF, the fused order is erased and "RRF failed" is unmeasurable. + Implementation rule (**Option A**): the normalized RRF score becomes + *one feature* in the existing reranker — + `final_score = norm_rrf_score + bounded_body_coverage_boost + + bounded_title_boost − hard_penalties` — with the boosts bounded so + they adjust *within* the RRF prior, not overwrite it. Hard filters + still drop candidates. Add `test_rrf_signal_survives_post_merge_rerank`. + Deterministic tie-break: `sort key = (−rrf_score, + source_role_priority, source_root_hex, chunk_id)` (exact key + doesn't matter; stable + documented does) — + `test_rrf_tie_break_is_deterministic`. + +3. **Split accept-path-5 into 5a vs 5b.** **5a — semantic-warrant + survival:** a low-title-overlap vec hit that earns a stronger + span-level warrant (span-contains-anchor / claim-pointer-warrant / + entity-predicate-date-quote / concept-synonym-bridge) may enter + normal context *and later become evidence-warranted*. **5b — + exploratory-`UNGROUNDED` survival:** a hit admitted only because the + answer is explicitly exploratory may enter context **only under the + exploratory label** and is **hard-excluded from verified-evidence + counting** — it can never support a STRICT / EVIDENCE-WARRANTED + output. Don't merge these into one path: an "exploratory" route must + not become a back-door bridge into warranted answer context. Tests: + `test_accept_path_5_warranted_span_survives_low_title_overlap`, + `test_accept_path_5_similarity_only_does_not_survive_low_title_overlap`, + `test_exploratory_ungrounded_vec_candidate_cannot_increment_verified_count`. + +4. **Mixed-fleet coverage is explicit, not silent.** When a hybrid run + spans shards where some have `chunk_vecs` and some don't, the + run-DAG records `vec_coverage = {shards_total, shards_with_vec, + shards_without_vec, unavailable_shards:[…], coverage_mode: + full|partial}`. **Bench-validity rule:** a hybrid bench result is + valid only at the declared coverage threshold — ideally *full* + coverage on the benchmark shards. **Production rule:** a + hybrid-labelled answer's audit metadata states full-vs-partial + coverage. Else an operator thinks they ran hybrid while most shards + silently fell back to FTS5. Tests: + `test_vec_absent_on_one_shard_records_partial_coverage`, + `test_hybrid_label_does_not_silently_mean_fts5_only`. + +5. **Bench needs exact denominators + a row schema.** Per-query bench + row: `{query_id, condition: fts5|vec|rrf_hybrid|union_no_rrf, + retrieval_backend_version, vec_coverage_mode, top_k, + proof_quality_label, primary_answer_source_rank, used_sources, + wrong_warrant: bool, ungrounded_output: bool, latency_ms, + context_chars}`. Derived: `target_fixture_lift = rate(C good) − + rate(A good)`; `lexical_regression = rate(C good | lexical) − + rate(A good | lexical)`; `semantic_false_neighbor_rate = + rate(confident-wrong-warranted | adversarial-neighbor)`; + `ungrounded_regression = rate(C UNGROUNDED out) − rate(A UNGROUNDED + out)`. "Good label" by mode: quote/span/entity/paraphrase → STRICT + *or* EVIDENCE-WARRANTED; lattice → EVIDENCE-WARRANTED (or the + current top verified label); exploratory mode → excluded from + proof-quality lift entirely. + +6. **Consume #000051 metadata in the run-DAG** (audit/replay, never + proof): `{vecpack_id, vecpack_manifest_root, vecpack_trust_tier, + vecpack_import_event}`. If a hybrid answer rested on gossiped + embeddings, replay needs to know which vecpack supplied them — and + #000050's cache-policy decides which *trust tiers* (#000051 §6a.4) + a *cached* hybrid answer may rest on (Tier 2+ once governance-hash + wiring is live; Tier 0/1 → raw search / experimental only). + +7. **Implementation order — governance + cache guard before any RRF + code can persist.** (1) `governance_policy_hash` wiring for vec + + fusion policy; (2) cache-write guard for `vec`/`hybrid` when the + policy hash is incomplete; (3) mixed-fleet fallback + `vec_coverage` + metadata; (4) run-DAG vec-stage metadata; (5) RRF merge as the + fifth route behind `--retrieval-backend hybrid`; (6) accept-path-5a; + (7) accept-path-5b (if still needed after 5a); (8) four-condition + bench harness rows; (9) adversarial semantic-neighbor fixture pack; + (10) the C-vs-A-vs-D decision (keep RRF / ship union / keep vec + opt-in). + +8. **Test additions** (append to §5.6): + `test_hybrid_governance_hash_includes_rrf_k_and_vec_top_k`, + `test_hybrid_cache_write_blocked_without_vec_policy_hash`, + `test_rrf_score_does_not_use_raw_bm25_or_raw_distance`, + `test_rrf_tie_break_is_deterministic`, + `test_rrf_signal_survives_post_merge_rerank`, + `test_vec_absent_on_one_shard_records_partial_coverage`, + `test_hybrid_label_does_not_silently_mean_fts5_only`, + `test_accept_path_5_warranted_span_survives_low_title_overlap`, + `test_accept_path_5_similarity_only_does_not_survive_low_title_overlap`, + `test_exploratory_ungrounded_vec_candidate_cannot_increment_verified_count`, + `test_rule8_rejects_wrong_franchise_semantic_neighbor`, + `test_candidate_union_control_written_to_bench_journal`. + ## 6. References - #000039 — the parent (Phase 0 + Phase 1, closed 2026-05-12); diff --git a/docs/tickets/ticket-000051-federated-vecpack-distribution.md b/docs/tickets/ticket-000051-federated-vecpack-distribution.md index 2e84e3c..e91db58 100644 --- a/docs/tickets/ticket-000051-federated-vecpack-distribution.md +++ b/docs/tickets/ticket-000051-federated-vecpack-distribution.md @@ -218,6 +218,148 @@ quant recipe in `vec_backend_version`, not the output bytes (the same + a pointer from #000050 §2 prereq #1 ("backfill once, distribute via #000051"). +## 6a. Review amendments — 2026-05-13 (Dav1d de-novo review) + +Fold these into the v1 spec before implementation. They do **not** +change the architecture (soft sidecar, `leaf_hash`-keyed, cheap +validation, never proof path) — they sharpen it. + +1. **Narrower "harmless" wording.** A bad/adversarial vecpack cannot + ground a claim, mutate hard roots, or forge an audit proof *if the + hard/soft boundary holds* — but it is **not** "only a recall nudge". + It can also bias candidate ranking, waste top-K on junk (hiding good + candidates), inflate latency/context, and skew bench conclusions — + i.e. a *denial-of-retrieval* surface. So: no proof-grade + embedding-*correctness* verification; **yes** structural, + compatibility, provenance, trust-tier, and import-integrity checks. + +2. **Manifest + roots (integrity, not correctness).** A vecpack carries + a canonical-JSON **manifest** — `format_version` (`vecpack-v1`), + `vecpack_id = sha256(canonical manifest)`, `shard_root`, + `vec_backend_version`, full `backend_config` (see #3), `count`, + `entry_root`, `part_count`, `compression`, `producer_id`, + `created_at`. Each row has a deterministic + `row_hash = H(leaf_hash || vec_backend_version || quant_recipe || + embedding_blob)`; `entry_root = MerkleRoot(row_hashes)`. This is + **transit integrity** ("did the bytes arrive unchanged?"), *not* + correctness ("was this embedding actually computed by the declared + model?") — #000051 needs the former, never the latter. Audit events + cite `manifest_root` / `entry_root`, not millions of row hashes. + +3. **`vec_backend_version` must commit the full reproducible input.** + "embedding = f(chunk bytes, embedder config)" is under-specified. + `backend_config` (in the manifest, summarized by the version token) + must pin: `embedder_name`, `embedder_revision` (digest, not just a + tag), `tokenizer_revision`, `preprocessing_policy` (which prose + normalization — `chunk_text_base@v1`), `pooling_policy`, + `normalization_policy` (`l2_normalize@v1`), `quantization_recipe` + (`int8sym127_global_clamp_round@v1` — *named*, not bare `int8`), + `dimension`, `dtype`, `distance_metric`, `ann_index`. The compact + token (`vec-v1-bge-small-en-v1.5-384int8sym127-l2norm-flat`) is a + derived summary; the manifest fields are the auditable payload. + (Producer profile CPU-vs-GPU is *not* pinned — outputs aren't + byte-stable across it and don't need to be; soft data.) + +4. **Trust tiers.** Tier 0 = local file / unknown producer (import OK + for raw vec search if structural checks pass; `vecpack_trust = + untrusted`; **no cached hybrid QA**). Tier 1 = integrity-checked + mesh sender (sender authenticated, manifest/roots check out; raw + + experimental hybrid; cached QA still gated on #000050 policy). + Tier 2 = trusted producer (identity accepted by local policy; + eligible for cached hybrid QA once #000050 governance-hash wiring is + live). Tier 3 = locally generated (strongest operational trust; + still soft data, still `UNGROUNDED`). #000050 decides which tiers a + *cached* hybrid answer may rest on. Trust tier is **audit metadata, + never proof identity** — a trusted vecpack saying "this chunk is + semantically close" is still just a *candidate proposal*; the + verifier must inspect the text. + +5. **Streaming, resumable import — never load 2.4 GB into RAM.** + Manifest-first validation → shard/backend compat check → disk-budget + check → stream entries part-by-part → stage valid rows in batches + (1k–10k rows) → commit in bounded transactions (WAL, `executemany`, + no per-row commits) → record progress → resume-or-rollback after + interruption (idempotent: re-import is a no-op). + +6. **Bad-row thresholds + duplicate policy.** `bad_row_rate == 0` → + clean; `0 < rate ≤ 0.01%` → import valid rows, warn; + `rate > 1%` → reject / quarantine the pack. `missing_chunk` is fine + for partial packs (record `coverage_ratio`). `duplicate_leaf_hash`: + identical blob → skip silently; conflicting blob → reject the pack + (or row, by policy). Thresholds are policy knobs. + +7. **Exception matrix (deterministic status for every failure).** + Wrong `shard_root` → `reject_wrong_shard_root`; missing `chunk_vecs` + → `created_vec_meta`; existing `vec_meta` mismatch → + `reject_backend_mismatch`; unsupported quant → + `reject_unsupported_quant`; metric/quant incompat → + `reject_metric_quant_mismatch`; bad blob length → + `skipped_bad_length`; non-finite norm → `skipped_bad_norm`; norm + out of sane range → `skipped_bad_norm_range`; missing local chunk → + `skipped_missing_chunk`; dup identical → `duplicate_identical`; dup + conflict → `duplicate_conflict`; mesh sig fail → + `reject_signature_fail`; AEAD fail → `reject_aead_fail`; missing + part → `incomplete_parts`; part-hash mismatch → + `reject_part_hash_mismatch`; disk budget exceeded → + `reject_insufficient_disk`; interrupted → `import_interrupted`; + bad-row rate over threshold → `reject_bad_row_rate`; unknown + producer → `untrusted_producer`. **Cache impact is `None` on every + row** — vecpack import never touches `providence_cache`. + +8. **Soft audit events.** Import/export/mesh-transfer write soft + provenance events: `vecpack_export_{started,completed}`, + `vecpack_import_{started,verified,completed,rejected,interrupted}`, + `vecpack_part_{received,rejected}`, + `vecpack_mesh_{announced,requested,served}` — each with + `{vecpack_id, manifest_root, shard_root, vec_backend_version, + trust_tier, counts:{received,loaded,skipped,bad}}`. Soft events, + not proof events. + +9. **CLI surface.** Add to §4's `export`/`import`: + `arborist vecpack inspect --in path` (manifest only); + `vecpack verify --shard --in path` (all structural checks, no + insert); `vecpack import --dry-run` (report expected status); + `vecpack import --replace` (explicit rewrite, same backend only — + never silently overwrite a *different* backend's vectors); + `vecpack import --trust-policy {reject-unknown,untrusted-ok,trusted-only}`. + +10. **SQD discipline now, π* module later.** Don't block on a + `vecpack@v1` π* canonicalizer — but the manifest must already be + canonical bytes: sorted keys, explicit scalar types, no implicit + defaults in hash-bearing fields, versioned quant recipe, stable + `manifest_root` / `entry_root`. Formalize as + `pi_star_ref ∈ {vecpack_manifest@v1, vecpack_entry@v1, + vec_backend_config@v1}` later. + +11. **Metadata #000050 consumes** (for run-DAG, audit, replay — never + proof): `{vecpack_id, vecpack_manifest_root, vec_backend_version, + coverage_ratio, trust_tier, import_event_hash}`. + +**Test additions** (append to §6.6): +`test_vecpack_manifest_canonical_hash_stable`, +`test_vecpack_export_import_cross_build_leaf_hash_mapping`, +`test_vecpack_import_rejects_{wrong_shard_root,backend_version_mismatch,metric_quant_mismatch}`, +`test_vecpack_import_skips_{missing_leaf_hash,bad_blob_length,nonfinite_vector}`, +`test_vecpack_import_rejects_duplicate_leaf_conflict`, +`test_vecpack_import_idempotent_same_pack`, +`test_vecpack_import_resume_after_interruption`, +`test_vecpack_import_respects_bad_row_threshold`, +`test_vecpack_import_rejects_insufficient_disk_budget`, +`test_vecpack_mesh_envelope_signature_failure_rejects`, +`test_vecpack_mesh_part_reassembly_requires_all_parts`, +`test_vecpack_bad_embedding_cannot_increment_verified_evidence_count`, +`test_vecpack_import_writes_soft_audit_event_only`, +`test_vecpack_import_does_not_modify_document_root_or_cache_key`. +The last two are the hard/soft-boundary proof. + +**Implementation order:** (1) file `vecpack export`/`import` + +manifest/roots; (2) structural validation + soft audit events; (3) +cross-build `leaf_hash`-mapping tests; (4) streaming/resume; (5) mesh +payload transport; (6) #000050 integration via vecpack metadata. +Default safety throughout: imported vecpacks support **raw vec search**; +**cached hybrid QA stays blocked** until #000050's governance-hash +wiring + cache-write guard land. + ## 7. References - #000039 — the local vec backend (Phase 0 + Phase 1, closed