#000051 §6a: vecpack manifest + entry_root (transit integrity, not embedding-correctness proof); vec_backend_version must commit the full reproducible input (model/tokenizer revisions, preprocessing/pooling/ normalization policies, named quant recipe); trust tiers 0-3 (untrusted OK for raw search, cached hybrid QA needs Tier 2+ and #000050 wiring); streaming/resumable/bounded-memory import + disk budget; bad-row thresholds + duplicate policy; exception matrix; soft audit events; extra CLI (inspect/verify/--dry-run/--replace/--trust-policy); SQD canonical-bytes discipline now, pi* module later; 15 test additions. #000050 §5a: governance_policy_hash must cover the fusion policy (retrieval_backend, vec_top_k, rrf_k, route set, accept_path_version, fallback/coverage policy, candidate budget) not just the 6 vec fields; RRF must survive the post-merge rerank (Option A: RRF score is one bounded feature in the existing reranker) + deterministic tie-break; split accept-path-5 into 5a warranted-survival vs 5b exploratory- UNGROUNDED-survival (5b hard-excluded from verified-evidence counting); mixed-fleet vec_coverage metadata + bench-validity rule; exact bench denominators + row schema + good-label-by-mode; consume #000051 metadata in run-DAG; implementation order (governance+cache-guard before RRF code can persist); 12 test additions.
This commit is contained in:
parent
e391f1f01c
commit
7ff37f8d70
2 changed files with 259 additions and 1 deletions
|
|
@ -145,7 +145,11 @@ because changing the embedder changes the candidates changes the
|
|||
context changes the answer, and an un-versioned cached hybrid answer is
|
||||
exactly the silent provenance drift the substrate exists to prevent.
|
||||
The 8-dim `cache_key` invariant is untouched (existing dim, not a new
|
||||
one).
|
||||
one). **The hash must also cover the *fusion policy* — `rrf_k`,
|
||||
`vec_top_k`, the route set, `accept_path_version`, the fallback /
|
||||
coverage policy, the candidate-budget policy — not just the six vec
|
||||
fields; see §5a.1** (changing `rrf_k` shifts the answer as much as
|
||||
changing the embedder).
|
||||
|
||||
**Run-DAG metadata.** A hybrid run's per-run Merkle-DAG records the vec
|
||||
retrieval stage: `route="vec"`, `VEC_BACKEND_VERSION`, the six config
|
||||
|
|
@ -223,6 +227,118 @@ a nice-to-have.
|
|||
filter chain + Rule-8 still reject adversarial semantic-neighbor
|
||||
hits; a bench-row in the QA-modes journal.
|
||||
|
||||
## 5a. Review amendments — 2026-05-13 (Dav1d de-novo review)
|
||||
|
||||
Fold into the Phase-2 spec before code. None change the architecture
|
||||
(vec additive, RRF rank-based, soft/`UNGROUNDED`, FTS5 default,
|
||||
8-dim cache-key untouched) — they close measurement and provenance
|
||||
gaps.
|
||||
|
||||
1. **`governance_policy_hash` must cover the *fusion policy*, not just
|
||||
the six vec-config fields.** Hybrid answer context depends just as
|
||||
much on `retrieval_backend` (`fts5` | `hybrid`), `vec_top_k`,
|
||||
`rrf_k` (default 60), the `hybrid_route_set` (which five routes),
|
||||
`accept_path_version` (the title-relevance / semantic-warrant gate
|
||||
version), the `fallback_policy` (what happens when `chunk_vecs` is
|
||||
absent), and the `candidate_budget_policy` (final top-K × per-source
|
||||
cap interaction). All of these + the six vec-config fields fold into
|
||||
`governance_policy_hash`. Still **no ninth cache-key dimension** —
|
||||
it all rides the existing `governance_policy_hash` dim. Serialize as
|
||||
canonical JSON (sorted keys, explicit scalar types, recipe-named
|
||||
quant, stable version token) → sha256 → the hash. (π* formalization
|
||||
as `vec_config@v1` is later, not a blocker.)
|
||||
|
||||
2. **RRF must survive the post-merge rerank — else the bench tests the
|
||||
old reranker, not RRF.** If the existing body-coverage `sqrt`
|
||||
rerank + title-token boost fully re-sort the candidate list after
|
||||
RRF, the fused order is erased and "RRF failed" is unmeasurable.
|
||||
Implementation rule (**Option A**): the normalized RRF score becomes
|
||||
*one feature* in the existing reranker —
|
||||
`final_score = norm_rrf_score + bounded_body_coverage_boost +
|
||||
bounded_title_boost − hard_penalties` — with the boosts bounded so
|
||||
they adjust *within* the RRF prior, not overwrite it. Hard filters
|
||||
still drop candidates. Add `test_rrf_signal_survives_post_merge_rerank`.
|
||||
Deterministic tie-break: `sort key = (−rrf_score,
|
||||
source_role_priority, source_root_hex, chunk_id)` (exact key
|
||||
doesn't matter; stable + documented does) —
|
||||
`test_rrf_tie_break_is_deterministic`.
|
||||
|
||||
3. **Split accept-path-5 into 5a vs 5b.** **5a — semantic-warrant
|
||||
survival:** a low-title-overlap vec hit that earns a stronger
|
||||
span-level warrant (span-contains-anchor / claim-pointer-warrant /
|
||||
entity-predicate-date-quote / concept-synonym-bridge) may enter
|
||||
normal context *and later become evidence-warranted*. **5b —
|
||||
exploratory-`UNGROUNDED` survival:** a hit admitted only because the
|
||||
answer is explicitly exploratory may enter context **only under the
|
||||
exploratory label** and is **hard-excluded from verified-evidence
|
||||
counting** — it can never support a STRICT / EVIDENCE-WARRANTED
|
||||
output. Don't merge these into one path: an "exploratory" route must
|
||||
not become a back-door bridge into warranted answer context. Tests:
|
||||
`test_accept_path_5_warranted_span_survives_low_title_overlap`,
|
||||
`test_accept_path_5_similarity_only_does_not_survive_low_title_overlap`,
|
||||
`test_exploratory_ungrounded_vec_candidate_cannot_increment_verified_count`.
|
||||
|
||||
4. **Mixed-fleet coverage is explicit, not silent.** When a hybrid run
|
||||
spans shards where some have `chunk_vecs` and some don't, the
|
||||
run-DAG records `vec_coverage = {shards_total, shards_with_vec,
|
||||
shards_without_vec, unavailable_shards:[…], coverage_mode:
|
||||
full|partial}`. **Bench-validity rule:** a hybrid bench result is
|
||||
valid only at the declared coverage threshold — ideally *full*
|
||||
coverage on the benchmark shards. **Production rule:** a
|
||||
hybrid-labelled answer's audit metadata states full-vs-partial
|
||||
coverage. Else an operator thinks they ran hybrid while most shards
|
||||
silently fell back to FTS5. Tests:
|
||||
`test_vec_absent_on_one_shard_records_partial_coverage`,
|
||||
`test_hybrid_label_does_not_silently_mean_fts5_only`.
|
||||
|
||||
5. **Bench needs exact denominators + a row schema.** Per-query bench
|
||||
row: `{query_id, condition: fts5|vec|rrf_hybrid|union_no_rrf,
|
||||
retrieval_backend_version, vec_coverage_mode, top_k,
|
||||
proof_quality_label, primary_answer_source_rank, used_sources,
|
||||
wrong_warrant: bool, ungrounded_output: bool, latency_ms,
|
||||
context_chars}`. Derived: `target_fixture_lift = rate(C good) −
|
||||
rate(A good)`; `lexical_regression = rate(C good | lexical) −
|
||||
rate(A good | lexical)`; `semantic_false_neighbor_rate =
|
||||
rate(confident-wrong-warranted | adversarial-neighbor)`;
|
||||
`ungrounded_regression = rate(C UNGROUNDED out) − rate(A UNGROUNDED
|
||||
out)`. "Good label" by mode: quote/span/entity/paraphrase → STRICT
|
||||
*or* EVIDENCE-WARRANTED; lattice → EVIDENCE-WARRANTED (or the
|
||||
current top verified label); exploratory mode → excluded from
|
||||
proof-quality lift entirely.
|
||||
|
||||
6. **Consume #000051 metadata in the run-DAG** (audit/replay, never
|
||||
proof): `{vecpack_id, vecpack_manifest_root, vecpack_trust_tier,
|
||||
vecpack_import_event}`. If a hybrid answer rested on gossiped
|
||||
embeddings, replay needs to know which vecpack supplied them — and
|
||||
#000050's cache-policy decides which *trust tiers* (#000051 §6a.4)
|
||||
a *cached* hybrid answer may rest on (Tier 2+ once governance-hash
|
||||
wiring is live; Tier 0/1 → raw search / experimental only).
|
||||
|
||||
7. **Implementation order — governance + cache guard before any RRF
|
||||
code can persist.** (1) `governance_policy_hash` wiring for vec +
|
||||
fusion policy; (2) cache-write guard for `vec`/`hybrid` when the
|
||||
policy hash is incomplete; (3) mixed-fleet fallback + `vec_coverage`
|
||||
metadata; (4) run-DAG vec-stage metadata; (5) RRF merge as the
|
||||
fifth route behind `--retrieval-backend hybrid`; (6) accept-path-5a;
|
||||
(7) accept-path-5b (if still needed after 5a); (8) four-condition
|
||||
bench harness rows; (9) adversarial semantic-neighbor fixture pack;
|
||||
(10) the C-vs-A-vs-D decision (keep RRF / ship union / keep vec
|
||||
opt-in).
|
||||
|
||||
8. **Test additions** (append to §5.6):
|
||||
`test_hybrid_governance_hash_includes_rrf_k_and_vec_top_k`,
|
||||
`test_hybrid_cache_write_blocked_without_vec_policy_hash`,
|
||||
`test_rrf_score_does_not_use_raw_bm25_or_raw_distance`,
|
||||
`test_rrf_tie_break_is_deterministic`,
|
||||
`test_rrf_signal_survives_post_merge_rerank`,
|
||||
`test_vec_absent_on_one_shard_records_partial_coverage`,
|
||||
`test_hybrid_label_does_not_silently_mean_fts5_only`,
|
||||
`test_accept_path_5_warranted_span_survives_low_title_overlap`,
|
||||
`test_accept_path_5_similarity_only_does_not_survive_low_title_overlap`,
|
||||
`test_exploratory_ungrounded_vec_candidate_cannot_increment_verified_count`,
|
||||
`test_rule8_rejects_wrong_franchise_semantic_neighbor`,
|
||||
`test_candidate_union_control_written_to_bench_journal`.
|
||||
|
||||
## 6. References
|
||||
|
||||
- #000039 — the parent (Phase 0 + Phase 1, closed 2026-05-12);
|
||||
|
|
|
|||
|
|
@ -218,6 +218,148 @@ quant recipe in `vec_backend_version`, not the output bytes (the same
|
|||
+ a pointer from #000050 §2 prereq #1 ("backfill once, distribute
|
||||
via #000051").
|
||||
|
||||
## 6a. Review amendments — 2026-05-13 (Dav1d de-novo review)
|
||||
|
||||
Fold these into the v1 spec before implementation. They do **not**
|
||||
change the architecture (soft sidecar, `leaf_hash`-keyed, cheap
|
||||
validation, never proof path) — they sharpen it.
|
||||
|
||||
1. **Narrower "harmless" wording.** A bad/adversarial vecpack cannot
|
||||
ground a claim, mutate hard roots, or forge an audit proof *if the
|
||||
hard/soft boundary holds* — but it is **not** "only a recall nudge".
|
||||
It can also bias candidate ranking, waste top-K on junk (hiding good
|
||||
candidates), inflate latency/context, and skew bench conclusions —
|
||||
i.e. a *denial-of-retrieval* surface. So: no proof-grade
|
||||
embedding-*correctness* verification; **yes** structural,
|
||||
compatibility, provenance, trust-tier, and import-integrity checks.
|
||||
|
||||
2. **Manifest + roots (integrity, not correctness).** A vecpack carries
|
||||
a canonical-JSON **manifest** — `format_version` (`vecpack-v1`),
|
||||
`vecpack_id = sha256(canonical manifest)`, `shard_root`,
|
||||
`vec_backend_version`, full `backend_config` (see #3), `count`,
|
||||
`entry_root`, `part_count`, `compression`, `producer_id`,
|
||||
`created_at`. Each row has a deterministic
|
||||
`row_hash = H(leaf_hash || vec_backend_version || quant_recipe ||
|
||||
embedding_blob)`; `entry_root = MerkleRoot(row_hashes)`. This is
|
||||
**transit integrity** ("did the bytes arrive unchanged?"), *not*
|
||||
correctness ("was this embedding actually computed by the declared
|
||||
model?") — #000051 needs the former, never the latter. Audit events
|
||||
cite `manifest_root` / `entry_root`, not millions of row hashes.
|
||||
|
||||
3. **`vec_backend_version` must commit the full reproducible input.**
|
||||
"embedding = f(chunk bytes, embedder config)" is under-specified.
|
||||
`backend_config` (in the manifest, summarized by the version token)
|
||||
must pin: `embedder_name`, `embedder_revision` (digest, not just a
|
||||
tag), `tokenizer_revision`, `preprocessing_policy` (which prose
|
||||
normalization — `chunk_text_base@v1`), `pooling_policy`,
|
||||
`normalization_policy` (`l2_normalize@v1`), `quantization_recipe`
|
||||
(`int8sym127_global_clamp_round@v1` — *named*, not bare `int8`),
|
||||
`dimension`, `dtype`, `distance_metric`, `ann_index`. The compact
|
||||
token (`vec-v1-bge-small-en-v1.5-384int8sym127-l2norm-flat`) is a
|
||||
derived summary; the manifest fields are the auditable payload.
|
||||
(Producer profile CPU-vs-GPU is *not* pinned — outputs aren't
|
||||
byte-stable across it and don't need to be; soft data.)
|
||||
|
||||
4. **Trust tiers.** Tier 0 = local file / unknown producer (import OK
|
||||
for raw vec search if structural checks pass; `vecpack_trust =
|
||||
untrusted`; **no cached hybrid QA**). Tier 1 = integrity-checked
|
||||
mesh sender (sender authenticated, manifest/roots check out; raw +
|
||||
experimental hybrid; cached QA still gated on #000050 policy).
|
||||
Tier 2 = trusted producer (identity accepted by local policy;
|
||||
eligible for cached hybrid QA once #000050 governance-hash wiring is
|
||||
live). Tier 3 = locally generated (strongest operational trust;
|
||||
still soft data, still `UNGROUNDED`). #000050 decides which tiers a
|
||||
*cached* hybrid answer may rest on. Trust tier is **audit metadata,
|
||||
never proof identity** — a trusted vecpack saying "this chunk is
|
||||
semantically close" is still just a *candidate proposal*; the
|
||||
verifier must inspect the text.
|
||||
|
||||
5. **Streaming, resumable import — never load 2.4 GB into RAM.**
|
||||
Manifest-first validation → shard/backend compat check → disk-budget
|
||||
check → stream entries part-by-part → stage valid rows in batches
|
||||
(1k–10k rows) → commit in bounded transactions (WAL, `executemany`,
|
||||
no per-row commits) → record progress → resume-or-rollback after
|
||||
interruption (idempotent: re-import is a no-op).
|
||||
|
||||
6. **Bad-row thresholds + duplicate policy.** `bad_row_rate == 0` →
|
||||
clean; `0 < rate ≤ 0.01%` → import valid rows, warn;
|
||||
`rate > 1%` → reject / quarantine the pack. `missing_chunk` is fine
|
||||
for partial packs (record `coverage_ratio`). `duplicate_leaf_hash`:
|
||||
identical blob → skip silently; conflicting blob → reject the pack
|
||||
(or row, by policy). Thresholds are policy knobs.
|
||||
|
||||
7. **Exception matrix (deterministic status for every failure).**
|
||||
Wrong `shard_root` → `reject_wrong_shard_root`; missing `chunk_vecs`
|
||||
→ `created_vec_meta`; existing `vec_meta` mismatch →
|
||||
`reject_backend_mismatch`; unsupported quant →
|
||||
`reject_unsupported_quant`; metric/quant incompat →
|
||||
`reject_metric_quant_mismatch`; bad blob length →
|
||||
`skipped_bad_length`; non-finite norm → `skipped_bad_norm`; norm
|
||||
out of sane range → `skipped_bad_norm_range`; missing local chunk →
|
||||
`skipped_missing_chunk`; dup identical → `duplicate_identical`; dup
|
||||
conflict → `duplicate_conflict`; mesh sig fail →
|
||||
`reject_signature_fail`; AEAD fail → `reject_aead_fail`; missing
|
||||
part → `incomplete_parts`; part-hash mismatch →
|
||||
`reject_part_hash_mismatch`; disk budget exceeded →
|
||||
`reject_insufficient_disk`; interrupted → `import_interrupted`;
|
||||
bad-row rate over threshold → `reject_bad_row_rate`; unknown
|
||||
producer → `untrusted_producer`. **Cache impact is `None` on every
|
||||
row** — vecpack import never touches `providence_cache`.
|
||||
|
||||
8. **Soft audit events.** Import/export/mesh-transfer write soft
|
||||
provenance events: `vecpack_export_{started,completed}`,
|
||||
`vecpack_import_{started,verified,completed,rejected,interrupted}`,
|
||||
`vecpack_part_{received,rejected}`,
|
||||
`vecpack_mesh_{announced,requested,served}` — each with
|
||||
`{vecpack_id, manifest_root, shard_root, vec_backend_version,
|
||||
trust_tier, counts:{received,loaded,skipped,bad}}`. Soft events,
|
||||
not proof events.
|
||||
|
||||
9. **CLI surface.** Add to §4's `export`/`import`:
|
||||
`arborist vecpack inspect --in path` (manifest only);
|
||||
`vecpack verify --shard <db> --in path` (all structural checks, no
|
||||
insert); `vecpack import --dry-run` (report expected status);
|
||||
`vecpack import --replace` (explicit rewrite, same backend only —
|
||||
never silently overwrite a *different* backend's vectors);
|
||||
`vecpack import --trust-policy {reject-unknown,untrusted-ok,trusted-only}`.
|
||||
|
||||
10. **SQD discipline now, π* module later.** Don't block on a
|
||||
`vecpack@v1` π* canonicalizer — but the manifest must already be
|
||||
canonical bytes: sorted keys, explicit scalar types, no implicit
|
||||
defaults in hash-bearing fields, versioned quant recipe, stable
|
||||
`manifest_root` / `entry_root`. Formalize as
|
||||
`pi_star_ref ∈ {vecpack_manifest@v1, vecpack_entry@v1,
|
||||
vec_backend_config@v1}` later.
|
||||
|
||||
11. **Metadata #000050 consumes** (for run-DAG, audit, replay — never
|
||||
proof): `{vecpack_id, vecpack_manifest_root, vec_backend_version,
|
||||
coverage_ratio, trust_tier, import_event_hash}`.
|
||||
|
||||
**Test additions** (append to §6.6):
|
||||
`test_vecpack_manifest_canonical_hash_stable`,
|
||||
`test_vecpack_export_import_cross_build_leaf_hash_mapping`,
|
||||
`test_vecpack_import_rejects_{wrong_shard_root,backend_version_mismatch,metric_quant_mismatch}`,
|
||||
`test_vecpack_import_skips_{missing_leaf_hash,bad_blob_length,nonfinite_vector}`,
|
||||
`test_vecpack_import_rejects_duplicate_leaf_conflict`,
|
||||
`test_vecpack_import_idempotent_same_pack`,
|
||||
`test_vecpack_import_resume_after_interruption`,
|
||||
`test_vecpack_import_respects_bad_row_threshold`,
|
||||
`test_vecpack_import_rejects_insufficient_disk_budget`,
|
||||
`test_vecpack_mesh_envelope_signature_failure_rejects`,
|
||||
`test_vecpack_mesh_part_reassembly_requires_all_parts`,
|
||||
`test_vecpack_bad_embedding_cannot_increment_verified_evidence_count`,
|
||||
`test_vecpack_import_writes_soft_audit_event_only`,
|
||||
`test_vecpack_import_does_not_modify_document_root_or_cache_key`.
|
||||
The last two are the hard/soft-boundary proof.
|
||||
|
||||
**Implementation order:** (1) file `vecpack export`/`import` +
|
||||
manifest/roots; (2) structural validation + soft audit events; (3)
|
||||
cross-build `leaf_hash`-mapping tests; (4) streaming/resume; (5) mesh
|
||||
payload transport; (6) #000050 integration via vecpack metadata.
|
||||
Default safety throughout: imported vecpacks support **raw vec search**;
|
||||
**cached hybrid QA stays blocked** until #000050's governance-hash
|
||||
wiring + cache-write guard land.
|
||||
|
||||
## 7. References
|
||||
|
||||
- #000039 — the local vec backend (Phase 0 + Phase 1, closed
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue