diff --git a/docs/TICKETS.md b/docs/TICKETS.md index e2d2065..95d5910 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -111,6 +111,7 @@ Newest first. Update on every open/close. | ID | Title | Status | Opened | Directive | |----------|------------------------------------------------|-----------------------|------------|-----------| +| #000066 | Cold-pack overlay / graft mode (pack-as-package, witness-pattern audit chain) | **scaffold-only · awaiting go/no-go** (2026-05-26; surfaced while running #000065 reshard, fox extension: "we could envision a pack for wikipedia 2010, wikipedia current, etc"). Extend #000061 cold-pack hydration with a second mode: overlay an existing pack onto a populated shard set instead of hydrating into empty. Doc/chunk/edge/concept overlay is trivial (`INSERT OR IGNORE` on content-addressed PKs collapses dupes); FTS5 overlay is trivial (new chunks → new rowids → new FTS rows). The interesting part is the audit chain — can't naively append the pack's events because `prev_event_hash` linkage breaks across the join. Chosen approach: **graft receipt**. Append one new `event_type='graft'` event to the host chain carrying `(pack_hash, snapshot_root, corpus_name, event_count, first_event_hash, last_event_hash, manifest_root)`; the pack file itself becomes the durable witness for the absorbed events (anyone can re-fetch the pack, walk its internal chain, and verify it matches the receipt). Host chain stays linear; pack chain is a "witnessed subgraph." This is the same witness pattern Merkle-AGI v8/v9 is heading toward, but bought at near-zero schema cost. Rejected alternatives: re-chain everything (breaks external refs to old event_hashes — cache_keys anchoring to old `audit_event_hash`, snapshots, etc. — silently invalid); chain forest with new `chain_id` column (right answer when graft dominates the lifecycle, but premature now). **Pack-as-package extension** (fox 2026-05-26): each pack carries a `corpus_name` field in its manifest (`wikipedia-2010`, `wikipedia-current`, `arxiv-cs`, `textbooks-undergrad`, …) so operators pick which corpora to graft — `arborist cold graft wikipedia-current` becomes as natural as `apt install firefox`. Multiple packs of the same corpus name: most-recent `snapshot_root` wins; older packs stay in the bucket until GC. URI conflicts across corpora (e.g., `wikipedia.org/wiki/Foo` in both 2010 and current): different content → different `document_root` → both stored, `supersedes` edges per CLAUDE.md invariant. Providence-cache conflicts: same `cache_key` with different answer → existing v9.8 falsification framework handles it (`state='stale'` or `quarantined`). Mesh-peer-corpus-merge: each peer's pack is a graftable package; partition reconciliation becomes "exchange the packs you each carry, graft what you lack". The mesh-of-arborists semantic. Sequence: (1) `corpus_name` field in #000061 manifest format + alias index in bucket (`corpora//latest.json` pointer to active pack_hash); (2) `arborist cold graft ` / `arborist cold graft --corpus ` mode in evict.py — read pack, INSERT OR IGNORE per-table, emit graft receipt; (3) conflict-policy flag (`--on-uri-conflict {supersedes,skip,fail}`, default `supersedes`); (4) `arborist cold list-corpora` shows available packages in a bucket. Scaffold first, code only when (a) #000065 reshard lands and stabilises (b) a second corpus exists (the wikipedia-current snapshot, or first textbook bundle ready to graft onto wikipedia-2010 base) (c) at least two peers want to exchange. | 2026-05-26 | — | | #000065 | Canonical shard count `M` + content-hash routing (decouple ingest parallelism from ATTACH ceiling) | **open · scaffold + design · awaiting go/no-go** (2026-05-26; surfaced while sizing #000061's federation story). Today shard count conflates two roles: producer ingest parallelism (wants vCPU count) + consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10 on stock python3 sqlite3). Producer with 16 vCPU → 16 shards → consumers fail to attach the 11th. Producer with 4 shards → 16-vCPU box runs 75% idle on ingest. Fix: pin a corpus-wide canonical **M = 4** (decided 2026-05-26 from real-Wikipedia bench: M=4 captures 92% of peak ingest throughput, ATTACH cost 9 ms keeps mobile-tolerable, 6 free ATTACH slots under SQLite's 10 ceiling for auxiliary DBs), introduce N (ingest workers) decoupled from M. Document → shard assignment becomes content-deterministic: `shard_idx = int(document_root[:8], 16) % M`. Same input → same output across every peer (today's "spray by ingest order" is non-deterministic across peers, a real federation weakness). Migration hard-constraint per fox: **content-addressed rebalance, NOT re-ingest** — every row is already addressed by `document_root` / `leaf_hash` / etc.; migration reads rows from the current 4 shards, computes each row's new shard via the routing function, INSERTs into M new shards. No source re-parse, no re-canonicalization, no re-chunking, no LLM. ~20–40 min I/O-bound vs. hours-to-days for true re-ingest. Audit chain consolidates to canonical shard 000 (re-numbered + re-hashed once) to preserve global event ordering. Phases: 0 design lock + pin M in meta table → 1 read path (connect_query honors M) → 2 ingest path (multi-shard write per worker) → 3 cold-pack restore re-routes on pull → 4 corpus migration tool. Open audit-chain re-numbering question (every shard has its own seq + event_hash; rebalancing splits a producer's chain across M consumer shards). Don't proliferate sub-tickets; the audit handling is part of this design lock. Out of scope: custom-built sqlite3 with higher MAX_ATTACHED (rejected: violates "python3 + venv + sqlite3 only" property from CLAUDE.md); topic-clustering shards (would break ingest determinism). | 2026-05-26 | — | | #000064 | Cold-object operations toolkit (verify/diff/doctor/repair-fts/gc-plan + audit taxonomy) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §11/§12/§14). Operator-facing observability + repair tools on top of #000061: `cold verify` (sample/full integrity check), `cold diff` (local vs remote manifest), `cold doctor` (one-shot health: connectivity / credentials / manifest age / missing-object count / tamper sample / audit-chain integrity), `cold repair-fts` (rebuild FTS5 from chunks.content), `cold gc-plan` (orphan bucket objects, read-only by default — destructive only with `--apply` + confirm). Plus expanded audit-event taxonomy: per-PUT/HEAD/GET success/failure events, manifest-pointer events, verify/doctor/gc events. All read-mostly; destructive ops require `--apply`. Bundled so the audit-taxonomy gets one design pass instead of five-way drift. Sequence: doctor → verify → diff → repair-fts → gc-plan. No code until #000061 closes. | 2026-05-26 | — | | #000063 | Cold-object private-ciphertext mode (mesh-keyed object keys) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §9 / response A §13.3). Adds private mode to #000061 cold-object format so chunk bodies + manifest can be uploaded to public-read bucket without leaking corpus membership. Two strategies: (A) deterministic `object_key = HMAC(group_key, leaf_hash)` + AEAD-encrypted body — supports lookup-by-leaf-hash given the key; (B) random-key ciphertext + encrypted private manifest — stronger membership hiding, needs manifest fetch first. Strategy A default; B opt-in. Group key from existing `arborist/mesh/crypto.py`; pack manifest carries `epoch_id` for rotation. Verifier path unchanged: consumer decrypts, then `hash_leaf(plaintext) == leaf_hash` as in public mode. No code until (1) a real non-public corpus needs cold-object shipping, (2) mesh group-key ABI is stable enough to reference, (3) threat-model split between A vs B is settled by real adversary. | 2026-05-26 | — | @@ -179,4 +180,4 @@ Newest first. Update on every open/close. ## Next ID -`000066` +`000067` diff --git a/docs/tickets/ticket-000066-cold-pack-overlay-graft-mode.md b/docs/tickets/ticket-000066-cold-pack-overlay-graft-mode.md new file mode 100644 index 0000000..79c6c07 --- /dev/null +++ b/docs/tickets/ticket-000066-cold-pack-overlay-graft-mode.md @@ -0,0 +1,328 @@ +# Ticket #000066 — Cold-pack overlay / graft mode (pack-as-package) + +**Opened:** 2026-05-26 +**Status:** scaffold-only · awaiting go/no-go +**Origin:** surfaced while running #000065 reshard cutover, fox extension + "we could envision a pack for wikipedia 2010, wikipedia current, etc" +**Related:** #000061 (cold-pack distribution tier — base mechanism), + #000065 (canonical shard count + content-hash routing — the + consumer side this graft mode plugs into), #000014/#000017 + (witness pattern alignment), v8/v9 substrate (chain forest as the + long-game upgrade path) + +## 1. The capability gap + +#000061 cold-pack hydration today is **one-shot, into-empty**. Pull a +pack from the bucket, decompress, write rows into target shards. +There is no way to take a pack and **merge** it onto an existing +populated shard set. + +That gap blocks three real workflows: + +1. **Pack-as-package** — operator picks which corpora to graft: + `arborist cold graft wikipedia-current` should feel like + `apt install firefox`. Each pack is a named bundle + (`wikipedia-2010`, `wikipedia-current`, `arxiv-cs`, + `textbooks-undergrad`, …). Operators compose corpora. + +2. **Incremental extension** — a peer already has the + `wikipedia-2010` baseline and wants to add `textbooks-undergrad` + without re-bootstrapping. Today they'd have to re-hydrate from + scratch. + +3. **Mesh-peer-corpus-merge** — two peers diverge over a partition. + Each has packs the other lacks. Reconciliation = exchange + graft + what's missing. This is what makes a *mesh* of arborists rather + than a *fleet*. + +## 2. What's actually hard + +Three concerns. Two trivial, one interesting. + +### 2.1 Doc / chunk / edge overlay (trivial) + +Every per-document table is keyed by content-addressed primary keys +(`document_root` is `sha256(merkle_root)`, `chunks` is unique on +`(document_root, idx)`, `edges` is the wide tuple PK). `INSERT OR +IGNORE` collapses dupes naturally — identical content = identical +row = no-op. No conflict resolution needed for the proof-bearing +tables. + +The only non-trivial sub-case: **URI conflicts**. A pack from +`wikipedia-current` carries `https://en.wikipedia.org/wiki/Obama` +with a different `document_root` than the same URI in the +`wikipedia-2010` baseline (different fetch time, different content). +Both rows get stored — different `document_root` means they're +different documents at the Merkle layer. A `supersedes` edge from +old to new captures the lineage (per CLAUDE.md schema invariants). + +Conflict-policy flag for the operator: +``` +--on-uri-conflict {supersedes,skip,fail} default: supersedes +``` +- `supersedes`: store both, emit `supersedes` edge old→new +- `skip`: keep existing, drop incoming (legacy-corpus-priority) +- `fail`: stop the graft, report the conflict count + +### 2.2 FTS5 overlay (trivial) + +Contentless FTS5 indexes (`chunks_fts`, `documents_fts`) are +populated by `INSERT INTO chunks_fts (rowid, content)`. New chunks +get new `chunk_id` values (the AUTOINCREMENT PK) → new FTS rowids → +new FTS rows. Existing FTS rows are untouched. Idempotent. + +### 2.3 Audit chain overlay (the interesting one) + +The audit chain is **linear**: +``` +event_hash = sha256(prev_event_hash || canonical(body)) +``` + +Naively appending the pack's events to the host chain doesn't work +because the pack's `prev_event_hash` linkages reference events that +exist only inside the pack, not in the host chain. + +Three coherent ways to handle this, ranked by cost: + +#### Approach A — Graft receipt (chosen) + +Append exactly **one** new audit event to the host chain: + +```json +{ + "event_type": "graft", + "body": { + "kind": "graft", + "pack_hash": "", + "snapshot_root": "", + "corpus_name": "wikipedia-current", + "manifest_root": "", + "event_count": 1_245_678, + "first_event_hash": "", + "last_event_hash": "", + "documents_imported": 866_881, + "chunks_imported": 1_560_021, + "edges_imported": 22_664_792, + "on_uri_conflict_policy": "supersedes", + "uri_conflicts_resolved": 4_321 + }, + "ts": +} +``` + +The host's chain stays linear. The pack file itself is the durable +witness for the absorbed events — anyone can: + +1. Re-fetch the pack by its `pack_hash`. +2. Walk the pack's internal audit chain. +3. Verify `first_event_hash` / `last_event_hash` / `event_count` + match the receipt. + +This is the same **witness pattern** v8/v9 substrate is converging +toward (mechanistic witnesses #000062, falsification proposals +#000037, etc.) — content-addressed evidence held outside the host +chain, referenced via a hash binding in the receipt event. + +Cost: ~zero schema change. One new `event_type='graft'`. Done. + +#### Approach B — Re-chain everything (rejected) + +Read both chains, merge by `ts`, recompute every `event_hash`. This +is what the #000065 Option-A reshard does today. + +**Why rejected for graft**: the reshard is a one-time topology +change with no external references to the audit chain. Graft is +expected to be frequent (every package install). Re-chaining +invalidates every external reference to old `event_hash` values — +cache_keys anchored to `audit_event_hash`, snapshots, falsification +records — and silently produces stale pointers. + +#### Approach C — Chain forest (rejected for now) + +Add a `chain_id` column to `audit_events`. Each pack becomes a new +chain in the forest. Cross-chain Merkle anchors bind one chain's +root into another's event body. + +**Why rejected for now**: this is the *right answer when graft is +the common case*, but the schema migration cost is high (every +chain-walker code path needs to handle multi-chain), and the +receipt-pattern (Approach A) gets us the capability at a fraction +of the cost while leaving the door open to upgrade. + +When to revisit Approach C: when graft frequency exceeds reshard +frequency by ≥10×, or when multi-peer corpus-merging produces +enough audit-chain-fan-in to make linear-chain queries painful. + +## 3. Pack-as-package surface + +### 3.1 Manifest extension + +Add one field to the #000061 manifest: + +```json +{ + ...existing fields..., + "corpus_name": "wikipedia-current", + "corpus_version": "2026-05-26", + "previous_pack_hash": "" +} +``` + +`corpus_name` is operator-set at pack time. `corpus_version` is the +operator's free-form label. `previous_pack_hash` chains successive +versions of the same corpus so an operator can walk "show me the +history of `wikipedia-current` in this bucket." + +### 3.2 Bucket alias index + +``` +corpora/ + wikipedia-2010/latest.json → {"pack_hash": "abc...", "as_of": "..."} + wikipedia-current/latest.json → {"pack_hash": "def...", "as_of": "..."} + textbooks-undergrad/latest.json +``` + +`arborist cold graft --corpus wikipedia-current` does a single HTTPS +GET against `corpora/wikipedia-current/latest.json` to find the +current pack_hash, then pulls and grafts. Same pattern as +`manifest/latest.json` (the #000061 gap-1 pointer). + +### 3.3 CLI surface + +``` +# install a named corpus +arborist cold graft --corpus wikipedia-current + +# install a specific pack (skip the alias indirection) +arborist cold graft + +# enumerate available packages in a bucket +arborist cold list-corpora + +# see which corpora are already grafted on this peer +arborist corpus list-grafted + +# conflict policy +arborist cold graft --corpus wikipedia-current \ + --on-uri-conflict supersedes +``` + +## 4. Mesh-peer-corpus-merge as the long-game payoff + +The graft mode is what makes "mesh of arborists" coherent. Two +peers, A and B, after a partition: + +1. A has packs `[wikipedia-2010, arxiv-cs]`. +2. B has packs `[wikipedia-2010, textbooks-undergrad]`. +3. They reconcile: A pulls `textbooks-undergrad`, grafts. B pulls + `arxiv-cs`, grafts. +4. Each now has all three. The graft receipts in each peer's audit + chain record exactly when + from where each package landed. + +No coordination protocol needed beyond "exchange your pack lists, +graft what you lack." Mesh state is the union of locally-stored +graft receipts. + +## 5. Sequence + +Code only when **all three** are true: + +1. #000065 reshard lands and stabilises. +2. A second corpus exists — either the `wikipedia-current` snapshot + captured from a real Wikipedia dump, or the first textbook bundle + ready to graft onto the `wikipedia-2010` base. (Until then there's + nothing to test against.) +3. At least two peers (real or simulated) want to exchange packages. + +Phases when it ships: + +1. **Manifest extension** — `corpus_name`, `corpus_version`, + `previous_pack_hash` in pack format. Backward-compatible + (existing packs default to `corpus_name="unknown"`). +2. **Bucket alias index** — `corpora//latest.json` pointer + + `put_latest_corpus_pointer` / `get_latest_corpus_pointer` helpers + in `arborist/cold_object.py` (mirrors gap-1's + `manifest/latest.json` pattern). +3. **Graft executor** — `arborist/evict.py:graft_pack(pack_hash, + on_uri_conflict)` reads pack, runs INSERT OR IGNORE per-table, + emits a single `event_type='graft'` audit event. +4. **CLI** — `arborist cold graft`, `arborist cold list-corpora`, + `arborist corpus list-grafted`. +5. **Conflict-policy tests** — three URI-conflict paths + (`supersedes`, `skip`, `fail`), validated on a tiny synthetic + two-corpus fixture. +6. **Mesh integration test** — simulate two peers, run reconcile, + verify both arrive at identical post-graft `snapshot_root`. + +## 6. Out of scope + +- **Custom audit-chain re-numbering** (Approach B above). +- **Chain forest schema migration** (Approach C above) — see + upgrade-path note in §2.3 if graft frequency demands it. +- **Cross-peer conflict resolution for `providence_cache`** — the + v9.8 falsification framework already handles this + (`state='stale'` / `quarantined`). Graft inherits, doesn't + reinvent. +- **Federated package discovery** ("which buckets host + `arxiv-cs`?") — operator-driven for now, manual bucket + configuration. A discovery layer can come later if it earns its + place. +- **Signed packages** — the pack file is already content-addressed + via `pack_hash` (#000061). Signature-based authenticity adds an + optional integrity layer; defer until a real adversary model + demands it. + +## 7. Witness-pattern alignment + +The graft receipt is structurally identical to the +content-addressed-evidence pattern Merkle-AGI v8/v9 is converging +on: + +- Mechanistic witnesses (#000062) — body holds a + `MechanisticWitnessRoot`, the captured artifacts live outside + the chain. +- Falsification proposals (#000037) — body holds the failure + fingerprint, the controller's full reasoning lives outside. +- Cold-pack graft (this ticket) — body holds `pack_hash`, the + imported events live in the pack file outside. + +In every case the pattern is: **host chain stays linear and small; +durable evidence is content-addressed and externalised; the chain +references the evidence by hash; anyone can verify the binding by +re-fetching the evidence.** + +This is intentional. The substrate is moving toward "audit chain as +index of witnesses" rather than "audit chain as transcript of +everything that ever happened." Graft mode is one more node in that +arc. + +## 8. Open questions + +- **Corpus-name namespace**: who owns `wikipedia-current`? Today + packs are operator-self-attested (the operator who builds the + pack picks the `corpus_name`). For mesh-peer-corpus-merge this is + fine within a trust group. For public bucket distribution + (Patch-the-Planet), name squatting becomes a concern eventually. + Defer until that's a real problem. +- **Graft idempotence**: grafting the same `pack_hash` twice should + be a no-op (every row already exists). Verify in tests; emit one + receipt or zero on the second attempt? +- **Receipt visibility for falsification**: if a grafted pack + carries content that the host later falsifies, the falsification + audit event references a `subject_root` from the grafted content. + Forensic recoverability of "which graft brought this in" requires + joining `falsifications.subject_root` ↔ `graft.documents_imported`. + Probably wants a tiny helper, not a schema change. + +## 9. References + +- #000061 (`docs/cold-object-store.md`) — base cold-pack format + this ticket extends. +- #000065 (`docs/tickets/ticket-000065-canonical-shard-count-content-hash-routing.md`) + — the consumer side; graft writes into M hash-routed shards via + the same `shard_for_document` routing. +- v8/v9 substrate Witness pattern — see #000062 §4.7 for the + mechanistic-witness instantiation of the same shape. +- CLAUDE.md "schema invariants" — `supersedes` edges, idempotent + re-ingest, content-addressed PKs — all already accommodate graft + semantically; this ticket just makes the workflow first-class.