#000066: cold-pack overlay/graft mode (pack-as-package)

Scaffold-only ticket. Captures the architecture for taking the
#000061 cold-pack format and adding a second mode beside hydrate:
overlay an existing pack onto a populated shard set ("graft").

Surfaced while running the #000065 reshard cutover and fox extended
the design: each pack carries a `corpus_name` field
(wikipedia-2010, wikipedia-current, arxiv-cs, ...), making
`arborist cold graft wikipedia-current` feel like `apt install`.

Three concerns analysed:
  doc/chunk/edge overlay   trivial (INSERT OR IGNORE on content-
                           addressed PKs collapses dupes)
  FTS5 overlay             trivial (new chunk_ids → new fts rows)
  audit chain overlay      the only hard part — three approaches:
                           A graft receipt (chosen): one event in
                             host chain carrying pack_hash +
                             event_count + first/last hashes; pack
                             file is the durable witness; zero
                             schema cost; aligned with v8/v9
                             witness pattern
                           B re-chain everything: rejected — graft
                             is frequent so invalidating external
                             refs is wrong tradeoff (different
                             story from the one-time reshard)
                           C chain forest with chain_id col: right
                             answer when graft dominates lifecycle
                             but premature now

Long-game payoff: mesh-peer-corpus-merge. Two peers diverge over a
partition, each carries packs the other lacks, reconciliation =
exchange + graft what's missing. Makes "mesh of arborists"
coherent rather than "fleet of arborists."

Scaffold gated on (a) #000065 lands+stabilises, (b) a second
corpus exists to graft, (c) at least two peers want to exchange.
No code until then; the design lock is what the ticket buys.

Index bumped Next ID 000066→000067.
This commit is contained in:
russell@unturf.com 2026-05-26 14:09:07 -04:00
parent c26d03956d
commit f4397a9217
No known key found for this signature in database
2 changed files with 330 additions and 1 deletions

View file

@ -111,6 +111,7 @@ Newest first. Update on every open/close.
| ID | Title | Status | Opened | Directive |
|----------|------------------------------------------------|-----------------------|------------|-----------|
| #000066 | Cold-pack overlay / graft mode (pack-as-package, witness-pattern audit chain) | **scaffold-only · awaiting go/no-go** (2026-05-26; surfaced while running #000065 reshard, fox extension: "we could envision a pack for wikipedia 2010, wikipedia current, etc"). Extend #000061 cold-pack hydration with a second mode: overlay an existing pack onto a populated shard set instead of hydrating into empty. Doc/chunk/edge/concept overlay is trivial (`INSERT OR IGNORE` on content-addressed PKs collapses dupes); FTS5 overlay is trivial (new chunks → new rowids → new FTS rows). The interesting part is the audit chain — can't naively append the pack's events because `prev_event_hash` linkage breaks across the join. Chosen approach: **graft receipt**. Append one new `event_type='graft'` event to the host chain carrying `(pack_hash, snapshot_root, corpus_name, event_count, first_event_hash, last_event_hash, manifest_root)`; the pack file itself becomes the durable witness for the absorbed events (anyone can re-fetch the pack, walk its internal chain, and verify it matches the receipt). Host chain stays linear; pack chain is a "witnessed subgraph." This is the same witness pattern Merkle-AGI v8/v9 is heading toward, but bought at near-zero schema cost. Rejected alternatives: re-chain everything (breaks external refs to old event_hashes — cache_keys anchoring to old `audit_event_hash`, snapshots, etc. — silently invalid); chain forest with new `chain_id` column (right answer when graft dominates the lifecycle, but premature now). **Pack-as-package extension** (fox 2026-05-26): each pack carries a `corpus_name` field in its manifest (`wikipedia-2010`, `wikipedia-current`, `arxiv-cs`, `textbooks-undergrad`, …) so operators pick which corpora to graft — `arborist cold graft wikipedia-current` becomes as natural as `apt install firefox`. Multiple packs of the same corpus name: most-recent `snapshot_root` wins; older packs stay in the bucket until GC. URI conflicts across corpora (e.g., `wikipedia.org/wiki/Foo` in both 2010 and current): different content → different `document_root` → both stored, `supersedes` edges per CLAUDE.md invariant. Providence-cache conflicts: same `cache_key` with different answer → existing v9.8 falsification framework handles it (`state='stale'` or `quarantined`). Mesh-peer-corpus-merge: each peer's pack is a graftable package; partition reconciliation becomes "exchange the packs you each carry, graft what you lack". The mesh-of-arborists semantic. Sequence: (1) `corpus_name` field in #000061 manifest format + alias index in bucket (`corpora/<name>/latest.json` pointer to active pack_hash); (2) `arborist cold graft <pack_hash>` / `arborist cold graft --corpus <name>` mode in evict.py — read pack, INSERT OR IGNORE per-table, emit graft receipt; (3) conflict-policy flag (`--on-uri-conflict {supersedes,skip,fail}`, default `supersedes`); (4) `arborist cold list-corpora` shows available packages in a bucket. Scaffold first, code only when (a) #000065 reshard lands and stabilises (b) a second corpus exists (the wikipedia-current snapshot, or first textbook bundle ready to graft onto wikipedia-2010 base) (c) at least two peers want to exchange. | 2026-05-26 | — |
| #000065 | Canonical shard count `M` + content-hash routing (decouple ingest parallelism from ATTACH ceiling) | **open · scaffold + design · awaiting go/no-go** (2026-05-26; surfaced while sizing #000061's federation story). Today shard count conflates two roles: producer ingest parallelism (wants vCPU count) + consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10 on stock python3 sqlite3). Producer with 16 vCPU → 16 shards → consumers fail to attach the 11th. Producer with 4 shards → 16-vCPU box runs 75% idle on ingest. Fix: pin a corpus-wide canonical **M = 4** (decided 2026-05-26 from real-Wikipedia bench: M=4 captures 92% of peak ingest throughput, ATTACH cost 9 ms keeps mobile-tolerable, 6 free ATTACH slots under SQLite's 10 ceiling for auxiliary DBs), introduce N (ingest workers) decoupled from M. Document → shard assignment becomes content-deterministic: `shard_idx = int(document_root[:8], 16) % M`. Same input → same output across every peer (today's "spray by ingest order" is non-deterministic across peers, a real federation weakness). Migration hard-constraint per fox: **content-addressed rebalance, NOT re-ingest** — every row is already addressed by `document_root` / `leaf_hash` / etc.; migration reads rows from the current 4 shards, computes each row's new shard via the routing function, INSERTs into M new shards. No source re-parse, no re-canonicalization, no re-chunking, no LLM. ~2040 min I/O-bound vs. hours-to-days for true re-ingest. Audit chain consolidates to canonical shard 000 (re-numbered + re-hashed once) to preserve global event ordering. Phases: 0 design lock + pin M in meta table → 1 read path (connect_query honors M) → 2 ingest path (multi-shard write per worker) → 3 cold-pack restore re-routes on pull → 4 corpus migration tool. Open audit-chain re-numbering question (every shard has its own seq + event_hash; rebalancing splits a producer's chain across M consumer shards). Don't proliferate sub-tickets; the audit handling is part of this design lock. Out of scope: custom-built sqlite3 with higher MAX_ATTACHED (rejected: violates "python3 + venv + sqlite3 only" property from CLAUDE.md); topic-clustering shards (would break ingest determinism). | 2026-05-26 | — |
| #000064 | Cold-object operations toolkit (verify/diff/doctor/repair-fts/gc-plan + audit taxonomy) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §11/§12/§14). Operator-facing observability + repair tools on top of #000061: `cold verify` (sample/full integrity check), `cold diff` (local vs remote manifest), `cold doctor` (one-shot health: connectivity / credentials / manifest age / missing-object count / tamper sample / audit-chain integrity), `cold repair-fts` (rebuild FTS5 from chunks.content), `cold gc-plan` (orphan bucket objects, read-only by default — destructive only with `--apply` + confirm). Plus expanded audit-event taxonomy: per-PUT/HEAD/GET success/failure events, manifest-pointer events, verify/doctor/gc events. All read-mostly; destructive ops require `--apply`. Bundled so the audit-taxonomy gets one design pass instead of five-way drift. Sequence: doctor → verify → diff → repair-fts → gc-plan. No code until #000061 closes. | 2026-05-26 | — |
| #000063 | Cold-object private-ciphertext mode (mesh-keyed object keys) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §9 / response A §13.3). Adds private mode to #000061 cold-object format so chunk bodies + manifest can be uploaded to public-read bucket without leaking corpus membership. Two strategies: (A) deterministic `object_key = HMAC(group_key, leaf_hash)` + AEAD-encrypted body — supports lookup-by-leaf-hash given the key; (B) random-key ciphertext + encrypted private manifest — stronger membership hiding, needs manifest fetch first. Strategy A default; B opt-in. Group key from existing `arborist/mesh/crypto.py`; pack manifest carries `epoch_id` for rotation. Verifier path unchanged: consumer decrypts, then `hash_leaf(plaintext) == leaf_hash` as in public mode. No code until (1) a real non-public corpus needs cold-object shipping, (2) mesh group-key ABI is stable enough to reference, (3) threat-model split between A vs B is settled by real adversary. | 2026-05-26 | — |
@ -179,4 +180,4 @@ Newest first. Update on every open/close.
## Next ID
`000066`
`000067`

View file

@ -0,0 +1,328 @@
# Ticket #000066 — Cold-pack overlay / graft mode (pack-as-package)
**Opened:** 2026-05-26
**Status:** scaffold-only · awaiting go/no-go
**Origin:** surfaced while running #000065 reshard cutover, fox extension
"we could envision a pack for wikipedia 2010, wikipedia current, etc"
**Related:** #000061 (cold-pack distribution tier — base mechanism),
#000065 (canonical shard count + content-hash routing — the
consumer side this graft mode plugs into), #000014/#000017
(witness pattern alignment), v8/v9 substrate (chain forest as the
long-game upgrade path)
## 1. The capability gap
#000061 cold-pack hydration today is **one-shot, into-empty**. Pull a
pack from the bucket, decompress, write rows into target shards.
There is no way to take a pack and **merge** it onto an existing
populated shard set.
That gap blocks three real workflows:
1. **Pack-as-package** — operator picks which corpora to graft:
`arborist cold graft wikipedia-current` should feel like
`apt install firefox`. Each pack is a named bundle
(`wikipedia-2010`, `wikipedia-current`, `arxiv-cs`,
`textbooks-undergrad`, …). Operators compose corpora.
2. **Incremental extension** — a peer already has the
`wikipedia-2010` baseline and wants to add `textbooks-undergrad`
without re-bootstrapping. Today they'd have to re-hydrate from
scratch.
3. **Mesh-peer-corpus-merge** — two peers diverge over a partition.
Each has packs the other lacks. Reconciliation = exchange + graft
what's missing. This is what makes a *mesh* of arborists rather
than a *fleet*.
## 2. What's actually hard
Three concerns. Two trivial, one interesting.
### 2.1 Doc / chunk / edge overlay (trivial)
Every per-document table is keyed by content-addressed primary keys
(`document_root` is `sha256(merkle_root)`, `chunks` is unique on
`(document_root, idx)`, `edges` is the wide tuple PK). `INSERT OR
IGNORE` collapses dupes naturally — identical content = identical
row = no-op. No conflict resolution needed for the proof-bearing
tables.
The only non-trivial sub-case: **URI conflicts**. A pack from
`wikipedia-current` carries `https://en.wikipedia.org/wiki/Obama`
with a different `document_root` than the same URI in the
`wikipedia-2010` baseline (different fetch time, different content).
Both rows get stored — different `document_root` means they're
different documents at the Merkle layer. A `supersedes` edge from
old to new captures the lineage (per CLAUDE.md schema invariants).
Conflict-policy flag for the operator:
```
--on-uri-conflict {supersedes,skip,fail} default: supersedes
```
- `supersedes`: store both, emit `supersedes` edge old→new
- `skip`: keep existing, drop incoming (legacy-corpus-priority)
- `fail`: stop the graft, report the conflict count
### 2.2 FTS5 overlay (trivial)
Contentless FTS5 indexes (`chunks_fts`, `documents_fts`) are
populated by `INSERT INTO chunks_fts (rowid, content)`. New chunks
get new `chunk_id` values (the AUTOINCREMENT PK) → new FTS rowids →
new FTS rows. Existing FTS rows are untouched. Idempotent.
### 2.3 Audit chain overlay (the interesting one)
The audit chain is **linear**:
```
event_hash = sha256(prev_event_hash || canonical(body))
```
Naively appending the pack's events to the host chain doesn't work
because the pack's `prev_event_hash` linkages reference events that
exist only inside the pack, not in the host chain.
Three coherent ways to handle this, ranked by cost:
#### Approach A — Graft receipt (chosen)
Append exactly **one** new audit event to the host chain:
```json
{
"event_type": "graft",
"body": {
"kind": "graft",
"pack_hash": "<sha256 of pack file>",
"snapshot_root": "<corpus snapshot root the pack pins>",
"corpus_name": "wikipedia-current",
"manifest_root": "<sha256 of manifest>",
"event_count": 1_245_678,
"first_event_hash": "<pack's chain genesis>",
"last_event_hash": "<pack's chain head>",
"documents_imported": 866_881,
"chunks_imported": 1_560_021,
"edges_imported": 22_664_792,
"on_uri_conflict_policy": "supersedes",
"uri_conflicts_resolved": 4_321
},
"ts": <ingest time>
}
```
The host's chain stays linear. The pack file itself is the durable
witness for the absorbed events — anyone can:
1. Re-fetch the pack by its `pack_hash`.
2. Walk the pack's internal audit chain.
3. Verify `first_event_hash` / `last_event_hash` / `event_count`
match the receipt.
This is the same **witness pattern** v8/v9 substrate is converging
toward (mechanistic witnesses #000062, falsification proposals
#000037, etc.) — content-addressed evidence held outside the host
chain, referenced via a hash binding in the receipt event.
Cost: ~zero schema change. One new `event_type='graft'`. Done.
#### Approach B — Re-chain everything (rejected)
Read both chains, merge by `ts`, recompute every `event_hash`. This
is what the #000065 Option-A reshard does today.
**Why rejected for graft**: the reshard is a one-time topology
change with no external references to the audit chain. Graft is
expected to be frequent (every package install). Re-chaining
invalidates every external reference to old `event_hash` values —
cache_keys anchored to `audit_event_hash`, snapshots, falsification
records — and silently produces stale pointers.
#### Approach C — Chain forest (rejected for now)
Add a `chain_id` column to `audit_events`. Each pack becomes a new
chain in the forest. Cross-chain Merkle anchors bind one chain's
root into another's event body.
**Why rejected for now**: this is the *right answer when graft is
the common case*, but the schema migration cost is high (every
chain-walker code path needs to handle multi-chain), and the
receipt-pattern (Approach A) gets us the capability at a fraction
of the cost while leaving the door open to upgrade.
When to revisit Approach C: when graft frequency exceeds reshard
frequency by ≥10×, or when multi-peer corpus-merging produces
enough audit-chain-fan-in to make linear-chain queries painful.
## 3. Pack-as-package surface
### 3.1 Manifest extension
Add one field to the #000061 manifest:
```json
{
...existing fields...,
"corpus_name": "wikipedia-current",
"corpus_version": "2026-05-26",
"previous_pack_hash": "<prior pack of same corpus, or null>"
}
```
`corpus_name` is operator-set at pack time. `corpus_version` is the
operator's free-form label. `previous_pack_hash` chains successive
versions of the same corpus so an operator can walk "show me the
history of `wikipedia-current` in this bucket."
### 3.2 Bucket alias index
```
corpora/
wikipedia-2010/latest.json → {"pack_hash": "abc...", "as_of": "..."}
wikipedia-current/latest.json → {"pack_hash": "def...", "as_of": "..."}
textbooks-undergrad/latest.json
```
`arborist cold graft --corpus wikipedia-current` does a single HTTPS
GET against `corpora/wikipedia-current/latest.json` to find the
current pack_hash, then pulls and grafts. Same pattern as
`manifest/latest.json` (the #000061 gap-1 pointer).
### 3.3 CLI surface
```
# install a named corpus
arborist cold graft --corpus wikipedia-current
# install a specific pack (skip the alias indirection)
arborist cold graft <pack_hash>
# enumerate available packages in a bucket
arborist cold list-corpora
# see which corpora are already grafted on this peer
arborist corpus list-grafted
# conflict policy
arborist cold graft --corpus wikipedia-current \
--on-uri-conflict supersedes
```
## 4. Mesh-peer-corpus-merge as the long-game payoff
The graft mode is what makes "mesh of arborists" coherent. Two
peers, A and B, after a partition:
1. A has packs `[wikipedia-2010, arxiv-cs]`.
2. B has packs `[wikipedia-2010, textbooks-undergrad]`.
3. They reconcile: A pulls `textbooks-undergrad`, grafts. B pulls
`arxiv-cs`, grafts.
4. Each now has all three. The graft receipts in each peer's audit
chain record exactly when + from where each package landed.
No coordination protocol needed beyond "exchange your pack lists,
graft what you lack." Mesh state is the union of locally-stored
graft receipts.
## 5. Sequence
Code only when **all three** are true:
1. #000065 reshard lands and stabilises.
2. A second corpus exists — either the `wikipedia-current` snapshot
captured from a real Wikipedia dump, or the first textbook bundle
ready to graft onto the `wikipedia-2010` base. (Until then there's
nothing to test against.)
3. At least two peers (real or simulated) want to exchange packages.
Phases when it ships:
1. **Manifest extension**`corpus_name`, `corpus_version`,
`previous_pack_hash` in pack format. Backward-compatible
(existing packs default to `corpus_name="unknown"`).
2. **Bucket alias index**`corpora/<name>/latest.json` pointer +
`put_latest_corpus_pointer` / `get_latest_corpus_pointer` helpers
in `arborist/cold_object.py` (mirrors gap-1's
`manifest/latest.json` pattern).
3. **Graft executor** — `arborist/evict.py:graft_pack(pack_hash,
on_uri_conflict)` reads pack, runs INSERT OR IGNORE per-table,
emits a single `event_type='graft'` audit event.
4. **CLI**`arborist cold graft`, `arborist cold list-corpora`,
`arborist corpus list-grafted`.
5. **Conflict-policy tests** — three URI-conflict paths
(`supersedes`, `skip`, `fail`), validated on a tiny synthetic
two-corpus fixture.
6. **Mesh integration test** — simulate two peers, run reconcile,
verify both arrive at identical post-graft `snapshot_root`.
## 6. Out of scope
- **Custom audit-chain re-numbering** (Approach B above).
- **Chain forest schema migration** (Approach C above) — see
upgrade-path note in §2.3 if graft frequency demands it.
- **Cross-peer conflict resolution for `providence_cache`** — the
v9.8 falsification framework already handles this
(`state='stale'` / `quarantined`). Graft inherits, doesn't
reinvent.
- **Federated package discovery** ("which buckets host
`arxiv-cs`?") — operator-driven for now, manual bucket
configuration. A discovery layer can come later if it earns its
place.
- **Signed packages** — the pack file is already content-addressed
via `pack_hash` (#000061). Signature-based authenticity adds an
optional integrity layer; defer until a real adversary model
demands it.
## 7. Witness-pattern alignment
The graft receipt is structurally identical to the
content-addressed-evidence pattern Merkle-AGI v8/v9 is converging
on:
- Mechanistic witnesses (#000062) — body holds a
`MechanisticWitnessRoot`, the captured artifacts live outside
the chain.
- Falsification proposals (#000037) — body holds the failure
fingerprint, the controller's full reasoning lives outside.
- Cold-pack graft (this ticket) — body holds `pack_hash`, the
imported events live in the pack file outside.
In every case the pattern is: **host chain stays linear and small;
durable evidence is content-addressed and externalised; the chain
references the evidence by hash; anyone can verify the binding by
re-fetching the evidence.**
This is intentional. The substrate is moving toward "audit chain as
index of witnesses" rather than "audit chain as transcript of
everything that ever happened." Graft mode is one more node in that
arc.
## 8. Open questions
- **Corpus-name namespace**: who owns `wikipedia-current`? Today
packs are operator-self-attested (the operator who builds the
pack picks the `corpus_name`). For mesh-peer-corpus-merge this is
fine within a trust group. For public bucket distribution
(Patch-the-Planet), name squatting becomes a concern eventually.
Defer until that's a real problem.
- **Graft idempotence**: grafting the same `pack_hash` twice should
be a no-op (every row already exists). Verify in tests; emit one
receipt or zero on the second attempt?
- **Receipt visibility for falsification**: if a grafted pack
carries content that the host later falsifies, the falsification
audit event references a `subject_root` from the grafted content.
Forensic recoverability of "which graft brought this in" requires
joining `falsifications.subject_root``graft.documents_imported`.
Probably wants a tiny helper, not a schema change.
## 9. References
- #000061 (`docs/cold-object-store.md`) — base cold-pack format
this ticket extends.
- #000065 (`docs/tickets/ticket-000065-canonical-shard-count-content-hash-routing.md`)
— the consumer side; graft writes into M hash-routed shards via
the same `shard_for_document` routing.
- v8/v9 substrate Witness pattern — see #000062 §4.7 for the
mechanistic-witness instantiation of the same shape.
- CLAUDE.md "schema invariants" — `supersedes` edges, idempotent
re-ingest, content-addressed PKs — all already accommodate graft
semantically; this ticket just makes the workflow first-class.