# Ticket #000066 — Cold-pack overlay / graft mode (pack-as-package)

**Opened:** 2026-05-26
**Status:** scaffold-only · awaiting go/no-go
**Origin:** surfaced while running #000065 reshard cutover, fox extension
  "we could envision a pack for wikipedia 2010, wikipedia current, etc"
**Related:** #000061 (cold-pack distribution tier — base mechanism),
  #000065 (canonical shard count + content-hash routing — the
  consumer side this graft mode plugs into), #000014/#000017
  (witness pattern alignment), v8/v9 substrate (chain forest as the
  long-game upgrade path)

## 1. The capability gap

#000061 cold-pack hydration today is **one-shot, into-empty**. Pull a
pack from the bucket, decompress, write rows into target shards.
There is no way to take a pack and **merge** it onto an existing
populated shard set.

That gap blocks three real workflows:

1. **Pack-as-package** — operator picks which corpora to graft:
   `arborist cold graft wikipedia-current` should feel like
   `apt install firefox`. Each pack is a named bundle
   (`wikipedia-2010`, `wikipedia-current`, `arxiv-cs`,
   `textbooks-undergrad`, …). Operators compose corpora.

2. **Incremental extension** — a peer already has the
   `wikipedia-2010` baseline and wants to add `textbooks-undergrad`
   without re-bootstrapping. Today they'd have to re-hydrate from
   scratch.

3. **Mesh-peer-corpus-merge** — two peers diverge over a partition.
   Each has packs the other lacks. Reconciliation = exchange + graft
   what's missing. This is what makes a *mesh* of arborists rather
   than a *fleet*.

## 2. What's actually hard

Three concerns. Two trivial, one interesting.

### 2.1 Doc / chunk / edge overlay (trivial)

Every per-document table is keyed by content-addressed primary keys
(`document_root` is `sha256(merkle_root)`, `chunks` is unique on
`(document_root, idx)`, `edges` is the wide tuple PK). `INSERT OR
IGNORE` collapses dupes naturally — identical content = identical
row = no-op. No conflict resolution needed for the proof-bearing
tables.

The only non-trivial sub-case: **URI conflicts**. A pack from
`wikipedia-current` carries `https://en.wikipedia.org/wiki/Obama`
with a different `document_root` than the same URI in the
`wikipedia-2010` baseline (different fetch time, different content).
Both rows get stored — different `document_root` means they're
different documents at the Merkle layer. A `supersedes` edge from
old to new captures the lineage (per CLAUDE.md schema invariants).

Conflict-policy flag for the operator:
```
--on-uri-conflict {supersedes,skip,fail}    default: supersedes
```
- `supersedes`: store both, emit `supersedes` edge old→new
- `skip`: keep existing, drop incoming (legacy-corpus-priority)
- `fail`: stop the graft, report the conflict count

### 2.2 FTS5 overlay (trivial)

Contentless FTS5 indexes (`chunks_fts`, `documents_fts`) are
populated by `INSERT INTO chunks_fts (rowid, content)`. New chunks
get new `chunk_id` values (the AUTOINCREMENT PK) → new FTS rowids →
new FTS rows. Existing FTS rows are untouched. Idempotent.

### 2.3 Audit chain overlay (the interesting one)

The audit chain is **linear**:
```
event_hash = sha256(prev_event_hash || canonical(body))
```

Naively appending the pack's events to the host chain doesn't work
because the pack's `prev_event_hash` linkages reference events that
exist only inside the pack, not in the host chain.

Three coherent ways to handle this, ranked by cost:

#### Approach A — Graft receipt (chosen)

Append exactly **one** new audit event to the host chain:

```json
{
  "event_type": "graft",
  "body": {
    "kind": "graft",
    "pack_hash": "<sha256 of pack file>",
    "snapshot_root": "<corpus snapshot root the pack pins>",
    "corpus_name": "wikipedia-current",
    "manifest_root": "<sha256 of manifest>",
    "event_count": 1_245_678,
    "first_event_hash": "<pack's chain genesis>",
    "last_event_hash": "<pack's chain head>",
    "documents_imported": 866_881,
    "chunks_imported": 1_560_021,
    "edges_imported": 22_664_792,
    "on_uri_conflict_policy": "supersedes",
    "uri_conflicts_resolved": 4_321
  },
  "ts": <ingest time>
}
```

The host's chain stays linear. The pack file itself is the durable
witness for the absorbed events — anyone can:

1. Re-fetch the pack by its `pack_hash`.
2. Walk the pack's internal audit chain.
3. Verify `first_event_hash` / `last_event_hash` / `event_count`
   match the receipt.

This is the same **witness pattern** v8/v9 substrate is converging
toward (mechanistic witnesses #000062, falsification proposals
#000037, etc.) — content-addressed evidence held outside the host
chain, referenced via a hash binding in the receipt event.

Cost: ~zero schema change. One new `event_type='graft'`. Done.

#### Approach B — Re-chain everything (rejected)

Read both chains, merge by `ts`, recompute every `event_hash`. This
is what the #000065 Option-A reshard does today.

**Why rejected for graft**: the reshard is a one-time topology
change with no external references to the audit chain. Graft is
expected to be frequent (every package install). Re-chaining
invalidates every external reference to old `event_hash` values —
cache_keys anchored to `audit_event_hash`, snapshots, falsification
records — and silently produces stale pointers.

#### Approach C — Chain forest (rejected for now)

Add a `chain_id` column to `audit_events`. Each pack becomes a new
chain in the forest. Cross-chain Merkle anchors bind one chain's
root into another's event body.

**Why rejected for now**: this is the *right answer when graft is
the common case*, but the schema migration cost is high (every
chain-walker code path needs to handle multi-chain), and the
receipt-pattern (Approach A) gets us the capability at a fraction
of the cost while leaving the door open to upgrade.

When to revisit Approach C: when graft frequency exceeds reshard
frequency by ≥10×, or when multi-peer corpus-merging produces
enough audit-chain-fan-in to make linear-chain queries painful.

## 3. Pack-as-package surface

### 3.1 Manifest extension

Add one field to the #000061 manifest:

```json
{
  ...existing fields...,
  "corpus_name": "wikipedia-current",
  "corpus_version": "2026-05-26",
  "previous_pack_hash": "<prior pack of same corpus, or null>"
}
```

`corpus_name` is operator-set at pack time. `corpus_version` is the
operator's free-form label. `previous_pack_hash` chains successive
versions of the same corpus so an operator can walk "show me the
history of `wikipedia-current` in this bucket."

### 3.2 Bucket alias index

```
corpora/
  wikipedia-2010/latest.json     → {"pack_hash": "abc...", "as_of": "..."}
  wikipedia-current/latest.json  → {"pack_hash": "def...", "as_of": "..."}
  textbooks-undergrad/latest.json
```

`arborist cold graft --corpus wikipedia-current` does a single HTTPS
GET against `corpora/wikipedia-current/latest.json` to find the
current pack_hash, then pulls and grafts. Same pattern as
`manifest/latest.json` (the #000061 gap-1 pointer).

### 3.3 CLI surface

```
# install a named corpus
arborist cold graft --corpus wikipedia-current

# install a specific pack (skip the alias indirection)
arborist cold graft <pack_hash>

# enumerate available packages in a bucket
arborist cold list-corpora

# see which corpora are already grafted on this peer
arborist corpus list-grafted

# conflict policy
arborist cold graft --corpus wikipedia-current \
  --on-uri-conflict supersedes
```

## 4. Mesh-peer-corpus-merge as the long-game payoff

The graft mode is what makes "mesh of arborists" coherent. Two
peers, A and B, after a partition:

1. A has packs `[wikipedia-2010, arxiv-cs]`.
2. B has packs `[wikipedia-2010, textbooks-undergrad]`.
3. They reconcile: A pulls `textbooks-undergrad`, grafts. B pulls
   `arxiv-cs`, grafts.
4. Each now has all three. The graft receipts in each peer's audit
   chain record exactly when + from where each package landed.

No coordination protocol needed beyond "exchange your pack lists,
graft what you lack." Mesh state is the union of locally-stored
graft receipts.

## 5. Sequence

Code only when **all three** are true:

1. #000065 reshard lands and stabilises.
2. A second corpus exists — either the `wikipedia-current` snapshot
   captured from a real Wikipedia dump, or the first textbook bundle
   ready to graft onto the `wikipedia-2010` base. (Until then there's
   nothing to test against.)
3. At least two peers (real or simulated) want to exchange packages.

Phases when it ships:

1. **Manifest extension** — `corpus_name`, `corpus_version`,
   `previous_pack_hash` in pack format. Backward-compatible
   (existing packs default to `corpus_name="unknown"`).
2. **Bucket alias index** — `corpora/<name>/latest.json` pointer +
   `put_latest_corpus_pointer` / `get_latest_corpus_pointer` helpers
   in `arborist/cold_object.py` (mirrors gap-1's
   `manifest/latest.json` pattern).
3. **Graft executor** — `arborist/evict.py:graft_pack(pack_hash,
   on_uri_conflict)` reads pack, runs INSERT OR IGNORE per-table,
   emits a single `event_type='graft'` audit event.
4. **CLI** — `arborist cold graft`, `arborist cold list-corpora`,
   `arborist corpus list-grafted`.
5. **Conflict-policy tests** — three URI-conflict paths
   (`supersedes`, `skip`, `fail`), validated on a tiny synthetic
   two-corpus fixture.
6. **Mesh integration test** — simulate two peers, run reconcile,
   verify both arrive at identical post-graft `snapshot_root`.

## 6. Out of scope

- **Custom audit-chain re-numbering** (Approach B above).
- **Chain forest schema migration** (Approach C above) — see
  upgrade-path note in §2.3 if graft frequency demands it.
- **Cross-peer conflict resolution for `providence_cache`** — the
  v9.8 falsification framework already handles this
  (`state='stale'` / `quarantined`). Graft inherits, doesn't
  reinvent.
- **Federated package discovery** ("which buckets host
  `arxiv-cs`?") — operator-driven for now, manual bucket
  configuration. A discovery layer can come later if it earns its
  place.
- **Signed packages** — the pack file is already content-addressed
  via `pack_hash` (#000061). Signature-based authenticity adds an
  optional integrity layer; defer until a real adversary model
  demands it.

## 7. Witness-pattern alignment

The graft receipt is structurally identical to the
content-addressed-evidence pattern Merkle-AGI v8/v9 is converging
on:

- Mechanistic witnesses (#000062) — body holds a
  `MechanisticWitnessRoot`, the captured artifacts live outside
  the chain.
- Falsification proposals (#000037) — body holds the failure
  fingerprint, the controller's full reasoning lives outside.
- Cold-pack graft (this ticket) — body holds `pack_hash`, the
  imported events live in the pack file outside.

In every case the pattern is: **host chain stays linear and small;
durable evidence is content-addressed and externalised; the chain
references the evidence by hash; anyone can verify the binding by
re-fetching the evidence.**

This is intentional. The substrate is moving toward "audit chain as
index of witnesses" rather than "audit chain as transcript of
everything that ever happened." Graft mode is one more node in that
arc.

## 8. Open questions

- **Corpus-name namespace**: who owns `wikipedia-current`? Today
  packs are operator-self-attested (the operator who builds the
  pack picks the `corpus_name`). For mesh-peer-corpus-merge this is
  fine within a trust group. For public bucket distribution
  (Patch-the-Planet), name squatting becomes a concern eventually.
  Defer until that's a real problem.
- **Graft idempotence**: grafting the same `pack_hash` twice should
  be a no-op (every row already exists). Verify in tests; emit one
  receipt or zero on the second attempt?
- **Receipt visibility for falsification**: if a grafted pack
  carries content that the host later falsifies, the falsification
  audit event references a `subject_root` from the grafted content.
  Forensic recoverability of "which graft brought this in" requires
  joining `falsifications.subject_root` ↔ `graft.documents_imported`.
  Probably wants a tiny helper, not a schema change.

## 9. References

- #000061 (`docs/cold-object-store.md`) — base cold-pack format
  this ticket extends.
- #000065 (`docs/tickets/ticket-000065-canonical-shard-count-content-hash-routing.md`)
  — the consumer side; graft writes into M hash-routed shards via
  the same `shard_for_document` routing.
- v8/v9 substrate Witness pattern — see #000062 §4.7 for the
  mechanistic-witness instantiation of the same shape.
- CLAUDE.md "schema invariants" — `supersedes` edges, idempotent
  re-ingest, content-addressed PKs — all already accommodate graft
  semantically; this ticket just makes the workflow first-class.
