docs: #000065 — correct re-ingest framing to teleport (content-addressed rebalance)

fox caught the imprecision: "the shards are teleported if we fix this
because we know the shard a doc will end up in."

Re-ingest = re-read source files, re-canonicalize, re-chunk, re-hash.
Hours-to-days from Wikipedia dumps.

What's actually needed = content-addressed rebalance. Every row in
the corpus is already addressed by content (document_root, leaf_hash,
merkle_nodes.hash, audit_event_hash). Migration reads existing rows,
computes new_shard_idx via the routing function, INSERTs into the new
M shards. No source-file work, no canonicalization, no LLM.

Updated the ticket's "Migration story" section with the per-table
teleport recipe:

  documents:            move by document_root → new shard
  chunks (+content):    follow doc to its new shard, re-assign chunk_id
  chunks_fts:           rebuild per new shard from chunks.content
  merkle_nodes:         move with document_root
  edges:                move by src_root
  concept_relations,    move by content-derived parent
    derivations,
    providence_cache,
    citation_aliases,
    term_aliases:
  audit_events:         ALL → shard 000 (canonical), re-numbered + re-hashed
  snapshots, mesh_*:    canonical shard 000

Audit chain handling: chose Option A (canonical shard 000) over
Option B (split-by-subject_root). Preserves global event ordering,
re-hashing happens once at migration time. The alternative loses
cross-shard event ordering for falsification + replay reasoning.

Estimated wall: ~20-40 min I/O-bound on current corpus
(35 GB across 4 shards → ~35 GB across 8 shards, moves once).

Hard discipline added: move into ~/.arborist/shards.v2/, keep
originals as rollback, atomic dir-swap when verified. Mistake in
routing function = corrupted federation, recoverable only by going
back to the originals.

docs/TICKETS.md

@@ -111,7 +111,7 @@ Newest first. Update on every open/close.
 
 | ID       | Title                                          | Status                | Opened     | Directive |
 |----------|------------------------------------------------|-----------------------|------------|-----------|
-| #000065  | Canonical shard count `M` + content-hash routing (decouple ingest parallelism from ATTACH ceiling) | **open · scaffold + design · awaiting go/no-go** (2026-05-26; surfaced while sizing #000061's federation story). Today shard count conflates two roles: producer ingest parallelism (wants vCPU count) + consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10 on stock python3 sqlite3). Producer with 16 vCPU → 16 shards → consumers fail to attach the 11th. Producer with 4 shards → 16-vCPU box runs 75% idle on ingest. Fix: pin a corpus-wide canonical M (default 8, ≤ ATTACH ceiling), introduce N (ingest workers) decoupled from M. Document → shard assignment becomes content-deterministic: `shard_idx = int(document_root[:8], 16) % M`. Same input → same output across every peer (today's "spray by ingest order" is non-deterministic across peers, a real federation weakness). Migration hard-constraint per fox: **requires re-ingest of every shard** — current layout is sprayed by ingest order, post-ticket is sprayed by content hash, the two are incompatible by construction. Phases: 0 design lock + pin M in meta table → 1 read path (connect_query honors M) → 2 ingest path (multi-shard write per worker) → 3 cold-pack restore re-routes on pull → 4 corpus migration tool. Open audit-chain re-numbering question (every shard has its own seq + event_hash; rebalancing splits a producer's chain across M consumer shards). Don't proliferate sub-tickets; the audit handling is part of this design lock. Out of scope: custom-built sqlite3 with higher MAX_ATTACHED (rejected: violates "python3 + venv + sqlite3 only" property from CLAUDE.md); topic-clustering shards (would break ingest determinism). | 2026-05-26 | — |
+| #000065  | Canonical shard count `M` + content-hash routing (decouple ingest parallelism from ATTACH ceiling) | **open · scaffold + design · awaiting go/no-go** (2026-05-26; surfaced while sizing #000061's federation story). Today shard count conflates two roles: producer ingest parallelism (wants vCPU count) + consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10 on stock python3 sqlite3). Producer with 16 vCPU → 16 shards → consumers fail to attach the 11th. Producer with 4 shards → 16-vCPU box runs 75% idle on ingest. Fix: pin a corpus-wide canonical M (default 8, ≤ ATTACH ceiling), introduce N (ingest workers) decoupled from M. Document → shard assignment becomes content-deterministic: `shard_idx = int(document_root[:8], 16) % M`. Same input → same output across every peer (today's "spray by ingest order" is non-deterministic across peers, a real federation weakness). Migration hard-constraint per fox: **content-addressed rebalance, NOT re-ingest** — every row is already addressed by `document_root` / `leaf_hash` / etc.; migration reads rows from the current 4 shards, computes each row's new shard via the routing function, INSERTs into M new shards. No source re-parse, no re-canonicalization, no re-chunking, no LLM. ~20–40 min I/O-bound vs. hours-to-days for true re-ingest. Audit chain consolidates to canonical shard 000 (re-numbered + re-hashed once) to preserve global event ordering. Phases: 0 design lock + pin M in meta table → 1 read path (connect_query honors M) → 2 ingest path (multi-shard write per worker) → 3 cold-pack restore re-routes on pull → 4 corpus migration tool. Open audit-chain re-numbering question (every shard has its own seq + event_hash; rebalancing splits a producer's chain across M consumer shards). Don't proliferate sub-tickets; the audit handling is part of this design lock. Out of scope: custom-built sqlite3 with higher MAX_ATTACHED (rejected: violates "python3 + venv + sqlite3 only" property from CLAUDE.md); topic-clustering shards (would break ingest determinism). | 2026-05-26 | — |
 | #000064  | Cold-object operations toolkit (verify/diff/doctor/repair-fts/gc-plan + audit taxonomy) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §11/§12/§14). Operator-facing observability + repair tools on top of #000061: `cold verify` (sample/full integrity check), `cold diff` (local vs remote manifest), `cold doctor` (one-shot health: connectivity / credentials / manifest age / missing-object count / tamper sample / audit-chain integrity), `cold repair-fts` (rebuild FTS5 from chunks.content), `cold gc-plan` (orphan bucket objects, read-only by default — destructive only with `--apply` + confirm). Plus expanded audit-event taxonomy: per-PUT/HEAD/GET success/failure events, manifest-pointer events, verify/doctor/gc events. All read-mostly; destructive ops require `--apply`. Bundled so the audit-taxonomy gets one design pass instead of five-way drift. Sequence: doctor → verify → diff → repair-fts → gc-plan. No code until #000061 closes. | 2026-05-26 | — |
 | #000063  | Cold-object private-ciphertext mode (mesh-keyed object keys) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §9 / response A §13.3). Adds private mode to #000061 cold-object format so chunk bodies + manifest can be uploaded to public-read bucket without leaking corpus membership. Two strategies: (A) deterministic `object_key = HMAC(group_key, leaf_hash)` + AEAD-encrypted body — supports lookup-by-leaf-hash given the key; (B) random-key ciphertext + encrypted private manifest — stronger membership hiding, needs manifest fetch first. Strategy A default; B opt-in. Group key from existing `arborist/mesh/crypto.py`; pack manifest carries `epoch_id` for rotation. Verifier path unchanged: consumer decrypts, then `hash_leaf(plaintext) == leaf_hash` as in public mode. No code until (1) a real non-public corpus needs cold-object shipping, (2) mesh group-key ABI is stable enough to reference, (3) threat-model split between A vs B is settled by real adversary. | 2026-05-26 | — |
 | #000062  | Mechanistic Witness: governed diagnostic sidecar (CNA/SAE/Neuronpedia) | **scaffold-only · awaiting go/no-go** (2026-05-26; Dav1d de-novo review §4.7 / §9.1.F). Specification of a mechanistic-interpretability sidecar that produces a content-addressed `MechanisticWitnessRoot` over (model, prompts, capture policy, neurons/features, intervention deltas), used as a **diagnostic input** to SelfModel (#000014/#000017) + benchmark-fixture generation. **Hard constraint:** soft signals never enter the hard proof path — `audit_mode` does NOT move based on witness output, `providence_cache` is untouched, `governance_policy_hash` only moves via explicit ForkScore ACCEPT with M+C+X axes passing (#000060 §7). Four guardrails (diagnostic-only by default · sandbox intervention only · no production steering without governance · feature labels never semantic proof). Witness root TLV-encodes `model_config_root | activation_capture_policy_root | contrastive_prompt_set_root | feature_or_neuron_set_root | intervention_result_root | behavioral_delta_root | safety_policy_root`. Scaffold only — no code until a real falsifier-in-hand use case exists + the four guardrails are restated in CLAUDE.md as rules + #000060 H-ABCDEFG-M+C+X harness exists to gate promotion. Captured to keep mechanistic-interp tooling out of the substrate unless and until it earns its place; the dual-use risk (Pan et al. 2025 CNA: 0.1% MLP ablation breaks refusal in 72B models) makes the governance-first framing load-bearing. | 2026-05-26 | — |

docs/tickets/ticket-000065-canonical-shard-count-content-hash-routing.md

@@ -10,13 +10,15 @@
   workers ≥ M without forcing consumers to attach > 10 shards.
 **Audience:** dav1d (architecture-level invariant change), every
   operator running arborist with `--shards-dir`.
-**Hard constraint:** **migration requires re-ingest** of every shard
-  in the corpus. Current shards are sprayed by ingest order (operator-
-  controlled, non-deterministic across peers). After this ticket,
-  documents are sprayed by content hash (deterministic, same across
-  peers). The two layouts are incompatible by construction —
-  rebalancing 14M chunks across 8 shards is functionally equivalent to
-  a full re-ingest.
+**Hard constraint:** migration is a **content-addressed rebalance,
+  NOT a re-ingest**. Every row in the corpus is already addressed by
+  content (`document_root`, `leaf_hash`, `merkle_nodes.hash`, etc.) —
+  the migration reads existing rows from the current 4 shards, computes
+  each row's new shard index via the content-hash routing function, and
+  INSERTs into the new M shards. No source-file re-parse, no
+  re-canonicalization, no re-chunking, no LLM. Wall time is I/O-bound
+  (~20–40 min for fox's current 4-shard / 14M-chunk corpus on local
+  disk) instead of hours-to-days for a true re-ingest.
 
 ## Problem
 
@@ -109,41 +111,82 @@ document_roots in the same shards. ATTACH count = M ≤ 8, well under
 the ceiling.
 ```
 
-## Migration story
+## Migration story — teleport, not re-ingest
 
 Current arborist has 4 shards (fox's corpus). Documents in each shard
 were assigned by `ingest_source` calls — the operator splits work,
 each call writes to one shard. No content-hash routing.
 
-Migration to content-hash routing requires:
+Migration is **a content-addressed rebalance**, not a re-ingest.
+Every row already has its content-derived identity. The migration
+just moves rows from their current (ingest-order-determined) shard
+to their (content-hash-determined) shard:
 
-1. **Pick M.** Recommended default: 8 (leaves headroom under
-   ATTACH=10 for the manifest pointer + qa.db + snapshots.db +
-   selfmodel-chain.db).
-2. **Cold-pack the current 4 shards** (already supported via
-   `make cold-pack-all` after this ticket lands).
-3. **Drop the old 4 shards** + create M new empty shards via the
-   updated `connect()` (which would honor a `--shards-count M`).
-4. **Re-ingest from packs** (or from upstream sources). At restore
-   time, each row is routed to its new content-hashed shard.
+1. **Pick M.** Default: 8 (leaves headroom under ATTACH=10 for
+   `snapshots.db`, `qa.db`, `selfmodel-chain.db`, the manifest
+   pointer connection).
+2. **Cold-pack the current 4 shards** as a recovery rollback
+   (already in bucket from #000061).
+3. **Create M new empty shards.**
+4. **Per-table teleport:**
 
-For fox's specific corpus (Wikipedia 2010 + textbooks): re-ingest
-from the original dumps is the cheapest path. The cold packs in
-DO Spaces also work but require the new restore code to honor M.
+   | Table | Move how | Notes |
+   |-------|----------|-------|
+   | `documents` | by `document_root` → new shard via routing fn | content-addressed PK, just INSERT |
+   | `chunks` (incl. content) | follow `chunks.document_root` to same shard as parent doc | re-assign `chunk_id` autoincrement local |
+   | `chunks_fts` | rebuild per new shard from `chunks.content` | contentless FTS5, bounded by chunk count |
+   | `merkle_nodes` | follow `document_root` | move by PK |
+   | `edges` | by `src_root` → new shard | content-addressed PK |
+   | `concept_relations` | follow `target_root` / etc. → new shard | re-assign local id |
+   | `derivations` | follow `core_root` | move |
+   | `providence_cache` | follow `source_root` | move |
+   | `citation_aliases`, `term_aliases` | content-addressed | move |
+   | `audit_events` | **all → shard 000 (canonical)** | preserves global ordering + chain integrity |
+   | `snapshots` | shard 000 (canonical) | snapshot is corpus-wide |
+   | `mesh_*` | shard 000 (canonical) | mesh state is per-peer, not per-document |
+   | `concept_token_idf` | follow token's owning shard via hash(token) % M | local IDs |
 
-Estimated wall-clock cost (4 shards × 8.8 GB each):
+5. **Rebuild `chunks_fts` per new shard.** Contentless FTS5 inserts
+   from `chunks.content`. Minutes-scale per shard.
+6. **VACUUM.** Reclaim free pages after the row moves.
+7. **`make chain-check-shards`.** Audit chain in shard 000 still
+   verifies end-to-end because rows weren't reordered, just moved
+   together.
+
+**Audit chain handling — chosen Option A (canonical shard 000):**
+
+Each shard's existing `audit_events.seq INTEGER PRIMARY KEY
+AUTOINCREMENT` chain stays intact when moved as a contiguous block.
+Consolidating four chains into one means re-numbering `seq` and
+re-hashing the chain end-to-end (interleaving by `ts`, recomputing
+`event_hash` per row). Loses per-producer-per-shard chain identity
+but preserves global event ordering. Alternative (Option B: split
+audit by `subject_root` to per-shard chains) was considered and
+rejected — loses cross-shard event ordering, harder to reason about
+for falsification + replay.
+
+Estimated wall-clock cost (4 shards × 8.8 GB each = ~35 GB total):
 
 | Phase | Cost |
 |-------|------|
-| Cold pack current state (already done; v3 SPV packs in bucket) | 0 — already done |
-| Reset shard directory | seconds |
-| Re-ingest from packs into M=8 layout | ~30 min if pack-restore is parallel; longer if serial |
-| Re-build FTS5 indexes | ~10 min |
+| Cold-pack current state (recovery rollback) | 0 — already in bucket |
+| Create M empty target shards | seconds |
+| Move documents + chunks + chunks.content | I/O bound, ~10 min |
+| Move edges (largest, 22M × 4 rows total) | ~10 min |
+| Move merkle + concepts + derivations + small tables | ~minutes |
+| Consolidate audit_events → shard 000 + re-hash chain | ~few min |
+| Rebuild `chunks_fts` per new shard | ~minutes per shard |
+| VACUUM | ~few min per shard |
 | `make chain-check-shards` | ~minutes |
+| **Total wall** | **~20–40 min** |
 
-Hard discipline: do NOT mutate the live corpus until the new pack
-format is ready to restore into the new layout. Mistake in the
-routing function = corrupted federation.
+Vs. a true re-ingest from Wikipedia dumps: ~hours.
+
+Hard discipline: do NOT mutate the live shards in place. Move into
+NEW shards (`~/.arborist/shards.v2/`), keep the originals as
+rollback until the move is verified, then atomically swap directories.
+Mistake in the routing function = corrupted federation, recoverable
+only by going back to the original 4 shards.
 
 ## Implementation phases