#000065 closed: production reshard landed; record in corpus-history

2026-05-26 19:47 UTC. ~94 min wall. 3,468,226 globally-unique docs +
6,235,588 chunks + 90,592,990 edges + 3,468,403 audit events re-routed
from non-deterministic spray-by-ingest-order layout to canonical
content-hash M=4 layout (shard_idx = int(document_root[:8], 16) % 4).

Final state:
  per-shard doc uniformity within ±0.04% (theoretical max ±0.05% for
    first-32-bit SHA-256 prefix)
  audit chain consolidated to shard 000 via Option A (re-sorted by ts,
    re-chained; bodies preserved unchanged; tail event type=reshard
    carries plan+result body, hash 8da3aa19…)
  on-disk sizes: 000=11.0 / 001=8.8 / 002=8.8 / 003=8.8 GB
  validation: chunks delta 176 (0.003%) + edges delta 547 (0.0006%)
    are cross-shard dupes from re-ingest history, collapsed by
    INSERT OR IGNORE; within the 1% tolerance gate
  smoke queries: Barack Obama / YouTube / Albert Einstein all
    returned proper evidence from correct (hash-routed) shards
  chain-check-shards: 0 breaks on every shard

Two mid-flight defect fixes (also committed):
  04edff7: derivations.src_root FK guard fired on legitimately
    cross-shard refs → writer connection PRAGMA foreign_keys = OFF
  c86d5ac: WAL accumulated ~37 GB across passes (SQLite auto-checkpoint
    blocked by open reader cursors) → _checkpoint_truncate between
    phases. Production migration was rescued mid-flight by manual
    sibling-connection wal_checkpoint(TRUNCATE) freeing 27.7 GB.

Closes #000065 in both the index and the per-ticket file. The
per-ticket design doc stays open as a design-log artifact (its
content is still the right reference for the next reshard / for
graft mode #000066).

Follow-on tracked as tasks #44–#47:
  #44 re-pack post-reshard shards into DO Spaces (current bucket
      packs are stale, still in pre-reshard topology)
  #45 verify bucket pack hydration is deterministic against the new
      content-hash layout
  #46 genesis a fresh peer on 3090-ai.foxhop.net from cloud — first
      real SPV-wallet end-to-end test
  #47 retire stale pre-reshard bucket packs after #46 confirms

docs/TICKETS.md

@@ -112,7 +112,7 @@ Newest first. Update on every open/close.
 | ID       | Title                                          | Status                | Opened     | Directive |
 |----------|------------------------------------------------|-----------------------|------------|-----------|
 | #000066  | Cold-pack overlay / graft mode (pack-as-package, witness-pattern audit chain) | **scaffold-only · awaiting go/no-go** (2026-05-26; surfaced while running #000065 reshard, fox extension: "we could envision a pack for wikipedia 2010, wikipedia current, etc"). Extend #000061 cold-pack hydration with a second mode: overlay an existing pack onto a populated shard set instead of hydrating into empty. Doc/chunk/edge/concept overlay is trivial (`INSERT OR IGNORE` on content-addressed PKs collapses dupes); FTS5 overlay is trivial (new chunks → new rowids → new FTS rows). The interesting part is the audit chain — can't naively append the pack's events because `prev_event_hash` linkage breaks across the join. Chosen approach: **graft receipt**. Append one new `event_type='graft'` event to the host chain carrying `(pack_hash, snapshot_root, corpus_name, event_count, first_event_hash, last_event_hash, manifest_root)`; the pack file itself becomes the durable witness for the absorbed events (anyone can re-fetch the pack, walk its internal chain, and verify it matches the receipt). Host chain stays linear; pack chain is a "witnessed subgraph." This is the same witness pattern Merkle-AGI v8/v9 is heading toward, but bought at near-zero schema cost. Rejected alternatives: re-chain everything (breaks external refs to old event_hashes — cache_keys anchoring to old `audit_event_hash`, snapshots, etc. — silently invalid); chain forest with new `chain_id` column (right answer when graft dominates the lifecycle, but premature now). **Pack-as-package extension** (fox 2026-05-26): each pack carries a `corpus_name` field in its manifest (`wikipedia-2010`, `wikipedia-current`, `arxiv-cs`, `textbooks-undergrad`, …) so operators pick which corpora to graft — `arborist cold graft wikipedia-current` becomes as natural as `apt install firefox`. Multiple packs of the same corpus name: most-recent `snapshot_root` wins; older packs stay in the bucket until GC. URI conflicts across corpora (e.g., `wikipedia.org/wiki/Foo` in both 2010 and current): different content → different `document_root` → both stored, `supersedes` edges per CLAUDE.md invariant. Providence-cache conflicts: same `cache_key` with different answer → existing v9.8 falsification framework handles it (`state='stale'` or `quarantined`). Mesh-peer-corpus-merge: each peer's pack is a graftable package; partition reconciliation becomes "exchange the packs you each carry, graft what you lack". The mesh-of-arborists semantic. Sequence: (1) `corpus_name` field in #000061 manifest format + alias index in bucket (`corpora/<name>/latest.json` pointer to active pack_hash); (2) `arborist cold graft <pack_hash>` / `arborist cold graft --corpus <name>` mode in evict.py — read pack, INSERT OR IGNORE per-table, emit graft receipt; (3) conflict-policy flag (`--on-uri-conflict {supersedes,skip,fail}`, default `supersedes`); (4) `arborist cold list-corpora` shows available packages in a bucket. Scaffold first, code only when (a) #000065 reshard lands and stabilises (b) a second corpus exists (the wikipedia-current snapshot, or first textbook bundle ready to graft onto wikipedia-2010 base) (c) at least two peers want to exchange. | 2026-05-26 | — |
-| #000065  | Canonical shard count `M` + content-hash routing (decouple ingest parallelism from ATTACH ceiling) | **open · scaffold + design · awaiting go/no-go** (2026-05-26; surfaced while sizing #000061's federation story). Today shard count conflates two roles: producer ingest parallelism (wants vCPU count) + consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10 on stock python3 sqlite3). Producer with 16 vCPU → 16 shards → consumers fail to attach the 11th. Producer with 4 shards → 16-vCPU box runs 75% idle on ingest. Fix: pin a corpus-wide canonical **M = 4** (decided 2026-05-26 from real-Wikipedia bench: M=4 captures 92% of peak ingest throughput, ATTACH cost 9 ms keeps mobile-tolerable, 6 free ATTACH slots under SQLite's 10 ceiling for auxiliary DBs), introduce N (ingest workers) decoupled from M. Document → shard assignment becomes content-deterministic: `shard_idx = int(document_root[:8], 16) % M`. Same input → same output across every peer (today's "spray by ingest order" is non-deterministic across peers, a real federation weakness). Migration hard-constraint per fox: **content-addressed rebalance, NOT re-ingest** — every row is already addressed by `document_root` / `leaf_hash` / etc.; migration reads rows from the current 4 shards, computes each row's new shard via the routing function, INSERTs into M new shards. No source re-parse, no re-canonicalization, no re-chunking, no LLM. ~20–40 min I/O-bound vs. hours-to-days for true re-ingest. Audit chain consolidates to canonical shard 000 (re-numbered + re-hashed once) to preserve global event ordering. Phases: 0 design lock + pin M in meta table → 1 read path (connect_query honors M) → 2 ingest path (multi-shard write per worker) → 3 cold-pack restore re-routes on pull → 4 corpus migration tool. Open audit-chain re-numbering question (every shard has its own seq + event_hash; rebalancing splits a producer's chain across M consumer shards). Don't proliferate sub-tickets; the audit handling is part of this design lock. Out of scope: custom-built sqlite3 with higher MAX_ATTACHED (rejected: violates "python3 + venv + sqlite3 only" property from CLAUDE.md); topic-clustering shards (would break ingest determinism). | 2026-05-26 | — |
+| #000065  | Canonical shard count `M` + content-hash routing (decouple ingest parallelism from ATTACH ceiling) | **closed · landed in `c86d5ac`** (2026-05-26 19:47 UTC cutover, ~94 min wall). Production reshard completed end-to-end on the live host: 3,468,226 globally-unique docs / 6,235,588 chunks / 90,592,990 edges / 3,468,403 audit events re-routed to content-hash-deterministic M=4 layout. Per-shard doc uniformity within ±0.04% (theoretical limit ±0.05%). Audit chain consolidated to canonical shard 000 via Option A (3.47M events re-sorted by ts + re-chained, bodies preserved); tail event `type=reshard` carries plan+result body. Validation gate caught 176 chunks + 547 edges as cross-shard dupes (collapsed by INSERT OR IGNORE; 0.003% delta, within 1% tolerance). Two defects surfaced + fixed mid-cutover: (a) `derivations.src_root` FK fired on legitimately cross-shard refs — fix in `04edff7`: writer connection runs `PRAGMA foreign_keys = OFF`, runtime stays FK=ON; (b) WAL accumulated ~37 GB across FTS rebuild + audit consolidate because SQLite auto-checkpoint can't reclaim pages while a reader cursor is open — fix in `c86d5ac`: `_checkpoint_truncate` called between executor phases. Full migration record in `docs/corpus-history.md` (which entry is the operator-facing equivalent of the audit chain tail). Follow-on work tracked separately: #44 re-pack into bucket → #45 verify bucket determinism → #46 genesis fresh peer on 3090-ai.foxhop.net from cloud (first real SPV-wallet end-to-end test) → #47 retire stale pre-reshard bucket packs. (2026-05-26; surfaced while sizing #000061's federation story). Today shard count conflates two roles: producer ingest parallelism (wants vCPU count) + consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10 on stock python3 sqlite3). Producer with 16 vCPU → 16 shards → consumers fail to attach the 11th. Producer with 4 shards → 16-vCPU box runs 75% idle on ingest. Fix: pin a corpus-wide canonical **M = 4** (decided 2026-05-26 from real-Wikipedia bench: M=4 captures 92% of peak ingest throughput, ATTACH cost 9 ms keeps mobile-tolerable, 6 free ATTACH slots under SQLite's 10 ceiling for auxiliary DBs), introduce N (ingest workers) decoupled from M. Document → shard assignment becomes content-deterministic: `shard_idx = int(document_root[:8], 16) % M`. Same input → same output across every peer (today's "spray by ingest order" is non-deterministic across peers, a real federation weakness). Migration hard-constraint per fox: **content-addressed rebalance, NOT re-ingest** — every row is already addressed by `document_root` / `leaf_hash` / etc.; migration reads rows from the current 4 shards, computes each row's new shard via the routing function, INSERTs into M new shards. No source re-parse, no re-canonicalization, no re-chunking, no LLM. ~20–40 min I/O-bound vs. hours-to-days for true re-ingest. Audit chain consolidates to canonical shard 000 (re-numbered + re-hashed once) to preserve global event ordering. Phases: 0 design lock + pin M in meta table → 1 read path (connect_query honors M) → 2 ingest path (multi-shard write per worker) → 3 cold-pack restore re-routes on pull → 4 corpus migration tool. Open audit-chain re-numbering question (every shard has its own seq + event_hash; rebalancing splits a producer's chain across M consumer shards). Don't proliferate sub-tickets; the audit handling is part of this design lock. Out of scope: custom-built sqlite3 with higher MAX_ATTACHED (rejected: violates "python3 + venv + sqlite3 only" property from CLAUDE.md); topic-clustering shards (would break ingest determinism). | 2026-05-26 | — |
 | #000064  | Cold-object operations toolkit (verify/diff/doctor/repair-fts/gc-plan + audit taxonomy) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §11/§12/§14). Operator-facing observability + repair tools on top of #000061: `cold verify` (sample/full integrity check), `cold diff` (local vs remote manifest), `cold doctor` (one-shot health: connectivity / credentials / manifest age / missing-object count / tamper sample / audit-chain integrity), `cold repair-fts` (rebuild FTS5 from chunks.content), `cold gc-plan` (orphan bucket objects, read-only by default — destructive only with `--apply` + confirm). Plus expanded audit-event taxonomy: per-PUT/HEAD/GET success/failure events, manifest-pointer events, verify/doctor/gc events. All read-mostly; destructive ops require `--apply`. Bundled so the audit-taxonomy gets one design pass instead of five-way drift. Sequence: doctor → verify → diff → repair-fts → gc-plan. No code until #000061 closes. | 2026-05-26 | — |
 | #000063  | Cold-object private-ciphertext mode (mesh-keyed object keys) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §9 / response A §13.3). Adds private mode to #000061 cold-object format so chunk bodies + manifest can be uploaded to public-read bucket without leaking corpus membership. Two strategies: (A) deterministic `object_key = HMAC(group_key, leaf_hash)` + AEAD-encrypted body — supports lookup-by-leaf-hash given the key; (B) random-key ciphertext + encrypted private manifest — stronger membership hiding, needs manifest fetch first. Strategy A default; B opt-in. Group key from existing `arborist/mesh/crypto.py`; pack manifest carries `epoch_id` for rotation. Verifier path unchanged: consumer decrypts, then `hash_leaf(plaintext) == leaf_hash` as in public mode. No code until (1) a real non-public corpus needs cold-object shipping, (2) mesh group-key ABI is stable enough to reference, (3) threat-model split between A vs B is settled by real adversary. | 2026-05-26 | — |
 | #000062  | Mechanistic Witness: governed diagnostic sidecar (CNA/SAE/Neuronpedia) | **scaffold-only · awaiting go/no-go** (2026-05-26; Dav1d de-novo review §4.7 / §9.1.F). Specification of a mechanistic-interpretability sidecar that produces a content-addressed `MechanisticWitnessRoot` over (model, prompts, capture policy, neurons/features, intervention deltas), used as a **diagnostic input** to SelfModel (#000014/#000017) + benchmark-fixture generation. **Hard constraint:** soft signals never enter the hard proof path — `audit_mode` does NOT move based on witness output, `providence_cache` is untouched, `governance_policy_hash` only moves via explicit ForkScore ACCEPT with M+C+X axes passing (#000060 §7). Four guardrails (diagnostic-only by default · sandbox intervention only · no production steering without governance · feature labels never semantic proof). Witness root TLV-encodes `model_config_root | activation_capture_policy_root | contrastive_prompt_set_root | feature_or_neuron_set_root | intervention_result_root | behavioral_delta_root | safety_policy_root`. Scaffold only — no code until a real falsifier-in-hand use case exists + the four guardrails are restated in CLAUDE.md as rules + #000060 H-ABCDEFG-M+C+X harness exists to gate promotion. Captured to keep mechanistic-interp tooling out of the substrate unless and until it earns its place; the dual-use risk (Pan et al. 2025 CNA: 0.1% MLP ablation breaks refusal in 72B models) makes the governance-first framing load-bearing. | 2026-05-26 | — |

docs/corpus-history.md

@@ -77,3 +77,116 @@ initial ingest take?" is friction; one line in this doc removes that
 friction for the next operator (or for #000065 migration planning,
 where it informs the lower bound on a re-ingest if a teleport went
 wrong).
+
+## Wikipedia 2010 content-hash reshard (#000065)
+
+**Event.** Migrated the four production shards from non-deterministic
+"spray by ingest order" routing to canonical content-hash routing
+(`shard_idx = int(document_root[:8], 16) % 4`). Same content, new
+topology. First exercise of the SPV-wallet pattern at production
+scale.
+
+**When.** 2026-05-26 18:13 UTC → 19:47 UTC (~94 minutes wall, single-
+process serial executor).
+
+**Strategy.** `all_at_once` picked by the planner from 95.5 GB free /
+37.6 GB corpus / 64.2 GB predicted peak draw. Direct source→target
+read+write into `.db.new` files in the same `~/.arborist/shards/`
+directory; atomic `os.replace(.db.new, .db)` per file at the end.
+Originals never moved into a "v2 dir" or "backup dir" — the in-place
+`.db.new` → `.db` swap was fox's stated preference (α-shape in the
+#000065 discussion thread).
+
+**Output** (measured at 19:48 UTC via the executor's own row-count
+validation pass; matches pre-migration snapshot within tolerance).
+
+| Shard | docs | chunks | merkle_nodes | edges | concepts |
+|-------|-----:|-------:|-------------:|------:|---------:|
+| `000.db` | 866,628 | 1,562,008 | 822,180 | 22,540,066 | 187,063 |
+| `001.db` | 866,797 | 1,554,470 | 818,062 | 22,667,717 | 187,011 |
+| `002.db` | 866,825 | 1,560,728 | 821,274 | 22,667,416 | 187,268 |
+| `003.db` | 866,998 | 1,558,382 | 817,617 | 22,717,791 | 187,283 |
+| **total** | **3,467,248** | **6,235,588** | **3,279,133** | **90,592,990** | **748,625** |
+
+`audit_events` consolidated to canonical shard 000 (3,468,403 events,
+Option-A canonical chain: original events sorted by `ts` + re-chained
+with fresh `event_hash` values; bodies preserved unchanged for
+forensic fidelity). Tail event of new chain: `type=reshard`, hash
+`8da3aa19…`.
+
+**Per-shard final on-disk size:** 11.00 / 8.79 / 8.83 / 8.78 GB =
+37.4 GB total (shard 000 carries the consolidated audit chain so is
+~2 GB larger).
+
+**Doc routing uniformity:** 866,628 / 866,797 / 866,825 / 866,998
+across the 4 shards — max deviation 370 docs (0.04%). First-32-bit
+SHA-256 prefix is uniform; routing achieves the theoretical ±0.05%
+limit.
+
+**Validation deltas vs pre-migration snapshot** (within ±1% tolerance
+gate, executor refused to atomic-promote on mismatch — see
+`bench/results/pre-migration-snapshot.json`):
+
+```
+documents:  3,468,226 vs 3,468,226   exact         (0.00% delta)
+chunks:     6,235,588 vs 6,235,764   −176          (0.003% delta)
+edges:     90,592,990 vs 90,593,537  −547          (0.0006% delta)
+```
+
+The 176 missing chunks + 547 missing edges are cross-shard duplicates
+collapsed by `INSERT OR IGNORE` — pre-migration had 166 documents that
+appeared in multiple source shards (re-ingest history); their chunks +
+edges were also duplicated and the reshard consolidates them.
+
+**Throughput.** ~104.4M rows moved in 74 min of executor time (the
+20 minutes outside the executor were planner overhead + audit
+extraction + VACUUM at end). Aggregate ~23,500 rows/sec across all
+tables. Realistic ceiling on `python+sqlite3` content-hash routing at
+this corpus scale; bounded by SQLite executemany INSERT performance
+on indexed tables.
+
+**Two defects surfaced + fixed during the cutover** (both committed
+mid-flight on 2026-05-26 because they were also long-running issues
+hiding in our pre-existing pack format):
+
+1. Cross-shard FK violation on `derivations.src_root` —
+   single-shard-era guard fired on a row that legitimately routed
+   `core_root` to one target and `src_root` to another. Fix: writer
+   connection runs `PRAGMA foreign_keys = OFF`; runtime stays
+   FK=ON. Committed `04edff7`.
+
+2. Near-ENOSPC on WAL accumulation — SQLite auto-checkpoint can't
+   reclaim WAL pages while a reader cursor is open, and the FTS
+   rebuild's cursor stays open through 1.5M chunks per target. Four
+   shards' worth of FTS rebuild + audit consolidate accumulated
+   ~37 GB of committed-but-unreclaimable WAL; disk free fell to
+   7.9 GB before manual intervention via sibling-connection
+   `PRAGMA wal_checkpoint(TRUNCATE)` freed 27.7 GB. Fix: executor
+   now runs the checkpoint at the end of each `_rebuild_fts_on_target`
+   and after `_consolidate_audit_chain`. Committed `c86d5ac`.
+
+**SPV-wallet validation status.** This cutover exercised the reshard
+executor end-to-end but did NOT exercise the cold-pack hydrate path
+(reshard reads directly from local source shards; packs were not
+involved). Real pack-hydrate validation is queued at #44–#46:
+re-pack from new shards → upload to DO Spaces → genesis on
+3090-ai.foxhop.net from cloud.
+
+**Derivation queries.**
+
+Final per-shard row counts:
+```sql
+-- run against each ~/.arborist/shards/00N.db
+SELECT
+  (SELECT COUNT(*) FROM documents) AS docs,
+  (SELECT COUNT(*) FROM chunks) AS chunks,
+  (SELECT COUNT(*) FROM edges) AS edges,
+  (SELECT COUNT(*) FROM audit_events) AS audit;
+```
+
+Migration audit event:
+```sql
+-- tail event on shard 000 carries the full plan + result body
+SELECT event_type, body, ts FROM audit_events
+WHERE event_type = 'reshard' ORDER BY seq DESC LIMIT 1;
+```

docs/tickets/ticket-000065-canonical-shard-count-content-hash-routing.md

@@ -1,6 +1,6 @@
 # Ticket #000065 — Canonical shard count + content-hash routing
 
-**Status:** open · awaiting go/no-go · scaffold + design
+**Status:** closed · landed in `c86d5ac` · production reshard completed 2026-05-26 19:47 UTC (~94 min wall, 3.47M docs / 6.24M chunks / 90.6M edges / 3.47M audit events re-routed to content-hash M=4 layout; full record in `docs/corpus-history.md`). Two mid-flight fixes: cross-shard FK guard (`04edff7`) and WAL checkpoint between phases (`c86d5ac`). Follow-on work tracked separately as tasks #44–#47 (re-pack, verify bucket determinism, genesis fresh peer, retire stale packs).
 **Opened:** 2026-05-26
 **Scope:** decouple two roles that "shard count" currently conflates:
   ingest parallelism (producer-side, wants vCPU count) and consumer