# Corpus history

State changes at scale that aren't otherwise captured outside the
audit chain. Append-only; oldest events first. Each entry says **what
happened**, **when**, **how derived** (audit-chain query, git log,
operator note), and links to the data of record.

The audit chain is the durable source of truth for ingests / migrations
/ falsifications / cold-pack pushes; this doc surfaces the headline
numbers so future operators don't have to re-derive them from
`sqlite3 audit_events`.

## Wikipedia 2010 ingest — production corpus origin

**Event.** Initial ingest of Wikipedia 2010 dump
(`data/enwiki-20101011-pages-articles.xml.bz2`, 6.65 GB compressed)
into the four production shards.

**When.** 2026-04-27 21:33:13 UTC → 2026-04-27 23:13:28 UTC
(~100 minutes wall clock, all four shards run in parallel).

**Source dump.** `enwiki-20101011-pages-articles.xml.bz2`, 6,652,983,189
bytes, mtime 2026-05-07 (fetched before ingest).

**Output** (measured 2026-05-26 via `bench/pre_migration_snapshot.py`;
authoritative reference for #000065 reshard verification).

| Shard | docs | chunks | edges | audit events | first ingest | last ingest |
|-------|-----:|-------:|------:|-------------:|--------------|-------------|
| `000.db` | 866,782 | 1,560,021 | 22,664,792 | 866,886 | 2026-04-27 21:33:13Z | 2026-04-27 23:13:20Z |
| `001.db` | 867,695 | 1,545,929 | 22,612,247 | 867,695 | 2026-04-27 21:33:13Z | 2026-04-27 23:13:01Z |
| `002.db` | 866,825 | 1,600,420 | 22,649,376 | 866,825 | 2026-04-27 21:33:13Z | 2026-04-27 23:13:26Z |
| `003.db` | 866,998 | 1,529,394 | 22,667,122 | 866,997 | 2026-04-27 21:33:13Z | 2026-04-27 23:13:28Z |
| **total** | **3,468,300** | **6,235,764** | **90,593,537** | **3,468,403** | | |

(Shard 000 has 104 additional ingest events on 2026-05-09 — a later
top-up of post-2010 fillers; doesn't change the bulk timing.
3,468,226 globally unique `document_root` values vs 3,468,392 raw rows
→ 166 cross-shard duplicates from re-ingest history.)

**Throughput.** ~580 docs/sec aggregate across 4 workers ≈ 145
docs/sec per shard ≈ 1,040 chunks/sec aggregate ≈ 15,000 edges/sec.
Realistic production rate for Wikipedia-XML workload (XML parse +
canonicalization + edge extraction + chunker pass per document).

The 2026-05-26 synthetic + 2003-cur-dump benchmarks
(`bench/shard_count_sweep.py`) measured ~6,400 chunks/sec aggregate at
M=4 on the same hardware — that rate excludes XML parsing because the
cur dump is pre-parsed SQL, so it overstates throughput by ~6× for
planning purposes against the XML pipeline. For #000065 reshard
planning the relevant rate is **INSERT throughput only** (no parse,
no canonicalize, no chunker), which should beat both XML and cur
ceilings.

**Hardware.** 8-vCPU box (cf. CLAUDE.md "Per-call model selection" +
the live bench observations from #000061).

**Derivation.** Run on shard `000.db`:

```sql
SELECT
  datetime(MIN(ts), 'unixepoch') AS first,
  datetime(MAX(ts), 'unixepoch') AS last,
  printf('%.2f', (MAX(ts) - MIN(ts)) / 60.0) AS minutes,
  COUNT(*) AS docs
FROM audit_events
WHERE event_type = 'ingest' AND ts < strftime('%s', '2026-05-01');
```

Repeat per shard. The ingest event body carries `document_uri` +
`chunks` + `source_type` per row so all the per-doc details are
recoverable forensically.

**Why captured here.** The audit chain holds this in the body of every
ingest event but reading 3.47M rows to answer "how long did the
initial ingest take?" is friction; one line in this doc removes that
friction for the next operator (or for #000065 migration planning,
where it informs the lower bound on a re-ingest if a teleport went
wrong).

## Wikipedia 2010 content-hash reshard (#000065)

**Event.** Migrated the four production shards from non-deterministic
"spray by ingest order" routing to canonical content-hash routing
(`shard_idx = int(document_root[:8], 16) % 4`). Same content, new
topology. First exercise of the SPV-wallet pattern at production
scale.

**When.** 2026-05-26 18:13 UTC → 19:47 UTC (~94 minutes wall, single-
process serial executor).

**Strategy.** `all_at_once` picked by the planner from 95.5 GB free /
37.6 GB corpus / 64.2 GB predicted peak draw. Direct source→target
read+write into `.db.new` files in the same `~/.arborist/shards/`
directory; atomic `os.replace(.db.new, .db)` per file at the end.
Originals never moved into a "v2 dir" or "backup dir" — the in-place
`.db.new` → `.db` swap was fox's stated preference (α-shape in the
#000065 discussion thread).

**Output** (measured at 19:48 UTC via the executor's own row-count
validation pass; matches pre-migration snapshot within tolerance).

| Shard | docs | chunks | merkle_nodes | edges | concepts |
|-------|-----:|-------:|-------------:|------:|---------:|
| `000.db` | 866,628 | 1,562,008 | 822,180 | 22,540,066 | 187,063 |
| `001.db` | 866,797 | 1,554,470 | 818,062 | 22,667,717 | 187,011 |
| `002.db` | 866,825 | 1,560,728 | 821,274 | 22,667,416 | 187,268 |
| `003.db` | 866,998 | 1,558,382 | 817,617 | 22,717,791 | 187,283 |
| **total** | **3,467,248** | **6,235,588** | **3,279,133** | **90,592,990** | **748,625** |

`audit_events` consolidated to canonical shard 000 (3,468,403 events,
Option-A canonical chain: original events sorted by `ts` + re-chained
with fresh `event_hash` values; bodies preserved unchanged for
forensic fidelity). Tail event of new chain: `type=reshard`, hash
`8da3aa19…`.

**Per-shard final on-disk size:** 11.00 / 8.79 / 8.83 / 8.78 GB =
37.4 GB total (shard 000 carries the consolidated audit chain so is
~2 GB larger).

**Doc routing uniformity:** 866,628 / 866,797 / 866,825 / 866,998
across the 4 shards — max deviation 370 docs (0.04%). First-32-bit
SHA-256 prefix is uniform; routing achieves the theoretical ±0.05%
limit.

**Validation deltas vs pre-migration snapshot** (within ±1% tolerance
gate, executor refused to atomic-promote on mismatch — see
`bench/results/pre-migration-snapshot.json`):

```
documents:  3,468,226 vs 3,468,226   exact         (0.00% delta)
chunks:     6,235,588 vs 6,235,764   −176          (0.003% delta)
edges:     90,592,990 vs 90,593,537  −547          (0.0006% delta)
```

The 176 missing chunks + 547 missing edges are cross-shard duplicates
collapsed by `INSERT OR IGNORE` — pre-migration had 166 documents that
appeared in multiple source shards (re-ingest history); their chunks +
edges were also duplicated and the reshard consolidates them.

**Throughput.** ~104.4M rows moved in 74 min of executor time (the
20 minutes outside the executor were planner overhead + audit
extraction + VACUUM at end). Aggregate ~23,500 rows/sec across all
tables. Realistic ceiling on `python+sqlite3` content-hash routing at
this corpus scale; bounded by SQLite executemany INSERT performance
on indexed tables.

**Two defects surfaced + fixed during the cutover** (both committed
mid-flight on 2026-05-26 because they were also long-running issues
hiding in our pre-existing pack format):

1. Cross-shard FK violation on `derivations.src_root` —
   single-shard-era guard fired on a row that legitimately routed
   `core_root` to one target and `src_root` to another. Fix: writer
   connection runs `PRAGMA foreign_keys = OFF`; runtime stays
   FK=ON. Committed `04edff7`.

2. Near-ENOSPC on WAL accumulation — SQLite auto-checkpoint can't
   reclaim WAL pages while a reader cursor is open, and the FTS
   rebuild's cursor stays open through 1.5M chunks per target. Four
   shards' worth of FTS rebuild + audit consolidate accumulated
   ~37 GB of committed-but-unreclaimable WAL; disk free fell to
   7.9 GB before manual intervention via sibling-connection
   `PRAGMA wal_checkpoint(TRUNCATE)` freed 27.7 GB. Fix: executor
   now runs the checkpoint at the end of each `_rebuild_fts_on_target`
   and after `_consolidate_audit_chain`. Committed `c86d5ac`.

**SPV-wallet validation status.** This cutover exercised the reshard
executor end-to-end but did NOT exercise the cold-pack hydrate path
(reshard reads directly from local source shards; packs were not
involved). Real pack-hydrate validation is queued at #44–#46:
re-pack from new shards → upload to DO Spaces → genesis on
3090-ai.foxhop.net from cloud.

**Derivation queries.**

Final per-shard row counts:
```sql
-- run against each ~/.arborist/shards/00N.db
SELECT
  (SELECT COUNT(*) FROM documents) AS docs,
  (SELECT COUNT(*) FROM chunks) AS chunks,
  (SELECT COUNT(*) FROM edges) AS edges,
  (SELECT COUNT(*) FROM audit_events) AS audit;
```

Migration audit event:
```sql
-- tail event on shard 000 carries the full plan + result body
SELECT event_type, body, ts FROM audit_events
WHERE event_type = 'reshard' ORDER BY seq DESC LIMIT 1;
```
