# Corpus history

State changes at scale that aren't otherwise captured outside the
audit chain. Append-only; oldest events first. Each entry says **what
happened**, **when**, **how derived** (audit-chain query, git log,
operator note), and links to the data of record.

The audit chain is the durable source of truth for ingests / migrations
/ falsifications / cold-pack pushes; this doc surfaces the headline
numbers so future operators don't have to re-derive them from
`sqlite3 audit_events`.

## Wikipedia 2010 ingest — production corpus origin

**Event.** Initial ingest of Wikipedia 2010 dump
(`data/enwiki-20101011-pages-articles.xml.bz2`, 6.65 GB compressed)
into the four production shards.

**When.** 2026-04-27 21:33:13 UTC → 2026-04-27 23:13:28 UTC
(~100 minutes wall clock, all four shards run in parallel).

**Source dump.** `enwiki-20101011-pages-articles.xml.bz2`, 6,652,983,189
bytes, mtime 2026-05-07 (fetched before ingest).

**Output** (measured 2026-05-26 via `bench/pre_migration_snapshot.py`;
authoritative reference for #000065 reshard verification).

| Shard | docs | chunks | edges | audit events | first ingest | last ingest |
|-------|-----:|-------:|------:|-------------:|--------------|-------------|
| `000.db` | 866,782 | 1,560,021 | 22,664,792 | 866,886 | 2026-04-27 21:33:13Z | 2026-04-27 23:13:20Z |
| `001.db` | 867,695 | 1,545,929 | 22,612,247 | 867,695 | 2026-04-27 21:33:13Z | 2026-04-27 23:13:01Z |
| `002.db` | 866,825 | 1,600,420 | 22,649,376 | 866,825 | 2026-04-27 21:33:13Z | 2026-04-27 23:13:26Z |
| `003.db` | 866,998 | 1,529,394 | 22,667,122 | 866,997 | 2026-04-27 21:33:13Z | 2026-04-27 23:13:28Z |
| **total** | **3,468,300** | **6,235,764** | **90,593,537** | **3,468,403** | | |

(Shard 000 has 104 additional ingest events on 2026-05-09 — a later
top-up of post-2010 fillers; doesn't change the bulk timing.
3,468,226 globally unique `document_root` values vs 3,468,392 raw rows
→ 166 cross-shard duplicates from re-ingest history.)

**Throughput.** ~580 docs/sec aggregate across 4 workers ≈ 145
docs/sec per shard ≈ 1,040 chunks/sec aggregate ≈ 15,000 edges/sec.
Realistic production rate for Wikipedia-XML workload (XML parse +
canonicalization + edge extraction + chunker pass per document).

The 2026-05-26 synthetic + 2003-cur-dump benchmarks
(`bench/shard_count_sweep.py`) measured ~6,400 chunks/sec aggregate at
M=4 on the same hardware — that rate excludes XML parsing because the
cur dump is pre-parsed SQL, so it overstates throughput by ~6× for
planning purposes against the XML pipeline. For #000065 reshard
planning the relevant rate is **INSERT throughput only** (no parse,
no canonicalize, no chunker), which should beat both XML and cur
ceilings.

**Hardware.** 8-vCPU box (cf. CLAUDE.md "Per-call model selection" +
the live bench observations from #000061).

**Derivation.** Run on shard `000.db`:

```sql
SELECT
  datetime(MIN(ts), 'unixepoch') AS first,
  datetime(MAX(ts), 'unixepoch') AS last,
  printf('%.2f', (MAX(ts) - MIN(ts)) / 60.0) AS minutes,
  COUNT(*) AS docs
FROM audit_events
WHERE event_type = 'ingest' AND ts < strftime('%s', '2026-05-01');
```

Repeat per shard. The ingest event body carries `document_uri` +
`chunks` + `source_type` per row so all the per-doc details are
recoverable forensically.

**Why captured here.** The audit chain holds this in the body of every
ingest event but reading 3.47M rows to answer "how long did the
initial ingest take?" is friction; one line in this doc removes that
friction for the next operator (or for #000065 migration planning,
where it informs the lower bound on a re-ingest if a teleport went
wrong).