engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: capture Wikipedia 2010 ingest origin (the 100-min, 4-way parallel run)

Browse source 
The original ingest of Wikipedia 2010 into fox's 4 production shards
was 2026-04-27 21:33-23:13 UTC — ~100 minutes wall, 4-way parallel,
3.47M docs / 14.12M chunks. Captured here because the audit chain is
the only durable record but querying 3.47M rows to recover the
headline number is friction; one line in a doc removes it.

Also informs #000065 reshard planning: ingest rate ceiling on real
XML workload is ~2,350 chunks/sec aggregate (4-way), vs the
~6,400 chunks/sec the M-sweep bench measured on the 2003 cur dump
(which skips XML parsing). The teleport-style reshard should beat
both ceilings because it's just SQLite INSERT throughput, no XML
parse + canonicalize + edge extraction.

New file: docs/corpus-history.md. Indexed in CLAUDE.md docs section.
Append-only convention; future migrations + cold-pack runs add
entries here so the operator log isn't only in the audit chain.

Derivation query (sqlite3 audit_events) embedded in the entry so
future re-derivation is one copy-paste.
This commit is contained in:
russell@unturf.com 2026-05-26 12:48:12 -04:00
parent 967fedbbe0
commit 0ddff43142
No known key found for this signature in database
2 changed files with 78 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

73
docs/corpus-history.md Normal file
View file

@ -0,0 +1,73 @@
# Corpus history
State changes at scale that aren't otherwise captured outside the
audit chain. Append-only; oldest events first. Each entry says **what
happened**, **when**, **how derived** (audit-chain query, git log,
operator note), and links to the data of record.
The audit chain is the durable source of truth for ingests / migrations
/ falsifications / cold-pack pushes; this doc surfaces the headline
numbers so future operators don't have to re-derive them from
`sqlite3 audit_events`.
## Wikipedia 2010 ingest — production corpus origin
**Event.** Initial ingest of Wikipedia 2010 dump
(`data/enwiki-20101011-pages-articles.xml.bz2`, 6.65 GB compressed)
into the four production shards.
**When.** 2026-04-27 21:33:13 UTC → 2026-04-27 23:13:28 UTC
(~100 minutes wall clock, all four shards run in parallel).
**Source dump.** `enwiki-20101011-pages-articles.xml.bz2`, 6,652,983,189
bytes, mtime 2026-05-07 (fetched before ingest).
**Output.**
| Shard | docs | chunks | first ingest event | last ingest event |
|-------|-----:|-------:|--------------------|-------------------|
| `000.db` | 866,782 | ~3.54M | 2026-04-27 21:33:13Z | 2026-04-27 23:13:20Z |
| `001.db` | 867,695 | ~3.51M | 2026-04-27 21:33:13Z | 2026-04-27 23:13:01Z |
| `002.db` | 866,825 | ~3.63M | 2026-04-27 21:33:13Z | 2026-04-27 23:13:26Z |
| `003.db` | 866,998 | ~3.44M | 2026-04-27 21:33:13Z | 2026-04-27 23:13:28Z |
| **total** | **3,468,300** | **~14.12M** | | |
(Shard 000 also has 92 additional ingest events on 2026-05-09 — a
later top-up of post-2010 fillers; doesn't change the bulk timing.)
**Throughput.** ~580 docs/sec aggregate across 4 workers ≈ 145
docs/sec per shard ≈ 2,350 chunks/sec aggregate. This is the realistic
production ingest rate for Wikipedia-XML workload (XML parsing +
canonicalization + edge extraction + chunker pass per document).
The 2026-05-26 synthetic + 2003-cur-dump benchmarks
(`bench/shard_count_sweep.py`) measured ~6,400 chunks/sec aggregate at
M=4 on the same hardware — that rate excludes XML parsing because the
cur dump is pre-parsed SQL, so it overstates throughput by ~2.7× for
planning purposes against the XML pipeline.
**Hardware.** 8-vCPU box (cf. CLAUDE.md "Per-call model selection" +
the live bench observations from #000061).
**Derivation.** Run on shard `000.db`:
```sql
SELECT
datetime(MIN(ts), 'unixepoch') AS first,
datetime(MAX(ts), 'unixepoch') AS last,
printf('%.2f', (MAX(ts) - MIN(ts)) / 60.0) AS minutes,
COUNT(*) AS docs
FROM audit_events
WHERE event_type = 'ingest' AND ts < strftime('%s', '2026-05-01');
```
Repeat per shard. The ingest event body carries `document_uri` +
`chunks` + `source_type` per row so all the per-doc details are
recoverable forensically.
**Why captured here.** The audit chain holds this in the body of every
ingest event but reading 3.47M rows to answer "how long did the
initial ingest take?" is friction; one line in this doc removes that
friction for the next operator (or for #000065 migration planning,
where it informs the lower bound on a re-ingest if a teleport went
wrong).