From 0ddff431429b257bf6b3600e29c47cfdf1f24a39 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Tue, 26 May 2026 12:48:12 -0400 Subject: [PATCH] docs: capture Wikipedia 2010 ingest origin (the 100-min, 4-way parallel run) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The original ingest of Wikipedia 2010 into fox's 4 production shards was 2026-04-27 21:33-23:13 UTC — ~100 minutes wall, 4-way parallel, 3.47M docs / 14.12M chunks. Captured here because the audit chain is the only durable record but querying 3.47M rows to recover the headline number is friction; one line in a doc removes it. Also informs #000065 reshard planning: ingest rate ceiling on real XML workload is ~2,350 chunks/sec aggregate (4-way), vs the ~6,400 chunks/sec the M-sweep bench measured on the 2003 cur dump (which skips XML parsing). The teleport-style reshard should beat both ceilings because it's just SQLite INSERT throughput, no XML parse + canonicalize + edge extraction. New file: docs/corpus-history.md. Indexed in CLAUDE.md docs section. Append-only convention; future migrations + cold-pack runs add entries here so the operator log isn't only in the audit chain. Derivation query (sqlite3 audit_events) embedded in the entry so future re-derivation is one copy-paste. --- CLAUDE.md | 5 +++ docs/corpus-history.md | 73 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+) create mode 100644 docs/corpus-history.md diff --git a/CLAUDE.md b/CLAUDE.md index 00d5f6b..83acaaa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -598,6 +598,11 @@ Architecture / ongoing work: - `docs/embedding.md` — embedding arborist as a library in another Python app (`arborist.embed`): produce `Document`s → ingest → dedup + FTS5 + audit chain. The neopig-backend seam. +- `docs/corpus-history.md` — durable note of state changes at scale + that aren't otherwise captured outside the audit chain (initial + Wikipedia 2010 ingest, future migrations, cold-pack runs). Append- + only; one entry per event. Surface for the headline numbers + a + pointer to the audit-chain query that derived them. - `docs/cold-object-store.md` — cold-pack distribution tier (#000061): serialize the corpus into `tar.zst` packs and ship them via S3-compatible buckets (DO Spaces / AWS S3 / R2 / B2 / GCS / MinIO via boto3) and/or diff --git a/docs/corpus-history.md b/docs/corpus-history.md new file mode 100644 index 0000000..962e4a0 --- /dev/null +++ b/docs/corpus-history.md @@ -0,0 +1,73 @@ +# Corpus history + +State changes at scale that aren't otherwise captured outside the +audit chain. Append-only; oldest events first. Each entry says **what +happened**, **when**, **how derived** (audit-chain query, git log, +operator note), and links to the data of record. + +The audit chain is the durable source of truth for ingests / migrations +/ falsifications / cold-pack pushes; this doc surfaces the headline +numbers so future operators don't have to re-derive them from +`sqlite3 audit_events`. + +## Wikipedia 2010 ingest — production corpus origin + +**Event.** Initial ingest of Wikipedia 2010 dump +(`data/enwiki-20101011-pages-articles.xml.bz2`, 6.65 GB compressed) +into the four production shards. + +**When.** 2026-04-27 21:33:13 UTC → 2026-04-27 23:13:28 UTC +(~100 minutes wall clock, all four shards run in parallel). + +**Source dump.** `enwiki-20101011-pages-articles.xml.bz2`, 6,652,983,189 +bytes, mtime 2026-05-07 (fetched before ingest). + +**Output.** + +| Shard | docs | chunks | first ingest event | last ingest event | +|-------|-----:|-------:|--------------------|-------------------| +| `000.db` | 866,782 | ~3.54M | 2026-04-27 21:33:13Z | 2026-04-27 23:13:20Z | +| `001.db` | 867,695 | ~3.51M | 2026-04-27 21:33:13Z | 2026-04-27 23:13:01Z | +| `002.db` | 866,825 | ~3.63M | 2026-04-27 21:33:13Z | 2026-04-27 23:13:26Z | +| `003.db` | 866,998 | ~3.44M | 2026-04-27 21:33:13Z | 2026-04-27 23:13:28Z | +| **total** | **3,468,300** | **~14.12M** | | | + +(Shard 000 also has 92 additional ingest events on 2026-05-09 — a +later top-up of post-2010 fillers; doesn't change the bulk timing.) + +**Throughput.** ~580 docs/sec aggregate across 4 workers ≈ 145 +docs/sec per shard ≈ 2,350 chunks/sec aggregate. This is the realistic +production ingest rate for Wikipedia-XML workload (XML parsing + +canonicalization + edge extraction + chunker pass per document). + +The 2026-05-26 synthetic + 2003-cur-dump benchmarks +(`bench/shard_count_sweep.py`) measured ~6,400 chunks/sec aggregate at +M=4 on the same hardware — that rate excludes XML parsing because the +cur dump is pre-parsed SQL, so it overstates throughput by ~2.7× for +planning purposes against the XML pipeline. + +**Hardware.** 8-vCPU box (cf. CLAUDE.md "Per-call model selection" + +the live bench observations from #000061). + +**Derivation.** Run on shard `000.db`: + +```sql +SELECT + datetime(MIN(ts), 'unixepoch') AS first, + datetime(MAX(ts), 'unixepoch') AS last, + printf('%.2f', (MAX(ts) - MIN(ts)) / 60.0) AS minutes, + COUNT(*) AS docs +FROM audit_events +WHERE event_type = 'ingest' AND ts < strftime('%s', '2026-05-01'); +``` + +Repeat per shard. The ingest event body carries `document_uri` + +`chunks` + `source_type` per row so all the per-doc details are +recoverable forensically. + +**Why captured here.** The audit chain holds this in the body of every +ingest event but reading 3.47M rows to answer "how long did the +initial ingest take?" is friction; one line in this doc removes that +friction for the next operator (or for #000065 migration planning, +where it informs the lower bound on a re-ingest if a teleport went +wrong).