russell@unturf.com
3aae8119a6
Three pieces, all read-only or additive — no shard mutation, no
schema-version bump:
1. shard_for_document(document_root, M) in arborist/document.py.
Pure function: int(document_root[:8], 16) % M. 22 tests cover
determinism, range-bounds, near-uniform distribution (±5pp at
N=20k), and seven lock-in fixtures so peers will disagree
loudly if anyone changes the formula.
2. corpus_shard_count meta field + get/set helpers in store.py.
Lives in the existing key/value meta table; SCHEMA_VERSION
stays at v9.8.0 (the DDL doesn't change and source_root is
layout-independent, so cache records survive a reshard).
Legacy shards (without the field) return None; reshard tool
populates it on every target shard at migration time.
3. Pre-migration snapshot captured to
bench/results/pre-migration-snapshot.json:
docs 3,468,392 (3,468,226 globally unique)
chunks 6,235,764
edges 90,593,537
audit 3,468,403
This is the reference set post-reshard row counts must match.
4. Audit-event extraction script writes all 3.47M events from
all 4 shards to /tmp/audit-events.ndjson (2.0 GB) for the
Option-A canonical-chain consolidation step. Verifies chain
integrity on extract — all 4 source chains report 0 breaks.
5. Fixed a wrong chunk count in docs/corpus-history.md
(had ~3.54M/shard; actual is ~1.56M/shard) and added the
edge-count column (~22.6M/shard, 90.6M total). 6.24M chunks
total, not 14.12M.
Tests: 29 new pass (22 routing + 7 meta). No existing tests
touched.
79 lines
3.6 KiB
Markdown
79 lines
3.6 KiB
Markdown
# Corpus history
|
||
|
||
State changes at scale that aren't otherwise captured outside the
|
||
audit chain. Append-only; oldest events first. Each entry says **what
|
||
happened**, **when**, **how derived** (audit-chain query, git log,
|
||
operator note), and links to the data of record.
|
||
|
||
The audit chain is the durable source of truth for ingests / migrations
|
||
/ falsifications / cold-pack pushes; this doc surfaces the headline
|
||
numbers so future operators don't have to re-derive them from
|
||
`sqlite3 audit_events`.
|
||
|
||
## Wikipedia 2010 ingest — production corpus origin
|
||
|
||
**Event.** Initial ingest of Wikipedia 2010 dump
|
||
(`data/enwiki-20101011-pages-articles.xml.bz2`, 6.65 GB compressed)
|
||
into the four production shards.
|
||
|
||
**When.** 2026-04-27 21:33:13 UTC → 2026-04-27 23:13:28 UTC
|
||
(~100 minutes wall clock, all four shards run in parallel).
|
||
|
||
**Source dump.** `enwiki-20101011-pages-articles.xml.bz2`, 6,652,983,189
|
||
bytes, mtime 2026-05-07 (fetched before ingest).
|
||
|
||
**Output** (measured 2026-05-26 via `bench/pre_migration_snapshot.py`;
|
||
authoritative reference for #000065 reshard verification).
|
||
|
||
| Shard | docs | chunks | edges | audit events | first ingest | last ingest |
|
||
|-------|-----:|-------:|------:|-------------:|--------------|-------------|
|
||
| `000.db` | 866,782 | 1,560,021 | 22,664,792 | 866,886 | 2026-04-27 21:33:13Z | 2026-04-27 23:13:20Z |
|
||
| `001.db` | 867,695 | 1,545,929 | 22,612,247 | 867,695 | 2026-04-27 21:33:13Z | 2026-04-27 23:13:01Z |
|
||
| `002.db` | 866,825 | 1,600,420 | 22,649,376 | 866,825 | 2026-04-27 21:33:13Z | 2026-04-27 23:13:26Z |
|
||
| `003.db` | 866,998 | 1,529,394 | 22,667,122 | 866,997 | 2026-04-27 21:33:13Z | 2026-04-27 23:13:28Z |
|
||
| **total** | **3,468,300** | **6,235,764** | **90,593,537** | **3,468,403** | | |
|
||
|
||
(Shard 000 has 104 additional ingest events on 2026-05-09 — a later
|
||
top-up of post-2010 fillers; doesn't change the bulk timing.
|
||
3,468,226 globally unique `document_root` values vs 3,468,392 raw rows
|
||
→ 166 cross-shard duplicates from re-ingest history.)
|
||
|
||
**Throughput.** ~580 docs/sec aggregate across 4 workers ≈ 145
|
||
docs/sec per shard ≈ 1,040 chunks/sec aggregate ≈ 15,000 edges/sec.
|
||
Realistic production rate for Wikipedia-XML workload (XML parse +
|
||
canonicalization + edge extraction + chunker pass per document).
|
||
|
||
The 2026-05-26 synthetic + 2003-cur-dump benchmarks
|
||
(`bench/shard_count_sweep.py`) measured ~6,400 chunks/sec aggregate at
|
||
M=4 on the same hardware — that rate excludes XML parsing because the
|
||
cur dump is pre-parsed SQL, so it overstates throughput by ~6× for
|
||
planning purposes against the XML pipeline. For #000065 reshard
|
||
planning the relevant rate is **INSERT throughput only** (no parse,
|
||
no canonicalize, no chunker), which should beat both XML and cur
|
||
ceilings.
|
||
|
||
**Hardware.** 8-vCPU box (cf. CLAUDE.md "Per-call model selection" +
|
||
the live bench observations from #000061).
|
||
|
||
**Derivation.** Run on shard `000.db`:
|
||
|
||
```sql
|
||
SELECT
|
||
datetime(MIN(ts), 'unixepoch') AS first,
|
||
datetime(MAX(ts), 'unixepoch') AS last,
|
||
printf('%.2f', (MAX(ts) - MIN(ts)) / 60.0) AS minutes,
|
||
COUNT(*) AS docs
|
||
FROM audit_events
|
||
WHERE event_type = 'ingest' AND ts < strftime('%s', '2026-05-01');
|
||
```
|
||
|
||
Repeat per shard. The ingest event body carries `document_uri` +
|
||
`chunks` + `source_type` per row so all the per-doc details are
|
||
recoverable forensically.
|
||
|
||
**Why captured here.** The audit chain holds this in the body of every
|
||
ingest event but reading 3.47M rows to answer "how long did the
|
||
initial ingest take?" is friction; one line in this doc removes that
|
||
friction for the next operator (or for #000065 migration planning,
|
||
where it informs the lower bound on a re-ingest if a teleport went
|
||
wrong).
|