arborist/docs/corpus-history.md

Corpus history

State changes at scale that aren't otherwise captured outside the audit chain. Append-only; oldest events first. Each entry says what happened, when, how derived (audit-chain query, git log, operator note), and links to the data of record.

The audit chain is the durable source of truth for ingests / migrations / falsifications / cold-pack pushes; this doc surfaces the headline numbers so future operators don't have to re-derive them from sqlite3 audit_events.

Wikipedia 2010 ingest — production corpus origin

Event. Initial ingest of Wikipedia 2010 dump (data/enwiki-20101011-pages-articles.xml.bz2, 6.65 GB compressed) into the four production shards.

When. 2026-04-27 21:33:13 UTC → 2026-04-27 23:13:28 UTC (~100 minutes wall clock, all four shards run in parallel).

Source dump. enwiki-20101011-pages-articles.xml.bz2, 6,652,983,189 bytes, mtime 2026-05-07 (fetched before ingest).

Output (measured 2026-05-26 via bench/pre_migration_snapshot.py; authoritative reference for #000065 reshard verification).

Shard	docs	chunks	edges	audit events	first ingest	last ingest
000.db	866,782	1,560,021	22,664,792	866,886	2026-04-27 21:33:13Z	2026-04-27 23:13:20Z
001.db	867,695	1,545,929	22,612,247	867,695	2026-04-27 21:33:13Z	2026-04-27 23:13:01Z
002.db	866,825	1,600,420	22,649,376	866,825	2026-04-27 21:33:13Z	2026-04-27 23:13:26Z
003.db	866,998	1,529,394	22,667,122	866,997	2026-04-27 21:33:13Z	2026-04-27 23:13:28Z
total	3,468,300	6,235,764	90,593,537	3,468,403

(Shard 000 has 104 additional ingest events on 2026-05-09 — a later top-up of post-2010 fillers; doesn't change the bulk timing. 3,468,226 globally unique document_root values vs 3,468,392 raw rows → 166 cross-shard duplicates from re-ingest history.)

Throughput. ~580 docs/sec aggregate across 4 workers ≈ 145 docs/sec per shard ≈ 1,040 chunks/sec aggregate ≈ 15,000 edges/sec. Realistic production rate for Wikipedia-XML workload (XML parse + canonicalization + edge extraction + chunker pass per document).

The 2026-05-26 synthetic + 2003-cur-dump benchmarks (bench/shard_count_sweep.py) measured ~6,400 chunks/sec aggregate at M=4 on the same hardware — that rate excludes XML parsing because the cur dump is pre-parsed SQL, so it overstates throughput by ~6× for planning purposes against the XML pipeline. For #000065 reshard planning the relevant rate is INSERT throughput only (no parse, no canonicalize, no chunker), which should beat both XML and cur ceilings.

Hardware. 8-vCPU box (cf. CLAUDE.md "Per-call model selection" + the live bench observations from #000061).

Derivation. Run on shard 000.db:

SELECT
  datetime(MIN(ts), 'unixepoch') AS first,
  datetime(MAX(ts), 'unixepoch') AS last,
  printf('%.2f', (MAX(ts) - MIN(ts)) / 60.0) AS minutes,
  COUNT(*) AS docs
FROM audit_events
WHERE event_type = 'ingest' AND ts < strftime('%s', '2026-05-01');

Repeat per shard. The ingest event body carries document_uri + chunks + source_type per row so all the per-doc details are recoverable forensically.

Why captured here. The audit chain holds this in the body of every ingest event but reading 3.47M rows to answer "how long did the initial ingest take?" is friction; one line in this doc removes that friction for the next operator (or for #000065 migration planning, where it informs the lower bound on a re-ingest if a teleport went wrong).

Wikipedia 2010 content-hash reshard (#000065)

Event. Migrated the four production shards from non-deterministic "spray by ingest order" routing to canonical content-hash routing (shard_idx = int(document_root[:8], 16) % 4). Same content, new topology. First exercise of the SPV-wallet pattern at production scale.

When. 2026-05-26 18:13 UTC → 19:47 UTC (~94 minutes wall, single- process serial executor).

Strategy. all_at_once picked by the planner from 95.5 GB free / 37.6 GB corpus / 64.2 GB predicted peak draw. Direct source→target read+write into .db.new files in the same ~/.arborist/shards/ directory; atomic os.replace(.db.new, .db) per file at the end. Originals never moved into a "v2 dir" or "backup dir" — the in-place .db.new → .db swap was fox's stated preference (α-shape in the #000065 discussion thread).

Output (measured at 19:48 UTC via the executor's own row-count validation pass; matches pre-migration snapshot within tolerance).

Shard	docs	chunks	merkle_nodes	edges	concepts
000.db	866,628	1,562,008	822,180	22,540,066	187,063
001.db	866,797	1,554,470	818,062	22,667,717	187,011
002.db	866,825	1,560,728	821,274	22,667,416	187,268
003.db	866,998	1,558,382	817,617	22,717,791	187,283
total	3,467,248	6,235,588	3,279,133	90,592,990	748,625

audit_events consolidated to canonical shard 000 (3,468,403 events, Option-A canonical chain: original events sorted by ts + re-chained with fresh event_hash values; bodies preserved unchanged for forensic fidelity). Tail event of new chain: type=reshard, hash 8da3aa19….

Per-shard final on-disk size: 11.00 / 8.79 / 8.83 / 8.78 GB = 37.4 GB total (shard 000 carries the consolidated audit chain so is ~2 GB larger).

Doc routing uniformity: 866,628 / 866,797 / 866,825 / 866,998 across the 4 shards — max deviation 370 docs (0.04%). First-32-bit SHA-256 prefix is uniform; routing achieves the theoretical ±0.05% limit.

Validation deltas vs pre-migration snapshot (within ±1% tolerance gate, executor refused to atomic-promote on mismatch — see bench/results/pre-migration-snapshot.json):

documents:  3,468,226 vs 3,468,226   exact         (0.00% delta)
chunks:     6,235,588 vs 6,235,764   −176          (0.003% delta)
edges:     90,592,990 vs 90,593,537  −547          (0.0006% delta)

The 176 missing chunks + 547 missing edges are cross-shard duplicates collapsed by INSERT OR IGNORE — pre-migration had 166 documents that appeared in multiple source shards (re-ingest history); their chunks + edges were also duplicated and the reshard consolidates them.

Throughput. ~104.4M rows moved in 74 min of executor time (the 20 minutes outside the executor were planner overhead + audit extraction + VACUUM at end). Aggregate ~23,500 rows/sec across all tables. Realistic ceiling on python+sqlite3 content-hash routing at this corpus scale; bounded by SQLite executemany INSERT performance on indexed tables.

Two defects surfaced + fixed during the cutover (both committed mid-flight on 2026-05-26 because they were also long-running issues hiding in our pre-existing pack format):

1. Cross-shard FK violation on derivations.src_root — single-shard-era guard fired on a row that legitimately routed core_root to one target and src_root to another. Fix: writer connection runs PRAGMA foreign_keys = OFF; runtime stays FK=ON. Committed 04edff7.

2. Near-ENOSPC on WAL accumulation — SQLite auto-checkpoint can't reclaim WAL pages while a reader cursor is open, and the FTS rebuild's cursor stays open through 1.5M chunks per target. Four shards' worth of FTS rebuild + audit consolidate accumulated ~37 GB of committed-but-unreclaimable WAL; disk free fell to 7.9 GB before manual intervention via sibling-connection PRAGMA wal_checkpoint(TRUNCATE) freed 27.7 GB. Fix: executor now runs the checkpoint at the end of each _rebuild_fts_on_target and after _consolidate_audit_chain. Committed c86d5ac.

SPV-wallet validation status. This cutover exercised the reshard executor end-to-end but did NOT exercise the cold-pack hydrate path (reshard reads directly from local source shards; packs were not involved). Real pack-hydrate validation is queued at #44–#46: re-pack from new shards → upload to DO Spaces → genesis on 3090-ai.foxhop.net from cloud.

Derivation queries.

Final per-shard row counts:

-- run against each ~/.arborist/shards/00N.db
SELECT
  (SELECT COUNT(*) FROM documents) AS docs,
  (SELECT COUNT(*) FROM chunks) AS chunks,
  (SELECT COUNT(*) FROM edges) AS edges,
  (SELECT COUNT(*) FROM audit_events) AS audit;

Migration audit event:

-- tail event on shard 000 carries the full plan + result body
SELECT event_type, body, ts FROM audit_events
WHERE event_type = 'reshard' ORDER BY seq DESC LIMIT 1;