engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
arborist/docs/corpus-history.md
russell@unturf.com 3aae8119a6
#000065 step 1: routing helper + meta field + pre-migration snapshot 
Three pieces, all read-only or additive — no shard mutation, no
schema-version bump:

1. shard_for_document(document_root, M) in arborist/document.py.
   Pure function: int(document_root[:8], 16) % M. 22 tests cover
   determinism, range-bounds, near-uniform distribution (±5pp at
   N=20k), and seven lock-in fixtures so peers will disagree
   loudly if anyone changes the formula.

2. corpus_shard_count meta field + get/set helpers in store.py.
   Lives in the existing key/value meta table; SCHEMA_VERSION
   stays at v9.8.0 (the DDL doesn't change and source_root is
   layout-independent, so cache records survive a reshard).
   Legacy shards (without the field) return None; reshard tool
   populates it on every target shard at migration time.

3. Pre-migration snapshot captured to
   bench/results/pre-migration-snapshot.json:
     docs        3,468,392  (3,468,226 globally unique)
     chunks      6,235,764
     edges      90,593,537
     audit       3,468,403
   This is the reference set post-reshard row counts must match.

4. Audit-event extraction script writes all 3.47M events from
   all 4 shards to /tmp/audit-events.ndjson (2.0 GB) for the
   Option-A canonical-chain consolidation step. Verifies chain
   integrity on extract — all 4 source chains report 0 breaks.

5. Fixed a wrong chunk count in docs/corpus-history.md
   (had ~3.54M/shard; actual is ~1.56M/shard) and added the
   edge-count column (~22.6M/shard, 90.6M total). 6.24M chunks
   total, not 14.12M.

Tests: 29 new pass (22 routing + 7 meta). No existing tests
touched.
2026-05-26 13:13:44 -04:00

3.6 KiB
Raw Blame History

Corpus history

State changes at scale that aren't otherwise captured outside the audit chain. Append-only; oldest events first. Each entry says what happened, when, how derived (audit-chain query, git log, operator note), and links to the data of record.

The audit chain is the durable source of truth for ingests / migrations / falsifications / cold-pack pushes; this doc surfaces the headline numbers so future operators don't have to re-derive them from sqlite3 audit_events.

Wikipedia 2010 ingest — production corpus origin

Event. Initial ingest of Wikipedia 2010 dump (data/enwiki-20101011-pages-articles.xml.bz2, 6.65 GB compressed) into the four production shards.

When. 2026-04-27 21:33:13 UTC → 2026-04-27 23:13:28 UTC (~100 minutes wall clock, all four shards run in parallel).

Source dump. enwiki-20101011-pages-articles.xml.bz2, 6,652,983,189 bytes, mtime 2026-05-07 (fetched before ingest).

Output (measured 2026-05-26 via bench/pre_migration_snapshot.py; authoritative reference for #000065 reshard verification).

Shard docs chunks edges audit events first ingest last ingest
000.db 866,782 1,560,021 22,664,792 866,886 2026-04-27 21:33:13Z 2026-04-27 23:13:20Z
001.db 867,695 1,545,929 22,612,247 867,695 2026-04-27 21:33:13Z 2026-04-27 23:13:01Z
002.db 866,825 1,600,420 22,649,376 866,825 2026-04-27 21:33:13Z 2026-04-27 23:13:26Z
003.db 866,998 1,529,394 22,667,122 866,997 2026-04-27 21:33:13Z 2026-04-27 23:13:28Z
total 3,468,300 6,235,764 90,593,537 3,468,403

(Shard 000 has 104 additional ingest events on 2026-05-09 — a later top-up of post-2010 fillers; doesn't change the bulk timing. 3,468,226 globally unique document_root values vs 3,468,392 raw rows → 166 cross-shard duplicates from re-ingest history.)

Throughput. ~580 docs/sec aggregate across 4 workers ≈ 145 docs/sec per shard ≈ 1,040 chunks/sec aggregate ≈ 15,000 edges/sec. Realistic production rate for Wikipedia-XML workload (XML parse + canonicalization + edge extraction + chunker pass per document).

The 2026-05-26 synthetic + 2003-cur-dump benchmarks (bench/shard_count_sweep.py) measured ~6,400 chunks/sec aggregate at M=4 on the same hardware — that rate excludes XML parsing because the cur dump is pre-parsed SQL, so it overstates throughput by ~6× for planning purposes against the XML pipeline. For #000065 reshard planning the relevant rate is INSERT throughput only (no parse, no canonicalize, no chunker), which should beat both XML and cur ceilings.

Hardware. 8-vCPU box (cf. CLAUDE.md "Per-call model selection" + the live bench observations from #000061).

Derivation. Run on shard 000.db:

SELECT
  datetime(MIN(ts), 'unixepoch') AS first,
  datetime(MAX(ts), 'unixepoch') AS last,
  printf('%.2f', (MAX(ts) - MIN(ts)) / 60.0) AS minutes,
  COUNT(*) AS docs
FROM audit_events
WHERE event_type = 'ingest' AND ts < strftime('%s', '2026-05-01');

Repeat per shard. The ingest event body carries document_uri + chunks + source_type per row so all the per-doc details are recoverable forensically.

Why captured here. The audit chain holds this in the body of every ingest event but reading 3.47M rows to answer "how long did the initial ingest take?" is friction; one line in this doc removes that friction for the next operator (or for #000065 migration planning, where it informs the lower bound on a re-ingest if a teleport went wrong).