russell@unturf.com
3aae8119a6
Three pieces, all read-only or additive — no shard mutation, no
schema-version bump:
1. shard_for_document(document_root, M) in arborist/document.py.
Pure function: int(document_root[:8], 16) % M. 22 tests cover
determinism, range-bounds, near-uniform distribution (±5pp at
N=20k), and seven lock-in fixtures so peers will disagree
loudly if anyone changes the formula.
2. corpus_shard_count meta field + get/set helpers in store.py.
Lives in the existing key/value meta table; SCHEMA_VERSION
stays at v9.8.0 (the DDL doesn't change and source_root is
layout-independent, so cache records survive a reshard).
Legacy shards (without the field) return None; reshard tool
populates it on every target shard at migration time.
3. Pre-migration snapshot captured to
bench/results/pre-migration-snapshot.json:
docs 3,468,392 (3,468,226 globally unique)
chunks 6,235,764
edges 90,593,537
audit 3,468,403
This is the reference set post-reshard row counts must match.
4. Audit-event extraction script writes all 3.47M events from
all 4 shards to /tmp/audit-events.ndjson (2.0 GB) for the
Option-A canonical-chain consolidation step. Verifies chain
integrity on extract — all 4 source chains report 0 breaks.
5. Fixed a wrong chunk count in docs/corpus-history.md
(had ~3.54M/shard; actual is ~1.56M/shard) and added the
edge-count column (~22.6M/shard, 90.6M total). 6.24M chunks
total, not 14.12M.
Tests: 29 new pass (22 routing + 7 meta). No existing tests
touched.
92 lines
No EOL
2.2 KiB
JSON
92 lines
No EOL
2.2 KiB
JSON
{
|
|
"captured_at_utc": "2026-05-26T17:05:23Z",
|
|
"shards_dir": "/home/fox/.arborist/shards",
|
|
"shards": [
|
|
{
|
|
"path": "/home/fox/.arborist/shards/000.db",
|
|
"size_bytes": 9399353344,
|
|
"docs": 866874,
|
|
"chunks": 1558523,
|
|
"edges": 22613200,
|
|
"audit_events": 866882,
|
|
"providence_cache": 0,
|
|
"schema_version": null,
|
|
"docs_by_source_type": {
|
|
"claim_pack": 92,
|
|
"wikipedia_xml": 866782
|
|
},
|
|
"audit_first_ts": 1777325593,
|
|
"audit_last_ts": 1779758541,
|
|
"audit_by_type": {
|
|
"cold_pack_pushed": 8,
|
|
"ingest": 866874
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/fox/.arborist/shards/001.db",
|
|
"size_bytes": 9360822272,
|
|
"docs": 867695,
|
|
"chunks": 1558131,
|
|
"edges": 22595130,
|
|
"audit_events": 867696,
|
|
"providence_cache": 0,
|
|
"schema_version": null,
|
|
"docs_by_source_type": {
|
|
"wikipedia_xml": 867695
|
|
},
|
|
"audit_first_ts": 1777325593,
|
|
"audit_last_ts": 1779758540,
|
|
"audit_by_type": {
|
|
"cold_pack_pushed": 1,
|
|
"ingest": 867695
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/fox/.arborist/shards/002.db",
|
|
"size_bytes": 9430011904,
|
|
"docs": 866825,
|
|
"chunks": 1560728,
|
|
"edges": 22667416,
|
|
"audit_events": 866826,
|
|
"providence_cache": 0,
|
|
"schema_version": null,
|
|
"docs_by_source_type": {
|
|
"wikipedia_xml": 866825
|
|
},
|
|
"audit_first_ts": 1777325593,
|
|
"audit_last_ts": 1779758486,
|
|
"audit_by_type": {
|
|
"cold_pack_pushed": 1,
|
|
"ingest": 866825
|
|
}
|
|
},
|
|
{
|
|
"path": "/home/fox/.arborist/shards/003.db",
|
|
"size_bytes": 9415876608,
|
|
"docs": 866998,
|
|
"chunks": 1558382,
|
|
"edges": 22717791,
|
|
"audit_events": 866999,
|
|
"providence_cache": 0,
|
|
"schema_version": null,
|
|
"docs_by_source_type": {
|
|
"wikipedia_xml": 866998
|
|
},
|
|
"audit_first_ts": 1777325593,
|
|
"audit_last_ts": 1779758540,
|
|
"audit_by_type": {
|
|
"cold_pack_pushed": 1,
|
|
"ingest": 866998
|
|
}
|
|
}
|
|
],
|
|
"totals": {
|
|
"size_bytes": 37606064128,
|
|
"docs": 3468392,
|
|
"chunks": 6235764,
|
|
"edges": 90593537,
|
|
"audit_events": 3468403,
|
|
"docs_globally_unique": 3468226
|
|
},
|
|
"elapsed_seconds": 43.52
|
|
} |