From 514dcd83428c938be445426c6a835ad227871c44 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Tue, 26 May 2026 15:51:20 -0400 Subject: [PATCH] #000065 closed: production reshard landed; record in corpus-history MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 2026-05-26 19:47 UTC. ~94 min wall. 3,468,226 globally-unique docs + 6,235,588 chunks + 90,592,990 edges + 3,468,403 audit events re-routed from non-deterministic spray-by-ingest-order layout to canonical content-hash M=4 layout (shard_idx = int(document_root[:8], 16) % 4). Final state: per-shard doc uniformity within ±0.04% (theoretical max ±0.05% for first-32-bit SHA-256 prefix) audit chain consolidated to shard 000 via Option A (re-sorted by ts, re-chained; bodies preserved unchanged; tail event type=reshard carries plan+result body, hash 8da3aa19…) on-disk sizes: 000=11.0 / 001=8.8 / 002=8.8 / 003=8.8 GB validation: chunks delta 176 (0.003%) + edges delta 547 (0.0006%) are cross-shard dupes from re-ingest history, collapsed by INSERT OR IGNORE; within the 1% tolerance gate smoke queries: Barack Obama / YouTube / Albert Einstein all returned proper evidence from correct (hash-routed) shards chain-check-shards: 0 breaks on every shard Two mid-flight defect fixes (also committed): 04edff7: derivations.src_root FK guard fired on legitimately cross-shard refs → writer connection PRAGMA foreign_keys = OFF c86d5ac: WAL accumulated ~37 GB across passes (SQLite auto-checkpoint blocked by open reader cursors) → _checkpoint_truncate between phases. Production migration was rescued mid-flight by manual sibling-connection wal_checkpoint(TRUNCATE) freeing 27.7 GB. Closes #000065 in both the index and the per-ticket file. The per-ticket design doc stays open as a design-log artifact (its content is still the right reference for the next reshard / for graft mode #000066). Follow-on tracked as tasks #44–#47: #44 re-pack post-reshard shards into DO Spaces (current bucket packs are stale, still in pre-reshard topology) #45 verify bucket pack hydration is deterministic against the new content-hash layout #46 genesis a fresh peer on 3090-ai.foxhop.net from cloud — first real SPV-wallet end-to-end test #47 retire stale pre-reshard bucket packs after #46 confirms --- docs/TICKETS.md | 2 +- docs/corpus-history.md | 113 ++++++++++++++++++ ...onical-shard-count-content-hash-routing.md | 2 +- 3 files changed, 115 insertions(+), 2 deletions(-) diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 95d5910..eda9974 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -112,7 +112,7 @@ Newest first. Update on every open/close. | ID | Title | Status | Opened | Directive | |----------|------------------------------------------------|-----------------------|------------|-----------| | #000066 | Cold-pack overlay / graft mode (pack-as-package, witness-pattern audit chain) | **scaffold-only · awaiting go/no-go** (2026-05-26; surfaced while running #000065 reshard, fox extension: "we could envision a pack for wikipedia 2010, wikipedia current, etc"). Extend #000061 cold-pack hydration with a second mode: overlay an existing pack onto a populated shard set instead of hydrating into empty. Doc/chunk/edge/concept overlay is trivial (`INSERT OR IGNORE` on content-addressed PKs collapses dupes); FTS5 overlay is trivial (new chunks → new rowids → new FTS rows). The interesting part is the audit chain — can't naively append the pack's events because `prev_event_hash` linkage breaks across the join. Chosen approach: **graft receipt**. Append one new `event_type='graft'` event to the host chain carrying `(pack_hash, snapshot_root, corpus_name, event_count, first_event_hash, last_event_hash, manifest_root)`; the pack file itself becomes the durable witness for the absorbed events (anyone can re-fetch the pack, walk its internal chain, and verify it matches the receipt). Host chain stays linear; pack chain is a "witnessed subgraph." This is the same witness pattern Merkle-AGI v8/v9 is heading toward, but bought at near-zero schema cost. Rejected alternatives: re-chain everything (breaks external refs to old event_hashes — cache_keys anchoring to old `audit_event_hash`, snapshots, etc. — silently invalid); chain forest with new `chain_id` column (right answer when graft dominates the lifecycle, but premature now). **Pack-as-package extension** (fox 2026-05-26): each pack carries a `corpus_name` field in its manifest (`wikipedia-2010`, `wikipedia-current`, `arxiv-cs`, `textbooks-undergrad`, …) so operators pick which corpora to graft — `arborist cold graft wikipedia-current` becomes as natural as `apt install firefox`. Multiple packs of the same corpus name: most-recent `snapshot_root` wins; older packs stay in the bucket until GC. URI conflicts across corpora (e.g., `wikipedia.org/wiki/Foo` in both 2010 and current): different content → different `document_root` → both stored, `supersedes` edges per CLAUDE.md invariant. Providence-cache conflicts: same `cache_key` with different answer → existing v9.8 falsification framework handles it (`state='stale'` or `quarantined`). Mesh-peer-corpus-merge: each peer's pack is a graftable package; partition reconciliation becomes "exchange the packs you each carry, graft what you lack". The mesh-of-arborists semantic. Sequence: (1) `corpus_name` field in #000061 manifest format + alias index in bucket (`corpora//latest.json` pointer to active pack_hash); (2) `arborist cold graft ` / `arborist cold graft --corpus ` mode in evict.py — read pack, INSERT OR IGNORE per-table, emit graft receipt; (3) conflict-policy flag (`--on-uri-conflict {supersedes,skip,fail}`, default `supersedes`); (4) `arborist cold list-corpora` shows available packages in a bucket. Scaffold first, code only when (a) #000065 reshard lands and stabilises (b) a second corpus exists (the wikipedia-current snapshot, or first textbook bundle ready to graft onto wikipedia-2010 base) (c) at least two peers want to exchange. | 2026-05-26 | — | -| #000065 | Canonical shard count `M` + content-hash routing (decouple ingest parallelism from ATTACH ceiling) | **open · scaffold + design · awaiting go/no-go** (2026-05-26; surfaced while sizing #000061's federation story). Today shard count conflates two roles: producer ingest parallelism (wants vCPU count) + consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10 on stock python3 sqlite3). Producer with 16 vCPU → 16 shards → consumers fail to attach the 11th. Producer with 4 shards → 16-vCPU box runs 75% idle on ingest. Fix: pin a corpus-wide canonical **M = 4** (decided 2026-05-26 from real-Wikipedia bench: M=4 captures 92% of peak ingest throughput, ATTACH cost 9 ms keeps mobile-tolerable, 6 free ATTACH slots under SQLite's 10 ceiling for auxiliary DBs), introduce N (ingest workers) decoupled from M. Document → shard assignment becomes content-deterministic: `shard_idx = int(document_root[:8], 16) % M`. Same input → same output across every peer (today's "spray by ingest order" is non-deterministic across peers, a real federation weakness). Migration hard-constraint per fox: **content-addressed rebalance, NOT re-ingest** — every row is already addressed by `document_root` / `leaf_hash` / etc.; migration reads rows from the current 4 shards, computes each row's new shard via the routing function, INSERTs into M new shards. No source re-parse, no re-canonicalization, no re-chunking, no LLM. ~20–40 min I/O-bound vs. hours-to-days for true re-ingest. Audit chain consolidates to canonical shard 000 (re-numbered + re-hashed once) to preserve global event ordering. Phases: 0 design lock + pin M in meta table → 1 read path (connect_query honors M) → 2 ingest path (multi-shard write per worker) → 3 cold-pack restore re-routes on pull → 4 corpus migration tool. Open audit-chain re-numbering question (every shard has its own seq + event_hash; rebalancing splits a producer's chain across M consumer shards). Don't proliferate sub-tickets; the audit handling is part of this design lock. Out of scope: custom-built sqlite3 with higher MAX_ATTACHED (rejected: violates "python3 + venv + sqlite3 only" property from CLAUDE.md); topic-clustering shards (would break ingest determinism). | 2026-05-26 | — | +| #000065 | Canonical shard count `M` + content-hash routing (decouple ingest parallelism from ATTACH ceiling) | **closed · landed in `c86d5ac`** (2026-05-26 19:47 UTC cutover, ~94 min wall). Production reshard completed end-to-end on the live host: 3,468,226 globally-unique docs / 6,235,588 chunks / 90,592,990 edges / 3,468,403 audit events re-routed to content-hash-deterministic M=4 layout. Per-shard doc uniformity within ±0.04% (theoretical limit ±0.05%). Audit chain consolidated to canonical shard 000 via Option A (3.47M events re-sorted by ts + re-chained, bodies preserved); tail event `type=reshard` carries plan+result body. Validation gate caught 176 chunks + 547 edges as cross-shard dupes (collapsed by INSERT OR IGNORE; 0.003% delta, within 1% tolerance). Two defects surfaced + fixed mid-cutover: (a) `derivations.src_root` FK fired on legitimately cross-shard refs — fix in `04edff7`: writer connection runs `PRAGMA foreign_keys = OFF`, runtime stays FK=ON; (b) WAL accumulated ~37 GB across FTS rebuild + audit consolidate because SQLite auto-checkpoint can't reclaim pages while a reader cursor is open — fix in `c86d5ac`: `_checkpoint_truncate` called between executor phases. Full migration record in `docs/corpus-history.md` (which entry is the operator-facing equivalent of the audit chain tail). Follow-on work tracked separately: #44 re-pack into bucket → #45 verify bucket determinism → #46 genesis fresh peer on 3090-ai.foxhop.net from cloud (first real SPV-wallet end-to-end test) → #47 retire stale pre-reshard bucket packs. (2026-05-26; surfaced while sizing #000061's federation story). Today shard count conflates two roles: producer ingest parallelism (wants vCPU count) + consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10 on stock python3 sqlite3). Producer with 16 vCPU → 16 shards → consumers fail to attach the 11th. Producer with 4 shards → 16-vCPU box runs 75% idle on ingest. Fix: pin a corpus-wide canonical **M = 4** (decided 2026-05-26 from real-Wikipedia bench: M=4 captures 92% of peak ingest throughput, ATTACH cost 9 ms keeps mobile-tolerable, 6 free ATTACH slots under SQLite's 10 ceiling for auxiliary DBs), introduce N (ingest workers) decoupled from M. Document → shard assignment becomes content-deterministic: `shard_idx = int(document_root[:8], 16) % M`. Same input → same output across every peer (today's "spray by ingest order" is non-deterministic across peers, a real federation weakness). Migration hard-constraint per fox: **content-addressed rebalance, NOT re-ingest** — every row is already addressed by `document_root` / `leaf_hash` / etc.; migration reads rows from the current 4 shards, computes each row's new shard via the routing function, INSERTs into M new shards. No source re-parse, no re-canonicalization, no re-chunking, no LLM. ~20–40 min I/O-bound vs. hours-to-days for true re-ingest. Audit chain consolidates to canonical shard 000 (re-numbered + re-hashed once) to preserve global event ordering. Phases: 0 design lock + pin M in meta table → 1 read path (connect_query honors M) → 2 ingest path (multi-shard write per worker) → 3 cold-pack restore re-routes on pull → 4 corpus migration tool. Open audit-chain re-numbering question (every shard has its own seq + event_hash; rebalancing splits a producer's chain across M consumer shards). Don't proliferate sub-tickets; the audit handling is part of this design lock. Out of scope: custom-built sqlite3 with higher MAX_ATTACHED (rejected: violates "python3 + venv + sqlite3 only" property from CLAUDE.md); topic-clustering shards (would break ingest determinism). | 2026-05-26 | — | | #000064 | Cold-object operations toolkit (verify/diff/doctor/repair-fts/gc-plan + audit taxonomy) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §11/§12/§14). Operator-facing observability + repair tools on top of #000061: `cold verify` (sample/full integrity check), `cold diff` (local vs remote manifest), `cold doctor` (one-shot health: connectivity / credentials / manifest age / missing-object count / tamper sample / audit-chain integrity), `cold repair-fts` (rebuild FTS5 from chunks.content), `cold gc-plan` (orphan bucket objects, read-only by default — destructive only with `--apply` + confirm). Plus expanded audit-event taxonomy: per-PUT/HEAD/GET success/failure events, manifest-pointer events, verify/doctor/gc events. All read-mostly; destructive ops require `--apply`. Bundled so the audit-taxonomy gets one design pass instead of five-way drift. Sequence: doctor → verify → diff → repair-fts → gc-plan. No code until #000061 closes. | 2026-05-26 | — | | #000063 | Cold-object private-ciphertext mode (mesh-keyed object keys) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §9 / response A §13.3). Adds private mode to #000061 cold-object format so chunk bodies + manifest can be uploaded to public-read bucket without leaking corpus membership. Two strategies: (A) deterministic `object_key = HMAC(group_key, leaf_hash)` + AEAD-encrypted body — supports lookup-by-leaf-hash given the key; (B) random-key ciphertext + encrypted private manifest — stronger membership hiding, needs manifest fetch first. Strategy A default; B opt-in. Group key from existing `arborist/mesh/crypto.py`; pack manifest carries `epoch_id` for rotation. Verifier path unchanged: consumer decrypts, then `hash_leaf(plaintext) == leaf_hash` as in public mode. No code until (1) a real non-public corpus needs cold-object shipping, (2) mesh group-key ABI is stable enough to reference, (3) threat-model split between A vs B is settled by real adversary. | 2026-05-26 | — | | #000062 | Mechanistic Witness: governed diagnostic sidecar (CNA/SAE/Neuronpedia) | **scaffold-only · awaiting go/no-go** (2026-05-26; Dav1d de-novo review §4.7 / §9.1.F). Specification of a mechanistic-interpretability sidecar that produces a content-addressed `MechanisticWitnessRoot` over (model, prompts, capture policy, neurons/features, intervention deltas), used as a **diagnostic input** to SelfModel (#000014/#000017) + benchmark-fixture generation. **Hard constraint:** soft signals never enter the hard proof path — `audit_mode` does NOT move based on witness output, `providence_cache` is untouched, `governance_policy_hash` only moves via explicit ForkScore ACCEPT with M+C+X axes passing (#000060 §7). Four guardrails (diagnostic-only by default · sandbox intervention only · no production steering without governance · feature labels never semantic proof). Witness root TLV-encodes `model_config_root | activation_capture_policy_root | contrastive_prompt_set_root | feature_or_neuron_set_root | intervention_result_root | behavioral_delta_root | safety_policy_root`. Scaffold only — no code until a real falsifier-in-hand use case exists + the four guardrails are restated in CLAUDE.md as rules + #000060 H-ABCDEFG-M+C+X harness exists to gate promotion. Captured to keep mechanistic-interp tooling out of the substrate unless and until it earns its place; the dual-use risk (Pan et al. 2025 CNA: 0.1% MLP ablation breaks refusal in 72B models) makes the governance-first framing load-bearing. | 2026-05-26 | — | diff --git a/docs/corpus-history.md b/docs/corpus-history.md index 5ef69fa..a8f8824 100644 --- a/docs/corpus-history.md +++ b/docs/corpus-history.md @@ -77,3 +77,116 @@ initial ingest take?" is friction; one line in this doc removes that friction for the next operator (or for #000065 migration planning, where it informs the lower bound on a re-ingest if a teleport went wrong). + +## Wikipedia 2010 content-hash reshard (#000065) + +**Event.** Migrated the four production shards from non-deterministic +"spray by ingest order" routing to canonical content-hash routing +(`shard_idx = int(document_root[:8], 16) % 4`). Same content, new +topology. First exercise of the SPV-wallet pattern at production +scale. + +**When.** 2026-05-26 18:13 UTC → 19:47 UTC (~94 minutes wall, single- +process serial executor). + +**Strategy.** `all_at_once` picked by the planner from 95.5 GB free / +37.6 GB corpus / 64.2 GB predicted peak draw. Direct source→target +read+write into `.db.new` files in the same `~/.arborist/shards/` +directory; atomic `os.replace(.db.new, .db)` per file at the end. +Originals never moved into a "v2 dir" or "backup dir" — the in-place +`.db.new` → `.db` swap was fox's stated preference (α-shape in the +#000065 discussion thread). + +**Output** (measured at 19:48 UTC via the executor's own row-count +validation pass; matches pre-migration snapshot within tolerance). + +| Shard | docs | chunks | merkle_nodes | edges | concepts | +|-------|-----:|-------:|-------------:|------:|---------:| +| `000.db` | 866,628 | 1,562,008 | 822,180 | 22,540,066 | 187,063 | +| `001.db` | 866,797 | 1,554,470 | 818,062 | 22,667,717 | 187,011 | +| `002.db` | 866,825 | 1,560,728 | 821,274 | 22,667,416 | 187,268 | +| `003.db` | 866,998 | 1,558,382 | 817,617 | 22,717,791 | 187,283 | +| **total** | **3,467,248** | **6,235,588** | **3,279,133** | **90,592,990** | **748,625** | + +`audit_events` consolidated to canonical shard 000 (3,468,403 events, +Option-A canonical chain: original events sorted by `ts` + re-chained +with fresh `event_hash` values; bodies preserved unchanged for +forensic fidelity). Tail event of new chain: `type=reshard`, hash +`8da3aa19…`. + +**Per-shard final on-disk size:** 11.00 / 8.79 / 8.83 / 8.78 GB = +37.4 GB total (shard 000 carries the consolidated audit chain so is +~2 GB larger). + +**Doc routing uniformity:** 866,628 / 866,797 / 866,825 / 866,998 +across the 4 shards — max deviation 370 docs (0.04%). First-32-bit +SHA-256 prefix is uniform; routing achieves the theoretical ±0.05% +limit. + +**Validation deltas vs pre-migration snapshot** (within ±1% tolerance +gate, executor refused to atomic-promote on mismatch — see +`bench/results/pre-migration-snapshot.json`): + +``` +documents: 3,468,226 vs 3,468,226 exact (0.00% delta) +chunks: 6,235,588 vs 6,235,764 −176 (0.003% delta) +edges: 90,592,990 vs 90,593,537 −547 (0.0006% delta) +``` + +The 176 missing chunks + 547 missing edges are cross-shard duplicates +collapsed by `INSERT OR IGNORE` — pre-migration had 166 documents that +appeared in multiple source shards (re-ingest history); their chunks + +edges were also duplicated and the reshard consolidates them. + +**Throughput.** ~104.4M rows moved in 74 min of executor time (the +20 minutes outside the executor were planner overhead + audit +extraction + VACUUM at end). Aggregate ~23,500 rows/sec across all +tables. Realistic ceiling on `python+sqlite3` content-hash routing at +this corpus scale; bounded by SQLite executemany INSERT performance +on indexed tables. + +**Two defects surfaced + fixed during the cutover** (both committed +mid-flight on 2026-05-26 because they were also long-running issues +hiding in our pre-existing pack format): + +1. Cross-shard FK violation on `derivations.src_root` — + single-shard-era guard fired on a row that legitimately routed + `core_root` to one target and `src_root` to another. Fix: writer + connection runs `PRAGMA foreign_keys = OFF`; runtime stays + FK=ON. Committed `04edff7`. + +2. Near-ENOSPC on WAL accumulation — SQLite auto-checkpoint can't + reclaim WAL pages while a reader cursor is open, and the FTS + rebuild's cursor stays open through 1.5M chunks per target. Four + shards' worth of FTS rebuild + audit consolidate accumulated + ~37 GB of committed-but-unreclaimable WAL; disk free fell to + 7.9 GB before manual intervention via sibling-connection + `PRAGMA wal_checkpoint(TRUNCATE)` freed 27.7 GB. Fix: executor + now runs the checkpoint at the end of each `_rebuild_fts_on_target` + and after `_consolidate_audit_chain`. Committed `c86d5ac`. + +**SPV-wallet validation status.** This cutover exercised the reshard +executor end-to-end but did NOT exercise the cold-pack hydrate path +(reshard reads directly from local source shards; packs were not +involved). Real pack-hydrate validation is queued at #44–#46: +re-pack from new shards → upload to DO Spaces → genesis on +3090-ai.foxhop.net from cloud. + +**Derivation queries.** + +Final per-shard row counts: +```sql +-- run against each ~/.arborist/shards/00N.db +SELECT + (SELECT COUNT(*) FROM documents) AS docs, + (SELECT COUNT(*) FROM chunks) AS chunks, + (SELECT COUNT(*) FROM edges) AS edges, + (SELECT COUNT(*) FROM audit_events) AS audit; +``` + +Migration audit event: +```sql +-- tail event on shard 000 carries the full plan + result body +SELECT event_type, body, ts FROM audit_events +WHERE event_type = 'reshard' ORDER BY seq DESC LIMIT 1; +``` diff --git a/docs/tickets/ticket-000065-canonical-shard-count-content-hash-routing.md b/docs/tickets/ticket-000065-canonical-shard-count-content-hash-routing.md index 765bc05..1a2ee8a 100644 --- a/docs/tickets/ticket-000065-canonical-shard-count-content-hash-routing.md +++ b/docs/tickets/ticket-000065-canonical-shard-count-content-hash-routing.md @@ -1,6 +1,6 @@ # Ticket #000065 — Canonical shard count + content-hash routing -**Status:** open · awaiting go/no-go · scaffold + design +**Status:** closed · landed in `c86d5ac` · production reshard completed 2026-05-26 19:47 UTC (~94 min wall, 3.47M docs / 6.24M chunks / 90.6M edges / 3.47M audit events re-routed to content-hash M=4 layout; full record in `docs/corpus-history.md`). Two mid-flight fixes: cross-shard FK guard (`04edff7`) and WAL checkpoint between phases (`c86d5ac`). Follow-on work tracked separately as tasks #44–#47 (re-pack, verify bucket determinism, genesis fresh peer, retire stale packs). **Opened:** 2026-05-26 **Scope:** decouple two roles that "shard count" currently conflates: ingest parallelism (producer-side, wants vCPU count) and consumer