From 7f29ee91e6bc2e0317ece51e8ffa452328930813 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Tue, 26 May 2026 11:59:05 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20open=20#000065=20=E2=80=94=20canonical?= =?UTF-8?q?=20shard=20count=20+=20content-hash=20routing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Surfaced while sizing #000061's federation story. Today shard count conflates two roles: - producer ingest parallelism (wants = vCPU count) - consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10) On default Python sqlite3, the ATTACH ceiling is 10 and can't be raised without a custom sqlite3 build (which violates CLAUDE.md's "python3 + venv + sqlite3 is enough" property). Producer with 16 shards → consumer fails to attach the 11th, federation silently breaks. Design: introduce M = canonical shard count (corpus-wide constant, default 8) decoupled from N = ingest workers. Document → shard assignment becomes content-deterministic: shard_idx = int(document_root[:8], 16) % M Same input → same output across every peer. Today's "spray by ingest order" is non-deterministic across peers — two peers re-ingesting the same corpus put the same document_root in different shards. That weakens federation more than it should. Migration hard-constraint (fox: "this implies we will need to reprocess all our data into shards"): re-ingest required. Current layout is sprayed by ingest order; post-ticket is sprayed by content hash. Two layouts are incompatible by construction. Captured in ticket §Migration as the load-bearing operational note. Phases laid out (0-4: design lock → read path → ingest path → pack-restore → corpus migration tool). Open audit-chain re-numbering question (per-shard event_hash chains break when rows rebalance across shards). Kept as one ticket — do-not-proliferate. Scaffold only. No code yet — design lock first. Next ID bumped to 000066. --- docs/TICKETS.md | 3 +- ...onical-shard-count-content-hash-routing.md | 230 ++++++++++++++++++ 2 files changed, 232 insertions(+), 1 deletion(-) create mode 100644 docs/tickets/ticket-000065-canonical-shard-count-content-hash-routing.md diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 62b0764..ca43c8d 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -111,6 +111,7 @@ Newest first. Update on every open/close. | ID | Title | Status | Opened | Directive | |----------|------------------------------------------------|-----------------------|------------|-----------| +| #000065 | Canonical shard count `M` + content-hash routing (decouple ingest parallelism from ATTACH ceiling) | **open · scaffold + design · awaiting go/no-go** (2026-05-26; surfaced while sizing #000061's federation story). Today shard count conflates two roles: producer ingest parallelism (wants vCPU count) + consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10 on stock python3 sqlite3). Producer with 16 vCPU → 16 shards → consumers fail to attach the 11th. Producer with 4 shards → 16-vCPU box runs 75% idle on ingest. Fix: pin a corpus-wide canonical M (default 8, ≤ ATTACH ceiling), introduce N (ingest workers) decoupled from M. Document → shard assignment becomes content-deterministic: `shard_idx = int(document_root[:8], 16) % M`. Same input → same output across every peer (today's "spray by ingest order" is non-deterministic across peers, a real federation weakness). Migration hard-constraint per fox: **requires re-ingest of every shard** — current layout is sprayed by ingest order, post-ticket is sprayed by content hash, the two are incompatible by construction. Phases: 0 design lock + pin M in meta table → 1 read path (connect_query honors M) → 2 ingest path (multi-shard write per worker) → 3 cold-pack restore re-routes on pull → 4 corpus migration tool. Open audit-chain re-numbering question (every shard has its own seq + event_hash; rebalancing splits a producer's chain across M consumer shards). Don't proliferate sub-tickets; the audit handling is part of this design lock. Out of scope: custom-built sqlite3 with higher MAX_ATTACHED (rejected: violates "python3 + venv + sqlite3 only" property from CLAUDE.md); topic-clustering shards (would break ingest determinism). | 2026-05-26 | — | | #000064 | Cold-object operations toolkit (verify/diff/doctor/repair-fts/gc-plan + audit taxonomy) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §11/§12/§14). Operator-facing observability + repair tools on top of #000061: `cold verify` (sample/full integrity check), `cold diff` (local vs remote manifest), `cold doctor` (one-shot health: connectivity / credentials / manifest age / missing-object count / tamper sample / audit-chain integrity), `cold repair-fts` (rebuild FTS5 from chunks.content), `cold gc-plan` (orphan bucket objects, read-only by default — destructive only with `--apply` + confirm). Plus expanded audit-event taxonomy: per-PUT/HEAD/GET success/failure events, manifest-pointer events, verify/doctor/gc events. All read-mostly; destructive ops require `--apply`. Bundled so the audit-taxonomy gets one design pass instead of five-way drift. Sequence: doctor → verify → diff → repair-fts → gc-plan. No code until #000061 closes. | 2026-05-26 | — | | #000063 | Cold-object private-ciphertext mode (mesh-keyed object keys) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §9 / response A §13.3). Adds private mode to #000061 cold-object format so chunk bodies + manifest can be uploaded to public-read bucket without leaking corpus membership. Two strategies: (A) deterministic `object_key = HMAC(group_key, leaf_hash)` + AEAD-encrypted body — supports lookup-by-leaf-hash given the key; (B) random-key ciphertext + encrypted private manifest — stronger membership hiding, needs manifest fetch first. Strategy A default; B opt-in. Group key from existing `arborist/mesh/crypto.py`; pack manifest carries `epoch_id` for rotation. Verifier path unchanged: consumer decrypts, then `hash_leaf(plaintext) == leaf_hash` as in public mode. No code until (1) a real non-public corpus needs cold-object shipping, (2) mesh group-key ABI is stable enough to reference, (3) threat-model split between A vs B is settled by real adversary. | 2026-05-26 | — | | #000062 | Mechanistic Witness: governed diagnostic sidecar (CNA/SAE/Neuronpedia) | **scaffold-only · awaiting go/no-go** (2026-05-26; Dav1d de-novo review §4.7 / §9.1.F). Specification of a mechanistic-interpretability sidecar that produces a content-addressed `MechanisticWitnessRoot` over (model, prompts, capture policy, neurons/features, intervention deltas), used as a **diagnostic input** to SelfModel (#000014/#000017) + benchmark-fixture generation. **Hard constraint:** soft signals never enter the hard proof path — `audit_mode` does NOT move based on witness output, `providence_cache` is untouched, `governance_policy_hash` only moves via explicit ForkScore ACCEPT with M+C+X axes passing (#000060 §7). Four guardrails (diagnostic-only by default · sandbox intervention only · no production steering without governance · feature labels never semantic proof). Witness root TLV-encodes `model_config_root | activation_capture_policy_root | contrastive_prompt_set_root | feature_or_neuron_set_root | intervention_result_root | behavioral_delta_root | safety_policy_root`. Scaffold only — no code until a real falsifier-in-hand use case exists + the four guardrails are restated in CLAUDE.md as rules + #000060 H-ABCDEFG-M+C+X harness exists to gate promotion. Captured to keep mechanistic-interp tooling out of the substrate unless and until it earns its place; the dual-use risk (Pan et al. 2025 CNA: 0.1% MLP ablation breaks refusal in 72B models) makes the governance-first framing load-bearing. | 2026-05-26 | — | @@ -178,4 +179,4 @@ Newest first. Update on every open/close. ## Next ID -`000065` +`000066` diff --git a/docs/tickets/ticket-000065-canonical-shard-count-content-hash-routing.md b/docs/tickets/ticket-000065-canonical-shard-count-content-hash-routing.md new file mode 100644 index 0000000..d584bd2 --- /dev/null +++ b/docs/tickets/ticket-000065-canonical-shard-count-content-hash-routing.md @@ -0,0 +1,230 @@ +# Ticket #000065 — Canonical shard count + content-hash routing + +**Status:** open · awaiting go/no-go · scaffold + design +**Opened:** 2026-05-26 +**Scope:** decouple two roles that "shard count" currently conflates: + ingest parallelism (producer-side, wants vCPU count) and consumer + ATTACH fan-out (capped at SQLITE_MAX_ATTACHED = 10). Introduce a + corpus-wide canonical `M` (default 8), route documents to shards by + `hash(document_root) % M` deterministically. Producer chooses N + workers ≥ M without forcing consumers to attach > 10 shards. +**Audience:** dav1d (architecture-level invariant change), every + operator running arborist with `--shards-dir`. +**Hard constraint:** **migration requires re-ingest** of every shard + in the corpus. Current shards are sprayed by ingest order (operator- + controlled, non-deterministic across peers). After this ticket, + documents are sprayed by content hash (deterministic, same across + peers). The two layouts are incompatible by construction — + rebalancing 14M chunks across 8 shards is functionally equivalent to + a full re-ingest. + +## Problem + +`arborist --shards-dir` ATTACHes every `*.db` under the directory and +runs queries via UNION ALL views. SQLite's compile-time +`SQLITE_MAX_ATTACHED = 10` is a hard ceiling — the 11th ATTACH refuses. +Default Python sqlite3 ships with that ceiling; bumping it requires a +custom build, which violates CLAUDE.md's "python3.12 + venv + sqlite3 +is enough" property. + +But the producer wants more shards. Ingest is single-writer-per-file +in SQLite — N parallel ingest workers want N shard files to write +without serializing on a WAL writer lock. On a 16-vCPU producer, +N=16 maximizes ingest throughput. That breaks every consumer with +default sqlite3. + +Today the two roles use the same number, so: + +- Producer with N=16 shards → consumer can't open the 11th. Federation + silently fails. +- Producer with N=4 shards → producer underutilizes available vCPU on + ingest (75% idle on a 16-vCPU box). + +There's no current way for the producer to be parallel AND the +consumer to stay under the ATTACH ceiling. + +Also: shard assignment today is **non-deterministic across peers**. +Operator picks which source files go to which ingest command. Two +peers re-ingesting the same corpus may put the same document_root in +different shards. This makes federation gossip + content-addressing +weaker than it should be — two peers' "shard 003.db" can have +different contents. + +## Design + +Introduce two numbers, both explicit: + +``` +N = ingest worker count — producer's vCPU choice +M = canonical shard count — corpus-wide constant, M ≤ 8 by convention +``` + +**M is part of the corpus identity.** Pinned in the snapshot manifest + +the metadata pack manifest. Every peer that reads the corpus must +adopt the same M; changing M is a corpus-level migration. + +**Document → shard assignment is content-deterministic:** + +```python +def shard_for_document(document_root: str, M: int) -> int: + # First 8 hex chars of the document_root, interpreted as int, + # taken mod M. Same input → same output across every peer. + return int(document_root[:8], 16) % M +``` + +Same for `chunks.document_root` (a chunk's shard = its document's +shard). FKs across shards aren't enforced (CASCADE is per-shard), so +the document → chunk routing must agree by construction. + +**Producer flow (post-ticket):** + +```text +N ingest workers, M canonical shards (N can be > or < M): + + worker_i opens connections to all M shards + for each row to insert: + target_shard = shard_for_document(document_root, M) + INSERT into shard M[target_shard] + parallelism on inserts = min(N, M) + on writer contention within a shard: SQLite WAL serializes +``` + +Throughput at min(N, M) is the new ingest-parallelism ceiling. Picking +M = 8 makes 8-way ingest the practical max; 16-vCPU producers run +~50% utilization on ingest. That's the price of universal-consumer +compatibility. + +If a producer needs >8-way ingest, the option is N > M with the +understanding that workers will queue behind writer locks within each +shard — still parallel-ish because each worker is mostly doing +canonicalization/embedding work between INSERTs. + +**Consumer flow (post-ticket):** + +```text +Pull metadata pack for each of M shards. +Restore into ~/.arborist/shards/00{0..M-1}.db. +Every peer with the same corpus has the same M files with the same +document_roots in the same shards. ATTACH count = M ≤ 8, well under +the ceiling. +``` + +## Migration story + +Current arborist has 4 shards (fox's corpus). Documents in each shard +were assigned by `ingest_source` calls — the operator splits work, +each call writes to one shard. No content-hash routing. + +Migration to content-hash routing requires: + +1. **Pick M.** Recommended default: 8 (leaves headroom under + ATTACH=10 for the manifest pointer + qa.db + snapshots.db + + selfmodel-chain.db). +2. **Cold-pack the current 4 shards** (already supported via + `make cold-pack-all` after this ticket lands). +3. **Drop the old 4 shards** + create M new empty shards via the + updated `connect()` (which would honor a `--shards-count M`). +4. **Re-ingest from packs** (or from upstream sources). At restore + time, each row is routed to its new content-hashed shard. + +For fox's specific corpus (Wikipedia 2010 + textbooks): re-ingest +from the original dumps is the cheapest path. The cold packs in +DO Spaces also work but require the new restore code to honor M. + +Estimated wall-clock cost (4 shards × 8.8 GB each): + +| Phase | Cost | +|-------|------| +| Cold pack current state (already done; v3 SPV packs in bucket) | 0 — already done | +| Reset shard directory | seconds | +| Re-ingest from packs into M=8 layout | ~30 min if pack-restore is parallel; longer if serial | +| Re-build FTS5 indexes | ~10 min | +| `make chain-check-shards` | ~minutes | + +Hard discipline: do NOT mutate the live corpus until the new pack +format is ready to restore into the new layout. Mistake in the +routing function = corrupted federation. + +## Implementation phases + +**Phase 0 — design lock.** Pin M = 8 as default. Document +`shard_for_document` exactly. Add `corpus_shard_count` field to +`meta` table + bump `schema_version` to 9.9 (or whatever's next). +This phase ships as a doc PR only — no code yet. + +**Phase 1 — read path.** `connect_query` learns to expect M shards +named `000.db` through `{M-1:03d}.db`. Refuses to attach if shard +count != M. Doesn't break the current 4-shard setup (M = 4 explicit +in the meta table). + +**Phase 2 — ingest path.** `ingest_source` opens M connections, +routes each row by `shard_for_document`. Tests verify two +independent ingest calls of the same docs land them in the same +shards. Multi-writer SQLite contention measured at M = 4 vs M = 8. + +**Phase 3 — cold-pack restore.** Pack-pull re-routes rows by the +manifest's M, not the producer's. Lets a peer pull a 16-shard +publisher's packs and reassemble into M = 8 locally. (Audit chain +re-numbering is a sub-problem here; see "Open questions" below.) + +**Phase 4 — corpus migration tooling.** `arborist corpus resharding +--from 4 --to 8` walks the current 4 shards, computes the new +shard_idx for every row, writes to the new M = 8 shards. Audit +chain rewritten (per shard) — this is the destructive step; ticket +discipline says "don't run until cold-packs of the old layout are +in S3 as recovery rollback." + +## Open questions + +- **Audit chain re-numbering across shards.** Each shard has its + own `audit_events.seq INTEGER PRIMARY KEY AUTOINCREMENT` and its + own `event_hash = sha256(prev || body)`. Rebalancing rows means + splitting a producer's audit chain across M consumer shards. + Each consumer-side shard gets a subset; the chain on each must + be re-hashed. Loses per-producer-per-shard chain identity. Open + whether this is acceptable or whether audit events should be + globally serialized + replicated to all M shards. + +- **`mesh_*` tables.** Currently scoped to whichever shard the + operator initialized. With M shards, where does mesh state live? + Probably one canonical shard (`000.db`) by convention, but needs + decision. + +- **`chunks_fts` on multi-shard write.** When a worker INSERTs a + chunk into shard k, does FTS5 update happen in shard k's + contentless FTS table? Yes by construction — FTS5 lives per-DB. + Easy. + +- **Cross-shard edges.** Wikilinks pointing between shards (article + in 003 → article in 005). Today the edges row carries dst_root + and the resolver can hop. After content-hash routing, dst's + shard is computable but the edges row still lives in src's + shard. No change needed. + +## Scope boundaries + +In scope: + +- Pinning M as corpus constant +- Content-hash routing function +- Multi-writer ingest path +- Corpus migration tool +- Tests of cross-peer convergence + +Out of scope: + +- Bumping `SQLITE_MAX_ATTACHED` via custom build (rejected: + violates "stock python3 + sqlite3 only") +- Multi-connection consumer-side fan-out (option 3 from the + discussion — kept as a fallback if M = 8 still doesn't fit + every consumer) +- Topic-clustering shards (sort docs by content similarity into + shards). Possible future ticket but breaks ingest determinism. + +## Status + +Open, scaffold-only. Document is the load-bearing artifact for now. +No code lands until the design questions (audit chain handling +especially) are resolved. fox: "this implies we will need to +reprocess all our data into shards" — captured as the migration +hard constraint above.