docs: open #000065 — canonical shard count + content-hash routing
Surfaced while sizing #000061's federation story. Today shard count conflates two roles: - producer ingest parallelism (wants = vCPU count) - consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10) On default Python sqlite3, the ATTACH ceiling is 10 and can't be raised without a custom sqlite3 build (which violates CLAUDE.md's "python3 + venv + sqlite3 is enough" property). Producer with 16 shards → consumer fails to attach the 11th, federation silently breaks. Design: introduce M = canonical shard count (corpus-wide constant, default 8) decoupled from N = ingest workers. Document → shard assignment becomes content-deterministic: shard_idx = int(document_root[:8], 16) % M Same input → same output across every peer. Today's "spray by ingest order" is non-deterministic across peers — two peers re-ingesting the same corpus put the same document_root in different shards. That weakens federation more than it should. Migration hard-constraint (fox: "this implies we will need to reprocess all our data into shards"): re-ingest required. Current layout is sprayed by ingest order; post-ticket is sprayed by content hash. Two layouts are incompatible by construction. Captured in ticket §Migration as the load-bearing operational note. Phases laid out (0-4: design lock → read path → ingest path → pack-restore → corpus migration tool). Open audit-chain re-numbering question (per-shard event_hash chains break when rows rebalance across shards). Kept as one ticket — do-not-proliferate. Scaffold only. No code yet — design lock first. Next ID bumped to 000066.
This commit is contained in:
parent
315e783fa1
commit
7f29ee91e6
2 changed files with 232 additions and 1 deletions
|
|
@ -111,6 +111,7 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000065 | Canonical shard count `M` + content-hash routing (decouple ingest parallelism from ATTACH ceiling) | **open · scaffold + design · awaiting go/no-go** (2026-05-26; surfaced while sizing #000061's federation story). Today shard count conflates two roles: producer ingest parallelism (wants vCPU count) + consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10 on stock python3 sqlite3). Producer with 16 vCPU → 16 shards → consumers fail to attach the 11th. Producer with 4 shards → 16-vCPU box runs 75% idle on ingest. Fix: pin a corpus-wide canonical M (default 8, ≤ ATTACH ceiling), introduce N (ingest workers) decoupled from M. Document → shard assignment becomes content-deterministic: `shard_idx = int(document_root[:8], 16) % M`. Same input → same output across every peer (today's "spray by ingest order" is non-deterministic across peers, a real federation weakness). Migration hard-constraint per fox: **requires re-ingest of every shard** — current layout is sprayed by ingest order, post-ticket is sprayed by content hash, the two are incompatible by construction. Phases: 0 design lock + pin M in meta table → 1 read path (connect_query honors M) → 2 ingest path (multi-shard write per worker) → 3 cold-pack restore re-routes on pull → 4 corpus migration tool. Open audit-chain re-numbering question (every shard has its own seq + event_hash; rebalancing splits a producer's chain across M consumer shards). Don't proliferate sub-tickets; the audit handling is part of this design lock. Out of scope: custom-built sqlite3 with higher MAX_ATTACHED (rejected: violates "python3 + venv + sqlite3 only" property from CLAUDE.md); topic-clustering shards (would break ingest determinism). | 2026-05-26 | — |
|
||||
| #000064 | Cold-object operations toolkit (verify/diff/doctor/repair-fts/gc-plan + audit taxonomy) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §11/§12/§14). Operator-facing observability + repair tools on top of #000061: `cold verify` (sample/full integrity check), `cold diff` (local vs remote manifest), `cold doctor` (one-shot health: connectivity / credentials / manifest age / missing-object count / tamper sample / audit-chain integrity), `cold repair-fts` (rebuild FTS5 from chunks.content), `cold gc-plan` (orphan bucket objects, read-only by default — destructive only with `--apply` + confirm). Plus expanded audit-event taxonomy: per-PUT/HEAD/GET success/failure events, manifest-pointer events, verify/doctor/gc events. All read-mostly; destructive ops require `--apply`. Bundled so the audit-taxonomy gets one design pass instead of five-way drift. Sequence: doctor → verify → diff → repair-fts → gc-plan. No code until #000061 closes. | 2026-05-26 | — |
|
||||
| #000063 | Cold-object private-ciphertext mode (mesh-keyed object keys) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §9 / response A §13.3). Adds private mode to #000061 cold-object format so chunk bodies + manifest can be uploaded to public-read bucket without leaking corpus membership. Two strategies: (A) deterministic `object_key = HMAC(group_key, leaf_hash)` + AEAD-encrypted body — supports lookup-by-leaf-hash given the key; (B) random-key ciphertext + encrypted private manifest — stronger membership hiding, needs manifest fetch first. Strategy A default; B opt-in. Group key from existing `arborist/mesh/crypto.py`; pack manifest carries `epoch_id` for rotation. Verifier path unchanged: consumer decrypts, then `hash_leaf(plaintext) == leaf_hash` as in public mode. No code until (1) a real non-public corpus needs cold-object shipping, (2) mesh group-key ABI is stable enough to reference, (3) threat-model split between A vs B is settled by real adversary. | 2026-05-26 | — |
|
||||
| #000062 | Mechanistic Witness: governed diagnostic sidecar (CNA/SAE/Neuronpedia) | **scaffold-only · awaiting go/no-go** (2026-05-26; Dav1d de-novo review §4.7 / §9.1.F). Specification of a mechanistic-interpretability sidecar that produces a content-addressed `MechanisticWitnessRoot` over (model, prompts, capture policy, neurons/features, intervention deltas), used as a **diagnostic input** to SelfModel (#000014/#000017) + benchmark-fixture generation. **Hard constraint:** soft signals never enter the hard proof path — `audit_mode` does NOT move based on witness output, `providence_cache` is untouched, `governance_policy_hash` only moves via explicit ForkScore ACCEPT with M+C+X axes passing (#000060 §7). Four guardrails (diagnostic-only by default · sandbox intervention only · no production steering without governance · feature labels never semantic proof). Witness root TLV-encodes `model_config_root | activation_capture_policy_root | contrastive_prompt_set_root | feature_or_neuron_set_root | intervention_result_root | behavioral_delta_root | safety_policy_root`. Scaffold only — no code until a real falsifier-in-hand use case exists + the four guardrails are restated in CLAUDE.md as rules + #000060 H-ABCDEFG-M+C+X harness exists to gate promotion. Captured to keep mechanistic-interp tooling out of the substrate unless and until it earns its place; the dual-use risk (Pan et al. 2025 CNA: 0.1% MLP ablation breaks refusal in 72B models) makes the governance-first framing load-bearing. | 2026-05-26 | — |
|
||||
|
|
@ -178,4 +179,4 @@ Newest first. Update on every open/close.
|
|||
|
||||
## Next ID
|
||||
|
||||
`000065`
|
||||
`000066`
|
||||
|
|
|
|||
|
|
@ -0,0 +1,230 @@
|
|||
# Ticket #000065 — Canonical shard count + content-hash routing
|
||||
|
||||
**Status:** open · awaiting go/no-go · scaffold + design
|
||||
**Opened:** 2026-05-26
|
||||
**Scope:** decouple two roles that "shard count" currently conflates:
|
||||
ingest parallelism (producer-side, wants vCPU count) and consumer
|
||||
ATTACH fan-out (capped at SQLITE_MAX_ATTACHED = 10). Introduce a
|
||||
corpus-wide canonical `M` (default 8), route documents to shards by
|
||||
`hash(document_root) % M` deterministically. Producer chooses N
|
||||
workers ≥ M without forcing consumers to attach > 10 shards.
|
||||
**Audience:** dav1d (architecture-level invariant change), every
|
||||
operator running arborist with `--shards-dir`.
|
||||
**Hard constraint:** **migration requires re-ingest** of every shard
|
||||
in the corpus. Current shards are sprayed by ingest order (operator-
|
||||
controlled, non-deterministic across peers). After this ticket,
|
||||
documents are sprayed by content hash (deterministic, same across
|
||||
peers). The two layouts are incompatible by construction —
|
||||
rebalancing 14M chunks across 8 shards is functionally equivalent to
|
||||
a full re-ingest.
|
||||
|
||||
## Problem
|
||||
|
||||
`arborist --shards-dir` ATTACHes every `*.db` under the directory and
|
||||
runs queries via UNION ALL views. SQLite's compile-time
|
||||
`SQLITE_MAX_ATTACHED = 10` is a hard ceiling — the 11th ATTACH refuses.
|
||||
Default Python sqlite3 ships with that ceiling; bumping it requires a
|
||||
custom build, which violates CLAUDE.md's "python3.12 + venv + sqlite3
|
||||
is enough" property.
|
||||
|
||||
But the producer wants more shards. Ingest is single-writer-per-file
|
||||
in SQLite — N parallel ingest workers want N shard files to write
|
||||
without serializing on a WAL writer lock. On a 16-vCPU producer,
|
||||
N=16 maximizes ingest throughput. That breaks every consumer with
|
||||
default sqlite3.
|
||||
|
||||
Today the two roles use the same number, so:
|
||||
|
||||
- Producer with N=16 shards → consumer can't open the 11th. Federation
|
||||
silently fails.
|
||||
- Producer with N=4 shards → producer underutilizes available vCPU on
|
||||
ingest (75% idle on a 16-vCPU box).
|
||||
|
||||
There's no current way for the producer to be parallel AND the
|
||||
consumer to stay under the ATTACH ceiling.
|
||||
|
||||
Also: shard assignment today is **non-deterministic across peers**.
|
||||
Operator picks which source files go to which ingest command. Two
|
||||
peers re-ingesting the same corpus may put the same document_root in
|
||||
different shards. This makes federation gossip + content-addressing
|
||||
weaker than it should be — two peers' "shard 003.db" can have
|
||||
different contents.
|
||||
|
||||
## Design
|
||||
|
||||
Introduce two numbers, both explicit:
|
||||
|
||||
```
|
||||
N = ingest worker count — producer's vCPU choice
|
||||
M = canonical shard count — corpus-wide constant, M ≤ 8 by convention
|
||||
```
|
||||
|
||||
**M is part of the corpus identity.** Pinned in the snapshot manifest +
|
||||
the metadata pack manifest. Every peer that reads the corpus must
|
||||
adopt the same M; changing M is a corpus-level migration.
|
||||
|
||||
**Document → shard assignment is content-deterministic:**
|
||||
|
||||
```python
|
||||
def shard_for_document(document_root: str, M: int) -> int:
|
||||
# First 8 hex chars of the document_root, interpreted as int,
|
||||
# taken mod M. Same input → same output across every peer.
|
||||
return int(document_root[:8], 16) % M
|
||||
```
|
||||
|
||||
Same for `chunks.document_root` (a chunk's shard = its document's
|
||||
shard). FKs across shards aren't enforced (CASCADE is per-shard), so
|
||||
the document → chunk routing must agree by construction.
|
||||
|
||||
**Producer flow (post-ticket):**
|
||||
|
||||
```text
|
||||
N ingest workers, M canonical shards (N can be > or < M):
|
||||
|
||||
worker_i opens connections to all M shards
|
||||
for each row to insert:
|
||||
target_shard = shard_for_document(document_root, M)
|
||||
INSERT into shard M[target_shard]
|
||||
parallelism on inserts = min(N, M)
|
||||
on writer contention within a shard: SQLite WAL serializes
|
||||
```
|
||||
|
||||
Throughput at min(N, M) is the new ingest-parallelism ceiling. Picking
|
||||
M = 8 makes 8-way ingest the practical max; 16-vCPU producers run
|
||||
~50% utilization on ingest. That's the price of universal-consumer
|
||||
compatibility.
|
||||
|
||||
If a producer needs >8-way ingest, the option is N > M with the
|
||||
understanding that workers will queue behind writer locks within each
|
||||
shard — still parallel-ish because each worker is mostly doing
|
||||
canonicalization/embedding work between INSERTs.
|
||||
|
||||
**Consumer flow (post-ticket):**
|
||||
|
||||
```text
|
||||
Pull metadata pack for each of M shards.
|
||||
Restore into ~/.arborist/shards/00{0..M-1}.db.
|
||||
Every peer with the same corpus has the same M files with the same
|
||||
document_roots in the same shards. ATTACH count = M ≤ 8, well under
|
||||
the ceiling.
|
||||
```
|
||||
|
||||
## Migration story
|
||||
|
||||
Current arborist has 4 shards (fox's corpus). Documents in each shard
|
||||
were assigned by `ingest_source` calls — the operator splits work,
|
||||
each call writes to one shard. No content-hash routing.
|
||||
|
||||
Migration to content-hash routing requires:
|
||||
|
||||
1. **Pick M.** Recommended default: 8 (leaves headroom under
|
||||
ATTACH=10 for the manifest pointer + qa.db + snapshots.db +
|
||||
selfmodel-chain.db).
|
||||
2. **Cold-pack the current 4 shards** (already supported via
|
||||
`make cold-pack-all` after this ticket lands).
|
||||
3. **Drop the old 4 shards** + create M new empty shards via the
|
||||
updated `connect()` (which would honor a `--shards-count M`).
|
||||
4. **Re-ingest from packs** (or from upstream sources). At restore
|
||||
time, each row is routed to its new content-hashed shard.
|
||||
|
||||
For fox's specific corpus (Wikipedia 2010 + textbooks): re-ingest
|
||||
from the original dumps is the cheapest path. The cold packs in
|
||||
DO Spaces also work but require the new restore code to honor M.
|
||||
|
||||
Estimated wall-clock cost (4 shards × 8.8 GB each):
|
||||
|
||||
| Phase | Cost |
|
||||
|-------|------|
|
||||
| Cold pack current state (already done; v3 SPV packs in bucket) | 0 — already done |
|
||||
| Reset shard directory | seconds |
|
||||
| Re-ingest from packs into M=8 layout | ~30 min if pack-restore is parallel; longer if serial |
|
||||
| Re-build FTS5 indexes | ~10 min |
|
||||
| `make chain-check-shards` | ~minutes |
|
||||
|
||||
Hard discipline: do NOT mutate the live corpus until the new pack
|
||||
format is ready to restore into the new layout. Mistake in the
|
||||
routing function = corrupted federation.
|
||||
|
||||
## Implementation phases
|
||||
|
||||
**Phase 0 — design lock.** Pin M = 8 as default. Document
|
||||
`shard_for_document` exactly. Add `corpus_shard_count` field to
|
||||
`meta` table + bump `schema_version` to 9.9 (or whatever's next).
|
||||
This phase ships as a doc PR only — no code yet.
|
||||
|
||||
**Phase 1 — read path.** `connect_query` learns to expect M shards
|
||||
named `000.db` through `{M-1:03d}.db`. Refuses to attach if shard
|
||||
count != M. Doesn't break the current 4-shard setup (M = 4 explicit
|
||||
in the meta table).
|
||||
|
||||
**Phase 2 — ingest path.** `ingest_source` opens M connections,
|
||||
routes each row by `shard_for_document`. Tests verify two
|
||||
independent ingest calls of the same docs land them in the same
|
||||
shards. Multi-writer SQLite contention measured at M = 4 vs M = 8.
|
||||
|
||||
**Phase 3 — cold-pack restore.** Pack-pull re-routes rows by the
|
||||
manifest's M, not the producer's. Lets a peer pull a 16-shard
|
||||
publisher's packs and reassemble into M = 8 locally. (Audit chain
|
||||
re-numbering is a sub-problem here; see "Open questions" below.)
|
||||
|
||||
**Phase 4 — corpus migration tooling.** `arborist corpus resharding
|
||||
--from 4 --to 8` walks the current 4 shards, computes the new
|
||||
shard_idx for every row, writes to the new M = 8 shards. Audit
|
||||
chain rewritten (per shard) — this is the destructive step; ticket
|
||||
discipline says "don't run until cold-packs of the old layout are
|
||||
in S3 as recovery rollback."
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Audit chain re-numbering across shards.** Each shard has its
|
||||
own `audit_events.seq INTEGER PRIMARY KEY AUTOINCREMENT` and its
|
||||
own `event_hash = sha256(prev || body)`. Rebalancing rows means
|
||||
splitting a producer's audit chain across M consumer shards.
|
||||
Each consumer-side shard gets a subset; the chain on each must
|
||||
be re-hashed. Loses per-producer-per-shard chain identity. Open
|
||||
whether this is acceptable or whether audit events should be
|
||||
globally serialized + replicated to all M shards.
|
||||
|
||||
- **`mesh_*` tables.** Currently scoped to whichever shard the
|
||||
operator initialized. With M shards, where does mesh state live?
|
||||
Probably one canonical shard (`000.db`) by convention, but needs
|
||||
decision.
|
||||
|
||||
- **`chunks_fts` on multi-shard write.** When a worker INSERTs a
|
||||
chunk into shard k, does FTS5 update happen in shard k's
|
||||
contentless FTS table? Yes by construction — FTS5 lives per-DB.
|
||||
Easy.
|
||||
|
||||
- **Cross-shard edges.** Wikilinks pointing between shards (article
|
||||
in 003 → article in 005). Today the edges row carries dst_root
|
||||
and the resolver can hop. After content-hash routing, dst's
|
||||
shard is computable but the edges row still lives in src's
|
||||
shard. No change needed.
|
||||
|
||||
## Scope boundaries
|
||||
|
||||
In scope:
|
||||
|
||||
- Pinning M as corpus constant
|
||||
- Content-hash routing function
|
||||
- Multi-writer ingest path
|
||||
- Corpus migration tool
|
||||
- Tests of cross-peer convergence
|
||||
|
||||
Out of scope:
|
||||
|
||||
- Bumping `SQLITE_MAX_ATTACHED` via custom build (rejected:
|
||||
violates "stock python3 + sqlite3 only")
|
||||
- Multi-connection consumer-side fan-out (option 3 from the
|
||||
discussion — kept as a fallback if M = 8 still doesn't fit
|
||||
every consumer)
|
||||
- Topic-clustering shards (sort docs by content similarity into
|
||||
shards). Possible future ticket but breaks ingest determinism.
|
||||
|
||||
## Status
|
||||
|
||||
Open, scaffold-only. Document is the load-bearing artifact for now.
|
||||
No code lands until the design questions (audit chain handling
|
||||
especially) are resolved. fox: "this implies we will need to
|
||||
reprocess all our data into shards" — captured as the migration
|
||||
hard constraint above.
|
||||
Loading…
Add table
Add a link
Reference in a new issue