# Ticket #000065 — Canonical shard count + content-hash routing

**Status:** closed · landed in `c86d5ac` · production reshard completed 2026-05-26 19:47 UTC (~94 min wall, 3.47M docs / 6.24M chunks / 90.6M edges / 3.47M audit events re-routed to content-hash M=4 layout; full record in `docs/corpus-history.md`). Two mid-flight fixes: cross-shard FK guard (`04edff7`) and WAL checkpoint between phases (`c86d5ac`). Follow-on work tracked separately as tasks #44–#47 (re-pack, verify bucket determinism, genesis fresh peer, retire stale packs).
**Opened:** 2026-05-26
**Scope:** decouple two roles that "shard count" currently conflates:
  ingest parallelism (producer-side, wants vCPU count) and consumer
  ATTACH fan-out (capped at SQLITE_MAX_ATTACHED = 10). Introduce a
  corpus-wide canonical `M` (default 8), route documents to shards by
  `hash(document_root) % M` deterministically. Producer chooses N
  workers ≥ M without forcing consumers to attach > 10 shards.
**Audience:** dav1d (architecture-level invariant change), every
  operator running arborist with `--shards-dir`.
**Hard constraint:** migration is a **content-addressed rebalance,
  NOT a re-ingest**. Every row in the corpus is already addressed by
  content (`document_root`, `leaf_hash`, `merkle_nodes.hash`, etc.) —
  the migration reads existing rows from the current 4 shards, computes
  each row's new shard index via the content-hash routing function, and
  INSERTs into the new M shards. No source-file re-parse, no
  re-canonicalization, no re-chunking, no LLM. Wall time is I/O-bound
  (~20–40 min for fox's current 4-shard / 14M-chunk corpus on local
  disk) instead of hours-to-days for a true re-ingest.

## Problem

`arborist --shards-dir` ATTACHes every `*.db` under the directory and
runs queries via UNION ALL views. SQLite's compile-time
`SQLITE_MAX_ATTACHED = 10` is a hard ceiling — the 11th ATTACH refuses.
Default Python sqlite3 ships with that ceiling; bumping it requires a
custom build, which violates CLAUDE.md's "python3.12 + venv + sqlite3
is enough" property.

But the producer wants more shards. Ingest is single-writer-per-file
in SQLite — N parallel ingest workers want N shard files to write
without serializing on a WAL writer lock. On a 16-vCPU producer,
N=16 maximizes ingest throughput. That breaks every consumer with
default sqlite3.

Today the two roles use the same number, so:

- Producer with N=16 shards → consumer can't open the 11th. Federation
  silently fails.
- Producer with N=4 shards → producer underutilizes available vCPU on
  ingest (75% idle on a 16-vCPU box).

There's no current way for the producer to be parallel AND the
consumer to stay under the ATTACH ceiling.

Also: shard assignment today is **non-deterministic across peers**.
Operator picks which source files go to which ingest command. Two
peers re-ingesting the same corpus may put the same document_root in
different shards. This makes federation gossip + content-addressing
weaker than it should be — two peers' "shard 003.db" can have
different contents.

## Canonical M = 4 (decided 2026-05-26 from real-corpus bench)

The corpus-wide canonical shard count is **M = 4**. Reasoning grounded
in two `bench/shard_count_sweep.py` runs (synthetic + real Wikipedia
2003 cur dump, 2000 docs each, M ∈ {1, 2, 4, 8}).

### Real Wikipedia numbers (2026-05-26)

```
 M   chunks/s    q_p50_ms    q_p99_ms   attach_ms   free ATTACH slots
 1    3,648        0.05        0.20        1.61              9
 2    5,649        0.03        0.19        3.24              8
 4    6,405        0.07        0.30        9.00              6
 8    6,959        0.03        0.28        9.65              2
```

Full CSV: `bench/results/shard-count-sweep-2026-05-26T16-31-34Z.csv`.
Synthetic-corpus comparison: `…T16-20-48Z.csv` (under-estimates per-
doc work because random-word docs skip wikitext parsing + edge
extraction — Wikipedia is the load-bearing measurement).

### Why M = 4

1. **Captures 92 % of peak ingest throughput** (6,405 / 6,959).
   Going from M=4 to M=8 buys only +9 % — diminishing-returns
   regime. The ingest bottleneck on real wikitext is Python CPU
   per worker (canonicalization + edge extraction), not SQLite
   writes; more shards don't unlock that.
2. **Keeps 6 ATTACH slots free** under SQLite's 10-shard ceiling for
   `snapshots.db`, `qa.db`, `selfmodel-chain.db`, `crawl_*.db`,
   future `mesh_*.db` if extracted, and the manifest-pointer
   connection. M=8 leaves only 2 slots — operationally tight.
3. **Mobile-tolerable.** Phone NAND attach is ~5–10× slower than
   NVMe. M=4 attach (9 ms on dev box) → ~45–90 ms cold-start on
   phone — feels instant. M=8 attach (9.65 ms) → ~50–100 ms; also
   fine but with no headroom for the slower mobile flash.
4. **Matches fox's current 4-shard layout** = the cheapest migration.
   Most rows already hash-route to their existing shard with
   probability 1/M = 25 % (vs. uniform reshuffling at any other M).
   The teleport is partial, not total.
5. **Query latency is flat across M within noise** at 2000-doc
   corpus. Larger query suites (n=1000+) on a populated corpus
   would tighten this; current data says shard count doesn't move
   p50 perceptibly when the FTS5 index fits in page cache.

### Why not M = 2

- Loses 13 % of peak ingest throughput vs M=4
- ATTACH headroom (8 slots) overkill for current arborist's
  auxiliary DBs
- Wider per-shard variance: 2 huge shards put more work behind
  one WAL writer lock during ingest spikes; 4 smaller shards
  spread the spike

### Why not M = 8

- +9 % ingest gain doesn't justify halving ATTACH headroom
- Mobile attach cliff (10 shards + 8 corpus + qa + snapshots +
  mesh = right at the ceiling)
- Federation peers with fewer vCPU than producer carry more
  per-shard work that doesn't pay back

### Design

Introduce two numbers, both explicit:

```
N = ingest worker count    — producer's vCPU choice
M = canonical shard count  — corpus-wide constant, M = 4 by default
```

**M is part of the corpus identity.** Pinned in the snapshot manifest +
the metadata pack manifest. Every peer that reads the corpus must
adopt the same M; changing M is a corpus-level migration.

**Document → shard assignment is content-deterministic:**

```python
def shard_for_document(document_root: str, M: int) -> int:
    # First 8 hex chars of the document_root, interpreted as int,
    # taken mod M. Same input → same output across every peer.
    return int(document_root[:8], 16) % M
```

Same for `chunks.document_root` (a chunk's shard = its document's
shard). FKs across shards aren't enforced (CASCADE is per-shard), so
the document → chunk routing must agree by construction.

**Producer flow (post-ticket):**

```text
N ingest workers, M canonical shards (N can be > or < M):

  worker_i opens connections to all M shards
  for each row to insert:
    target_shard = shard_for_document(document_root, M)
    INSERT into shard M[target_shard]
  parallelism on inserts = min(N, M)
  on writer contention within a shard: SQLite WAL serializes
```

Throughput at min(N, M) is the new ingest-parallelism ceiling. Picking
M = 8 makes 8-way ingest the practical max; 16-vCPU producers run
~50% utilization on ingest. That's the price of universal-consumer
compatibility.

If a producer needs >8-way ingest, the option is N > M with the
understanding that workers will queue behind writer locks within each
shard — still parallel-ish because each worker is mostly doing
canonicalization/embedding work between INSERTs.

**Consumer flow (post-ticket):**

```text
Pull metadata pack for each of M shards.
Restore into ~/.arborist/shards/00{0..M-1}.db.
Every peer with the same corpus has the same M files with the same
document_roots in the same shards. ATTACH count = M ≤ 8, well under
the ceiling.
```

## Migration story — teleport, not re-ingest

Current arborist has 4 shards (fox's corpus). Documents in each shard
were assigned by `ingest_source` calls — the operator splits work,
each call writes to one shard. No content-hash routing.

Migration is **a content-addressed rebalance**, not a re-ingest.
Every row already has its content-derived identity. The migration
just moves rows from their current (ingest-order-determined) shard
to their (content-hash-determined) shard:

1. **Pick M.** Default: 8 (leaves headroom under ATTACH=10 for
   `snapshots.db`, `qa.db`, `selfmodel-chain.db`, the manifest
   pointer connection).
2. **Cold-pack the current 4 shards** as a recovery rollback
   (already in bucket from #000061).
3. **Create M new empty shards.**
4. **Per-table teleport:**

   | Table | Move how | Notes |
   |-------|----------|-------|
   | `documents` | by `document_root` → new shard via routing fn | content-addressed PK, just INSERT |
   | `chunks` (incl. content) | follow `chunks.document_root` to same shard as parent doc | re-assign `chunk_id` autoincrement local |
   | `chunks_fts` | rebuild per new shard from `chunks.content` | contentless FTS5, bounded by chunk count |
   | `merkle_nodes` | follow `document_root` | move by PK |
   | `edges` | by `src_root` → new shard | content-addressed PK |
   | `concept_relations` | follow `target_root` / etc. → new shard | re-assign local id |
   | `derivations` | follow `core_root` | move |
   | `providence_cache` | follow `source_root` | move |
   | `citation_aliases`, `term_aliases` | content-addressed | move |
   | `audit_events` | **all → shard 000 (canonical)** | preserves global ordering + chain integrity |
   | `snapshots` | shard 000 (canonical) | snapshot is corpus-wide |
   | `mesh_*` | shard 000 (canonical) | mesh state is per-peer, not per-document |
   | `concept_token_idf` | follow token's owning shard via hash(token) % M | local IDs |

5. **Rebuild `chunks_fts` per new shard.** Contentless FTS5 inserts
   from `chunks.content`. Minutes-scale per shard.
6. **VACUUM.** Reclaim free pages after the row moves.
7. **`make chain-check-shards`.** Audit chain in shard 000 still
   verifies end-to-end because rows weren't reordered, just moved
   together.

**Audit chain handling — chosen Option A (canonical shard 000):**

Each shard's existing `audit_events.seq INTEGER PRIMARY KEY
AUTOINCREMENT` chain stays intact when moved as a contiguous block.
Consolidating four chains into one means re-numbering `seq` and
re-hashing the chain end-to-end (interleaving by `ts`, recomputing
`event_hash` per row). Loses per-producer-per-shard chain identity
but preserves global event ordering. Alternative (Option B: split
audit by `subject_root` to per-shard chains) was considered and
rejected — loses cross-shard event ordering, harder to reason about
for falsification + replay.

Estimated wall-clock cost (4 shards × 8.8 GB each = ~35 GB total):

| Phase | Cost |
|-------|------|
| Cold-pack current state (recovery rollback) | 0 — already in bucket |
| Create M empty target shards | seconds |
| Move documents + chunks + chunks.content | I/O bound, ~10 min |
| Move edges (largest, 22M × 4 rows total) | ~10 min |
| Move merkle + concepts + derivations + small tables | ~minutes |
| Consolidate audit_events → shard 000 + re-hash chain | ~few min |
| Rebuild `chunks_fts` per new shard | ~minutes per shard |
| VACUUM | ~few min per shard |
| `make chain-check-shards` | ~minutes |
| **Total wall** | **~20–40 min** |

Vs. a true re-ingest from Wikipedia dumps: ~hours.

Hard discipline: do NOT mutate the live shards in place. Move into
NEW shards (`~/.arborist/shards.v2/`), keep the originals as
rollback until the move is verified, then atomically swap directories.
Mistake in the routing function = corrupted federation, recoverable
only by going back to the original 4 shards.

## Implementation phases

**Phase 0 — design lock.** Pin M = 8 as default. Document
`shard_for_document` exactly. Add `corpus_shard_count` field to
`meta` table + bump `schema_version` to 9.9 (or whatever's next).
This phase ships as a doc PR only — no code yet.

**Phase 1 — read path.** `connect_query` learns to expect M shards
named `000.db` through `{M-1:03d}.db`. Refuses to attach if shard
count != M. Doesn't break the current 4-shard setup (M = 4 explicit
in the meta table).

**Phase 2 — ingest path.** `ingest_source` opens M connections,
routes each row by `shard_for_document`. Tests verify two
independent ingest calls of the same docs land them in the same
shards. Multi-writer SQLite contention measured at M = 4 vs M = 8.

**Phase 3 — cold-pack restore.** Pack-pull re-routes rows by the
manifest's M, not the producer's. Lets a peer pull a 16-shard
publisher's packs and reassemble into M = 8 locally. (Audit chain
re-numbering is a sub-problem here; see "Open questions" below.)

**Phase 4 — corpus migration tooling.** `arborist corpus resharding
--from 4 --to 8` walks the current 4 shards, computes the new
shard_idx for every row, writes to the new M = 8 shards. Audit
chain rewritten (per shard) — this is the destructive step; ticket
discipline says "don't run until cold-packs of the old layout are
in S3 as recovery rollback."

## Open questions

- **Audit chain re-numbering across shards.** Each shard has its
  own `audit_events.seq INTEGER PRIMARY KEY AUTOINCREMENT` and its
  own `event_hash = sha256(prev || body)`. Rebalancing rows means
  splitting a producer's audit chain across M consumer shards.
  Each consumer-side shard gets a subset; the chain on each must
  be re-hashed. Loses per-producer-per-shard chain identity. Open
  whether this is acceptable or whether audit events should be
  globally serialized + replicated to all M shards.

- **`mesh_*` tables.** Currently scoped to whichever shard the
  operator initialized. With M shards, where does mesh state live?
  Probably one canonical shard (`000.db`) by convention, but needs
  decision.

- **`chunks_fts` on multi-shard write.** When a worker INSERTs a
  chunk into shard k, does FTS5 update happen in shard k's
  contentless FTS table? Yes by construction — FTS5 lives per-DB.
  Easy.

- **Cross-shard edges.** Wikilinks pointing between shards (article
  in 003 → article in 005). Today the edges row carries dst_root
  and the resolver can hop. After content-hash routing, dst's
  shard is computable but the edges row still lives in src's
  shard. No change needed.

## When the SQLite-default substrate stops being right

The M=4 choice is correct **conditional on staying on stock python3
sqlite3 with SQLITE_MAX_ATTACHED=10**. Three thresholds where that
assumption breaks; each names what bench would justify moving:

### Threshold A: ATTACH-ceiling pressure forces a forked SQLite

**Signal:** auxiliary `.db` files (`qa.db`, `snapshots.db`,
`selfmodel-chain.db`, `crawl_*.db`, future `mesh_*.db`) plus M=4
canonical shards plus operator-facing connections push past 10 in
practice. Today: 4 canonical + 3-4 auxiliary = 7-8, fits with M=4
headroom. If auxiliary grows past 5, M=4 becomes the binding
constraint.

**What to bench before the fork:**

1. Compile sqlite3 with `SQLITE_MAX_ATTACHED=125` (the spec maximum).
2. Re-run `bench/shard_count_sweep.py` with M ∈ {16, 32, 64} on the
   real corpus.
3. Measure: ATTACH cost growth (linear vs. super-linear?), query
   latency under wider UNION ALL, ingest throughput at higher M.
4. **Decision criteria:**
   - If ATTACH cost stays roughly linear past M=10 → forked SQLite
     viable. Costs: a custom build to ship with arborist or a
     dependency on system-sqlite-with-this-flag (Debian / Fedora
     packages, mobile builds).
   - If ATTACH cost goes super-linear past M=10 → SQLite's
     architectural design doesn't expect wide multi-attach; need a
     different substrate.

**Cost of the fork:** ~1 day of build infrastructure. ~Permanent tax
on every install: arborist no longer "just works" on stock sqlite3.
Violates the CLAUDE.md "python3 + venv + sqlite3 is enough" property.
Don't do it unless the bench says we have to.

### Threshold B: ingest throughput hits a wall

**Signal:** workload requires real-time ingest at > 10k chunks/s
sustained (e.g., live mesh-replicated state across many peers). On
this benchmark, real-Wikipedia ingest peaks at ~7,000 chunks/s
regardless of M past 4. SQLite's single-writer-per-file model is the
ceiling — adding more workers behind the same writer lock doesn't
help.

**What to bench before considering alternatives:**

1. **Page size tuning.** Default 4 KB; arborist could rebuild shards
   with 16 KB or 64 KB pages. Bench: 4 KB vs 16 KB vs 64 KB on the
   same M=4 layout. Larger pages help wide-row tables (`edges`)
   especially.
2. **WAL checkpoint interval.** Default auto-checkpoint at 1000
   frames. Bench: ingest throughput at 100, 1000, 10000-frame
   intervals.
3. **`mmap_size` sweep.** Default 256 MB in arborist's `connect`.
   Bench: 0, 64 MB, 256 MB, 1 GB.
4. **`synchronous=NORMAL` vs `OFF` for ingest workers.** NORMAL is
   the default + safe under WAL. OFF removes the per-commit fsync
   entirely and is faster but loses durability on power loss. For
   ingest from a deterministic dump (re-runnable), `OFF` might be
   acceptable.

**Decision criteria:**
- If tuning gets us 2-5× more throughput → stay on SQLite, document
  the tuned pragmas in `connect()`.
- If still ceiling-limited at 10k+ chunks/s → genuine substrate
  question. Candidates:
  - **DuckDB** (columnar, MVCC, multiple concurrent writers,
    supports SQLite-compatible SQL surface, has its own FTS).
    Bench against same harness.
  - **libmdbx / LMDB** (B+tree, MVCC, very fast reads). No FTS;
    we'd build it. Probably overkill for arborist's read patterns.
  - **In-house DB.** Too ambitious without a specific failure of
    the above to justify.

### Threshold C: federation needs multi-writer-same-shard

**Signal:** the mesh state evolves so that multiple peers can write
to the same shard simultaneously (today: single-writer-per-shard
is the assumption — mesh gossip pulls audit events but doesn't
co-write). SQLite's writer lock serializes across peers — would
become a federation bottleneck.

**What to bench before alternatives:**

1. Two peers writing to a network filesystem (NFS) backed shard with
   WAL — does SQLite's locking actually work, and what's throughput?
   (Spoiler: SQLite famously hates NFS; this fails.)
2. Two peers writing to a shared block device (SAN) with WAL —
   similar story but better.
3. Application-level coordination: only one peer is "leader" for a
   shard at a time; followers replicate the WAL frames.

**Decision criteria:**
- If single-leader-per-shard works → stay on SQLite, build the
  leader-election layer above it.
- If we need true multi-writer-same-shard → SQLite is wrong; needs
  a substrate with built-in MVCC + conflict resolution. **DuckDB
  doesn't solve this either** — its MVCC is single-process. The
  candidate becomes something like FoundationDB or a CRDT layer on
  any key-value store.

### Honest summary

For the current arborist workload (single-writer-per-shard, M=4,
read-mostly federation), **stock python3 sqlite3 is the right
substrate.** None of the three thresholds are close to firing. The
bench discipline above exists so we know what to measure when
something changes, not as a roadmap to leave SQLite.

## Scope boundaries

In scope:

- Pinning M as corpus constant
- Content-hash routing function
- Multi-writer ingest path
- Corpus migration tool
- Tests of cross-peer convergence

Out of scope:

- Bumping `SQLITE_MAX_ATTACHED` via custom build (rejected:
  violates "stock python3 + sqlite3 only")
- Multi-connection consumer-side fan-out (option 3 from the
  discussion — kept as a fallback if M = 8 still doesn't fit
  every consumer)
- Topic-clustering shards (sort docs by content similarity into
  shards). Possible future ticket but breaks ingest determinism.

## Status

Open, scaffold-only. Document is the load-bearing artifact for now.
No code lands until the design questions (audit chain handling
especially) are resolved. fox: "this implies we will need to
reprocess all our data into shards" — captured as the migration
hard constraint above.
