push_pack now does a HEAD object_size check against the bucket
before every PUT. If the object exists at the content-addressed
key AND Content-Length matches the local pack body size, skip the
upload. Applies to chunks / fts / metadata pack bodies.
Why content-addressed-key + size is the right check:
pack_hash = hash_leaf(manifest_bytes)
object_key = packs/<pack_hash>.<kind>.tar.zst
The pack_hash IS the cryptographic identity of the body via its
manifest. If the bucket has an object at this exact key, the
content is provably identical (otherwise the producer that wrote
it computed a different pack_hash → different key). Content-Length
match validates against a half-uploaded multipart that a prior
crash might have left.
Why not ETag: S3 ETag for multipart-uploaded objects is the MD5
of per-part MD5s concatenated, then MD5'd, with a "-N" suffix for
part count. Two clients using different multipart_chunksize end
up with different ETags for the same content. Unreliable for
cross-client equivalence. Content-Length is stable.
Result fields gain `skipped_reupload: bool` so callers / the
audit chain can tell whether the bucket PUT happened or was a
no-op.
Real-world impact for tonight's re-pack (--include-fts):
before: 4 chunks PUTs (7 GB re-upload, ~24 min)
+ 4 metadata PUTs (new hashes because manifest now has
_fts_pack_hashes — these MUST upload)
+ 4 fts PUTs (new)
~16 GB uplink, ~50 min wall
after: 4 chunks SKIPPED (same content-addressed key already
in bucket)
+ 4 metadata PUTs (new hashes, MUST upload)
+ 4 fts PUTs (new)
~6 GB uplink, ~20 min wall
cold_pending bookkeeping still fires in the upload path so a
half-uploaded crash recovers cleanly; skip path leaves no trace
because nothing was started.
4 cold-unpack-routed tests pass unchanged.
The 3090 genesis bench at 28-min progress mark showed all 4 workers
sitting in jbd2_log_wait_commit / do_get_write_access — ext4
filesystem journal serializing every page-extension event across
the 4 parallel writers. SQLite synchronous=OFF and journal_mode=
MEMORY skip SQLite's own fsync/WAL costs, but file growth still
goes through ext4's journal layer.
Fix: bump SQLite page_size from default 4 KB to 16 KB on each
hydrate target. Bigger pages mean ~4× fewer page writes during
chunk-fill, which means ~4× fewer file-extension events through
ext4's journal. Direct attack on the measured bottleneck.
page_size can only be set on an EMPTY database; VACUUM finalises
the change. The unpack CLI now opens each target file raw,
PRAGMA page_size = 16384 + PRAGMA secure_delete = OFF + VACUUM,
closes, then connect() applies schema as normal on the
pre-sized empty DB. Idempotent on re-runs since hydrate dirs
are always clean.
Also sets secure_delete = OFF (default is OFF in most builds but
explicit for hydrate writers): no overwrite-with-zeros on row
deletion, less write traffic during the chunk UPDATEs that might
free overflow pages.
Expected impact: ~30-50% reduction in chunk-fill wall time vs the
killed 30-min v2 run. Combined with the prior bench-max round
(journal_mode=MEMORY, deferred FTS, batched UPDATEs) and the
upcoming fts-pack-skip-rebuild path, full genesis should land
under 30 min total instead of the 75-min v2 extrapolation.
4 cold-unpack-routed tests pass unchanged.
Each producer shard now optionally emits a THIRD pack alongside its
metadata and chunks packs: an "fts" pack containing the FTS5 shadow
tables (chunks_fts_data, chunks_fts_idx, chunks_fts_docsize,
chunks_fts_config + the documents_fts_* counterparts) packed as a
fresh SQLite file inside the tar so BLOB columns round-trip natively.
Consumer detects fts_pack_hashes in the metadata pack's manifest,
pulls each fts pack, ATTACHes the embedded sqlite, INSERTs every
shadow-table row into its target's empty shadow tables, and SKIPS
the local FTS rebuild entirely.
Producer side:
arborist/cold_object.py
+ PACK_KIND_FTS = "fts"
+ FTS_SHADOW_TABLES tuple (8 shadow tables)
+ build_fts_pack(src_db_path, ...)
creates a temp sqlite, applies SCHEMA_SQL (so destination
has FTS virtual tables → shadow tables auto-created), copies
every shadow-table row from src via cursor iteration, packs
the sqlite file into tar.zst
+ ParsedManifest.fts_pack_hashes
+ parse_manifest reads _fts_pack_hashes records
+ build_metadata_pack accepts fts_pack_hashes parameter and
writes the new manifest record
arborist/evict.py:push_pack
+ include_fts: bool = True parameter (CLI --no-fts opts out)
+ Phase B.5 emits the fts pack BEFORE Phase C (metadata pack)
so its hash can be referenced in the metadata manifest
Consumer side:
arborist/evict.py
+ _pull_fts_pack_into_targets() — pulls fts pack body, extracts
embedded sqlite, ATTACHes into each target, INSERT OR IGNORE
every shadow-table row. INSERT OR IGNORE protects against
rowid collisions on other targets that don't own these chunks.
+ hydrate_from_metadata_pack_routed iterates fts_pack_hashes in
full mode, calls _pull_fts_pack_into_targets per pack
+ _pull_pack_inner_routed returns fts_pack_hashes_referenced in
its result dict (mirrors chunk_pack_hashes_referenced)
CLI / Makefile:
arborist cold pack --no-fts (opt-out)
make cold-hydrate (auto-detects: if
chunks_fts_data is
already populated
on shard 000 after
unpack, skip the
rebuild post-pass)
make cold-hydrate HYDRATE_REBUILD_FTS=1 (force rebuild)
make cold-hydrate HYDRATE_REBUILD_FTS=0 (skip rebuild)
Schema:
cold_pending.kind CHECK extended to include 'fts'
pack_key() accepts kind="fts" → packs/<hash>.fts.tar.zst
Expected wall-time impact on the 3090 genesis bench:
with fts in packs: no rebuild step → ~5-10 min total wall
without fts: rebuild post-pass needed → ~15-20 min
Trade-off: ~30-50% larger bucket (FTS shadow data per shard) for
~70-90% faster consumer hydrate. Producer flips the trade via
--no-fts. The fts pack is optional in the manifest (empty list →
consumer falls back to rebuild) so old bucket data without fts
packs continues to work unchanged.
34 cold-unpack-routed + migrate + planner tests pass.
Real-measured bottleneck during the 3090 genesis run (2026-05-26):
all 4 parallel workers sitting in jbd2_log_wait_commit /
do_get_write_access — fighting for ext4's single filesystem journal.
The reshard executor was fast because it ran FTS5 rebuild
SEQUENTIALLY (one process per target in turn); the parallel 4-way
unpack collapsed that into journal contention.
Two coupled changes:
1. arborist/evict.py — _pull_pack_inner_routed
* Drop the _rebuild_fts_on_target call from phase 2c entirely.
Parallel inner loop now does: download → metadata route →
chunks fill. NO FTS5 writes during the hot parallel phase.
* Bulk-load PRAGMA tuning on every target connection:
synchronous=OFF no fsync per commit (already had this)
journal_mode=MEMORY WAL in RAM, not on disk (was WAL)
temp_store=MEMORY sort scratch in RAM
cache_size=-524288 512 MB page cache per connection
mmap_size=536870912 512 MB read mmap
Genesis crash recoverability = re-pull from bucket, so
durability of intermediate state has no value — these tunings
trade durability for throughput.
2. arborist/cli.py — new `arborist cold rebuild-fts --shards-dir DIR`
subcommand. Sequentially rebuilds chunks_fts + documents_fts on
every shard, one at a time. Each shard gets the full filesystem
journal in its turn. Total wall = sum of single-shard FTS
rebuild times, NOT 4× contention.
3. Makefile — `cold-hydrate` chains the FTS rebuild automatically
after the parallel unpack (only when HYDRATE_MODE=full, since
just-enough has no chunk bodies to index anyway). Operator can
skip with HYDRATE_REBUILD_FTS=0 for a 2-pass workflow.
Expected wall time on 3090:
parallel hydrate (download + chunks fill): ~5-10 min
serial FTS rebuild × 4 shards: ~5-15 min total
total: ~15-25 min
vs the killed 2.5-hour run.
34 existing cold-unpack-routed + migrate + planner tests still pass
(the tests' FTS check runs against the reshard path, which still
calls _rebuild_fts_on_target inside the executor — that path is one
process, no parallel contention).
Two genesis paths the SPV-wallet design supports, now both exposed
via the same make target:
make cold-hydrate HYDRATE_DIR=… HYDRATE_M=4 HYDRATE_MODE=full
pulls metadata packs + every referenced chunk pack. Full corpus
offline-queryable post-hydrate.
make cold-hydrate HYDRATE_DIR=… HYDRATE_M=4 HYDRATE_MODE=just-enough
pulls only metadata packs (~4 GB on a 37 GB corpus → ~10×
bandwidth reduction). chunks rows land with content=NULL, every
chunk-body query misses local cache and falls through to a
future JIT-fetch path (cache miss → CDN / mesh / re-pull). The
SPV-wallet headline shape.
Defaults to full for backward compatibility. The mode-flag validation
guards against typos at recipe time (invalid mode → exit 2 before
firing parallel boto3 traffic).
Bench target: measure both paths back-to-back on 3090. Just-enough
should bound the network phase only (no chunk-body fill, no FTS5
rebuild); full adds the chunk-restore phase that #000067's bench-
max round just optimized.
Real-world measure 2026-05-26 18:43 UTC: hydrating an 11 GB
post-reshard corpus (3.47M docs / 6.24M chunks) into a fresh 4-shard
peer ran at ~32 MB/min per shard — extrapolating to ~4 hours total,
the same speed as a fresh XML ingest. The whole point of a cold-pack
restore is being MUCH faster than re-ingest; killed mid-flight and
shipped these three optimizations:
(1) Build leaf_hash → (target_idx, chunk_id) map once up front
from each target's chunks table (M scans, total ~6M rows).
Replaces M=4 SELECTs per incoming chunk_body — was 24M
lookups, now 6M scan-once. ~4× win on the dispatch step.
(2) Batched executemany UPDATE per target (BATCH=5000) — replaces
the per-row `with transaction(target): UPDATE; INSERT_FTS`
pattern that opened 6M tiny transactions. ~10× win on disk
write throughput.
(3) Skip chunks_fts during the fill loop entirely; defer to one
bulk rebuild per target after all chunks are in place. Reuses
the existing arborist.migrate._rebuild_fts_on_target primitive
(decompresses via unpack_chunk, streams chunks → FTS via
batched executemany). Each per-row chunks_fts insert costs an
inverted-index update; bulk rebuild is ~10× faster than
incremental.
Plus PRAGMA synchronous=OFF on each target connection: genesis is
end-to-end rebuildable (a crash mid-hydrate leaves empty shards we
re-pull from the bucket), so durability of intermediate WAL pages
is not required. WAL stays bounded by PRAGMA wal_checkpoint(TRUNCATE)
at the end of phase 2b (already done implicitly by _rebuild_fts_on_target
in phase 2c).
Expected speedup: ~30-60× combined. Real number lands when the
re-run on 3090-ai.foxhop.net completes.
All 34 cold-unpack-routed + migrate tests pass unchanged.
Adds two missing pieces to the cold-pack lifecycle:
1. ALLOW_LICENSE_CLASS pass-through on cold-pack and cold-pack-all.
Without this, packs from a corpus containing any `unknown`-class
docs (html crawls / textbooks) refuse to push to a public bucket
per #000061 Gap 2. Real-world bench on 2026-05-26 hit this
because the production corpus has crawl + textbook content.
Usage:
make cold-pack-all ALLOW_LICENSE_CLASS=unknown
make cold-pack DB=~/.arborist/shards/000.db \
ALLOW_LICENSE_CLASS=unknown
2. cold-hydrate — M-aware genesis-from-bucket as a make target.
Discovers every metadata pack in the bucket via `cold list
--no-manifest` + a tiny python json filter, then fires
$(JOBS)-way parallel `arborist cold unpack` workers (one per
metadata pack) into the same target shards directory. Each
worker writes to one disjoint target shard because the bucket's
packs come from the post-reshard #000065 source layout (each
producer shard's docs all hash to one consumer target) — so
zero write contention across the parallel workers.
Usage:
make cold-hydrate \
HYDRATE_DIR=~/.arborist/shards-genesis-test \
HYDRATE_M=4
Why these belong here (not in /tmp wrapper scripts that get tossed):
fox 2026-05-26 — "make sure the pack and upload and the pull and
ingest all happen with makefile targets". The cold-pack lifecycle
is a first-class operations surface; throwaway tmp scripts hide
the real interface from future operators.
Both targets are listed in `make help`. The .PHONY line covers
cold-hydrate so a directory named cold-hydrate can't shadow it.
Sequence to repeat today's run end-to-end:
producer: make cold-pack-all ALLOW_LICENSE_CLASS=unknown
consumer: make cold-hydrate HYDRATE_DIR=~/.arborist/shards \
HYDRATE_M=4
Today's in-flight 3090 hydrate (started before this commit) used
the equivalent shell loop; from now on every operator uses the
make target.
Self-contained benchmark for the SPV-wallet validation. Records:
PRODUCER bucket state + pack count + compressed bytes
CONSUMER wall time, exit status, post-hydrate shard sizes,
per-shard documents/chunks/edges counts
Driver runs against the real DO Spaces bucket + the real fresh peer
on 3090-ai.foxhop.net. Writes one JSON artifact per run to
bench/results/cold-pack-roundtrip-<ISO>.json so a future operator
can diff hydrate times across pack-format changes (#000061 v3 →
graft mode #000066 → mesh-pull future).
Consumer command uses /usr/bin/time -v wrapped around
`arborist cold unpack --hydrate-shards-dir … --hydrate-M 4 --full`.
Hydrates into ~/.arborist/shards-genesis-test/ so it doesn't
clobber anything on the 3090.
No new ticket — this is task #45/#46 instrumentation. Existing
tests untouched.
Today's hydrate_from_metadata_pack writes every row into one shard.
With the corpus in M=4 hash-routed topology (#000065), a fresh peer
pulling packs must land each doc on shard_for_document(root, M) —
same routing function as the producer — or the consumer's M=4
ATTACH-and-route assumption is just decoration over a single-shard
reality.
Code (new entry points alongside the existing single-conn ones):
arborist/cold_pack_metadata.py
+ restore_shard_metadata_routed(targets, M, table_dir)
+ _restore_routed_table (per-document tables)
+ _restore_edges_fan_out_routed (edges by src_root)
+ _shard_idx_for_root helper (mirrors arborist.document)
+ routing rules: _ROUTED_BY_COL / _CONSOLIDATED_TO_SHARD_0
(mirror migrate.py's ROUTED_BY_DOCUMENT_ROOT / CONSOLIDATED_TABLES)
arborist/evict.py
+ hydrate_from_metadata_pack_routed(targets, backend, hash, M=, mode=)
+ _pull_pack_inner_routed (mirrors _pull_pack_inner; phase 2
chunk-body fill iterates every target shard — leaf_hash lookup
naturally hits at most one since each chunk's metadata row
landed on exactly one target during phase 1)
arborist/cli.py
arborist cold unpack
+ --hydrate-shards-dir DIR (M-aware genesis path)
+ --hydrate-M N (default 4 matches #000065)
legacy --db / --global-shards-dir path unchanged
Behaviour notes:
- FK enforcement off on target writes (cross-shard refs are valid
under hash routing, same fix as #000065 reshard executor)
- audit_events lands on target 0 (Option A consolidation)
- corpus-wide tables (snapshots / concepts / aliases / providence)
consolidate to target 0
- per-document tables route by document_root (documents,
document_http_meta, chunks, merkle_nodes)
- edges route by src_root (matches migrate.py)
- derivations route by core_root (matches migrate.py)
- existing single-conn API unchanged — callers that didn't pass a
shards-dir get the legacy single-shard behaviour
4 new tests:
test_pack_then_hydrate_routed — end-to-end pack → hydrate → assert
every doc on its hash-routed target, no doc on the wrong shard
test_corpus_shard_count_set_on_routed_hydrate — meta plumbing
test_M_mismatch_rejected
test_empty_targets_rejected
All 59 prior tests still pass.
Refactor opportunity (not taken): _ROUTED_BY_COL duplicates
migrate.py's ROUTED_BY_DOCUMENT_ROOT. A shared arborist/multi_shard.py
module would serve both reshard and graft (#000066). Left as
follow-up since the duplication is small and graft is still scaffold.
Unblocks #46 genesis on 3090: that's now a single arborist cold unpack
--hydrate-shards-dir ~/.arborist/shards --hydrate-M 4 invocation
instead of the α two-step kludge (hydrate-then-reshard).
Open ticket. Today's hydrate_from_metadata_pack takes one conn and
writes every incoming row into one shard — fine when the corpus
was a single shard, broken now that #000065 put the producer in
M=4 hash-routed topology. A fresh peer pulling packs must land each
doc on `shard_for_document(document_root, M)` — same routing
function as the producer — or the consumer's M=4 ATTACH-and-route
assumption is just decoration over a single-shard reality.
Plan:
1. Add corpus_shard_count to pack manifest (read from source meta
during dump_shard_metadata) — pack carries the M it was built
against.
2. restore_shard_metadata_routed(targets, M, table_dir) in
cold_pack_metadata.py — mirrors _route_per_doc_table from
migrate.py (per-document tables route by document_root /
src_root / core_root; consolidated tables all go to target 0).
3. hydrate_from_metadata_pack gains a targets / shards_dir param.
4. arborist cold unpack --shards-dir DIR initialises M target
shards from the manifest's corpus_shard_count and routes.
5. Regression test: pack 2 shards → hydrate into fresh 4 shards
→ assert every doc on its hash-routed target.
Refactor question (raised, not decided): the routing rules
(ROUTED_BY_DOCUMENT_ROOT, CONSOLIDATED_TABLES) currently live in
migrate.py. Either duplicate them in cold_pack_metadata.py (fast)
or factor into arborist/multi_shard.py (cleaner, also serves
#000066 graft mode). Shared module is more honest.
Prerequisite for #46 (genesis on 3090 from cloud). Without this,
genesis is a 2-step α-kludge (hydrate-then-reshard) that wastes
~30 min and treats packed shards as if from an arbitrary topology.
Index entry bumped; next-id 67→68. Per-ticket spec doc to follow
when the implementation gates open.
2026-05-26 19:47 UTC. ~94 min wall. 3,468,226 globally-unique docs +
6,235,588 chunks + 90,592,990 edges + 3,468,403 audit events re-routed
from non-deterministic spray-by-ingest-order layout to canonical
content-hash M=4 layout (shard_idx = int(document_root[:8], 16) % 4).
Final state:
per-shard doc uniformity within ±0.04% (theoretical max ±0.05% for
first-32-bit SHA-256 prefix)
audit chain consolidated to shard 000 via Option A (re-sorted by ts,
re-chained; bodies preserved unchanged; tail event type=reshard
carries plan+result body, hash 8da3aa19…)
on-disk sizes: 000=11.0 / 001=8.8 / 002=8.8 / 003=8.8 GB
validation: chunks delta 176 (0.003%) + edges delta 547 (0.0006%)
are cross-shard dupes from re-ingest history, collapsed by
INSERT OR IGNORE; within the 1% tolerance gate
smoke queries: Barack Obama / YouTube / Albert Einstein all
returned proper evidence from correct (hash-routed) shards
chain-check-shards: 0 breaks on every shard
Two mid-flight defect fixes (also committed):
04edff7: derivations.src_root FK guard fired on legitimately
cross-shard refs → writer connection PRAGMA foreign_keys = OFF
c86d5ac: WAL accumulated ~37 GB across passes (SQLite auto-checkpoint
blocked by open reader cursors) → _checkpoint_truncate between
phases. Production migration was rescued mid-flight by manual
sibling-connection wal_checkpoint(TRUNCATE) freeing 27.7 GB.
Closes#000065 in both the index and the per-ticket file. The
per-ticket design doc stays open as a design-log artifact (its
content is still the right reference for the next reshard / for
graft mode #000066).
Follow-on tracked as tasks #44–#47:
#44 re-pack post-reshard shards into DO Spaces (current bucket
packs are stale, still in pre-reshard topology)
#45 verify bucket pack hydration is deterministic against the new
content-hash layout
#46 genesis a fresh peer on 3090-ai.foxhop.net from cloud — first
real SPV-wallet end-to-end test
#47 retire stale pre-reshard bucket packs after #46 confirms
Add PRAGMA wal_checkpoint(TRUNCATE) at two points in
_execute_all_at_once so committed WAL pages don't pin disk through
subsequent passes. Production migration on 2026-05-26 hit 7 GB free
disk (down from 89) because SQLite's auto-checkpoint can't run while
a reader cursor is open, and FTS rebuild keeps a SELECT cursor open
through 1.5M chunks per target. Across 4 targets the FTS rebuild
plus audit consolidate held ~37 GB of committed-but-unreclaimed WAL.
Manual sibling-connection wal_checkpoint(TRUNCATE) freed 27 GB
mid-migration.
Checkpoints land at:
* end of _rebuild_fts_on_target (after the SELECT cursor is
explicitly cur.close()'d so the TRUNCATE checkpoint can actually
fire — TRUNCATE/RESTART block on active readers)
* end of _consolidate_audit_chain (after the 3.47M-row giant
transaction commits, before the next phase touches the same
connection)
VACUUM is already implicitly a checkpoint, so the existing per-
target VACUUM pass continues to handle the final checkpoint
naturally.
Helper _checkpoint_truncate(conn) returns the (busy, log_frames,
checkpointed) tuple SQLite emits; for the serial executor, busy=1
is improbable since each phase finishes before moving on.
Regression test
(TestWalCheckpointing.test_no_large_wal_after_migration) asserts
no WAL file exceeds 4 MB after migration completes. Without the
checkpoint calls this would fail on real-sized corpora; with them
the test passes deterministically.
Doesn't affect the running migration (it loaded the module from
memory before this commit). Future reshards run with bounded WAL —
no near-ENOSPC scares.
The 2026-05-26 cutover crashed mid-build with sqlite3.IntegrityError
"FOREIGN KEY constraint failed" inside _route_per_doc_table on the
derivations table.
Root cause: derivations.src_root carries a FK to
documents.document_root, but under content-hash routing a derivation
row's src_root can legitimately reference a surface doc that hashes
to a DIFFERENT target shard than the derivation's core_root. The FK
is a single-shard-era guard; it must stay live for the runtime
write path (to catch typo'd inserts into the wrong shard) but must
be OFF for the migration writer which legitimately produces
cross-shard refs.
Fix: arborist/migrate.py _connect_target now applies
`PRAGMA foreign_keys = OFF` after SCHEMA_SQL executescript runs.
Schema's own `PRAGMA foreign_keys = ON` still applies to the schema
DDL pass (and runtime connect() / connect_query() still get FK=ON
since they don't touch this helper). Only the migration writer is
relaxed. Documented inline.
Regression test
(TestCrossShardForeignKeys.test_cross_shard_derivation_succeeds)
synthesizes a derivation row whose core_root and src_root hash to
different M=4 target shards, runs the migration, asserts the row
lands on core_root's target with src_root pointing cross-shard. Pre-
fix this raised IntegrityError; post-fix it passes.
Originals untouched on the production host — the executor crashed
BEFORE the atomic-promote step, so .db files are intact;
~/.arborist/shards/00X.db.new files from the failed run will be
cleared by the next attempt's "stale .new before opening" cleanup
hook (already in _execute_all_at_once).
Scaffold-only ticket. Captures the architecture for taking the
#000061 cold-pack format and adding a second mode beside hydrate:
overlay an existing pack onto a populated shard set ("graft").
Surfaced while running the #000065 reshard cutover and fox extended
the design: each pack carries a `corpus_name` field
(wikipedia-2010, wikipedia-current, arxiv-cs, ...), making
`arborist cold graft wikipedia-current` feel like `apt install`.
Three concerns analysed:
doc/chunk/edge overlay trivial (INSERT OR IGNORE on content-
addressed PKs collapses dupes)
FTS5 overlay trivial (new chunk_ids → new fts rows)
audit chain overlay the only hard part — three approaches:
A graft receipt (chosen): one event in
host chain carrying pack_hash +
event_count + first/last hashes; pack
file is the durable witness; zero
schema cost; aligned with v8/v9
witness pattern
B re-chain everything: rejected — graft
is frequent so invalidating external
refs is wrong tradeoff (different
story from the one-time reshard)
C chain forest with chain_id col: right
answer when graft dominates lifecycle
but premature now
Long-game payoff: mesh-peer-corpus-merge. Two peers diverge over a
partition, each carries packs the other lacks, reconciliation =
exchange + graft what's missing. Makes "mesh of arborists"
coherent rather than "fleet of arborists."
Scaffold gated on (a) #000065 lands+stabilises, (b) a second
corpus exists to graft, (c) at least two peers want to exchange.
No code until then; the design lock is what the ticket buys.
Index bumped Next ID 000066→000067.
Reshard no longer creates a parallel "shards.v2/" directory. Builds
write to "<target_dir>/00X.db.new" alongside originals; when the
executor's validation passes, each .db.new is atomically renamed to
its final 00X.db name via os.replace (POSIX-atomic per file).
For an in-place migration (target_dir == source_dir, which is the
canonical use case), the rename REPLACES the original shard files.
~/.arborist/shards/ never contains a parallel "v2" or "next" or
"backup" directory — it always holds exactly one corpus, just with
the new topology after promote completes.
Validation gate (refuses to promote on mismatch):
* --expected-row-counts <snapshot.json> threads the pre-migration
snapshot's documents/chunks/edges totals through to the executor.
* Tolerance: ±1% to absorb the dupe-collapse from INSERT OR IGNORE
on cross-shard duplicate document_roots (166 dupes measured
pre-migration; <0.005% drift).
* Mismatch → RuntimeError, .db.new files left in place for
inspection, no rename performed.
CLI:
arborist corpus reshard --to 4 \
--source-dir ~/.arborist/shards \
--target-dir ~/.arborist/shards \
--audit-events-ndjson /tmp/audit-events.ndjson \
--expected-row-counts bench/results/pre-migration-snapshot.json
3 new tests:
* test_target_dir_only_has_db_files_after_completion — no .db.new
sidecars survive a successful run
* test_inplace_reshard_overwrites_originals — target_dir ==
source_dir works end-to-end; final files are the new hash-routed
shards
* test_validation_failure_leaves_new_files — bad expected counts
trip the validation guard; .db.new files survive; no .db files
promoted
Stale .db.new files from a prior failed run are unlinked before
opening fresh targets, so a partial-fail-and-retry is idempotent.
WAL/SHM sidecars removed at promote time so the new live .db
produces fresh sidecars on next open.
Build the 'all_at_once' executor and wire it into the CLI as
`arborist corpus reshard`. Plan-only preview against fox's host
confirms the planner picks all_at_once at 95.5 GB free, 64.2 GB
peak draw, 31.3 GB free at peak (well above the 4 GB safety).
Executor (arborist/migrate.py):
* ROUTED_BY_DOCUMENT_ROOT — documents, document_http_meta, chunks,
merkle_nodes, edges (by src_root), derivations (by core_root).
Each row hashed to shard_for_document(root, M) and INSERT'd into
the chosen target.
* CONSOLIDATED_TABLES — snapshots, concept_relations,
concept_token_idf, citation_aliases, term_aliases,
providence_cache, falsifications all land on canonical
target shard 000.
* REBUILT_ON_TARGET — chunks_fts + documents_fts rebuilt from the
materialized data after content moves. chunks.content is
zstd-packed at rest so the rebuilder decompresses via
arborist.compress.unpack_chunk before inserting plaintext into
the FTS5 index.
* Audit chain consolidation (Option A) — all rows from
/tmp/audit-events.ndjson are sorted by (ts, src_shard, src_seq),
re-chained with fresh event_hash values, and INSERT'd into
target shard 000. Bodies preserved unchanged for forensic
fidelity.
* One 'reshard' audit event appended at the tail, carrying the
plan + result as the body. An operator months later can answer
"where did this corpus topology come from" from this one row.
* corpus_shard_count meta stamped on every target shard.
* VACUUM each target at end to reclaim INSERT-pattern fragmentation.
CLI: `arborist corpus reshard --to M --source-dir DIR --target-dir DIR
[--plan-only | --force-strategy X
| --allow-in-place | --dry-run]`
11 integration tests build a tiny 2-shard corpus, run the migration,
verify per-doc routing, chunk-follows-doc, audit chain integrity,
reshard event at tail, corpus_shard_count meta on every target,
chunks_fts searchability, doc count preservation, dry-run no-op,
error handling.
Also: get_meta() now indexes by position so callers without
row_factory=sqlite3.Row don't trip. No behavior change for callers
that DO use Row factory.
Strategy B (per_source_shard) and C (streaming_row) are stubbed in
the planner (peak-draw estimators wired) but execute_plan raises
NotImplementedError for them. Not needed at fox's current disk
(88+ GB free); skipping the build until that's ever the constrained
path.
Disk-aware strategy picker for the corpus reshard. Pure-compute API:
feed it (corpus facts, free bytes) and it returns a HydrationPlan
naming one of three strategies + a peak-draw estimate.
Strategies (preference order):
all_at_once - direct source→target with overlap; fastest;
peak draw = corpus + WAL × M_target + VACUUM
per_source_shard - pack→delete→hydrate per round; safest;
peak draw = pack + WAL + 1 target VACUUM
streaming_row - in-place mutation, --allow-in-place opt-in;
peak draw = WAL only
Safety budget: 4 GB margin above the strategy's peak draw — the
buffer that survives one bad sort + one badly-sized WAL grow.
Production-host preview at 95 GB free / 38 GB corpus:
strategy: all_at_once
peak draw est: 64.2 GB
free at peak: 31.2 GB (well above safety)
rationale: all_at_once: peak draw 64.2 GB + 4.3 GB safety ≤ free 95.4 GB
The plan's full readout (strategy, free bytes, peak draw, rationale)
is JSON-serializable so it goes into the migration audit event as a
single forensic record.
14 tests cover: strategy pick by free-disk level, --force_strategy
override, --allow-in-place gating, skewed shard sizes, json-
serializable audit body, target_M validation.
No execution yet — this is just the planner. Next: the strategy A
executor (direct source→target read+write).
Three pieces, all read-only or additive — no shard mutation, no
schema-version bump:
1. shard_for_document(document_root, M) in arborist/document.py.
Pure function: int(document_root[:8], 16) % M. 22 tests cover
determinism, range-bounds, near-uniform distribution (±5pp at
N=20k), and seven lock-in fixtures so peers will disagree
loudly if anyone changes the formula.
2. corpus_shard_count meta field + get/set helpers in store.py.
Lives in the existing key/value meta table; SCHEMA_VERSION
stays at v9.8.0 (the DDL doesn't change and source_root is
layout-independent, so cache records survive a reshard).
Legacy shards (without the field) return None; reshard tool
populates it on every target shard at migration time.
3. Pre-migration snapshot captured to
bench/results/pre-migration-snapshot.json:
docs 3,468,392 (3,468,226 globally unique)
chunks 6,235,764
edges 90,593,537
audit 3,468,403
This is the reference set post-reshard row counts must match.
4. Audit-event extraction script writes all 3.47M events from
all 4 shards to /tmp/audit-events.ndjson (2.0 GB) for the
Option-A canonical-chain consolidation step. Verifies chain
integrity on extract — all 4 source chains report 0 breaks.
5. Fixed a wrong chunk count in docs/corpus-history.md
(had ~3.54M/shard; actual is ~1.56M/shard) and added the
edge-count column (~22.6M/shard, 90.6M total). 6.24M chunks
total, not 14.12M.
Tests: 29 new pass (22 routing + 7 meta). No existing tests
touched.
The original ingest of Wikipedia 2010 into fox's 4 production shards
was 2026-04-27 21:33-23:13 UTC — ~100 minutes wall, 4-way parallel,
3.47M docs / 14.12M chunks. Captured here because the audit chain is
the only durable record but querying 3.47M rows to recover the
headline number is friction; one line in a doc removes it.
Also informs #000065 reshard planning: ingest rate ceiling on real
XML workload is ~2,350 chunks/sec aggregate (4-way), vs the
~6,400 chunks/sec the M-sweep bench measured on the 2003 cur dump
(which skips XML parsing). The teleport-style reshard should beat
both ceilings because it's just SQLite INSERT throughput, no XML
parse + canonicalize + edge extraction.
New file: docs/corpus-history.md. Indexed in CLAUDE.md docs section.
Append-only convention; future migrations + cold-pack runs add
entries here so the operator log isn't only in the audit chain.
Derivation query (sqlite3 audit_events) embedded in the entry so
future re-derivation is one copy-paste.
Pinned the canonical shard count at M = 4 based on real-Wikipedia
ingest + query benchmark (bench/shard_count_sweep.py). Captured the
"when does SQLite stop being the right substrate" decision tree so
future operators know what bench would justify a fork or replacement.
Bench numbers (Wikipedia 2003 cur dump, 2000 docs, 4 cells of
M ∈ {1, 2, 4, 8}, 50 FTS queries per cell):
M chunks/s q_p50_ms q_p99_ms attach_ms
1 3,648 0.05 0.20 1.61
2 5,649 0.03 0.19 3.24
4 6,405 0.07 0.30 9.00
8 6,959 0.03 0.28 9.65
Key observations:
- M=1→M=2 is the biggest ingest win (+55%). Most gain happens there.
- M=2→M=4 is +13%. M=4→M=8 is only +9% — diminishing returns.
- Real wikitext canonicalization is per-worker Python CPU bound, not
SQLite-writer-lock bound. More shards don't unlock more CPU.
- Query p50/p99 is flat across M within noise (50 queries small).
- ATTACH cost grows linearly: 1.6 / 3.2 / 9.0 / 9.7 ms.
Why M=4 specifically:
- Captures 92% of peak ingest throughput (6,405 / 6,959).
- 6 ATTACH slots free under SQLite's 10 ceiling for aux DBs
(qa.db, snapshots.db, selfmodel-chain.db, crawl_*.db, future
mesh_*.db) — comfortable headroom. M=8 leaves only 2 slots.
- Mobile-tolerable: phone NAND attach is 5-10x slower than NVMe;
M=4 = 45-90 ms cold start (instant), M=8 = 50-100 ms (sluggish
with no headroom).
- Matches fox's current 4-shard layout = cheapest migration.
Decision tree for when SQLite stops being right (full text in
ticket §"When the SQLite-default substrate stops being right"):
A. ATTACH ceiling pressure (auxiliary DBs grow past 5) → bench
forked SQLite with SQLITE_MAX_ATTACHED=125, M ∈ {16, 32, 64};
if attach cost stays linear past M=10, fork viable but pays
permanent "no longer stock sqlite3" tax.
B. Ingest hits >10k chunks/s sustained ceiling → first tune
page_size / WAL checkpoint / mmap_size / synchronous. If
tuning gets 2-5x, stay on SQLite. If still ceiling-limited,
candidates: DuckDB (columnar, MVCC, FTS), libmdbx (B+tree no
FTS; we'd build it). In-house DB rejected without specific
failure of those.
C. Federation needs multi-writer-same-shard → SQLite writer-lock
serializes peers, becomes federation bottleneck. First try
leader-election (single-writer-per-shard with WAL replication
to followers). If true multi-writer required, SQLite is wrong;
candidates: FoundationDB, CRDT-on-KV-store. DuckDB does NOT
solve this — its MVCC is single-process.
Honest verdict: for current arborist workload (single-writer-per-
shard, read-mostly federation), stock python3 sqlite3 is the right
substrate. None of A/B/C are close to firing. The bench discipline
exists to know what to measure when something changes.
bench/results/shard-count-sweep-2026-05-26T16-20-48Z.csv (synthetic
baseline) + 2026-05-26T16-31-34Z.csv (real Wikipedia) committed as
the load-bearing measurement for the M=4 choice.
fox caught the imprecision: "the shards are teleported if we fix this
because we know the shard a doc will end up in."
Re-ingest = re-read source files, re-canonicalize, re-chunk, re-hash.
Hours-to-days from Wikipedia dumps.
What's actually needed = content-addressed rebalance. Every row in
the corpus is already addressed by content (document_root, leaf_hash,
merkle_nodes.hash, audit_event_hash). Migration reads existing rows,
computes new_shard_idx via the routing function, INSERTs into the new
M shards. No source-file work, no canonicalization, no LLM.
Updated the ticket's "Migration story" section with the per-table
teleport recipe:
documents: move by document_root → new shard
chunks (+content): follow doc to its new shard, re-assign chunk_id
chunks_fts: rebuild per new shard from chunks.content
merkle_nodes: move with document_root
edges: move by src_root
concept_relations, move by content-derived parent
derivations,
providence_cache,
citation_aliases,
term_aliases:
audit_events: ALL → shard 000 (canonical), re-numbered + re-hashed
snapshots, mesh_*: canonical shard 000
Audit chain handling: chose Option A (canonical shard 000) over
Option B (split-by-subject_root). Preserves global event ordering,
re-hashing happens once at migration time. The alternative loses
cross-shard event ordering for falsification + replay reasoning.
Estimated wall: ~20-40 min I/O-bound on current corpus
(35 GB across 4 shards → ~35 GB across 8 shards, moves once).
Hard discipline added: move into ~/.arborist/shards.v2/, keep
originals as rollback, atomic dir-swap when verified. Mistake in
routing function = corrupted federation, recoverable only by going
back to the originals.
Surfaced while sizing #000061's federation story. Today shard count
conflates two roles:
- producer ingest parallelism (wants = vCPU count)
- consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10)
On default Python sqlite3, the ATTACH ceiling is 10 and can't be
raised without a custom sqlite3 build (which violates CLAUDE.md's
"python3 + venv + sqlite3 is enough" property). Producer with 16
shards → consumer fails to attach the 11th, federation silently
breaks.
Design: introduce M = canonical shard count (corpus-wide constant,
default 8) decoupled from N = ingest workers. Document → shard
assignment becomes content-deterministic:
shard_idx = int(document_root[:8], 16) % M
Same input → same output across every peer. Today's "spray by ingest
order" is non-deterministic across peers — two peers re-ingesting
the same corpus put the same document_root in different shards. That
weakens federation more than it should.
Migration hard-constraint (fox: "this implies we will need to
reprocess all our data into shards"): re-ingest required. Current
layout is sprayed by ingest order; post-ticket is sprayed by content
hash. Two layouts are incompatible by construction. Captured in
ticket §Migration as the load-bearing operational note.
Phases laid out (0-4: design lock → read path → ingest path →
pack-restore → corpus migration tool). Open audit-chain re-numbering
question (per-shard event_hash chains break when rows rebalance
across shards). Kept as one ticket — do-not-proliferate.
Scaffold only. No code yet — design lock first.
Next ID bumped to 000066.
The 3.15 GB-per-shard index I added in bc7efe4 bloated every live
shard 33% to optimize a transient dump-time operation. Wrong trade —
the shards exist for arborist's query/retrieval/falsification path,
not for cold-pack's convenience. fox: "wtf you added an index to
fucking make the shard bigger?"
Refactor: instead of building a permanent on-disk index to avoid the
sort, configure the dump connection to spill the sort to a temp file:
conn.execute("PRAGMA temp_store = FILE")
conn.execute("PRAGMA cache_size = -10000") # 10 MB
SQLite's sort still happens, but it spills to /tmp instead of growing
the process heap. cache_size caps the page cache the sort works
inside. No permanent disk cost; transient temp disk only during the
dump phase.
DROP INDEX idx_edges_dst_uri_type_anchor + VACUUM ran across the 4
live shards before this commit (out-of-band ops):
000.db: 12.48 → 8.75 GB (~3 GB reclaimed)
001.db: 12.43 → 8.71 GB
002.db: 12.53 → 8.78 GB
003.db: 12.51 → 8.76 GB
total: ~12.6 GB reclaimed across the corpus
For a freshly-restored shard from a metadata pack, the index never
gets created (it was only in _dump_edges_fan_in, not in SCHEMA_SQL),
so consumers get the lean version from day one.
33/33 cold + evict tests pass.
The license_class gate landed in 576cb0e but only push_pack() the
function takes allow_license_class — the CLI couldn't pass it through.
Adds --allow-license-class {public_redistributable,unknown,private}
to `arborist cold pack`. Default stays at public_redistributable;
operators override only after confirming bucket ACL + source license
permit redistribution.
33/33 tests pass.
Dav1d's reviews of #000061 (Response A + Response B/FINAL in
~/Downloads, 2026-05-26) flagged a long list of items — most already
shipped in the SPV-split work. Three were genuine gaps worth folding
into #000061 before close:
Gap 1: manifest/latest pointer for new-peer discovery.
A fresh peer doing `cold list` got a list of metadata-pack hashes
but no obvious "which one is current for shard X." Added
get_latest_pointer + update_latest_pointer to the backend ABC.
push_pack writes manifest/latest.json on every successful metadata
pack push (read-modify-write keyed by snapshot_root). Mutable
pointer; content addressing of the packs themselves preserves the
trust root. Last-writer-wins on contention.
Gap 2: license_class field + producer-side refuse for public buckets.
Maps documents.source_type to a license bucket (wikipedia_cur /
textbook_tex → public_redistributable; html / grok / vcs → unknown;
anything else → unknown). Strictness order: public < unknown <
private. compute_shard_license_class() walks DISTINCT source_type
in documents. push_pack now refuses to upload if the shard's
strictest license is more restrictive than the operator's
allow_license_class (default: public_redistributable). The
metadata pack's manifest carries _license_class so consumers /
auditors can see the producer's classification without inspecting
source documents. ValueError on refusal — the bucket ACL is the
operator's call, but arborist refuses to participate in a
licensing/membership leak unless explicitly opted in.
Gap 3: cold_pending table for resumable uploads.
Killed mid-upload, push_pack left orphan multi-GB tempfiles in
/tmp with no DB trace. Added schema:
CREATE TABLE cold_pending (
tempfile_path TEXT PRIMARY KEY,
pack_hash TEXT NOT NULL,
kind TEXT NOT NULL,
backend_endpoint TEXT NOT NULL,
backend_bucket TEXT NOT NULL,
object_key TEXT NOT NULL,
started_at INTEGER NOT NULL,
state TEXT NOT NULL DEFAULT 'pending'
);
push_pack INSERTs a row before each upload + DELETEs on success.
A killed process leaves the row pointing at the orphan tempfile;
a recovery script (future) reads cold_pending, checks bucket for
the object, either deletes the row + tempfile (success was just
unreported) or re-uploads from the tempfile if it still exists.
Matches the same pattern as the audit chain — explicit state
rows beat inferring from chunks.content IS NULL.
Sibling tickets opened for the larger items the reviews flagged
(scaffold-only, no code; opening them captures the design in the
log without proliferating, per CLAUDE.md):
- #000063 Cold-object private-ciphertext mode (mesh-keyed object
keys for non-public corpora on public-read buckets). Needs mesh
group-key ABI + real non-public corpus before code.
- #000064 Cold-object operations toolkit (verify / diff / doctor /
repair-fts / gc-plan CLI + expanded audit-event taxonomy).
Bundled so the audit-event vocabulary gets one design pass.
5 new tests:
test_gap2_license_gate_refuses_unknown_class_to_public_bucket
test_gap2_license_class_in_metadata_manifest
test_gap1_latest_pointer_resolves_metadata_pack_per_snapshot
test_gap3_cold_pending_clears_on_successful_upload
test_gap3_cold_pending_records_inflight_upload
26 cold-object + 7 evict tests pass (33/33 green incl. boto3 wire).
Next ID bumped to 000065.
Live v3 SPV corpus run (bmq47x6t3) completed cleanly during this work.
Will report sizing + memory profile in the next message.
Three doc-only housekeeping items from Dav1d's de-novo reconciliation
of the architecture stack (no code changes; the active build is
#000061 cold-pack work, which is unaffected by this review):
1. Accepted-error formula. Dav1d corrected p_raw × (1-d)(1-r) to
p_raw × (1 - dr) where r is conditional on detection. Searched
docs/ and arborist/ — the wrong form does NOT appear in this
tree (it lives in the external recursive-truth-maintenance / RCO
manuscripts Dav1d cited). Nothing to fix here; recorded for the
manuscript authors.
2. #000060 H-ABCDEFG split. Folded the M/C/X axis split into the
harness scope: M (mechanism — does the substrate's plumbing work),
C (capability — does it improve task performance), X (external
adversarial — does it generalize outside author-designed fixtures).
ACCEPT requires clearing all three. Without X, the harness risks
self-validating benchmark theology — passing tests its own designers
picked. Tagged at fixture-definition time; aggregator emits per-axis
pass rates + combined ACCEPT verdict. Doesn't change the existing
BatteryResult row schema.
3. #000062 Mechanistic Witness — new scaffold-only ticket. Specifies
a content-addressed MechanisticWitnessRoot over (model_config,
capture_policy, contrastive_prompts, features/neurons, intervention
result, behavioral delta, safety policy) as a DIAGNOSTIC sidecar
feeding SelfModel + benchmark fixtures. Four hard guardrails:
diagnostic-only by default; sandbox intervention only; no production
steering without governance ACCEPT via #000060 M+C+X; feature labels
never become semantic proof. No code until a real falsifier use case
exists + guardrails are CLAUDE.md rules + #000060 harness gates
promotion. The dual-use risk (Pan et al. 2025: 0.1% MLP ablation
breaks refusal in 72B models) makes governance-first framing
load-bearing.
Next ID bumped to 000063.
No code change to arborist/. The in-flight v3 SPV corpus pack
(bmq47x6t3) continues unaffected.
Previously: COLD_PACK_JOBS hardcoded to 4. Caused the v3 corpus run to
fail at fork time on a memory-pressured box (4 GB available, each worker
needed ~5 GB peak → kernel swapped, SQLite executescript exceeded
busy_timeout, errored as "database is locked"). Killing the run + waiting
for memory to free → 2-way run succeeded.
Now: COLD_PACK_JOBS is auto-computed from `free -m` at run time. Reads
available RAM, subtracts headroom for OS + buff/cache, divides by the
observed per-worker peak, caps at COLD_PACK_JOBS_MAX. An explicit
override (COLD_PACK_JOBS=N on the make invocation) still wins for
operators who know better.
Tunables:
COLD_PACK_PER_WORKER_MB 5500 observed peak from real corpus run
COLD_PACK_HEADROOM_MB 2000 keep this RAM for OS / other procs
COLD_PACK_JOBS_MAX 8 ceiling regardless of RAM headroom
Same logic applied to cold-pack-all-dvd (local-dir output for burning).
Both targets print the chosen concurrency + the math behind it before
launching workers. If the auto-computed value is 1, that's a signal to
wait for RAM to free before trying parallelism.
Doesn't fix the underlying fragility of using fork-time RAM availability
to predict peak demand — workers grow over their lifetime as metadata
dumps materialize — but it's a real improvement over a static cap that
ignored system state entirely.
Bidirectional sync: producer always emits both kinds; consumer chooses
how much to pull.
packs/<hash>.metadata.tar.zst — one per shard (~0.6 GB compressed)
packs/<hash>.metadata.manifest.ndjson
packs/<hash>.chunks.tar.zst — N per shard (each ≤ 4.4 GB cap)
packs/<hash>.chunks.manifest.ndjson
Each artifact is independently content-addressed by its own manifest
hash. The metadata pack's manifest carries _chunk_pack_hashes — every
chunk pack covering this shard's content — so a "full" consumer can
iterate them. Chunks packs are anonymous from the consumer side
(reachable only via the metadata pack's reference list).
Consumer sync modes:
arborist cold unpack <metadata_hash> # default: just-enough
arborist cold unpack <metadata_hash> --full # also pulls chunks
just-enough: pull only the metadata pack. Schema fully restored; every
chunks row has content=NULL. Node is immediately queryable for
metadata operations (documents, edges, audit chain, Merkle); chunk-body
queries return null until a future JIT-fetch path fills them on cache
miss.
full: after the metadata pack lands, iterate _chunk_pack_hashes and pull
every chunk pack. Final state: full corpus offline-queryable.
This is fox's original SPV-wallet framing — was right from day one.
Key API changes:
pack_key(hash, *, kind="chunks", manifest=False) — kind in bucket path
stream_packs(chunks, *, max_compressed_bytes, ...) -> Iterator[FilePack]
now emits chunk-only packs (no extra_members)
build_metadata_pack(table_files, *, snapshot_root, chunk_pack_hashes, ...)
single FilePack with v3 metadata manifest
pull_metadata_pack(conn, backend, hash) -> dict
pull_chunk_pack(conn, backend, hash) -> dict
hydrate_from_metadata_pack(conn, backend, metadata_hash, *, mode) -> dict
push_pack orchestrates: dump tables → stream chunk packs (collect hashes)
→ build metadata pack with chunk_pack_hashes → upload all. Returns
{metadata_pack_hash, chunk_pack_hashes, packs: [...]}.
Manifest format v3:
metadata pack:
{"_format_version": 3}
{"_kind": "metadata"}
{"_snapshot_root": "..."}
{"_chunk_pack_hashes": [...]}
{"table_file": "tables/X.jsonl", "hash": "...", "size": N} per table
chunks pack:
{"_format_version": 3}
{"_kind": "chunks"}
{"leaf_hash": "...", "size": N} per chunk
pack_hash for each = hash_leaf(manifest_bytes); content-addressed at
both layers. Two writers with the same shard state produce identical
metadata_pack_hash AND identical chunk_pack_hashes.
Cap-and-split applies only to chunk packs (chunks are bounded N).
Metadata pack is one file per shard; if a single table file is larger
than the cap, that's noted as future row-level split work.
cold list now surfaces kind per artifact (metadata vs chunks), plus
table_count + chunk_pack_hashes (for metadata packs) or chunk_count
(for chunks packs). Operators can quickly find the metadata pack hash
to feed `cold unpack`.
28 cold-object + evict + boto3 tests pass (3 new SPV-shape tests + 1
v3 manifest-shape test + tampered-metadata-pack test).
Caught a real defect on the live v2 corpus run: pack_hash was computed
from the chunk-only manifest, so two packs with the same chunk set but
different table content collided on pack_hash. Observed live: v1 packs
and v2 packs for the same shard produced the SAME pack_hash:
v1 (chunks-only) b3c427baaf9b40e3c33ace8438b4254e3abdc7b38bdcb7b1a8012588cb32848a
v2 (tables + chunks) b3c427baaf9b40e3c33ace8438b4254e3abdc7b38bdcb7b1a8012588cb32848a
↑ identical, different bytes
Consequences if shipped: bucket overwrites swap whose tables you get,
mesh peers A and B disagree on what "pack X" is while both claim to
have it, content-addressing story is broken for the metadata half.
Fix: manifest format v2 prepends a `{"_format_version": 2}` header,
then one row per shipped table:
{"table_file": "tables/audit_events.jsonl", "hash": "...", "size": N}
{"table_file": "tables/chunks.jsonl", "hash": "...", "size": N}
...
{"leaf_hash": "...", "size": N} # chunks come after
{"leaf_hash": "...", "size": N}
pack_hash = hash_leaf(manifest_bytes) now binds the full pack content.
Two writers with the same shard state produce the same pack_hash; same
chunks + different tables produce different pack_hashes.
Implementation:
- hash_file_leaf(path) — streaming sha256 with leaf 0x00 prefix, 1 MB
reads, used to fingerprint multi-GB tables/<name>.jsonl files without
loading them into RAM.
- stream_packs' extra_members now takes (member_name, source_path,
content_hash) tuples. push_pack computes the hash with hash_file_leaf
after the dump finishes.
- _finalize_pack writes the format header + table refs (sorted by name
for determinism) before chunk entries.
- parse_manifest returns a ParsedManifest (format_version, tables[],
chunks[]) instead of just list[PackEntry]. Backward-compatible with
v1 manifests (no header → format_version=1, tables=()).
- pull_pack pulls the manifest aside during tar walk, then after
extraction verifies each tables/<name>.jsonl file via hash_file_leaf
against the manifest's content_hash. Mismatch → ValueError, no rows
reach the live schema.
New tests:
- test_v2_pack_hash_binds_table_contents — asserts manifest format +
table refs + per-table hash & size.
- test_pull_pack_rejects_tampered_table — corrupts one table's bytes
in a real pack, confirms pull_pack raises on hash mismatch.
27 cold-object + evict tests pass.
Live v2 corpus run showed ~5 GB RSS per worker — RssAnon dominant, so
process heap, not mmap. Traced to two unfixed memory pits in the edges
fan-in dump path:
1. SQLite ORDER BY on edges (22M rows, no covering index for the v2
sort order dst_uri+edge_type+anchor) allocates a multi-GB in-memory
sort area before spilling. Adding:
CREATE INDEX IF NOT EXISTS idx_edges_dst_uri_type_anchor
ON edges(dst_uri, edge_type, anchor, dst_root, src_root)
means the ORDER BY walks the index in order — no in-memory sort.
First create takes ~30-60 s on a 22M-row shard; idempotent on
subsequent dumps. Disk cost ~1 GB per shard (4 shards × 1 GB ≈
2-3 % corpus footprint increase). Worth it.
2. Python groupby accumulator: src_roots = [row[4] for row in group]
materializes the entire src_root list per destination. For
en.wikipedia.org/wiki/* destinations with millions of inbound links,
this list is itself ~GB-sized. Switch to bounded batches:
_FAN_IN_BATCH = 10_000 # max src_roots per fan-in JSON row
A destination with N inbound links splits into ceil(N / batch) rows.
Restore path (INSERT OR IGNORE) handles multi-row destinations
correctly because PK includes src_root — accidental duplicates
collapse cleanly.
New regression test test_edges_fan_in_batches_huge_destinations builds
an edges table with FAN_IN_BATCH+137 rows pointing at one dst_uri,
verifies the dump produces the expected number of split rows and the
restore reconstructs all N edges with no loss or duplication.
25 cold-object + evict tests pass.
The extra_members handling in stream_packs did:
with open(source_path, 'rb') as f:
source_bytes = f.read() # loads entire file into RAM
info = tarfile.TarInfo(...)
info.size = len(source_bytes)
state['tar'].addfile(info, io.BytesIO(source_bytes))
For a 4.3 GB edges.jsonl this is 4.3 GB of RAM per worker — exactly the
failure mode the prior streaming-to-disk refactor was supposed to
eliminate (57f8989). 4-way parallel would hit ~17 GB peak RAM, the same
OOM-risk number we saw on the v1 corpus run.
Fix: build the TarInfo by hand (size from .stat(), other fields zeroed),
then pass the open file handle directly to tar.addfile. tarfile reads
the body in chunks (~16 KB at a time) and feeds them into the zstd
stream_writer. Peak RAM per worker stays at ~tens of KB regardless of
metadata file size.
Determinism preserved: mtime/uid/gid default to 0 (NOT taken from the
filesystem via gettarinfo), so two writers' packs converge byte-for-byte
regardless of when they ran or who owns the temp file.
24 cold-object + evict tests pass.
v1 packs (chunks-only) were under-engineered: a new peer landing on
v1 packs would have chunk bodies indexed by leaf_hash but no documents
table, no audit chain, no merkle interior, no edges — couldn't actually
hydrate. fox: "isn't what I wanted you under engineered..."
v2 packs ship every load-bearing shard table alongside chunk bodies in
the same tar.zst:
manifest.jsonl # chunk catalog (unchanged)
tables/documents.jsonl # array-per-line columnar JSONL
tables/chunks.jsonl # without content column
tables/merkle_nodes.jsonl
tables/edges.jsonl # FAN-IN restructured
tables/audit_events.jsonl
tables/derivations.jsonl
tables/concept_relations.jsonl
tables/concept_token_idf.jsonl
tables/providence_cache.jsonl
tables/citation_aliases.jsonl
tables/term_aliases.jsonl
tables/snapshots.jsonl
tables/document_http_meta.jsonl
blobs/<hash[:2]>/<hash[2:]> # raw UTF-8 chunk bodies
Two compression strategies inside the pack:
1. Array-per-line JSONL ({"_columns": [...]} header line + ["v1","v2",...]
data lines) drops ~30% of uncompressed bytes vs object-per-row JSONL.
zstd recovers most of that on its own, but smaller uncompressed
footprint also speeds up stream-restore.
2. Edges fan-in restructure at pack-build time: 22M rows of
(src_root, edge_type, dst_root, dst_uri, anchor) → ~500k unique
(dst_uri, edge_type, anchor, dst_root) groups with src_roots as an
array. ~5-10x compressed savings on the dominant table. Reverses on
unpack into the per-edge live schema. Live queries unchanged.
NOT shipped (per-peer state): mesh_*, selfmodel_*, capital_ledger,
memory_*, controller_events, fork_score_branches, adapter_loss_reports,
falsifications, schema_meta, meta. NOT shipped (rebuildable): chunks_fts*,
documents_fts* — restored from chunks.content + documents.title on
unpack.
push_pack no longer appends `cold_pack_pushed` to the audit chain.
That event leaked into the next push's audit_events.jsonl dump and
broke the "two writers at the same corpus state produce identical
pack_hash" determinism property. The bucket/disc file IS the receipt;
the snapshot_root pinned inside the pack metadata binds it to a corpus
state. No load-bearing consumer of the audit row.
pull_pack restored to handle both v1 (chunks-only) and v2 (tables +
chunks) packs. For v2 it extracts tables/*.jsonl to a temp dir,
calls restore_shard_metadata (which INSERT OR IGNOREs into the live
schema and expands edges back to per-edge rows), then fills chunk
content for every leaf_hash in blobs/. Idempotent against populated
DBs (INSERT OR IGNORE all the way down). Self-cleaning temp dir.
Sizing measured 2026-05-26: ~2.1 GB per shard pack compressed (chunk
content 1.78 GB + metadata ~0.3 GB), ~8.5 GB total across 4 shards.
~20% more than v1 chunks-only for self-sufficient hydration.
24 cold-object + evict tests pass (+1 new test_push_pack_v2_hydrates_fresh_empty_db
that builds a pack from a populated DB and unpacks into a completely
empty DB to verify all tables restored). Full suite: 2558 passed,
28 skipped, 1 xfailed.
The in-memory BytesIO design held a whole 4.4 GB pack in RAM per worker
until finalization. 4-way parallel hit ~17 GB peak RAM — observed live
on the full-corpus run (21 GB used, 0 free, 3 GB swap in use, headed
toward OOM territory). Architectural defect, not just an inconvenience:
parallelism is capped by RAM rather than CPU/network.
stream_packs now:
- Opens a NamedTemporaryFile in work_dir (default tempfile.gettempdir()).
- Wraps it with ZstdCompressor.stream_writer + tarfile mode "w|".
- After each chunk: writer.flush(FLUSH_BLOCK) + file.flush(), check
file.tell() against the cap.
- On finalize: write the manifest tar member, close everything, yield
a FilePack carrying the path (not the bytes).
- On exception: close + unlink the in-flight temp file (no /tmp litter).
ObjectStoreBackend gains put_file(key, path). S3 impl uses boto3
upload_file which streams parts from disk during multipart upload —
memory bounded to max_concurrency × part_size = 10 × 8 MB = 80 MB
regardless of pack size. MemoryBackend reads the file into its dict
(tests only).
push_pack consumes FilePack:
- --push: backend.put_pack_file(path), then unlink the temp
- --local-dir: shutil.move from temp to local_dir/arborist-pack-<hash>
(cheap if same filesystem; one-pass read+write if not)
- both: shutil.move first, copy to bucket via put_pack_file from final
local path? — actually current code moves to local_dir
AFTER the bucket put; bucket put still works from the
temp path before move. Audit row carries pack.body_size
from FilePack (no len(bytes) on a 4 GB body).
Worker memory bound goes from O(pack_size) to O(zstd_buffer + tar_header)
≈ tens of MB. 4-way parallel now uses ~tens of MB total instead of
~17 GB. Disk requirement: work_dir needs (parallelism × cap) of free
space — 4 × 4.4 GB = 18 GB for a default full-cap parallel run.
23 passed in tests/test_cold_object.py + tests/test_evict.py.
Shards are independent SQLite files; the ORDER BY landed in 6f0ceab
makes each shard's chunk-to-pack assignment deterministic. Two new
Makefile targets fan out cold pack across every numbered shard via
xargs -P:
make cold-pack-all # parallel push to S3
make cold-pack-all-dvd LOCAL_DIR= # parallel write to disk (no S3)
Tunables:
SHARDS_DIR defaults to ~/.arborist/shards
SHARDS_PATTERN defaults to [0-9][0-9][0-9].db (numbered shards only,
skipping qa.db / snapshots.db / crawl_*)
COLD_PACK_JOBS defaults to 4
Expected wall-clock improvement: ~4x on full-corpus runs (4 shards
running in parallel saturate either CPU or network before serializing).
Also verified end-to-end pack → list → unpack round trip against the
live DO Spaces NYC3 endpoint (small pack, 20 chunks): pushed, listed,
unpacked into a fresh receiving DB, content restored byte-identical,
hashes matched.
New peers landing on a public bucket need to enumerate pack_hashes
before they can `arborist cold unpack <hash>`. `cold stats` only gives
a count; `cold list` returns the per-pack details:
- pack_hash (taken from the key)
- compressed_bytes (one HEAD round-trip via new object_size method)
- chunk_count (parsed from the small manifest sidecar; skippable
via --no-manifest for huge-bucket fast listing)
`cold stats` also now reports total bucket footprint (sums HEAD sizes).
Backend ABC gains `object_size(key) -> int | None` so both subcommands
get sizes without paying egress for the body. S3 impl uses HEAD;
MemoryBackend reads from the dict.
Verified live against DO Spaces NYC3: list-empty → push tiny pack →
list-with-manifest (pack_hash, size=5311 B, chunk_count=5) → list
--no-manifest → cold stats → cleanup.
23 passed in tests/test_cold_object.py + tests/test_evict.py.
Three improvements after the first DO Spaces smoke + bench:
1. ORDER BY c.leaf_hash on the chunk-selection SQL. Two writers running
cold pack against the same DB at the same snapshot now produce the
same pack_hashes — chunk-to-pack assignment is a function of (chunk
set, cap) and nothing else. Prerequisite for parallel per-shard pack
workers and for two replicas to converge on byte-identical bucket
state. Costs ~25% on build wall (real-bench 31s → 40s on 100k chunks)
due to sort over the leaf_hash index + documents JOIN; worth it.
New test pins the determinism property.
2. boto3 multipart upload via TransferConfig (8 MB threshold + 8 MB
parts + 10-way concurrency) on every put. Required anyway for packs
> 5 GB (DO Spaces single-PUT limit). Measured 5.5 MB/s → 9.0 MB/s
on 121 MB pack to DO Spaces NYC3 (1.6x; ceiling is closer to network
than to boto3 serialization).
3. Stream the SQL cursor in push_pack instead of fetchall(). At 14M
chunks × ~700 bytes/row the prior fetchall materialized ~10 GB of
Python heap before stream_packs ever ran. Cursor iteration bounds
memory by the in-progress pack (~few hundred MB at the 4.4 GB cap).
Full corpus extrapolation revises ~125 min (single-PUT) → ~104 min
(multipart, sequential per-shard). Real wins live in parallel per-shard
pack workers — deferred; the determinism work landed here is the
prerequisite.
22 passed in tests/test_cold_object.py + tests/test_evict.py.
Ship arborist corpus state to new peers and DVD-R archival via
point-in-time tar.zst packs. One artifact serves both channels —
bucket+CDN delivery and physical-media archival.
Bucket holds packs only. Pack key = hash_leaf(manifest_bytes), so same
chunk set on two writers produces the same pack_hash and upload is
idempotent. Each pack pins the corpus snapshot_root it covers in audit
+ result body — packs are delayed snapshots, not live mirrors;
falsifications between repacks produce new pack_hashes.
stream_packs runs streaming zstd over tarfile, peeking compressed-buffer
size after each chunk via FLUSH_BLOCK (preserves dictionary). Default
cap 4_400_000_000 — 4.4 GB DVD-R safe-fit, ~6.5% buffer below the
4.7 GB marketing capacity to absorb ISO9660 overhead, growisofs
lead-in/lead-out, media variance, and drive-edge refusal. Each disc
fills to ~4.4 GB recorded data, not the ~1.5 GB an uncompressed cap
produced.
One backend class (S3CompatibleBackend via boto3 + endpoint_url) covers
AWS S3, DO Spaces, R2, B2, GCS S3-interop, MinIO. Optional dep
[object-store] = boto3>=1.34; dev extras pull moto for the wire test.
Voyeur: credentials via AWS_ACCESS_KEY_ID/_SECRET_ACCESS_KEY env or
~/.aws/credentials, never printed; only endpoint URL + bucket name
surface in logs.
CLI: arborist cold {pack,unpack,stats}. Makefile: cold-pack,
cold-pack-dvd (local-dir output for growisofs), cold-unpack, cold-stats.
Sizing for current shards (14.1M chunks, ~17 GB compressed): ~4 packs
at the default cap, ~\$0.34/mo DO Spaces storage, ~\$0.0001/fresh-peer
hydrate.
Always-on raw-UTF-8 leaf store (per ticket "Hard invariants") deferred
— packs-only for now, backfill later.
2557 passed, 28 skipped, 1 xfailed.
Address Grok's two minor-improvement flags on the docs.
New docs/diagrams/{three-layer-stack,cache-key-8dim,falsification-states}.{dot,svg,png}
embedded into docs/_source/concepts.rst — visual scaffolding for the
3-layer stack, 8-dim cache_key composition, and falsification state
machine (previously prose+tables only).
docs/_source/cookbook.rst gains a "Use arborist as a Python library"
section: open_store + ingest_documents, custom Source subclass,
audit-chain walk + verify, Merkle proof round-trip, programmatic
arborist.qa.query() with OpenAICompatibleClient + StubClient swap.
Every Python recipe smoke-tested against a scratch DB before publish.
make docs-api: 0 new warnings. make test: 2557 passed.
The §12.1 pipeline I added was second-hand from benchmark-matrix.md
and got several things wrong against the code:
- listed 4 verdicts; actual is 5 (missing FABRICATED — the
fabrication-vs-WRONG split that energy-cogs §5.5 leans on for the
qwen-fabricates / hermes-abstains finding)
- "θ=0.85" was right by accident — but it's the code-judge-pinned
_CODE_JUDGE_THETA_CONTRA constant, raised from the manifest 0.5
default after measuring 114 FPs in the 0.5-0.75 band
- omitted the short-answer entity-grounding fast path (which runs
BEFORE NLI per the 2026-05-19 Poland-Tusk smoke)
- omitted the HYBRID rescue ladder (NLI entail / entity rescue /
2026-05-21 verbatim-quote-on-topic rescue)
- conflated WRONG and FABRICATED (the subject-in-gold split is what
distinguishes "source has the topic but a different value" from
"source silent on the topic")
Rewrote §12.1 grounded in `bench/judge_code.py:judge()` (its own
docstring at line 501-528 is the truth on rule order), with file:line
citations and the verdict-mapping in full.
Also fixed a real artifact-vs-doc drift INSIDE the judge: the
module-top docstring still claimed θ_contra default 0.5 and omitted
the short-path and the WRONG/FABRICATED split. Updated to match the
authoritative judge() docstring + current code.
No behavior change — docstring + benchmarks doc only.
A stable façade so another Python app can use arborist as a
content-addressed / Merkle / audit-chained store without the CLI or a
wire protocol. Import from arborist.embed, not internal modules, so
refactors don't break embedders.
Surface: open_store(path), ingest_documents(conn, docs), search(conn, q),
plus re-exported Document/Edge/Source/Hit/IngestStats. Core only
(python+sqlite3) — no extras. _IterableSource adapts a plain doc iterable
into the Source contract.
This is the seam for using arborist as neopig's optional provenance
backend: neopig produces Documents from crawled pages, arborist gives
content-dedup (document_root) + FTS5 + an append-only audit chain
alongside neopig's existing md5/FileVault storage. Docs in
docs/embedding.md. 6 tests pin open/ingest/dedup/idempotence/edges/search.
New docs/crawler.md covering the crawl path we actually run: BFS
same-host discovery, robots/feed/sitemap handling, polite vs --fast,
the shared-session + no-HEAD + crawl-delay-fix speedups, and the
content-addressed payoff. Leads on the two store-derived diagnostics:
duplicate detection (group by document_root — body, not URI) and
partial-overlap (shared chunk leaves), plus orphan finding
(sitemap − BFS-reached) and the planned crawl-report webmaster tools.
Honest pros/cons: orphans invisible to crawl by design, single-host,
no JS execution, --fast is anti-social off your own turf.
Full --fast crawl of russell.ballestrini.net (242 URIs): 26s -> ~5s.
Three changes, biggest first:
1. fast_mode now actually ignores crawl-delay (the ~5x). The delay was
only zeroed on the robots-200 path; a site with no robots.txt (404)
or a robots fetch error fell back to default_crawl_delay (2s). Under
--fast that made every concurrent fetch wave sleep ~2s — ~10 waves
x 2s dominated the wall time. _enforce_crawl_delay now short-circuits
when fast_mode, matching the documented "ignore crawl-delay"
contract regardless of robots status. Disallow is still honored
(separate path).
2. One shared ClientSession for the fetcher's lifetime (keepalive TCP
connector sized to page-worker width) instead of a fresh session per
fetch — ~3x on a 24-page wave. Lazily built in-loop via _get_session;
the bridge closes it in a finally (guarded on owning the fetcher).
3. Drop the per-page preflight HEAD. aiohttp exposes response headers
before the body is read, so the existing content-type binary guard
skips images/video/audio without downloading them — the HEAD was a
redundant round trip that doubled per-page latency.
Diverges arborist's AsyncWebFetcher from the agents.ai.unturf.com/core
verbatim lift (fox-approved); candidate to upstream. Regression tests
pin fast=no-delay / polite=delay, shared-session lifecycle, and bridge
session teardown (owned vs injected).
Two crawler-discovery changes surfaced while chasing fast-crawl wall
time on russell.ballestrini.net:
1. Feed-skip in BFS discovery: the bridge fetched feed/sitemap URLs
(a multi-MB atom.xml among them) only for ingest_crawled to discard
them. Gate enqueue on the existing _looks_like_feed_url so we never
fetch crawl-infrastructure URLs — less wasted work and one fewer
slow wave straggler.
2. lxml link extraction, DRY'd: the three duplicated BeautifulSoup
html.parser closures (fresh fetch + 2 cache paths) collapse into one
module-level extract_page_links() backed by lxml.html (C parser,
releases the GIL so to_thread actually parallelises) with a BS4
fallback for markup lxml rejects. Parse on a 24-page wave 3.5s->2.5s.
Honest scope: neither moves full-crawl wall time much — measurement
showed the dominant cost is the per-page HEAD+GET double round-trip on
a per-call ClientSession, not parsing. These are correct-and-cleaner
on their own; the wall-time lever (shared session + drop redundant
HEAD) is a separate change. lxml extraction is regression-pinned
against the BS4 fallback for parity.
The bridge BFS fetched pages one-at-a-time, so --fast only dropped the
crawl-delay (sequential, zero-wait). Fast_mode's CPU*3 page-worker
budget never reached the path operators actually run.
Replace the popleft loop with a wave loop: each iteration pulls up to
`fetcher.max_page_workers` URLs off the queue front and fetches them
with asyncio.gather. Width is CPU*3 under fast_mode, 1 otherwise, so
the polite path stays byte-for-byte sequential and the per-page
crawl-delay still serialises same-domain fetches. Wave size is capped
to the remaining max_pages budget; dedup moves from pop-time to
enqueue-time so a URL linked from two parents in one wave is fetched
exactly once.
Measured on russell.ballestrini.net (own host, robots 404): same
12-page work 23.1s polite -> 4.0s fast (5.7x); full 243-page crawl
~25s vs the ~486s polite floor (19x). Disallow still honored; only
the rate limit is lifted.
Tests: peak-in-flight pins (>1 fast, ==1 polite) plus all existing
BFS bound / dedup / depth / max-pages cases on the width=1 path.
When a re-crawl detects a real content delta (a just-ingested root that
supersedes a prior version — content hash changed, not redeploy/ETag
noise the idempotent ingest already no-op'd), the pipeline now surfaces
the page's document chain over time instead of just 'something changed'.
bridge.py: version_chain(conn, uri) walks a URI's documents by ingest_ts
(each content change = new content-addressed doc + supersedes edge);
delta_report() adds the word-level similarity of the latest change;
render_delta_report() prints it. ingest_crawled() detects superseding
roots, emits the lineage report to stderr per changed page, and returns
'deltas' in its summary. Validated on the live russell.ballestrini.net
re-crawl: 223 pages, full redeploy, exactly 1 content change (/about/),
rendered as a 2-version chain (90% similar to prior). 2 tests; suite
2551 passed.
- '.. class:: center' parsed as a Python class named 'center' (duplicate
across one-pager + two-pager) -> '.. rst-class:: center' (styling, no
object). Clears the duplicate-object-description warnings.
- two v8-consensus section underlines were shorter than their titles
('underline too short') -> extended to title length.
Pager + v8 pages now build with zero warnings.
The page was a markdown+LaTeX paper dumped into .rst — 499 errors/warnings,
big chunks rendering broken. Mechanical, content-preserving fixes:
- markdown code fences (```lang) -> RST .. code-block:: (48 blocks; the
dominant error source — RST read each ``` as an unclosed inline literal)
- display math [ ... ] -> literal blocks (34)
- code-block:: json -> text where bodies are schemas with <...>/[m,n]
placeholders, not valid JSON (10; same as the v8 fix)
- escape inline | in prose (math like |Z|, |pred(v)|) read as RST
substitution refs (37 lines; no pipe-tables in the file, so safe)
Residual 62 are scattered indented-math derivation lines RST treats as
block quotes — they still render, just warn; diminishing returns on an
advanced/optional theory page.
The simple-table (=== separators) had a wrapped multi-line cell that
docutils rejected as malformed -> the floor table rendered broken on the
live site. Converted to a list-table (alignment-proof). Also switched the
proposer-submission block from 'code-block:: json' to 'text' — it's a
schema with <...> placeholders, not valid JSON, so the json lexer warned.
Page now builds with no errors/warnings on these blocks.
