Add PRAGMA wal_checkpoint(TRUNCATE) at two points in
_execute_all_at_once so committed WAL pages don't pin disk through
subsequent passes. Production migration on 2026-05-26 hit 7 GB free
disk (down from 89) because SQLite's auto-checkpoint can't run while
a reader cursor is open, and FTS rebuild keeps a SELECT cursor open
through 1.5M chunks per target. Across 4 targets the FTS rebuild
plus audit consolidate held ~37 GB of committed-but-unreclaimed WAL.
Manual sibling-connection wal_checkpoint(TRUNCATE) freed 27 GB
mid-migration.
Checkpoints land at:
* end of _rebuild_fts_on_target (after the SELECT cursor is
explicitly cur.close()'d so the TRUNCATE checkpoint can actually
fire — TRUNCATE/RESTART block on active readers)
* end of _consolidate_audit_chain (after the 3.47M-row giant
transaction commits, before the next phase touches the same
connection)
VACUUM is already implicitly a checkpoint, so the existing per-
target VACUUM pass continues to handle the final checkpoint
naturally.
Helper _checkpoint_truncate(conn) returns the (busy, log_frames,
checkpointed) tuple SQLite emits; for the serial executor, busy=1
is improbable since each phase finishes before moving on.
Regression test
(TestWalCheckpointing.test_no_large_wal_after_migration) asserts
no WAL file exceeds 4 MB after migration completes. Without the
checkpoint calls this would fail on real-sized corpora; with them
the test passes deterministically.
Doesn't affect the running migration (it loaded the module from
memory before this commit). Future reshards run with bounded WAL —
no near-ENOSPC scares.
The 2026-05-26 cutover crashed mid-build with sqlite3.IntegrityError
"FOREIGN KEY constraint failed" inside _route_per_doc_table on the
derivations table.
Root cause: derivations.src_root carries a FK to
documents.document_root, but under content-hash routing a derivation
row's src_root can legitimately reference a surface doc that hashes
to a DIFFERENT target shard than the derivation's core_root. The FK
is a single-shard-era guard; it must stay live for the runtime
write path (to catch typo'd inserts into the wrong shard) but must
be OFF for the migration writer which legitimately produces
cross-shard refs.
Fix: arborist/migrate.py _connect_target now applies
`PRAGMA foreign_keys = OFF` after SCHEMA_SQL executescript runs.
Schema's own `PRAGMA foreign_keys = ON` still applies to the schema
DDL pass (and runtime connect() / connect_query() still get FK=ON
since they don't touch this helper). Only the migration writer is
relaxed. Documented inline.
Regression test
(TestCrossShardForeignKeys.test_cross_shard_derivation_succeeds)
synthesizes a derivation row whose core_root and src_root hash to
different M=4 target shards, runs the migration, asserts the row
lands on core_root's target with src_root pointing cross-shard. Pre-
fix this raised IntegrityError; post-fix it passes.
Originals untouched on the production host — the executor crashed
BEFORE the atomic-promote step, so .db files are intact;
~/.arborist/shards/00X.db.new files from the failed run will be
cleared by the next attempt's "stale .new before opening" cleanup
hook (already in _execute_all_at_once).
Scaffold-only ticket. Captures the architecture for taking the
#000061 cold-pack format and adding a second mode beside hydrate:
overlay an existing pack onto a populated shard set ("graft").
Surfaced while running the #000065 reshard cutover and fox extended
the design: each pack carries a `corpus_name` field
(wikipedia-2010, wikipedia-current, arxiv-cs, ...), making
`arborist cold graft wikipedia-current` feel like `apt install`.
Three concerns analysed:
doc/chunk/edge overlay trivial (INSERT OR IGNORE on content-
addressed PKs collapses dupes)
FTS5 overlay trivial (new chunk_ids → new fts rows)
audit chain overlay the only hard part — three approaches:
A graft receipt (chosen): one event in
host chain carrying pack_hash +
event_count + first/last hashes; pack
file is the durable witness; zero
schema cost; aligned with v8/v9
witness pattern
B re-chain everything: rejected — graft
is frequent so invalidating external
refs is wrong tradeoff (different
story from the one-time reshard)
C chain forest with chain_id col: right
answer when graft dominates lifecycle
but premature now
Long-game payoff: mesh-peer-corpus-merge. Two peers diverge over a
partition, each carries packs the other lacks, reconciliation =
exchange + graft what's missing. Makes "mesh of arborists"
coherent rather than "fleet of arborists."
Scaffold gated on (a) #000065 lands+stabilises, (b) a second
corpus exists to graft, (c) at least two peers want to exchange.
No code until then; the design lock is what the ticket buys.
Index bumped Next ID 000066→000067.
Reshard no longer creates a parallel "shards.v2/" directory. Builds
write to "<target_dir>/00X.db.new" alongside originals; when the
executor's validation passes, each .db.new is atomically renamed to
its final 00X.db name via os.replace (POSIX-atomic per file).
For an in-place migration (target_dir == source_dir, which is the
canonical use case), the rename REPLACES the original shard files.
~/.arborist/shards/ never contains a parallel "v2" or "next" or
"backup" directory — it always holds exactly one corpus, just with
the new topology after promote completes.
Validation gate (refuses to promote on mismatch):
* --expected-row-counts <snapshot.json> threads the pre-migration
snapshot's documents/chunks/edges totals through to the executor.
* Tolerance: ±1% to absorb the dupe-collapse from INSERT OR IGNORE
on cross-shard duplicate document_roots (166 dupes measured
pre-migration; <0.005% drift).
* Mismatch → RuntimeError, .db.new files left in place for
inspection, no rename performed.
CLI:
arborist corpus reshard --to 4 \
--source-dir ~/.arborist/shards \
--target-dir ~/.arborist/shards \
--audit-events-ndjson /tmp/audit-events.ndjson \
--expected-row-counts bench/results/pre-migration-snapshot.json
3 new tests:
* test_target_dir_only_has_db_files_after_completion — no .db.new
sidecars survive a successful run
* test_inplace_reshard_overwrites_originals — target_dir ==
source_dir works end-to-end; final files are the new hash-routed
shards
* test_validation_failure_leaves_new_files — bad expected counts
trip the validation guard; .db.new files survive; no .db files
promoted
Stale .db.new files from a prior failed run are unlinked before
opening fresh targets, so a partial-fail-and-retry is idempotent.
WAL/SHM sidecars removed at promote time so the new live .db
produces fresh sidecars on next open.
Build the 'all_at_once' executor and wire it into the CLI as
`arborist corpus reshard`. Plan-only preview against fox's host
confirms the planner picks all_at_once at 95.5 GB free, 64.2 GB
peak draw, 31.3 GB free at peak (well above the 4 GB safety).
Executor (arborist/migrate.py):
* ROUTED_BY_DOCUMENT_ROOT — documents, document_http_meta, chunks,
merkle_nodes, edges (by src_root), derivations (by core_root).
Each row hashed to shard_for_document(root, M) and INSERT'd into
the chosen target.
* CONSOLIDATED_TABLES — snapshots, concept_relations,
concept_token_idf, citation_aliases, term_aliases,
providence_cache, falsifications all land on canonical
target shard 000.
* REBUILT_ON_TARGET — chunks_fts + documents_fts rebuilt from the
materialized data after content moves. chunks.content is
zstd-packed at rest so the rebuilder decompresses via
arborist.compress.unpack_chunk before inserting plaintext into
the FTS5 index.
* Audit chain consolidation (Option A) — all rows from
/tmp/audit-events.ndjson are sorted by (ts, src_shard, src_seq),
re-chained with fresh event_hash values, and INSERT'd into
target shard 000. Bodies preserved unchanged for forensic
fidelity.
* One 'reshard' audit event appended at the tail, carrying the
plan + result as the body. An operator months later can answer
"where did this corpus topology come from" from this one row.
* corpus_shard_count meta stamped on every target shard.
* VACUUM each target at end to reclaim INSERT-pattern fragmentation.
CLI: `arborist corpus reshard --to M --source-dir DIR --target-dir DIR
[--plan-only | --force-strategy X
| --allow-in-place | --dry-run]`
11 integration tests build a tiny 2-shard corpus, run the migration,
verify per-doc routing, chunk-follows-doc, audit chain integrity,
reshard event at tail, corpus_shard_count meta on every target,
chunks_fts searchability, doc count preservation, dry-run no-op,
error handling.
Also: get_meta() now indexes by position so callers without
row_factory=sqlite3.Row don't trip. No behavior change for callers
that DO use Row factory.
Strategy B (per_source_shard) and C (streaming_row) are stubbed in
the planner (peak-draw estimators wired) but execute_plan raises
NotImplementedError for them. Not needed at fox's current disk
(88+ GB free); skipping the build until that's ever the constrained
path.
Disk-aware strategy picker for the corpus reshard. Pure-compute API:
feed it (corpus facts, free bytes) and it returns a HydrationPlan
naming one of three strategies + a peak-draw estimate.
Strategies (preference order):
all_at_once - direct source→target with overlap; fastest;
peak draw = corpus + WAL × M_target + VACUUM
per_source_shard - pack→delete→hydrate per round; safest;
peak draw = pack + WAL + 1 target VACUUM
streaming_row - in-place mutation, --allow-in-place opt-in;
peak draw = WAL only
Safety budget: 4 GB margin above the strategy's peak draw — the
buffer that survives one bad sort + one badly-sized WAL grow.
Production-host preview at 95 GB free / 38 GB corpus:
strategy: all_at_once
peak draw est: 64.2 GB
free at peak: 31.2 GB (well above safety)
rationale: all_at_once: peak draw 64.2 GB + 4.3 GB safety ≤ free 95.4 GB
The plan's full readout (strategy, free bytes, peak draw, rationale)
is JSON-serializable so it goes into the migration audit event as a
single forensic record.
14 tests cover: strategy pick by free-disk level, --force_strategy
override, --allow-in-place gating, skewed shard sizes, json-
serializable audit body, target_M validation.
No execution yet — this is just the planner. Next: the strategy A
executor (direct source→target read+write).
Three pieces, all read-only or additive — no shard mutation, no
schema-version bump:
1. shard_for_document(document_root, M) in arborist/document.py.
Pure function: int(document_root[:8], 16) % M. 22 tests cover
determinism, range-bounds, near-uniform distribution (±5pp at
N=20k), and seven lock-in fixtures so peers will disagree
loudly if anyone changes the formula.
2. corpus_shard_count meta field + get/set helpers in store.py.
Lives in the existing key/value meta table; SCHEMA_VERSION
stays at v9.8.0 (the DDL doesn't change and source_root is
layout-independent, so cache records survive a reshard).
Legacy shards (without the field) return None; reshard tool
populates it on every target shard at migration time.
3. Pre-migration snapshot captured to
bench/results/pre-migration-snapshot.json:
docs 3,468,392 (3,468,226 globally unique)
chunks 6,235,764
edges 90,593,537
audit 3,468,403
This is the reference set post-reshard row counts must match.
4. Audit-event extraction script writes all 3.47M events from
all 4 shards to /tmp/audit-events.ndjson (2.0 GB) for the
Option-A canonical-chain consolidation step. Verifies chain
integrity on extract — all 4 source chains report 0 breaks.
5. Fixed a wrong chunk count in docs/corpus-history.md
(had ~3.54M/shard; actual is ~1.56M/shard) and added the
edge-count column (~22.6M/shard, 90.6M total). 6.24M chunks
total, not 14.12M.
Tests: 29 new pass (22 routing + 7 meta). No existing tests
touched.
The original ingest of Wikipedia 2010 into fox's 4 production shards
was 2026-04-27 21:33-23:13 UTC — ~100 minutes wall, 4-way parallel,
3.47M docs / 14.12M chunks. Captured here because the audit chain is
the only durable record but querying 3.47M rows to recover the
headline number is friction; one line in a doc removes it.
Also informs #000065 reshard planning: ingest rate ceiling on real
XML workload is ~2,350 chunks/sec aggregate (4-way), vs the
~6,400 chunks/sec the M-sweep bench measured on the 2003 cur dump
(which skips XML parsing). The teleport-style reshard should beat
both ceilings because it's just SQLite INSERT throughput, no XML
parse + canonicalize + edge extraction.
New file: docs/corpus-history.md. Indexed in CLAUDE.md docs section.
Append-only convention; future migrations + cold-pack runs add
entries here so the operator log isn't only in the audit chain.
Derivation query (sqlite3 audit_events) embedded in the entry so
future re-derivation is one copy-paste.
Pinned the canonical shard count at M = 4 based on real-Wikipedia
ingest + query benchmark (bench/shard_count_sweep.py). Captured the
"when does SQLite stop being the right substrate" decision tree so
future operators know what bench would justify a fork or replacement.
Bench numbers (Wikipedia 2003 cur dump, 2000 docs, 4 cells of
M ∈ {1, 2, 4, 8}, 50 FTS queries per cell):
M chunks/s q_p50_ms q_p99_ms attach_ms
1 3,648 0.05 0.20 1.61
2 5,649 0.03 0.19 3.24
4 6,405 0.07 0.30 9.00
8 6,959 0.03 0.28 9.65
Key observations:
- M=1→M=2 is the biggest ingest win (+55%). Most gain happens there.
- M=2→M=4 is +13%. M=4→M=8 is only +9% — diminishing returns.
- Real wikitext canonicalization is per-worker Python CPU bound, not
SQLite-writer-lock bound. More shards don't unlock more CPU.
- Query p50/p99 is flat across M within noise (50 queries small).
- ATTACH cost grows linearly: 1.6 / 3.2 / 9.0 / 9.7 ms.
Why M=4 specifically:
- Captures 92% of peak ingest throughput (6,405 / 6,959).
- 6 ATTACH slots free under SQLite's 10 ceiling for aux DBs
(qa.db, snapshots.db, selfmodel-chain.db, crawl_*.db, future
mesh_*.db) — comfortable headroom. M=8 leaves only 2 slots.
- Mobile-tolerable: phone NAND attach is 5-10x slower than NVMe;
M=4 = 45-90 ms cold start (instant), M=8 = 50-100 ms (sluggish
with no headroom).
- Matches fox's current 4-shard layout = cheapest migration.
Decision tree for when SQLite stops being right (full text in
ticket §"When the SQLite-default substrate stops being right"):
A. ATTACH ceiling pressure (auxiliary DBs grow past 5) → bench
forked SQLite with SQLITE_MAX_ATTACHED=125, M ∈ {16, 32, 64};
if attach cost stays linear past M=10, fork viable but pays
permanent "no longer stock sqlite3" tax.
B. Ingest hits >10k chunks/s sustained ceiling → first tune
page_size / WAL checkpoint / mmap_size / synchronous. If
tuning gets 2-5x, stay on SQLite. If still ceiling-limited,
candidates: DuckDB (columnar, MVCC, FTS), libmdbx (B+tree no
FTS; we'd build it). In-house DB rejected without specific
failure of those.
C. Federation needs multi-writer-same-shard → SQLite writer-lock
serializes peers, becomes federation bottleneck. First try
leader-election (single-writer-per-shard with WAL replication
to followers). If true multi-writer required, SQLite is wrong;
candidates: FoundationDB, CRDT-on-KV-store. DuckDB does NOT
solve this — its MVCC is single-process.
Honest verdict: for current arborist workload (single-writer-per-
shard, read-mostly federation), stock python3 sqlite3 is the right
substrate. None of A/B/C are close to firing. The bench discipline
exists to know what to measure when something changes.
bench/results/shard-count-sweep-2026-05-26T16-20-48Z.csv (synthetic
baseline) + 2026-05-26T16-31-34Z.csv (real Wikipedia) committed as
the load-bearing measurement for the M=4 choice.
fox caught the imprecision: "the shards are teleported if we fix this
because we know the shard a doc will end up in."
Re-ingest = re-read source files, re-canonicalize, re-chunk, re-hash.
Hours-to-days from Wikipedia dumps.
What's actually needed = content-addressed rebalance. Every row in
the corpus is already addressed by content (document_root, leaf_hash,
merkle_nodes.hash, audit_event_hash). Migration reads existing rows,
computes new_shard_idx via the routing function, INSERTs into the new
M shards. No source-file work, no canonicalization, no LLM.
Updated the ticket's "Migration story" section with the per-table
teleport recipe:
documents: move by document_root → new shard
chunks (+content): follow doc to its new shard, re-assign chunk_id
chunks_fts: rebuild per new shard from chunks.content
merkle_nodes: move with document_root
edges: move by src_root
concept_relations, move by content-derived parent
derivations,
providence_cache,
citation_aliases,
term_aliases:
audit_events: ALL → shard 000 (canonical), re-numbered + re-hashed
snapshots, mesh_*: canonical shard 000
Audit chain handling: chose Option A (canonical shard 000) over
Option B (split-by-subject_root). Preserves global event ordering,
re-hashing happens once at migration time. The alternative loses
cross-shard event ordering for falsification + replay reasoning.
Estimated wall: ~20-40 min I/O-bound on current corpus
(35 GB across 4 shards → ~35 GB across 8 shards, moves once).
Hard discipline added: move into ~/.arborist/shards.v2/, keep
originals as rollback, atomic dir-swap when verified. Mistake in
routing function = corrupted federation, recoverable only by going
back to the originals.
Surfaced while sizing #000061's federation story. Today shard count
conflates two roles:
- producer ingest parallelism (wants = vCPU count)
- consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10)
On default Python sqlite3, the ATTACH ceiling is 10 and can't be
raised without a custom sqlite3 build (which violates CLAUDE.md's
"python3 + venv + sqlite3 is enough" property). Producer with 16
shards → consumer fails to attach the 11th, federation silently
breaks.
Design: introduce M = canonical shard count (corpus-wide constant,
default 8) decoupled from N = ingest workers. Document → shard
assignment becomes content-deterministic:
shard_idx = int(document_root[:8], 16) % M
Same input → same output across every peer. Today's "spray by ingest
order" is non-deterministic across peers — two peers re-ingesting
the same corpus put the same document_root in different shards. That
weakens federation more than it should.
Migration hard-constraint (fox: "this implies we will need to
reprocess all our data into shards"): re-ingest required. Current
layout is sprayed by ingest order; post-ticket is sprayed by content
hash. Two layouts are incompatible by construction. Captured in
ticket §Migration as the load-bearing operational note.
Phases laid out (0-4: design lock → read path → ingest path →
pack-restore → corpus migration tool). Open audit-chain re-numbering
question (per-shard event_hash chains break when rows rebalance
across shards). Kept as one ticket — do-not-proliferate.
Scaffold only. No code yet — design lock first.
Next ID bumped to 000066.
The 3.15 GB-per-shard index I added in bc7efe4 bloated every live
shard 33% to optimize a transient dump-time operation. Wrong trade —
the shards exist for arborist's query/retrieval/falsification path,
not for cold-pack's convenience. fox: "wtf you added an index to
fucking make the shard bigger?"
Refactor: instead of building a permanent on-disk index to avoid the
sort, configure the dump connection to spill the sort to a temp file:
conn.execute("PRAGMA temp_store = FILE")
conn.execute("PRAGMA cache_size = -10000") # 10 MB
SQLite's sort still happens, but it spills to /tmp instead of growing
the process heap. cache_size caps the page cache the sort works
inside. No permanent disk cost; transient temp disk only during the
dump phase.
DROP INDEX idx_edges_dst_uri_type_anchor + VACUUM ran across the 4
live shards before this commit (out-of-band ops):
000.db: 12.48 → 8.75 GB (~3 GB reclaimed)
001.db: 12.43 → 8.71 GB
002.db: 12.53 → 8.78 GB
003.db: 12.51 → 8.76 GB
total: ~12.6 GB reclaimed across the corpus
For a freshly-restored shard from a metadata pack, the index never
gets created (it was only in _dump_edges_fan_in, not in SCHEMA_SQL),
so consumers get the lean version from day one.
33/33 cold + evict tests pass.
The license_class gate landed in 576cb0e but only push_pack() the
function takes allow_license_class — the CLI couldn't pass it through.
Adds --allow-license-class {public_redistributable,unknown,private}
to `arborist cold pack`. Default stays at public_redistributable;
operators override only after confirming bucket ACL + source license
permit redistribution.
33/33 tests pass.
Dav1d's reviews of #000061 (Response A + Response B/FINAL in
~/Downloads, 2026-05-26) flagged a long list of items — most already
shipped in the SPV-split work. Three were genuine gaps worth folding
into #000061 before close:
Gap 1: manifest/latest pointer for new-peer discovery.
A fresh peer doing `cold list` got a list of metadata-pack hashes
but no obvious "which one is current for shard X." Added
get_latest_pointer + update_latest_pointer to the backend ABC.
push_pack writes manifest/latest.json on every successful metadata
pack push (read-modify-write keyed by snapshot_root). Mutable
pointer; content addressing of the packs themselves preserves the
trust root. Last-writer-wins on contention.
Gap 2: license_class field + producer-side refuse for public buckets.
Maps documents.source_type to a license bucket (wikipedia_cur /
textbook_tex → public_redistributable; html / grok / vcs → unknown;
anything else → unknown). Strictness order: public < unknown <
private. compute_shard_license_class() walks DISTINCT source_type
in documents. push_pack now refuses to upload if the shard's
strictest license is more restrictive than the operator's
allow_license_class (default: public_redistributable). The
metadata pack's manifest carries _license_class so consumers /
auditors can see the producer's classification without inspecting
source documents. ValueError on refusal — the bucket ACL is the
operator's call, but arborist refuses to participate in a
licensing/membership leak unless explicitly opted in.
Gap 3: cold_pending table for resumable uploads.
Killed mid-upload, push_pack left orphan multi-GB tempfiles in
/tmp with no DB trace. Added schema:
CREATE TABLE cold_pending (
tempfile_path TEXT PRIMARY KEY,
pack_hash TEXT NOT NULL,
kind TEXT NOT NULL,
backend_endpoint TEXT NOT NULL,
backend_bucket TEXT NOT NULL,
object_key TEXT NOT NULL,
started_at INTEGER NOT NULL,
state TEXT NOT NULL DEFAULT 'pending'
);
push_pack INSERTs a row before each upload + DELETEs on success.
A killed process leaves the row pointing at the orphan tempfile;
a recovery script (future) reads cold_pending, checks bucket for
the object, either deletes the row + tempfile (success was just
unreported) or re-uploads from the tempfile if it still exists.
Matches the same pattern as the audit chain — explicit state
rows beat inferring from chunks.content IS NULL.
Sibling tickets opened for the larger items the reviews flagged
(scaffold-only, no code; opening them captures the design in the
log without proliferating, per CLAUDE.md):
- #000063 Cold-object private-ciphertext mode (mesh-keyed object
keys for non-public corpora on public-read buckets). Needs mesh
group-key ABI + real non-public corpus before code.
- #000064 Cold-object operations toolkit (verify / diff / doctor /
repair-fts / gc-plan CLI + expanded audit-event taxonomy).
Bundled so the audit-event vocabulary gets one design pass.
5 new tests:
test_gap2_license_gate_refuses_unknown_class_to_public_bucket
test_gap2_license_class_in_metadata_manifest
test_gap1_latest_pointer_resolves_metadata_pack_per_snapshot
test_gap3_cold_pending_clears_on_successful_upload
test_gap3_cold_pending_records_inflight_upload
26 cold-object + 7 evict tests pass (33/33 green incl. boto3 wire).
Next ID bumped to 000065.
Live v3 SPV corpus run (bmq47x6t3) completed cleanly during this work.
Will report sizing + memory profile in the next message.
Three doc-only housekeeping items from Dav1d's de-novo reconciliation
of the architecture stack (no code changes; the active build is
#000061 cold-pack work, which is unaffected by this review):
1. Accepted-error formula. Dav1d corrected p_raw × (1-d)(1-r) to
p_raw × (1 - dr) where r is conditional on detection. Searched
docs/ and arborist/ — the wrong form does NOT appear in this
tree (it lives in the external recursive-truth-maintenance / RCO
manuscripts Dav1d cited). Nothing to fix here; recorded for the
manuscript authors.
2. #000060 H-ABCDEFG split. Folded the M/C/X axis split into the
harness scope: M (mechanism — does the substrate's plumbing work),
C (capability — does it improve task performance), X (external
adversarial — does it generalize outside author-designed fixtures).
ACCEPT requires clearing all three. Without X, the harness risks
self-validating benchmark theology — passing tests its own designers
picked. Tagged at fixture-definition time; aggregator emits per-axis
pass rates + combined ACCEPT verdict. Doesn't change the existing
BatteryResult row schema.
3. #000062 Mechanistic Witness — new scaffold-only ticket. Specifies
a content-addressed MechanisticWitnessRoot over (model_config,
capture_policy, contrastive_prompts, features/neurons, intervention
result, behavioral delta, safety policy) as a DIAGNOSTIC sidecar
feeding SelfModel + benchmark fixtures. Four hard guardrails:
diagnostic-only by default; sandbox intervention only; no production
steering without governance ACCEPT via #000060 M+C+X; feature labels
never become semantic proof. No code until a real falsifier use case
exists + guardrails are CLAUDE.md rules + #000060 harness gates
promotion. The dual-use risk (Pan et al. 2025: 0.1% MLP ablation
breaks refusal in 72B models) makes governance-first framing
load-bearing.
Next ID bumped to 000063.
No code change to arborist/. The in-flight v3 SPV corpus pack
(bmq47x6t3) continues unaffected.
Previously: COLD_PACK_JOBS hardcoded to 4. Caused the v3 corpus run to
fail at fork time on a memory-pressured box (4 GB available, each worker
needed ~5 GB peak → kernel swapped, SQLite executescript exceeded
busy_timeout, errored as "database is locked"). Killing the run + waiting
for memory to free → 2-way run succeeded.
Now: COLD_PACK_JOBS is auto-computed from `free -m` at run time. Reads
available RAM, subtracts headroom for OS + buff/cache, divides by the
observed per-worker peak, caps at COLD_PACK_JOBS_MAX. An explicit
override (COLD_PACK_JOBS=N on the make invocation) still wins for
operators who know better.
Tunables:
COLD_PACK_PER_WORKER_MB 5500 observed peak from real corpus run
COLD_PACK_HEADROOM_MB 2000 keep this RAM for OS / other procs
COLD_PACK_JOBS_MAX 8 ceiling regardless of RAM headroom
Same logic applied to cold-pack-all-dvd (local-dir output for burning).
Both targets print the chosen concurrency + the math behind it before
launching workers. If the auto-computed value is 1, that's a signal to
wait for RAM to free before trying parallelism.
Doesn't fix the underlying fragility of using fork-time RAM availability
to predict peak demand — workers grow over their lifetime as metadata
dumps materialize — but it's a real improvement over a static cap that
ignored system state entirely.
Bidirectional sync: producer always emits both kinds; consumer chooses
how much to pull.
packs/<hash>.metadata.tar.zst — one per shard (~0.6 GB compressed)
packs/<hash>.metadata.manifest.ndjson
packs/<hash>.chunks.tar.zst — N per shard (each ≤ 4.4 GB cap)
packs/<hash>.chunks.manifest.ndjson
Each artifact is independently content-addressed by its own manifest
hash. The metadata pack's manifest carries _chunk_pack_hashes — every
chunk pack covering this shard's content — so a "full" consumer can
iterate them. Chunks packs are anonymous from the consumer side
(reachable only via the metadata pack's reference list).
Consumer sync modes:
arborist cold unpack <metadata_hash> # default: just-enough
arborist cold unpack <metadata_hash> --full # also pulls chunks
just-enough: pull only the metadata pack. Schema fully restored; every
chunks row has content=NULL. Node is immediately queryable for
metadata operations (documents, edges, audit chain, Merkle); chunk-body
queries return null until a future JIT-fetch path fills them on cache
miss.
full: after the metadata pack lands, iterate _chunk_pack_hashes and pull
every chunk pack. Final state: full corpus offline-queryable.
This is fox's original SPV-wallet framing — was right from day one.
Key API changes:
pack_key(hash, *, kind="chunks", manifest=False) — kind in bucket path
stream_packs(chunks, *, max_compressed_bytes, ...) -> Iterator[FilePack]
now emits chunk-only packs (no extra_members)
build_metadata_pack(table_files, *, snapshot_root, chunk_pack_hashes, ...)
single FilePack with v3 metadata manifest
pull_metadata_pack(conn, backend, hash) -> dict
pull_chunk_pack(conn, backend, hash) -> dict
hydrate_from_metadata_pack(conn, backend, metadata_hash, *, mode) -> dict
push_pack orchestrates: dump tables → stream chunk packs (collect hashes)
→ build metadata pack with chunk_pack_hashes → upload all. Returns
{metadata_pack_hash, chunk_pack_hashes, packs: [...]}.
Manifest format v3:
metadata pack:
{"_format_version": 3}
{"_kind": "metadata"}
{"_snapshot_root": "..."}
{"_chunk_pack_hashes": [...]}
{"table_file": "tables/X.jsonl", "hash": "...", "size": N} per table
chunks pack:
{"_format_version": 3}
{"_kind": "chunks"}
{"leaf_hash": "...", "size": N} per chunk
pack_hash for each = hash_leaf(manifest_bytes); content-addressed at
both layers. Two writers with the same shard state produce identical
metadata_pack_hash AND identical chunk_pack_hashes.
Cap-and-split applies only to chunk packs (chunks are bounded N).
Metadata pack is one file per shard; if a single table file is larger
than the cap, that's noted as future row-level split work.
cold list now surfaces kind per artifact (metadata vs chunks), plus
table_count + chunk_pack_hashes (for metadata packs) or chunk_count
(for chunks packs). Operators can quickly find the metadata pack hash
to feed `cold unpack`.
28 cold-object + evict + boto3 tests pass (3 new SPV-shape tests + 1
v3 manifest-shape test + tampered-metadata-pack test).
Caught a real defect on the live v2 corpus run: pack_hash was computed
from the chunk-only manifest, so two packs with the same chunk set but
different table content collided on pack_hash. Observed live: v1 packs
and v2 packs for the same shard produced the SAME pack_hash:
v1 (chunks-only) b3c427baaf9b40e3c33ace8438b4254e3abdc7b38bdcb7b1a8012588cb32848a
v2 (tables + chunks) b3c427baaf9b40e3c33ace8438b4254e3abdc7b38bdcb7b1a8012588cb32848a
↑ identical, different bytes
Consequences if shipped: bucket overwrites swap whose tables you get,
mesh peers A and B disagree on what "pack X" is while both claim to
have it, content-addressing story is broken for the metadata half.
Fix: manifest format v2 prepends a `{"_format_version": 2}` header,
then one row per shipped table:
{"table_file": "tables/audit_events.jsonl", "hash": "...", "size": N}
{"table_file": "tables/chunks.jsonl", "hash": "...", "size": N}
...
{"leaf_hash": "...", "size": N} # chunks come after
{"leaf_hash": "...", "size": N}
pack_hash = hash_leaf(manifest_bytes) now binds the full pack content.
Two writers with the same shard state produce the same pack_hash; same
chunks + different tables produce different pack_hashes.
Implementation:
- hash_file_leaf(path) — streaming sha256 with leaf 0x00 prefix, 1 MB
reads, used to fingerprint multi-GB tables/<name>.jsonl files without
loading them into RAM.
- stream_packs' extra_members now takes (member_name, source_path,
content_hash) tuples. push_pack computes the hash with hash_file_leaf
after the dump finishes.
- _finalize_pack writes the format header + table refs (sorted by name
for determinism) before chunk entries.
- parse_manifest returns a ParsedManifest (format_version, tables[],
chunks[]) instead of just list[PackEntry]. Backward-compatible with
v1 manifests (no header → format_version=1, tables=()).
- pull_pack pulls the manifest aside during tar walk, then after
extraction verifies each tables/<name>.jsonl file via hash_file_leaf
against the manifest's content_hash. Mismatch → ValueError, no rows
reach the live schema.
New tests:
- test_v2_pack_hash_binds_table_contents — asserts manifest format +
table refs + per-table hash & size.
- test_pull_pack_rejects_tampered_table — corrupts one table's bytes
in a real pack, confirms pull_pack raises on hash mismatch.
27 cold-object + evict tests pass.
Live v2 corpus run showed ~5 GB RSS per worker — RssAnon dominant, so
process heap, not mmap. Traced to two unfixed memory pits in the edges
fan-in dump path:
1. SQLite ORDER BY on edges (22M rows, no covering index for the v2
sort order dst_uri+edge_type+anchor) allocates a multi-GB in-memory
sort area before spilling. Adding:
CREATE INDEX IF NOT EXISTS idx_edges_dst_uri_type_anchor
ON edges(dst_uri, edge_type, anchor, dst_root, src_root)
means the ORDER BY walks the index in order — no in-memory sort.
First create takes ~30-60 s on a 22M-row shard; idempotent on
subsequent dumps. Disk cost ~1 GB per shard (4 shards × 1 GB ≈
2-3 % corpus footprint increase). Worth it.
2. Python groupby accumulator: src_roots = [row[4] for row in group]
materializes the entire src_root list per destination. For
en.wikipedia.org/wiki/* destinations with millions of inbound links,
this list is itself ~GB-sized. Switch to bounded batches:
_FAN_IN_BATCH = 10_000 # max src_roots per fan-in JSON row
A destination with N inbound links splits into ceil(N / batch) rows.
Restore path (INSERT OR IGNORE) handles multi-row destinations
correctly because PK includes src_root — accidental duplicates
collapse cleanly.
New regression test test_edges_fan_in_batches_huge_destinations builds
an edges table with FAN_IN_BATCH+137 rows pointing at one dst_uri,
verifies the dump produces the expected number of split rows and the
restore reconstructs all N edges with no loss or duplication.
25 cold-object + evict tests pass.
The extra_members handling in stream_packs did:
with open(source_path, 'rb') as f:
source_bytes = f.read() # loads entire file into RAM
info = tarfile.TarInfo(...)
info.size = len(source_bytes)
state['tar'].addfile(info, io.BytesIO(source_bytes))
For a 4.3 GB edges.jsonl this is 4.3 GB of RAM per worker — exactly the
failure mode the prior streaming-to-disk refactor was supposed to
eliminate (57f8989). 4-way parallel would hit ~17 GB peak RAM, the same
OOM-risk number we saw on the v1 corpus run.
Fix: build the TarInfo by hand (size from .stat(), other fields zeroed),
then pass the open file handle directly to tar.addfile. tarfile reads
the body in chunks (~16 KB at a time) and feeds them into the zstd
stream_writer. Peak RAM per worker stays at ~tens of KB regardless of
metadata file size.
Determinism preserved: mtime/uid/gid default to 0 (NOT taken from the
filesystem via gettarinfo), so two writers' packs converge byte-for-byte
regardless of when they ran or who owns the temp file.
24 cold-object + evict tests pass.
v1 packs (chunks-only) were under-engineered: a new peer landing on
v1 packs would have chunk bodies indexed by leaf_hash but no documents
table, no audit chain, no merkle interior, no edges — couldn't actually
hydrate. fox: "isn't what I wanted you under engineered..."
v2 packs ship every load-bearing shard table alongside chunk bodies in
the same tar.zst:
manifest.jsonl # chunk catalog (unchanged)
tables/documents.jsonl # array-per-line columnar JSONL
tables/chunks.jsonl # without content column
tables/merkle_nodes.jsonl
tables/edges.jsonl # FAN-IN restructured
tables/audit_events.jsonl
tables/derivations.jsonl
tables/concept_relations.jsonl
tables/concept_token_idf.jsonl
tables/providence_cache.jsonl
tables/citation_aliases.jsonl
tables/term_aliases.jsonl
tables/snapshots.jsonl
tables/document_http_meta.jsonl
blobs/<hash[:2]>/<hash[2:]> # raw UTF-8 chunk bodies
Two compression strategies inside the pack:
1. Array-per-line JSONL ({"_columns": [...]} header line + ["v1","v2",...]
data lines) drops ~30% of uncompressed bytes vs object-per-row JSONL.
zstd recovers most of that on its own, but smaller uncompressed
footprint also speeds up stream-restore.
2. Edges fan-in restructure at pack-build time: 22M rows of
(src_root, edge_type, dst_root, dst_uri, anchor) → ~500k unique
(dst_uri, edge_type, anchor, dst_root) groups with src_roots as an
array. ~5-10x compressed savings on the dominant table. Reverses on
unpack into the per-edge live schema. Live queries unchanged.
NOT shipped (per-peer state): mesh_*, selfmodel_*, capital_ledger,
memory_*, controller_events, fork_score_branches, adapter_loss_reports,
falsifications, schema_meta, meta. NOT shipped (rebuildable): chunks_fts*,
documents_fts* — restored from chunks.content + documents.title on
unpack.
push_pack no longer appends `cold_pack_pushed` to the audit chain.
That event leaked into the next push's audit_events.jsonl dump and
broke the "two writers at the same corpus state produce identical
pack_hash" determinism property. The bucket/disc file IS the receipt;
the snapshot_root pinned inside the pack metadata binds it to a corpus
state. No load-bearing consumer of the audit row.
pull_pack restored to handle both v1 (chunks-only) and v2 (tables +
chunks) packs. For v2 it extracts tables/*.jsonl to a temp dir,
calls restore_shard_metadata (which INSERT OR IGNOREs into the live
schema and expands edges back to per-edge rows), then fills chunk
content for every leaf_hash in blobs/. Idempotent against populated
DBs (INSERT OR IGNORE all the way down). Self-cleaning temp dir.
Sizing measured 2026-05-26: ~2.1 GB per shard pack compressed (chunk
content 1.78 GB + metadata ~0.3 GB), ~8.5 GB total across 4 shards.
~20% more than v1 chunks-only for self-sufficient hydration.
24 cold-object + evict tests pass (+1 new test_push_pack_v2_hydrates_fresh_empty_db
that builds a pack from a populated DB and unpacks into a completely
empty DB to verify all tables restored). Full suite: 2558 passed,
28 skipped, 1 xfailed.
The in-memory BytesIO design held a whole 4.4 GB pack in RAM per worker
until finalization. 4-way parallel hit ~17 GB peak RAM — observed live
on the full-corpus run (21 GB used, 0 free, 3 GB swap in use, headed
toward OOM territory). Architectural defect, not just an inconvenience:
parallelism is capped by RAM rather than CPU/network.
stream_packs now:
- Opens a NamedTemporaryFile in work_dir (default tempfile.gettempdir()).
- Wraps it with ZstdCompressor.stream_writer + tarfile mode "w|".
- After each chunk: writer.flush(FLUSH_BLOCK) + file.flush(), check
file.tell() against the cap.
- On finalize: write the manifest tar member, close everything, yield
a FilePack carrying the path (not the bytes).
- On exception: close + unlink the in-flight temp file (no /tmp litter).
ObjectStoreBackend gains put_file(key, path). S3 impl uses boto3
upload_file which streams parts from disk during multipart upload —
memory bounded to max_concurrency × part_size = 10 × 8 MB = 80 MB
regardless of pack size. MemoryBackend reads the file into its dict
(tests only).
push_pack consumes FilePack:
- --push: backend.put_pack_file(path), then unlink the temp
- --local-dir: shutil.move from temp to local_dir/arborist-pack-<hash>
(cheap if same filesystem; one-pass read+write if not)
- both: shutil.move first, copy to bucket via put_pack_file from final
local path? — actually current code moves to local_dir
AFTER the bucket put; bucket put still works from the
temp path before move. Audit row carries pack.body_size
from FilePack (no len(bytes) on a 4 GB body).
Worker memory bound goes from O(pack_size) to O(zstd_buffer + tar_header)
≈ tens of MB. 4-way parallel now uses ~tens of MB total instead of
~17 GB. Disk requirement: work_dir needs (parallelism × cap) of free
space — 4 × 4.4 GB = 18 GB for a default full-cap parallel run.
23 passed in tests/test_cold_object.py + tests/test_evict.py.
Shards are independent SQLite files; the ORDER BY landed in 6f0ceab
makes each shard's chunk-to-pack assignment deterministic. Two new
Makefile targets fan out cold pack across every numbered shard via
xargs -P:
make cold-pack-all # parallel push to S3
make cold-pack-all-dvd LOCAL_DIR= # parallel write to disk (no S3)
Tunables:
SHARDS_DIR defaults to ~/.arborist/shards
SHARDS_PATTERN defaults to [0-9][0-9][0-9].db (numbered shards only,
skipping qa.db / snapshots.db / crawl_*)
COLD_PACK_JOBS defaults to 4
Expected wall-clock improvement: ~4x on full-corpus runs (4 shards
running in parallel saturate either CPU or network before serializing).
Also verified end-to-end pack → list → unpack round trip against the
live DO Spaces NYC3 endpoint (small pack, 20 chunks): pushed, listed,
unpacked into a fresh receiving DB, content restored byte-identical,
hashes matched.
New peers landing on a public bucket need to enumerate pack_hashes
before they can `arborist cold unpack <hash>`. `cold stats` only gives
a count; `cold list` returns the per-pack details:
- pack_hash (taken from the key)
- compressed_bytes (one HEAD round-trip via new object_size method)
- chunk_count (parsed from the small manifest sidecar; skippable
via --no-manifest for huge-bucket fast listing)
`cold stats` also now reports total bucket footprint (sums HEAD sizes).
Backend ABC gains `object_size(key) -> int | None` so both subcommands
get sizes without paying egress for the body. S3 impl uses HEAD;
MemoryBackend reads from the dict.
Verified live against DO Spaces NYC3: list-empty → push tiny pack →
list-with-manifest (pack_hash, size=5311 B, chunk_count=5) → list
--no-manifest → cold stats → cleanup.
23 passed in tests/test_cold_object.py + tests/test_evict.py.
Three improvements after the first DO Spaces smoke + bench:
1. ORDER BY c.leaf_hash on the chunk-selection SQL. Two writers running
cold pack against the same DB at the same snapshot now produce the
same pack_hashes — chunk-to-pack assignment is a function of (chunk
set, cap) and nothing else. Prerequisite for parallel per-shard pack
workers and for two replicas to converge on byte-identical bucket
state. Costs ~25% on build wall (real-bench 31s → 40s on 100k chunks)
due to sort over the leaf_hash index + documents JOIN; worth it.
New test pins the determinism property.
2. boto3 multipart upload via TransferConfig (8 MB threshold + 8 MB
parts + 10-way concurrency) on every put. Required anyway for packs
> 5 GB (DO Spaces single-PUT limit). Measured 5.5 MB/s → 9.0 MB/s
on 121 MB pack to DO Spaces NYC3 (1.6x; ceiling is closer to network
than to boto3 serialization).
3. Stream the SQL cursor in push_pack instead of fetchall(). At 14M
chunks × ~700 bytes/row the prior fetchall materialized ~10 GB of
Python heap before stream_packs ever ran. Cursor iteration bounds
memory by the in-progress pack (~few hundred MB at the 4.4 GB cap).
Full corpus extrapolation revises ~125 min (single-PUT) → ~104 min
(multipart, sequential per-shard). Real wins live in parallel per-shard
pack workers — deferred; the determinism work landed here is the
prerequisite.
22 passed in tests/test_cold_object.py + tests/test_evict.py.
Ship arborist corpus state to new peers and DVD-R archival via
point-in-time tar.zst packs. One artifact serves both channels —
bucket+CDN delivery and physical-media archival.
Bucket holds packs only. Pack key = hash_leaf(manifest_bytes), so same
chunk set on two writers produces the same pack_hash and upload is
idempotent. Each pack pins the corpus snapshot_root it covers in audit
+ result body — packs are delayed snapshots, not live mirrors;
falsifications between repacks produce new pack_hashes.
stream_packs runs streaming zstd over tarfile, peeking compressed-buffer
size after each chunk via FLUSH_BLOCK (preserves dictionary). Default
cap 4_400_000_000 — 4.4 GB DVD-R safe-fit, ~6.5% buffer below the
4.7 GB marketing capacity to absorb ISO9660 overhead, growisofs
lead-in/lead-out, media variance, and drive-edge refusal. Each disc
fills to ~4.4 GB recorded data, not the ~1.5 GB an uncompressed cap
produced.
One backend class (S3CompatibleBackend via boto3 + endpoint_url) covers
AWS S3, DO Spaces, R2, B2, GCS S3-interop, MinIO. Optional dep
[object-store] = boto3>=1.34; dev extras pull moto for the wire test.
Voyeur: credentials via AWS_ACCESS_KEY_ID/_SECRET_ACCESS_KEY env or
~/.aws/credentials, never printed; only endpoint URL + bucket name
surface in logs.
CLI: arborist cold {pack,unpack,stats}. Makefile: cold-pack,
cold-pack-dvd (local-dir output for growisofs), cold-unpack, cold-stats.
Sizing for current shards (14.1M chunks, ~17 GB compressed): ~4 packs
at the default cap, ~\$0.34/mo DO Spaces storage, ~\$0.0001/fresh-peer
hydrate.
Always-on raw-UTF-8 leaf store (per ticket "Hard invariants") deferred
— packs-only for now, backfill later.
2557 passed, 28 skipped, 1 xfailed.
Address Grok's two minor-improvement flags on the docs.
New docs/diagrams/{three-layer-stack,cache-key-8dim,falsification-states}.{dot,svg,png}
embedded into docs/_source/concepts.rst — visual scaffolding for the
3-layer stack, 8-dim cache_key composition, and falsification state
machine (previously prose+tables only).
docs/_source/cookbook.rst gains a "Use arborist as a Python library"
section: open_store + ingest_documents, custom Source subclass,
audit-chain walk + verify, Merkle proof round-trip, programmatic
arborist.qa.query() with OpenAICompatibleClient + StubClient swap.
Every Python recipe smoke-tested against a scratch DB before publish.
make docs-api: 0 new warnings. make test: 2557 passed.
The §12.1 pipeline I added was second-hand from benchmark-matrix.md
and got several things wrong against the code:
- listed 4 verdicts; actual is 5 (missing FABRICATED — the
fabrication-vs-WRONG split that energy-cogs §5.5 leans on for the
qwen-fabricates / hermes-abstains finding)
- "θ=0.85" was right by accident — but it's the code-judge-pinned
_CODE_JUDGE_THETA_CONTRA constant, raised from the manifest 0.5
default after measuring 114 FPs in the 0.5-0.75 band
- omitted the short-answer entity-grounding fast path (which runs
BEFORE NLI per the 2026-05-19 Poland-Tusk smoke)
- omitted the HYBRID rescue ladder (NLI entail / entity rescue /
2026-05-21 verbatim-quote-on-topic rescue)
- conflated WRONG and FABRICATED (the subject-in-gold split is what
distinguishes "source has the topic but a different value" from
"source silent on the topic")
Rewrote §12.1 grounded in `bench/judge_code.py:judge()` (its own
docstring at line 501-528 is the truth on rule order), with file:line
citations and the verdict-mapping in full.
Also fixed a real artifact-vs-doc drift INSIDE the judge: the
module-top docstring still claimed θ_contra default 0.5 and omitted
the short-path and the WRONG/FABRICATED split. Updated to match the
authoritative judge() docstring + current code.
No behavior change — docstring + benchmarks doc only.
A stable façade so another Python app can use arborist as a
content-addressed / Merkle / audit-chained store without the CLI or a
wire protocol. Import from arborist.embed, not internal modules, so
refactors don't break embedders.
Surface: open_store(path), ingest_documents(conn, docs), search(conn, q),
plus re-exported Document/Edge/Source/Hit/IngestStats. Core only
(python+sqlite3) — no extras. _IterableSource adapts a plain doc iterable
into the Source contract.
This is the seam for using arborist as neopig's optional provenance
backend: neopig produces Documents from crawled pages, arborist gives
content-dedup (document_root) + FTS5 + an append-only audit chain
alongside neopig's existing md5/FileVault storage. Docs in
docs/embedding.md. 6 tests pin open/ingest/dedup/idempotence/edges/search.
New docs/crawler.md covering the crawl path we actually run: BFS
same-host discovery, robots/feed/sitemap handling, polite vs --fast,
the shared-session + no-HEAD + crawl-delay-fix speedups, and the
content-addressed payoff. Leads on the two store-derived diagnostics:
duplicate detection (group by document_root — body, not URI) and
partial-overlap (shared chunk leaves), plus orphan finding
(sitemap − BFS-reached) and the planned crawl-report webmaster tools.
Honest pros/cons: orphans invisible to crawl by design, single-host,
no JS execution, --fast is anti-social off your own turf.
Full --fast crawl of russell.ballestrini.net (242 URIs): 26s -> ~5s.
Three changes, biggest first:
1. fast_mode now actually ignores crawl-delay (the ~5x). The delay was
only zeroed on the robots-200 path; a site with no robots.txt (404)
or a robots fetch error fell back to default_crawl_delay (2s). Under
--fast that made every concurrent fetch wave sleep ~2s — ~10 waves
x 2s dominated the wall time. _enforce_crawl_delay now short-circuits
when fast_mode, matching the documented "ignore crawl-delay"
contract regardless of robots status. Disallow is still honored
(separate path).
2. One shared ClientSession for the fetcher's lifetime (keepalive TCP
connector sized to page-worker width) instead of a fresh session per
fetch — ~3x on a 24-page wave. Lazily built in-loop via _get_session;
the bridge closes it in a finally (guarded on owning the fetcher).
3. Drop the per-page preflight HEAD. aiohttp exposes response headers
before the body is read, so the existing content-type binary guard
skips images/video/audio without downloading them — the HEAD was a
redundant round trip that doubled per-page latency.
Diverges arborist's AsyncWebFetcher from the agents.ai.unturf.com/core
verbatim lift (fox-approved); candidate to upstream. Regression tests
pin fast=no-delay / polite=delay, shared-session lifecycle, and bridge
session teardown (owned vs injected).
Two crawler-discovery changes surfaced while chasing fast-crawl wall
time on russell.ballestrini.net:
1. Feed-skip in BFS discovery: the bridge fetched feed/sitemap URLs
(a multi-MB atom.xml among them) only for ingest_crawled to discard
them. Gate enqueue on the existing _looks_like_feed_url so we never
fetch crawl-infrastructure URLs — less wasted work and one fewer
slow wave straggler.
2. lxml link extraction, DRY'd: the three duplicated BeautifulSoup
html.parser closures (fresh fetch + 2 cache paths) collapse into one
module-level extract_page_links() backed by lxml.html (C parser,
releases the GIL so to_thread actually parallelises) with a BS4
fallback for markup lxml rejects. Parse on a 24-page wave 3.5s->2.5s.
Honest scope: neither moves full-crawl wall time much — measurement
showed the dominant cost is the per-page HEAD+GET double round-trip on
a per-call ClientSession, not parsing. These are correct-and-cleaner
on their own; the wall-time lever (shared session + drop redundant
HEAD) is a separate change. lxml extraction is regression-pinned
against the BS4 fallback for parity.
The bridge BFS fetched pages one-at-a-time, so --fast only dropped the
crawl-delay (sequential, zero-wait). Fast_mode's CPU*3 page-worker
budget never reached the path operators actually run.
Replace the popleft loop with a wave loop: each iteration pulls up to
`fetcher.max_page_workers` URLs off the queue front and fetches them
with asyncio.gather. Width is CPU*3 under fast_mode, 1 otherwise, so
the polite path stays byte-for-byte sequential and the per-page
crawl-delay still serialises same-domain fetches. Wave size is capped
to the remaining max_pages budget; dedup moves from pop-time to
enqueue-time so a URL linked from two parents in one wave is fetched
exactly once.
Measured on russell.ballestrini.net (own host, robots 404): same
12-page work 23.1s polite -> 4.0s fast (5.7x); full 243-page crawl
~25s vs the ~486s polite floor (19x). Disallow still honored; only
the rate limit is lifted.
Tests: peak-in-flight pins (>1 fast, ==1 polite) plus all existing
BFS bound / dedup / depth / max-pages cases on the width=1 path.
When a re-crawl detects a real content delta (a just-ingested root that
supersedes a prior version — content hash changed, not redeploy/ETag
noise the idempotent ingest already no-op'd), the pipeline now surfaces
the page's document chain over time instead of just 'something changed'.
bridge.py: version_chain(conn, uri) walks a URI's documents by ingest_ts
(each content change = new content-addressed doc + supersedes edge);
delta_report() adds the word-level similarity of the latest change;
render_delta_report() prints it. ingest_crawled() detects superseding
roots, emits the lineage report to stderr per changed page, and returns
'deltas' in its summary. Validated on the live russell.ballestrini.net
re-crawl: 223 pages, full redeploy, exactly 1 content change (/about/),
rendered as a 2-version chain (90% similar to prior). 2 tests; suite
2551 passed.
- '.. class:: center' parsed as a Python class named 'center' (duplicate
across one-pager + two-pager) -> '.. rst-class:: center' (styling, no
object). Clears the duplicate-object-description warnings.
- two v8-consensus section underlines were shorter than their titles
('underline too short') -> extended to title length.
Pager + v8 pages now build with zero warnings.
The page was a markdown+LaTeX paper dumped into .rst — 499 errors/warnings,
big chunks rendering broken. Mechanical, content-preserving fixes:
- markdown code fences (```lang) -> RST .. code-block:: (48 blocks; the
dominant error source — RST read each ``` as an unclosed inline literal)
- display math [ ... ] -> literal blocks (34)
- code-block:: json -> text where bodies are schemas with <...>/[m,n]
placeholders, not valid JSON (10; same as the v8 fix)
- escape inline | in prose (math like |Z|, |pred(v)|) read as RST
substitution refs (37 lines; no pipe-tables in the file, so safe)
Residual 62 are scattered indented-math derivation lines RST treats as
block quotes — they still render, just warn; diminishing returns on an
advanced/optional theory page.
The simple-table (=== separators) had a wrapped multi-line cell that
docutils rejected as malformed -> the floor table rendered broken on the
live site. Converted to a list-table (alignment-proof). Also switched the
proposer-submission block from 'code-block:: json' to 'text' — it's a
schema with <...> placeholders, not valid JSON, so the json lexer warned.
Page now builds with no errors/warnings on these blocks.
The Substrate group is Dav1d's formal Merkle-AGI research that drove the
design — valuable, but it sat third in the nav and read like required
reading. IA-only fix (no content touched):
- re-caption 'Substrate' -> 'Substrate theory (advanced)' + a preamble
marking it optional (nothing in Getting started / API depends on it);
- reorder it BELOW the practical sections (now: Summary -> Getting started
-> API -> Substrate theory -> Project);
- pull 'bench' out of Substrate into Getting started (it's the practical
benchmark surface, not theory).
Practitioner path comes first; theory stays intact + credited, just last.
- Frame the solution as a Reverse RAG (Merkle Providence Reverse RAG) with
a link to the whitepaper (unfirehose.com/merkle-providence-reverse-rag-
whitepaper — note: published on unfirehose, not uncloseai).
- New differentiator: NO vector embeddings — retrieval is lexical-first
(FTS5 BM25 + Merkle), dense-vector optional + off by default; embedding
10M docs costs 10-100x more/doc + a vector index to store/maintain. A
big part of why COGS is low.
- COGS framed per 1,000 answers, labeled @ $0.33/kWh (intro + diagram cost
node). Cost node clarified: no embeddings, NO reasoning (reasoning is the
thing that would cost 4-6x, which we skip). Retrieval node + mapping
table updated to lexical-first / no vector index.
ranksep=1.2 over-stretched it vertically (1193x1483, lots of whitespace).
Drop to 0.4 + nodesep 0.25: 1202x1063, ~28% less area, no vertical sprawl,
stays roughly square. Width is floored by the widest node label; click-to-
zoom (zoom.js) covers reading detail.
Graphviz renders SVG as <object>, which swallows clicks and can't be
lightboxed. Vendored, dependency-free zoom.js converts each graphviz
<object> to a clickable <img> (crisp vector) and adds a fullscreen
overlay: click to open, scroll to zoom toward the cursor, drag to pan,
Esc / dbl-click / background-click to close. Works for any img.zoomable,
so future charts get it free. Degrades gracefully (no JS -> inline image
still renders). Wired via html_css_files / html_js_files.
TB alone left it landscape (1202x1101, h/w 0.92). Width is floored by the
widest node label (can't shrink without trimming labels), so bump
ranksep to stretch vertically: 1193x1483, h/w 1.24 — clearly top-down/
vertical for the docs column.
rankdir LR -> TB so the diagram reads top-down and fits the docs column
(roughly square ~1200x1100, fox: 'square is good') instead of a wide
horizontal strip.
Graphviz DOT diagram of the arborist pipeline as the answer to the Google
L5 system-design prompt 'design a RAG pipeline for 10M docs with zero
hallucination'. Maps our components onto the canonical 10-box RAG design
and shows the three extensions that buy zero-hallucination + near-zero
cost: deterministic verifier (not a model confidence score) -> honest
UNGROUNDED; Merkle-bound cache that skips the GPU; measured energy COGS
(~$0.07-0.16/1k answers, non-reasoning). Includes the mapping table +
the 3.47M->10M scaling math (+77GB, sourcing/storage not redesign).
Enables sphinx.ext.graphviz (SVG output) + graphviz apt package on the
RTD build; adds the page to the Summary-pages toctree. Builds clean
(page renders, DOT validates via dot -Tsvg).
Main tweet + follow-up (Merkle cache hit skips GPU, doesn't increment the
per-1k). Numbers are the measured claim_lattice figures from the energy
report; note attached to hold the arbitrage/forcing-function framing until
the value side is hardened (higher N + blinded judge).
$0.10/1k-q overstated the qwen-27B case. Honest range: ~$0.07-0.16 per
1,000 queries of GPU electricity — hermes-8B $0.07-0.09 (under a dime),
qwen-27B $0.12-0.16 (over a dime; claim_lattice dearer than quote from
more prefilled context). Fixes the §5.4 'either rig' claim.
The report is now cost AND value (quality-per-dollar), not cost-only.
§5.5 quality delta: substrate-vs-solo (code judge, n=30) on numeral +
stale fixtures, hermes-8B vs qwen-27B-nothink. Substrate lifts both
2-13x and they nearly converge (hermes 18/21, qwen 19/26 grounded-
correct) — grounding comes from retrieval+verification, not parametric
size, so the base model matters far less. Bare-model failure styles
diverge (hermes abstains, qwen fabricates). Quality-per-dollar: ~18-26/30
grounded for $0.085-0.158/1k-q.
Honest caveats recorded: CORRECT=grounded-in-2010-corpus (not current);
the stale fixture is no longer post-corpus; the qwen thinking bug (39c040c)
that voided the first run and the judge strengthening (2d31866). §1
reframed; §8 repro + §9 next updated (post-2010 fixture, SOTA judge for
residue).
The code judge bailed to JUDGE_ERROR on 40% of in-corpus answers: HYBRID
(partial grounding) with low NLI entail, where the entity-grounding
rescue needs ZERO unsourced specifics. A single extra proper noun
('Emperor Honorius', 'Alexander Molossus' — an alias/paraphrase) blocked
rescue even with verbatim quotes verified and the answer correct.
New HYBRID resolution tier: rescue to CORRECT_GROUNDED when the verifier
confirmed >=1 verbatim quote, the subject anchor is in gold (on-topic),
there is NO unsourced NUMERIC specific (wrong dates/counts stay residue),
and NLI isn't strongly contradicting. Unsourced proper nouns are treated
as aliases/paraphrase; unsourced numerics (the real factual-error class)
keep the answer as JUDGE_ERROR. Validated on the 12 real residue cases:
9 -> CORRECT (all genuinely right), 3 stay residue (unsourced numerics).
JUDGE_ERROR 40% -> ~10%. self-test 4/4; 2 new tier tests; suite 2549.
