Five sections updated to match the post-2026-05-27 substrate state
(the user-payload-layout work, sibling ticket #000068, and the
2026-05-27 bench evidence are all in tree).
Verdict block (top): ADD: companion missed-answer guard -> DONE:
companion guard shipped as #000068 Phase 1+2+3 (default OFF; Phase
4 default flip NO-GO until wider bench + human spot-check).
Companion missed-answer guard section: renamed from "(proposed
sidecar)" to "(shipped 2026-05-27)". Carries the implementation
location (arborist/qa/inspect.py:diagnose_missed_answer), the full
output schema (diagnostic_version / confidence_class /
triggered_clauses / subject_tokens / missed_answer_candidate_spans
with offset_start/end/basis), the Phase 2 bench headline (2/228
fires, both Ballestrini, 100% precision, 0 FPs across 226 non-
Ballestrini), the Phase 3 demote-flag CLI surface
(--demote-on-missed-answer, default OFF), and corrected hash
discipline: Phase 1 sidecar fields fold into governance_policy_hash
only; the Phase 3 demote flag (answerability_demote_enabled) ALSO
folds into verifier_policy_hash because flipping it changes the
rendered audit_mode (a verifier-output property).
Future hardening list: "companion missed-answer falsification
guard" entry now points at #000068 instead of describing a
deterministic-sidecar to be built.
Roadmap Phase 5: "DESIGN OPEN" -> "DONE 2026-05-27" with all four
sub-phases of #000068 named individually (Phase 1 sidecar commit
2ab11d2, Phase 2 bench + Phase 3 demote commit ec55db5, Phase 4
default flip NO-GO per Dav1d 2026-05-27 §3.4). The #000068 phase
numbering is internal to that ticket; this roadmap names the
external-facing milestones.
Related links: past-tense the missed-answer guard ("when it lands"
-> "shipped 2026-05-27"); added cross-reference to
docs/tickets/ticket-000068-*.md; named both Dav1d review files by
path so a re-read can locate the inputs.
AUTOCOUNT tag (76 fixture-rows in bench/qa_questions.txt) still
matches; no test changes.
Phase 2 — bench instrumentation + measurement run
bench/qa_sweep.py picks up the answerability sidecar projection per row
(answerability_fired, answerability_confidence, answerability_denial_
pattern, answerability_answer_type, answerability_candidate_count) and
aggregates per-mode (answerability_fires + S/M/W confidence breakdown)
into a new column in the markdown summary table.
Measurement run on bench/qa_results/phase2-sidecar-on/2026-05-27T14-
16-22Z (76 questions × n=3 × claim_lattice × Hermes-3-8B × tail layout,
228 runs). Headline:
sidecar fires 2/228 (0.88%)
confidence dist 2 strong / 0 medium / 0 weak
precision 100% (2/2 fires were the Ballestrini fixture)
recall on Ballestrini 2/3 across n=3 (third run model extracted
correctly -> sidecar silent,
correct behavior)
false positives 0/226 non-Ballestrini runs
verifier verdict both fires labeled STRICT by the binary
verifier (the verifier-blind class, exactly
as predicted)
Detection rule's three-clause conjunction (denial + extraction-shape +
candidate proximity near cleaned subject tokens) is operating at the
precision floor. The strong-confidence-only firing pattern is what
calibrates Phase 3's demote threshold.
Phase 3 — opt-in demote flag (default OFF per Dav1d Phase 4 NO-GO)
arborist/qa/keys.py: answerability_demote_enabled added to
_VERIFIER_POLICY_FIELDS so flipping the flag partitions cache via
verifier_policy_hash. Justification: when on, the rendered audit_mode
changes (EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL), which IS a
verifier-output property; verifier hash must move accordingly. The
other answerability_* fields stay governance-only (sidecar
diagnostic, no audit_mode mutation).
arborist/cli.py:_render_audit_label extended with answerability +
demote_enabled kwargs. Logic:
demote_triggers = (
demote_enabled
and answerability["answerability_warning"] is True
and answerability["confidence_class"] in ("strong", "medium")
)
lattice modes:
EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL (rung transition)
POINTER-LINKED / ANCHOR-WARRANTED -> "rung · missed-answer"
(tail tag; rung itself already
signals degradation)
non-lattice modes (quote/span/entity/paraphrase):
audit_mode token unchanged + "· missed-answer" tail tag
weak confidence: NEVER demotes (Phase 2 saw zero weak fires on real
failures; reserved for future expanded detection ladder)
CLI flag --demote-on-missed-answer on both `arborist query` and
`arborist ask`, default OFF. Flows into call_policy[
"answerability_demote_enabled"] and through to result[
"answerability_demote_enabled"] so the renderer reads it without
needing the policy dict.
End-to-end verified live: 4 fresh Hermes-3-8B runs with --demote-on-
missed-answer on `songs by veronica ballestrini`, all 4 rendered
EVIDENCE-MISSED-PARTIAL · via claim_lattice (Hermes hit the failure
mode in all 4, sidecar fired strong, demote logic transformed the
label).
Phase 4 (default flip to demote-on) — NO-GO per Dav1d 2026-05-27 §3.4:
"a false sidecar warning is tolerable; a false audit-label demotion
can damage trust in correct abstentions." Phase 2 precision is 100%
but n=2 fires is too few samples to claim precision floor empirically.
Default flip blocks on wider bench + human spot-check of the warnings.
Tests: 47 total (36 Phase 1 + 11 new Phase 3 covering hash partitioning
discipline + render-label projection across all four rung/confidence
matrices). Full suite 2794 passed (delta +22 from prior 2772).
Bench output (bench/qa_results/phase2-sidecar-on/) intentionally not
committed — bench/qa_results/ is gitignored per existing convention;
the ticket carries the headline numbers + path for re-inspection.
Adds a deterministic read-only sidecar to detect a class of failure the
binary verifier is structurally blind to:
Evidence contains the answer.
Model says the evidence does not contain the answer.
Verifier sees no unsupported positive claim -> marks run clean.
User receives a false negative under EVIDENCE-WARRANTED.
The motivating case: "songs by veronica ballestrini" against the 2010
Wikipedia corpus. Hermes-3-8B under user_payload_layout=tail returned
"the specific songs by her are not mentioned in the provided evidence
blocks" when evidence E2 literally contained "Amazing", "Out There
Somewhere", "Fascinated", "What's Up With That", "Don't Say". Verifier
correctly returned EVIDENCE-WARRANTED 2/2 because the existing layered
verifier (quote / span / entity / paraphrase + Rule 8 title-relevance +
Rule 9 subject-tokens-absent + claim-count ceiling) guards unsupported
*presence*, has no hook for unsupported *absence*.
Layout fixes attention placement on the specific instance (the 5/27
n=3x75q bench confirms bookend/per_chunk recover Ballestrini); layout
alone cannot close the class -- adversarial phrasing or a bigger prompt
resurfaces the failure under any layout. The right substrate move is to
falsify "not mentioned" as a testable claim.
Detection rule (three-clause conjunction, all must fire):
A. Denial pattern in answer (sealed v1 phrase list: "not mentioned",
"not provided", "the evidence does not say", "does not mention",
"no specific", "no evidence", "cannot determine from the provided
evidence", "is not stated", "is not specified"). Casefolded +
whitespace-normalized substring match.
B. Question is extraction/list-shaped. Either a surface cue ("songs
by", "works by", "books by", "who wrote", "who composed", "what
year", "list of", "name all", ...) matches, OR the existing
arborist.qa.quantifier classifier returns intensity in {ALL,
COMPREHENSIVE, OPEN_REQUEST, MANY, PLURAL}.
C. Evidence contains candidate spans matching the answer_type within
a proximity window (default 600 chars) of cleaned subject tokens.
Candidate kinds aligned to answer_type:
title_like -> quoted_string, title_case_span, comma_list_item
person -> title_case_span
date -> year, date
Hardenings folded in from the 2026-05-27 Dav1d de-novo review:
1. Subject tokens strip cue/relation/stop words. For "songs by
veronica ballestrini" the cleaned subject is ["veronica",
"ballestrini"], NOT all four tokens. Without this the guard
false-triggers on "Harvard University" or "New York" near
proper-noun subjects.
2. Answer-type alignment. Candidate span kind must match query type
so "songs by John Smith" + evidence about Harvard/NY does not
strong-trigger.
3. Confidence class is deterministic (weak | medium | strong), not
boolean. Strong requires quoted_string near exact subject mention
+ multiple type-matched candidates. Phase 3 demote will gate on
confidence_class.
4. Cap output at 10 candidates (the per_chunk-quote-inflation
lesson). Prevents the guard becoming another claim amplifier.
5. Offsets are offset_start + offset_end + offset_basis=
"evidence_object_text", never an ambiguous single offset.
6. Cache-hit path returns answerability: None. Cached records do not
carry the evidence_map, only the rendered sources summary, so the
sidecar cannot recompute candidate spans without re-running
retrieval. Operators wanting fresh diagnostics use --burn.
7. Phase 1 stays out of verifier_policy_hash. The
answerability_sidecar_enabled / answerability_threshold /
denial_patterns_version / extraction_cues_version fields fold
into governance_policy_hash only. Phase 3 demote flag
(answerability_demote_enabled, default False) will move the
verifier hash WHEN ON because it changes the rendered audit_mode
(EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL).
Sidecar discipline (matches arborist.qa.inspect.diagnose_* sister
functions deflection / coherence / title_relevance):
- no model calls (no LLM-as-judge, no NLI, no translation)
- no audit_events writes
- no providence_cache writes
- no answer text mutation
- no claim promotion -- the trigger conjunction makes promotion
structurally impossible (only fires on denial answers)
- byte-deterministic: same (question, answer, evidence, policy) ->
same output every time
Result-dict integration: result["answerability"] is None when the
guard did not fire, or a structured diagnostic dict when it did
(diagnostic_version, confidence_class, triggered_clauses,
denial_pattern_matched, extraction_cue_matched, extraction_shape,
answer_type, subject_tokens, candidate_count, threshold,
missed_answer_candidate_spans). Read by bench_qa (Phase 2 will add
warning-count aggregate to bench rows) and CLI render.
Three return points carry the key:
- miss-path (full retrieval + verify): computed from evidence_map
- cache-hit: None (Dav1d cache-hit recompute discipline -- evidence
not stored, recompute requires re-retrieval)
- reject-broad early-return: None (no evidence examined)
Tests: 36 new pinning the three-clause logic, positive (Ballestrini)
regression, negative control (John Smith + Harvard/NY), each-clause-
alone silence, schema integrity, byte-determinism, sidecar-disabled
short-circuit, dict-shaped evidence support. All pass; existing
inspect tests (60) all pass.
End-to-end verified live via the CLI on the real corpus (2010 ~/.arborist
/shards): 3 fresh Hermes-3-8B runs on "songs by veronica ballestrini",
run 1 hit the failure (sidecar fired with confidence: strong, 351
candidates, denial: "not mentioned"), runs 2-3 model extracted
correctly and sidecar correctly silent.
Phase 2 (bench + threshold tuning) and Phase 3 (opt-in demote flag)
are open as follow-ups. Per Dav1d: NO-GO on default demote-on until
benchmark + human spot-check confirms low false-positive rate.
Full spec in docs/tickets/ticket-000068-verifier-blind-missed-answer-
guard.md (post-review hardenings section at top names the seven
load-bearing changes from the Dav1d 2026-05-27 review).
format_user_payload helper in arborist/qa/prompts.py becomes the single
source of truth for the user-turn payload. Three layouts:
tail (default) evidence first, question at end (prior behavior)
bookend question repeated before AND after evidence — counters
lost-in-the-middle on small models (≤8B)
per_chunk bookend + a one-line [for: <q>] reminder before each
evidence block; for list/extraction queries
USER_PAYLOAD_LAYOUTS constant exported; unknown layout raises ValueError.
The six _user_payload closures in query.py (3) and runner.py (3) all
delegate to format_user_payload. Quote-mode passes per_chunk_marker=None
to fall back to bookend on flat document/sources context.
Wired through both DEFAULT_QUERY_POLICY and DEFAULT_POLICY. Folds into
governance_policy_hash (the layout changes the user-turn content the
model sees, so the policy hash partitions cleanly per layout); does NOT
fold into verifier_policy_hash (verifier rules unchanged).
Makefile gets LAYOUT_DEFAULT ?= tail and LAYOUT ?= $(LAYOUT_DEFAULT) so
operators can flip per-call (LAYOUT=bookend make query Q="...") or
session-wide (LAYOUT_DEFAULT=bookend make query Q="..."). Recommendation
matrix in the Makefile comment block above the query target encodes the
2026-05-27 bench finding.
Motivating case: the Veronica-Ballestrini "songs by" failure. Hermes-3-
8B under tail layout returned "specific songs by her are not mentioned
in the provided evidence" when evidence E2 literally contained the song
names. Same query under bookend recovered the answer (with conflation
between Veronica Ballestrini and The Veronicas); under per_chunk
recovered AND disambiguated three entities. Qwen-27B unaffected by
layout. The Ballestrini case is added to bench/qa_questions.txt as a
regression fixture under "entity list", with a 4-line comment pointing
to docs/user-payload-layout.md.
2026-05-27 bench (n=3 × 75q, claim_lattice mode, Hermes-3-8B):
tail STRICT 94/225 (0.418) — control
bookend STRICT 95/225 (0.422) — +0.44pp (noise, 5pp floor)
per_chunk STRICT 72/225 (0.320) — -9.78pp (significant regression)
Verdict: tail stays default (cache-preserving and bench-confirmed
neutral). Bookend/per_chunk available as opt-in operator knobs. Per_chunk
regresses in aggregate because the per-chunk reminder over-anchors the
model on every chunk (TOO_MANY_EVIDENCE_IDS violations rose from 20 →
54; mean answer chars in 32-64KB bucket doubled from 720 → 1660). The
Ballestrini-class failure is real but rare across the curated set; a
layout fix that helps the rare case at the cost of 10pp aggregate is a
bad default trade. Documented in full in docs/user-payload-layout.md
along with the Dav1d 2026-05-27 review framing (GO for opt-in, NO-GO
for default promotion, ADD companion missed-answer guard).
CLI changes (--user-payload-layout flag on `query` and `ask`) landed
separately in commit e5ee283 alongside the #54 busy_timeout fix.
2026-05-27 v4-fixed bench on 3090: 3 of 4 parallel workers crashed with
sqlite3.OperationalError: database is locked at PRAGMA journal_mode =
MEMORY. Result: only shard 000 fully hydrated, shards 001/002/003 had
chunks rows landed but content NULL (~99.99% empty), no FTS data.
Peer functionally usable for only ~25% of corpus.
Root cause: parallel hydrate (xargs -P N) spawns N separate processes
that each open connections to all M target shards. Multiple processes
attempting PRAGMA journal_mode change on the same .db file at the same
instant serialize on a brief exclusive lock — without busy_timeout
SQLite throws BUSY immediately and the worker exception-propagates
out of _pull_pack_inner_routed before phase 2 (chunks fill) can run.
Fix: `PRAGMA busy_timeout = 30000` as the FIRST statement on every
hydrate writer connection. SQLite then waits up to 30s for any lock
instead of throwing — slowest worker gets its lock, fastest writes
go through immediately. No measurable cost when there's no
contention (busy_timeout is a wait, not a poll).
Set in TWO places:
arborist/cli.py:_cmd_cold_unpack
Immediately after connect(p), before any other PRAGMA. Covers
the PRAGMA foreign_keys = OFF that runs before the hydrate
pipeline.
arborist/evict.py:_pull_pack_inner_routed
Belt-and-suspenders: in case the function is called with
externally-built connections that didn't set busy_timeout, the
bulk-tuning loop sets it before journal_mode/synchronous/etc.
6 cold-unpack-routed tests pass (no contention in single-process
test fixture; the fix is invisible there). Real validation is the
next 3090 v5 bench against re-packed (#53 pre-sized chunks) bucket.
Real consumer-side bottleneck for cold-pack genesis is the phase-2
chunks-content UPDATE loop: each `UPDATE chunks SET content=? WHERE
chunk_id=?` grows the row from NULL to ~500 bytes, triggering SQLite
page splits, which become ext4 metadata-journal events. With 6M
chunks × 4 parallel writers, those journal events serialize and
dominate consumer wall time (~130 of the 164-min v3-revert run).
Fix: producer dumps a synthetic `_content_size` column in chunks.jsonl
carrying the on-disk byte length of each chunk's content (constant-
time SQLite `length(content)`). Consumer's phase 1 INSERT pre-allocates
the row with `content = bytes(_content_size)` instead of letting it
default to NULL. Phase 2's UPDATE then replaces same-size bytes
in-place — no row growth, no page splits, no per-row journal events.
Implementation:
arborist/cold_pack_metadata.py
_dump_generic_table for `chunks`:
Emit synthetic `_content_size` column = length(content) at the
end of the columnar header. Underscore prefix avoids collision
with any future schema column.
_restore_routed_table:
Detect `_content_size` in the JSONL header; if present (and
table == chunks), build INSERT against [chunks_cols] + ['content']
and substitute a bytes(_content_size) placeholder for the
content position. Phase 2 UPDATE later replaces those bytes.
Forward compatibility:
- Old packs (no _content_size): consumer uses today's NULL-content
INSERT path. No behavior change.
- New packs: consumer auto-detects, uses pre-sized path.
SPV-wallet trade-off:
In just-enough mode the consumer pulls only the metadata pack so
chunks land with the placeholder bytes (NOT NULL anymore). That's
a SEMANTIC CHANGE for SPV — `chunks.content IS NULL` no longer
means "JIT-fetch later." Documented in code; if SPV-mode JIT-fetch
ever ships, it must distinguish placeholder bytes (where every
byte is 0) from real content.
New test (TestPreSizedChunks.test_chunks_content_pre_sized_after_metadata_restore):
Hydrate just-enough → chunks rows have non-NULL bytes content of
correct size. Catches the regression if a future change reverts
the placeholder logic.
Expected impact: ~50-70% reduction in phase-2 wall time. Real number
lands when the next 3090 bench-max iteration runs against re-packed
bucket. 6 cold-unpack-routed tests pass.
v4 bench (2026-05-27 03:09 UTC) measured a corrupt outcome:
chunks_fts=1,561,604 on EVERY target shard regardless of chunks
count. _pull_fts_pack_into_targets had been iterating all M targets
and INSERT'ing each fts pack's shadow tables into every one of them.
Each fts pack's chunks_fts_docsize entries reference chunk_ids that
were independently auto-assigned in its source producer shard (each
source DB has chunk_ids 1..1.56M independently). Inserting all 4
packs into all 4 targets → 4× the per-target FTS rows, pointing at
chunk_ids the target doesn't own. Body searches would return garbage.
Fix: sample one id from the fts pack's chunks_fts_docsize, look it
up in each target's chunks table, INSERT only into the target where
it's found. The other M-1 targets stay untouched and receive their
FTS data from their corresponding fts pack(s) in later iterations.
In post-reshard production topology, each producer source shard's
docs all hash to ONE consumer target, so this 1:1 mapping is exact.
The test fixture is artificial (single-shard producer with docs
hash-distributed across M=4 targets) but still validates the core
property: only one target receives FTS data; the others stay empty.
FTS5 shadow tables aren't subsettable per-row (segment data is
opaque, mixed entries for many docs in one segment) so we can't
filter FTS rows to "only chunks that exist on this target" — we
copy all-or-nothing per pack. That's why the producer's post-reshard
shape (each pack scoped to one target's docs) is the structural
prerequisite for fts packs to make sense.
New regression test
(TestFtsPackRoutingRegression.test_fts_pack_only_on_owning_target):
exactly 1 of M targets has chunks_fts_docsize > 0; the others
must have 0. Pre-fix this asserted on all-4 targets having FTS
data → failed. Post-fix passes.
Returns now include owning_target_idx for forensic visibility into
which target the fts pack landed on.
5 cold-unpack-routed tests pass.
_cmd_cold_list only recognized .metadata.tar.zst and .chunks.tar.zst
key suffixes; .fts.tar.zst keys hit the else-continue and were
invisible. 2026-05-26 producer pushed 4 fts packs and cold list
reported 0 of them.
Trivial extension: add an `elif key.endswith('.fts.tar.zst')` branch
that classifies as kind='fts'. Now cold list shows all 3 kinds.
Also unblocks the make cold-hydrate target's metadata-pack discovery
filter (`kind=='metadata'`) from accidentally matching mis-classified
fts packs.
3090 bench-max v3 run at 13:25 elapsed had shards at 696-760 MB; v2
at 8:46 elapsed (default 4 KB pages) had 549-662 MB. Larger pages
made the chunk-content UPDATE workload SLOWER, not faster.
Reason: chunk-content payloads average ~500 bytes after zstd. With
4 KB pages each UPDATE rewrites a 4 KB page; with 16 KB pages the
same UPDATE rewrites a 16 KB page → 4× write amplification per
row. ext4 journal traffic increases roughly proportionally.
Bigger pages help SCAN-heavy workloads (fewer page reads to walk
a B-tree). For narrow-row-UPDATE-heavy workloads like cold-pack
chunk fill, they hurt. Comment in code captures the measurement
so a future operator doesn't re-attempt this.
The other bench-max measures stay in place:
- synchronous = OFF (per-write fsync removed)
- journal_mode = MEMORY (WAL in RAM)
- temp_store = MEMORY (sort scratch in RAM)
- cache_size = -524288 (512 MB page cache)
- mmap_size = 536870912 (512 MB read mmap)
- foreign_keys = OFF (no FK verify per row)
- deferred FTS5 rebuild post-pass
- batched 5000-row executemany UPDATEs
- bench-max #5 (this commit's revert): page_size stays at 4 KB
Next bench point: v4 against the new bucket packs that include
fts (--include-fts producer run in flight). That eliminates the
FTS rebuild step entirely on the consumer.
push_pack now does a HEAD object_size check against the bucket
before every PUT. If the object exists at the content-addressed
key AND Content-Length matches the local pack body size, skip the
upload. Applies to chunks / fts / metadata pack bodies.
Why content-addressed-key + size is the right check:
pack_hash = hash_leaf(manifest_bytes)
object_key = packs/<pack_hash>.<kind>.tar.zst
The pack_hash IS the cryptographic identity of the body via its
manifest. If the bucket has an object at this exact key, the
content is provably identical (otherwise the producer that wrote
it computed a different pack_hash → different key). Content-Length
match validates against a half-uploaded multipart that a prior
crash might have left.
Why not ETag: S3 ETag for multipart-uploaded objects is the MD5
of per-part MD5s concatenated, then MD5'd, with a "-N" suffix for
part count. Two clients using different multipart_chunksize end
up with different ETags for the same content. Unreliable for
cross-client equivalence. Content-Length is stable.
Result fields gain `skipped_reupload: bool` so callers / the
audit chain can tell whether the bucket PUT happened or was a
no-op.
Real-world impact for tonight's re-pack (--include-fts):
before: 4 chunks PUTs (7 GB re-upload, ~24 min)
+ 4 metadata PUTs (new hashes because manifest now has
_fts_pack_hashes — these MUST upload)
+ 4 fts PUTs (new)
~16 GB uplink, ~50 min wall
after: 4 chunks SKIPPED (same content-addressed key already
in bucket)
+ 4 metadata PUTs (new hashes, MUST upload)
+ 4 fts PUTs (new)
~6 GB uplink, ~20 min wall
cold_pending bookkeeping still fires in the upload path so a
half-uploaded crash recovers cleanly; skip path leaves no trace
because nothing was started.
4 cold-unpack-routed tests pass unchanged.
The 3090 genesis bench at 28-min progress mark showed all 4 workers
sitting in jbd2_log_wait_commit / do_get_write_access — ext4
filesystem journal serializing every page-extension event across
the 4 parallel writers. SQLite synchronous=OFF and journal_mode=
MEMORY skip SQLite's own fsync/WAL costs, but file growth still
goes through ext4's journal layer.
Fix: bump SQLite page_size from default 4 KB to 16 KB on each
hydrate target. Bigger pages mean ~4× fewer page writes during
chunk-fill, which means ~4× fewer file-extension events through
ext4's journal. Direct attack on the measured bottleneck.
page_size can only be set on an EMPTY database; VACUUM finalises
the change. The unpack CLI now opens each target file raw,
PRAGMA page_size = 16384 + PRAGMA secure_delete = OFF + VACUUM,
closes, then connect() applies schema as normal on the
pre-sized empty DB. Idempotent on re-runs since hydrate dirs
are always clean.
Also sets secure_delete = OFF (default is OFF in most builds but
explicit for hydrate writers): no overwrite-with-zeros on row
deletion, less write traffic during the chunk UPDATEs that might
free overflow pages.
Expected impact: ~30-50% reduction in chunk-fill wall time vs the
killed 30-min v2 run. Combined with the prior bench-max round
(journal_mode=MEMORY, deferred FTS, batched UPDATEs) and the
upcoming fts-pack-skip-rebuild path, full genesis should land
under 30 min total instead of the 75-min v2 extrapolation.
4 cold-unpack-routed tests pass unchanged.
Each producer shard now optionally emits a THIRD pack alongside its
metadata and chunks packs: an "fts" pack containing the FTS5 shadow
tables (chunks_fts_data, chunks_fts_idx, chunks_fts_docsize,
chunks_fts_config + the documents_fts_* counterparts) packed as a
fresh SQLite file inside the tar so BLOB columns round-trip natively.
Consumer detects fts_pack_hashes in the metadata pack's manifest,
pulls each fts pack, ATTACHes the embedded sqlite, INSERTs every
shadow-table row into its target's empty shadow tables, and SKIPS
the local FTS rebuild entirely.
Producer side:
arborist/cold_object.py
+ PACK_KIND_FTS = "fts"
+ FTS_SHADOW_TABLES tuple (8 shadow tables)
+ build_fts_pack(src_db_path, ...)
creates a temp sqlite, applies SCHEMA_SQL (so destination
has FTS virtual tables → shadow tables auto-created), copies
every shadow-table row from src via cursor iteration, packs
the sqlite file into tar.zst
+ ParsedManifest.fts_pack_hashes
+ parse_manifest reads _fts_pack_hashes records
+ build_metadata_pack accepts fts_pack_hashes parameter and
writes the new manifest record
arborist/evict.py:push_pack
+ include_fts: bool = True parameter (CLI --no-fts opts out)
+ Phase B.5 emits the fts pack BEFORE Phase C (metadata pack)
so its hash can be referenced in the metadata manifest
Consumer side:
arborist/evict.py
+ _pull_fts_pack_into_targets() — pulls fts pack body, extracts
embedded sqlite, ATTACHes into each target, INSERT OR IGNORE
every shadow-table row. INSERT OR IGNORE protects against
rowid collisions on other targets that don't own these chunks.
+ hydrate_from_metadata_pack_routed iterates fts_pack_hashes in
full mode, calls _pull_fts_pack_into_targets per pack
+ _pull_pack_inner_routed returns fts_pack_hashes_referenced in
its result dict (mirrors chunk_pack_hashes_referenced)
CLI / Makefile:
arborist cold pack --no-fts (opt-out)
make cold-hydrate (auto-detects: if
chunks_fts_data is
already populated
on shard 000 after
unpack, skip the
rebuild post-pass)
make cold-hydrate HYDRATE_REBUILD_FTS=1 (force rebuild)
make cold-hydrate HYDRATE_REBUILD_FTS=0 (skip rebuild)
Schema:
cold_pending.kind CHECK extended to include 'fts'
pack_key() accepts kind="fts" → packs/<hash>.fts.tar.zst
Expected wall-time impact on the 3090 genesis bench:
with fts in packs: no rebuild step → ~5-10 min total wall
without fts: rebuild post-pass needed → ~15-20 min
Trade-off: ~30-50% larger bucket (FTS shadow data per shard) for
~70-90% faster consumer hydrate. Producer flips the trade via
--no-fts. The fts pack is optional in the manifest (empty list →
consumer falls back to rebuild) so old bucket data without fts
packs continues to work unchanged.
34 cold-unpack-routed + migrate + planner tests pass.
Real-measured bottleneck during the 3090 genesis run (2026-05-26):
all 4 parallel workers sitting in jbd2_log_wait_commit /
do_get_write_access — fighting for ext4's single filesystem journal.
The reshard executor was fast because it ran FTS5 rebuild
SEQUENTIALLY (one process per target in turn); the parallel 4-way
unpack collapsed that into journal contention.
Two coupled changes:
1. arborist/evict.py — _pull_pack_inner_routed
* Drop the _rebuild_fts_on_target call from phase 2c entirely.
Parallel inner loop now does: download → metadata route →
chunks fill. NO FTS5 writes during the hot parallel phase.
* Bulk-load PRAGMA tuning on every target connection:
synchronous=OFF no fsync per commit (already had this)
journal_mode=MEMORY WAL in RAM, not on disk (was WAL)
temp_store=MEMORY sort scratch in RAM
cache_size=-524288 512 MB page cache per connection
mmap_size=536870912 512 MB read mmap
Genesis crash recoverability = re-pull from bucket, so
durability of intermediate state has no value — these tunings
trade durability for throughput.
2. arborist/cli.py — new `arborist cold rebuild-fts --shards-dir DIR`
subcommand. Sequentially rebuilds chunks_fts + documents_fts on
every shard, one at a time. Each shard gets the full filesystem
journal in its turn. Total wall = sum of single-shard FTS
rebuild times, NOT 4× contention.
3. Makefile — `cold-hydrate` chains the FTS rebuild automatically
after the parallel unpack (only when HYDRATE_MODE=full, since
just-enough has no chunk bodies to index anyway). Operator can
skip with HYDRATE_REBUILD_FTS=0 for a 2-pass workflow.
Expected wall time on 3090:
parallel hydrate (download + chunks fill): ~5-10 min
serial FTS rebuild × 4 shards: ~5-15 min total
total: ~15-25 min
vs the killed 2.5-hour run.
34 existing cold-unpack-routed + migrate + planner tests still pass
(the tests' FTS check runs against the reshard path, which still
calls _rebuild_fts_on_target inside the executor — that path is one
process, no parallel contention).
Two genesis paths the SPV-wallet design supports, now both exposed
via the same make target:
make cold-hydrate HYDRATE_DIR=… HYDRATE_M=4 HYDRATE_MODE=full
pulls metadata packs + every referenced chunk pack. Full corpus
offline-queryable post-hydrate.
make cold-hydrate HYDRATE_DIR=… HYDRATE_M=4 HYDRATE_MODE=just-enough
pulls only metadata packs (~4 GB on a 37 GB corpus → ~10×
bandwidth reduction). chunks rows land with content=NULL, every
chunk-body query misses local cache and falls through to a
future JIT-fetch path (cache miss → CDN / mesh / re-pull). The
SPV-wallet headline shape.
Defaults to full for backward compatibility. The mode-flag validation
guards against typos at recipe time (invalid mode → exit 2 before
firing parallel boto3 traffic).
Bench target: measure both paths back-to-back on 3090. Just-enough
should bound the network phase only (no chunk-body fill, no FTS5
rebuild); full adds the chunk-restore phase that #000067's bench-
max round just optimized.
Real-world measure 2026-05-26 18:43 UTC: hydrating an 11 GB
post-reshard corpus (3.47M docs / 6.24M chunks) into a fresh 4-shard
peer ran at ~32 MB/min per shard — extrapolating to ~4 hours total,
the same speed as a fresh XML ingest. The whole point of a cold-pack
restore is being MUCH faster than re-ingest; killed mid-flight and
shipped these three optimizations:
(1) Build leaf_hash → (target_idx, chunk_id) map once up front
from each target's chunks table (M scans, total ~6M rows).
Replaces M=4 SELECTs per incoming chunk_body — was 24M
lookups, now 6M scan-once. ~4× win on the dispatch step.
(2) Batched executemany UPDATE per target (BATCH=5000) — replaces
the per-row `with transaction(target): UPDATE; INSERT_FTS`
pattern that opened 6M tiny transactions. ~10× win on disk
write throughput.
(3) Skip chunks_fts during the fill loop entirely; defer to one
bulk rebuild per target after all chunks are in place. Reuses
the existing arborist.migrate._rebuild_fts_on_target primitive
(decompresses via unpack_chunk, streams chunks → FTS via
batched executemany). Each per-row chunks_fts insert costs an
inverted-index update; bulk rebuild is ~10× faster than
incremental.
Plus PRAGMA synchronous=OFF on each target connection: genesis is
end-to-end rebuildable (a crash mid-hydrate leaves empty shards we
re-pull from the bucket), so durability of intermediate WAL pages
is not required. WAL stays bounded by PRAGMA wal_checkpoint(TRUNCATE)
at the end of phase 2b (already done implicitly by _rebuild_fts_on_target
in phase 2c).
Expected speedup: ~30-60× combined. Real number lands when the
re-run on 3090-ai.foxhop.net completes.
All 34 cold-unpack-routed + migrate tests pass unchanged.
Adds two missing pieces to the cold-pack lifecycle:
1. ALLOW_LICENSE_CLASS pass-through on cold-pack and cold-pack-all.
Without this, packs from a corpus containing any `unknown`-class
docs (html crawls / textbooks) refuse to push to a public bucket
per #000061 Gap 2. Real-world bench on 2026-05-26 hit this
because the production corpus has crawl + textbook content.
Usage:
make cold-pack-all ALLOW_LICENSE_CLASS=unknown
make cold-pack DB=~/.arborist/shards/000.db \
ALLOW_LICENSE_CLASS=unknown
2. cold-hydrate — M-aware genesis-from-bucket as a make target.
Discovers every metadata pack in the bucket via `cold list
--no-manifest` + a tiny python json filter, then fires
$(JOBS)-way parallel `arborist cold unpack` workers (one per
metadata pack) into the same target shards directory. Each
worker writes to one disjoint target shard because the bucket's
packs come from the post-reshard #000065 source layout (each
producer shard's docs all hash to one consumer target) — so
zero write contention across the parallel workers.
Usage:
make cold-hydrate \
HYDRATE_DIR=~/.arborist/shards-genesis-test \
HYDRATE_M=4
Why these belong here (not in /tmp wrapper scripts that get tossed):
fox 2026-05-26 — "make sure the pack and upload and the pull and
ingest all happen with makefile targets". The cold-pack lifecycle
is a first-class operations surface; throwaway tmp scripts hide
the real interface from future operators.
Both targets are listed in `make help`. The .PHONY line covers
cold-hydrate so a directory named cold-hydrate can't shadow it.
Sequence to repeat today's run end-to-end:
producer: make cold-pack-all ALLOW_LICENSE_CLASS=unknown
consumer: make cold-hydrate HYDRATE_DIR=~/.arborist/shards \
HYDRATE_M=4
Today's in-flight 3090 hydrate (started before this commit) used
the equivalent shell loop; from now on every operator uses the
make target.
Self-contained benchmark for the SPV-wallet validation. Records:
PRODUCER bucket state + pack count + compressed bytes
CONSUMER wall time, exit status, post-hydrate shard sizes,
per-shard documents/chunks/edges counts
Driver runs against the real DO Spaces bucket + the real fresh peer
on 3090-ai.foxhop.net. Writes one JSON artifact per run to
bench/results/cold-pack-roundtrip-<ISO>.json so a future operator
can diff hydrate times across pack-format changes (#000061 v3 →
graft mode #000066 → mesh-pull future).
Consumer command uses /usr/bin/time -v wrapped around
`arborist cold unpack --hydrate-shards-dir … --hydrate-M 4 --full`.
Hydrates into ~/.arborist/shards-genesis-test/ so it doesn't
clobber anything on the 3090.
No new ticket — this is task #45/#46 instrumentation. Existing
tests untouched.
Today's hydrate_from_metadata_pack writes every row into one shard.
With the corpus in M=4 hash-routed topology (#000065), a fresh peer
pulling packs must land each doc on shard_for_document(root, M) —
same routing function as the producer — or the consumer's M=4
ATTACH-and-route assumption is just decoration over a single-shard
reality.
Code (new entry points alongside the existing single-conn ones):
arborist/cold_pack_metadata.py
+ restore_shard_metadata_routed(targets, M, table_dir)
+ _restore_routed_table (per-document tables)
+ _restore_edges_fan_out_routed (edges by src_root)
+ _shard_idx_for_root helper (mirrors arborist.document)
+ routing rules: _ROUTED_BY_COL / _CONSOLIDATED_TO_SHARD_0
(mirror migrate.py's ROUTED_BY_DOCUMENT_ROOT / CONSOLIDATED_TABLES)
arborist/evict.py
+ hydrate_from_metadata_pack_routed(targets, backend, hash, M=, mode=)
+ _pull_pack_inner_routed (mirrors _pull_pack_inner; phase 2
chunk-body fill iterates every target shard — leaf_hash lookup
naturally hits at most one since each chunk's metadata row
landed on exactly one target during phase 1)
arborist/cli.py
arborist cold unpack
+ --hydrate-shards-dir DIR (M-aware genesis path)
+ --hydrate-M N (default 4 matches #000065)
legacy --db / --global-shards-dir path unchanged
Behaviour notes:
- FK enforcement off on target writes (cross-shard refs are valid
under hash routing, same fix as #000065 reshard executor)
- audit_events lands on target 0 (Option A consolidation)
- corpus-wide tables (snapshots / concepts / aliases / providence)
consolidate to target 0
- per-document tables route by document_root (documents,
document_http_meta, chunks, merkle_nodes)
- edges route by src_root (matches migrate.py)
- derivations route by core_root (matches migrate.py)
- existing single-conn API unchanged — callers that didn't pass a
shards-dir get the legacy single-shard behaviour
4 new tests:
test_pack_then_hydrate_routed — end-to-end pack → hydrate → assert
every doc on its hash-routed target, no doc on the wrong shard
test_corpus_shard_count_set_on_routed_hydrate — meta plumbing
test_M_mismatch_rejected
test_empty_targets_rejected
All 59 prior tests still pass.
Refactor opportunity (not taken): _ROUTED_BY_COL duplicates
migrate.py's ROUTED_BY_DOCUMENT_ROOT. A shared arborist/multi_shard.py
module would serve both reshard and graft (#000066). Left as
follow-up since the duplication is small and graft is still scaffold.
Unblocks #46 genesis on 3090: that's now a single arborist cold unpack
--hydrate-shards-dir ~/.arborist/shards --hydrate-M 4 invocation
instead of the α two-step kludge (hydrate-then-reshard).
Open ticket. Today's hydrate_from_metadata_pack takes one conn and
writes every incoming row into one shard — fine when the corpus
was a single shard, broken now that #000065 put the producer in
M=4 hash-routed topology. A fresh peer pulling packs must land each
doc on `shard_for_document(document_root, M)` — same routing
function as the producer — or the consumer's M=4 ATTACH-and-route
assumption is just decoration over a single-shard reality.
Plan:
1. Add corpus_shard_count to pack manifest (read from source meta
during dump_shard_metadata) — pack carries the M it was built
against.
2. restore_shard_metadata_routed(targets, M, table_dir) in
cold_pack_metadata.py — mirrors _route_per_doc_table from
migrate.py (per-document tables route by document_root /
src_root / core_root; consolidated tables all go to target 0).
3. hydrate_from_metadata_pack gains a targets / shards_dir param.
4. arborist cold unpack --shards-dir DIR initialises M target
shards from the manifest's corpus_shard_count and routes.
5. Regression test: pack 2 shards → hydrate into fresh 4 shards
→ assert every doc on its hash-routed target.
Refactor question (raised, not decided): the routing rules
(ROUTED_BY_DOCUMENT_ROOT, CONSOLIDATED_TABLES) currently live in
migrate.py. Either duplicate them in cold_pack_metadata.py (fast)
or factor into arborist/multi_shard.py (cleaner, also serves
#000066 graft mode). Shared module is more honest.
Prerequisite for #46 (genesis on 3090 from cloud). Without this,
genesis is a 2-step α-kludge (hydrate-then-reshard) that wastes
~30 min and treats packed shards as if from an arbitrary topology.
Index entry bumped; next-id 67→68. Per-ticket spec doc to follow
when the implementation gates open.
2026-05-26 19:47 UTC. ~94 min wall. 3,468,226 globally-unique docs +
6,235,588 chunks + 90,592,990 edges + 3,468,403 audit events re-routed
from non-deterministic spray-by-ingest-order layout to canonical
content-hash M=4 layout (shard_idx = int(document_root[:8], 16) % 4).
Final state:
per-shard doc uniformity within ±0.04% (theoretical max ±0.05% for
first-32-bit SHA-256 prefix)
audit chain consolidated to shard 000 via Option A (re-sorted by ts,
re-chained; bodies preserved unchanged; tail event type=reshard
carries plan+result body, hash 8da3aa19…)
on-disk sizes: 000=11.0 / 001=8.8 / 002=8.8 / 003=8.8 GB
validation: chunks delta 176 (0.003%) + edges delta 547 (0.0006%)
are cross-shard dupes from re-ingest history, collapsed by
INSERT OR IGNORE; within the 1% tolerance gate
smoke queries: Barack Obama / YouTube / Albert Einstein all
returned proper evidence from correct (hash-routed) shards
chain-check-shards: 0 breaks on every shard
Two mid-flight defect fixes (also committed):
04edff7: derivations.src_root FK guard fired on legitimately
cross-shard refs → writer connection PRAGMA foreign_keys = OFF
c86d5ac: WAL accumulated ~37 GB across passes (SQLite auto-checkpoint
blocked by open reader cursors) → _checkpoint_truncate between
phases. Production migration was rescued mid-flight by manual
sibling-connection wal_checkpoint(TRUNCATE) freeing 27.7 GB.
Closes#000065 in both the index and the per-ticket file. The
per-ticket design doc stays open as a design-log artifact (its
content is still the right reference for the next reshard / for
graft mode #000066).
Follow-on tracked as tasks #44–#47:
#44 re-pack post-reshard shards into DO Spaces (current bucket
packs are stale, still in pre-reshard topology)
#45 verify bucket pack hydration is deterministic against the new
content-hash layout
#46 genesis a fresh peer on 3090-ai.foxhop.net from cloud — first
real SPV-wallet end-to-end test
#47 retire stale pre-reshard bucket packs after #46 confirms
Add PRAGMA wal_checkpoint(TRUNCATE) at two points in
_execute_all_at_once so committed WAL pages don't pin disk through
subsequent passes. Production migration on 2026-05-26 hit 7 GB free
disk (down from 89) because SQLite's auto-checkpoint can't run while
a reader cursor is open, and FTS rebuild keeps a SELECT cursor open
through 1.5M chunks per target. Across 4 targets the FTS rebuild
plus audit consolidate held ~37 GB of committed-but-unreclaimed WAL.
Manual sibling-connection wal_checkpoint(TRUNCATE) freed 27 GB
mid-migration.
Checkpoints land at:
* end of _rebuild_fts_on_target (after the SELECT cursor is
explicitly cur.close()'d so the TRUNCATE checkpoint can actually
fire — TRUNCATE/RESTART block on active readers)
* end of _consolidate_audit_chain (after the 3.47M-row giant
transaction commits, before the next phase touches the same
connection)
VACUUM is already implicitly a checkpoint, so the existing per-
target VACUUM pass continues to handle the final checkpoint
naturally.
Helper _checkpoint_truncate(conn) returns the (busy, log_frames,
checkpointed) tuple SQLite emits; for the serial executor, busy=1
is improbable since each phase finishes before moving on.
Regression test
(TestWalCheckpointing.test_no_large_wal_after_migration) asserts
no WAL file exceeds 4 MB after migration completes. Without the
checkpoint calls this would fail on real-sized corpora; with them
the test passes deterministically.
Doesn't affect the running migration (it loaded the module from
memory before this commit). Future reshards run with bounded WAL —
no near-ENOSPC scares.
The 2026-05-26 cutover crashed mid-build with sqlite3.IntegrityError
"FOREIGN KEY constraint failed" inside _route_per_doc_table on the
derivations table.
Root cause: derivations.src_root carries a FK to
documents.document_root, but under content-hash routing a derivation
row's src_root can legitimately reference a surface doc that hashes
to a DIFFERENT target shard than the derivation's core_root. The FK
is a single-shard-era guard; it must stay live for the runtime
write path (to catch typo'd inserts into the wrong shard) but must
be OFF for the migration writer which legitimately produces
cross-shard refs.
Fix: arborist/migrate.py _connect_target now applies
`PRAGMA foreign_keys = OFF` after SCHEMA_SQL executescript runs.
Schema's own `PRAGMA foreign_keys = ON` still applies to the schema
DDL pass (and runtime connect() / connect_query() still get FK=ON
since they don't touch this helper). Only the migration writer is
relaxed. Documented inline.
Regression test
(TestCrossShardForeignKeys.test_cross_shard_derivation_succeeds)
synthesizes a derivation row whose core_root and src_root hash to
different M=4 target shards, runs the migration, asserts the row
lands on core_root's target with src_root pointing cross-shard. Pre-
fix this raised IntegrityError; post-fix it passes.
Originals untouched on the production host — the executor crashed
BEFORE the atomic-promote step, so .db files are intact;
~/.arborist/shards/00X.db.new files from the failed run will be
cleared by the next attempt's "stale .new before opening" cleanup
hook (already in _execute_all_at_once).
Scaffold-only ticket. Captures the architecture for taking the
#000061 cold-pack format and adding a second mode beside hydrate:
overlay an existing pack onto a populated shard set ("graft").
Surfaced while running the #000065 reshard cutover and fox extended
the design: each pack carries a `corpus_name` field
(wikipedia-2010, wikipedia-current, arxiv-cs, ...), making
`arborist cold graft wikipedia-current` feel like `apt install`.
Three concerns analysed:
doc/chunk/edge overlay trivial (INSERT OR IGNORE on content-
addressed PKs collapses dupes)
FTS5 overlay trivial (new chunk_ids → new fts rows)
audit chain overlay the only hard part — three approaches:
A graft receipt (chosen): one event in
host chain carrying pack_hash +
event_count + first/last hashes; pack
file is the durable witness; zero
schema cost; aligned with v8/v9
witness pattern
B re-chain everything: rejected — graft
is frequent so invalidating external
refs is wrong tradeoff (different
story from the one-time reshard)
C chain forest with chain_id col: right
answer when graft dominates lifecycle
but premature now
Long-game payoff: mesh-peer-corpus-merge. Two peers diverge over a
partition, each carries packs the other lacks, reconciliation =
exchange + graft what's missing. Makes "mesh of arborists"
coherent rather than "fleet of arborists."
Scaffold gated on (a) #000065 lands+stabilises, (b) a second
corpus exists to graft, (c) at least two peers want to exchange.
No code until then; the design lock is what the ticket buys.
Index bumped Next ID 000066→000067.
Reshard no longer creates a parallel "shards.v2/" directory. Builds
write to "<target_dir>/00X.db.new" alongside originals; when the
executor's validation passes, each .db.new is atomically renamed to
its final 00X.db name via os.replace (POSIX-atomic per file).
For an in-place migration (target_dir == source_dir, which is the
canonical use case), the rename REPLACES the original shard files.
~/.arborist/shards/ never contains a parallel "v2" or "next" or
"backup" directory — it always holds exactly one corpus, just with
the new topology after promote completes.
Validation gate (refuses to promote on mismatch):
* --expected-row-counts <snapshot.json> threads the pre-migration
snapshot's documents/chunks/edges totals through to the executor.
* Tolerance: ±1% to absorb the dupe-collapse from INSERT OR IGNORE
on cross-shard duplicate document_roots (166 dupes measured
pre-migration; <0.005% drift).
* Mismatch → RuntimeError, .db.new files left in place for
inspection, no rename performed.
CLI:
arborist corpus reshard --to 4 \
--source-dir ~/.arborist/shards \
--target-dir ~/.arborist/shards \
--audit-events-ndjson /tmp/audit-events.ndjson \
--expected-row-counts bench/results/pre-migration-snapshot.json
3 new tests:
* test_target_dir_only_has_db_files_after_completion — no .db.new
sidecars survive a successful run
* test_inplace_reshard_overwrites_originals — target_dir ==
source_dir works end-to-end; final files are the new hash-routed
shards
* test_validation_failure_leaves_new_files — bad expected counts
trip the validation guard; .db.new files survive; no .db files
promoted
Stale .db.new files from a prior failed run are unlinked before
opening fresh targets, so a partial-fail-and-retry is idempotent.
WAL/SHM sidecars removed at promote time so the new live .db
produces fresh sidecars on next open.
Build the 'all_at_once' executor and wire it into the CLI as
`arborist corpus reshard`. Plan-only preview against fox's host
confirms the planner picks all_at_once at 95.5 GB free, 64.2 GB
peak draw, 31.3 GB free at peak (well above the 4 GB safety).
Executor (arborist/migrate.py):
* ROUTED_BY_DOCUMENT_ROOT — documents, document_http_meta, chunks,
merkle_nodes, edges (by src_root), derivations (by core_root).
Each row hashed to shard_for_document(root, M) and INSERT'd into
the chosen target.
* CONSOLIDATED_TABLES — snapshots, concept_relations,
concept_token_idf, citation_aliases, term_aliases,
providence_cache, falsifications all land on canonical
target shard 000.
* REBUILT_ON_TARGET — chunks_fts + documents_fts rebuilt from the
materialized data after content moves. chunks.content is
zstd-packed at rest so the rebuilder decompresses via
arborist.compress.unpack_chunk before inserting plaintext into
the FTS5 index.
* Audit chain consolidation (Option A) — all rows from
/tmp/audit-events.ndjson are sorted by (ts, src_shard, src_seq),
re-chained with fresh event_hash values, and INSERT'd into
target shard 000. Bodies preserved unchanged for forensic
fidelity.
* One 'reshard' audit event appended at the tail, carrying the
plan + result as the body. An operator months later can answer
"where did this corpus topology come from" from this one row.
* corpus_shard_count meta stamped on every target shard.
* VACUUM each target at end to reclaim INSERT-pattern fragmentation.
CLI: `arborist corpus reshard --to M --source-dir DIR --target-dir DIR
[--plan-only | --force-strategy X
| --allow-in-place | --dry-run]`
11 integration tests build a tiny 2-shard corpus, run the migration,
verify per-doc routing, chunk-follows-doc, audit chain integrity,
reshard event at tail, corpus_shard_count meta on every target,
chunks_fts searchability, doc count preservation, dry-run no-op,
error handling.
Also: get_meta() now indexes by position so callers without
row_factory=sqlite3.Row don't trip. No behavior change for callers
that DO use Row factory.
Strategy B (per_source_shard) and C (streaming_row) are stubbed in
the planner (peak-draw estimators wired) but execute_plan raises
NotImplementedError for them. Not needed at fox's current disk
(88+ GB free); skipping the build until that's ever the constrained
path.
Disk-aware strategy picker for the corpus reshard. Pure-compute API:
feed it (corpus facts, free bytes) and it returns a HydrationPlan
naming one of three strategies + a peak-draw estimate.
Strategies (preference order):
all_at_once - direct source→target with overlap; fastest;
peak draw = corpus + WAL × M_target + VACUUM
per_source_shard - pack→delete→hydrate per round; safest;
peak draw = pack + WAL + 1 target VACUUM
streaming_row - in-place mutation, --allow-in-place opt-in;
peak draw = WAL only
Safety budget: 4 GB margin above the strategy's peak draw — the
buffer that survives one bad sort + one badly-sized WAL grow.
Production-host preview at 95 GB free / 38 GB corpus:
strategy: all_at_once
peak draw est: 64.2 GB
free at peak: 31.2 GB (well above safety)
rationale: all_at_once: peak draw 64.2 GB + 4.3 GB safety ≤ free 95.4 GB
The plan's full readout (strategy, free bytes, peak draw, rationale)
is JSON-serializable so it goes into the migration audit event as a
single forensic record.
14 tests cover: strategy pick by free-disk level, --force_strategy
override, --allow-in-place gating, skewed shard sizes, json-
serializable audit body, target_M validation.
No execution yet — this is just the planner. Next: the strategy A
executor (direct source→target read+write).
Three pieces, all read-only or additive — no shard mutation, no
schema-version bump:
1. shard_for_document(document_root, M) in arborist/document.py.
Pure function: int(document_root[:8], 16) % M. 22 tests cover
determinism, range-bounds, near-uniform distribution (±5pp at
N=20k), and seven lock-in fixtures so peers will disagree
loudly if anyone changes the formula.
2. corpus_shard_count meta field + get/set helpers in store.py.
Lives in the existing key/value meta table; SCHEMA_VERSION
stays at v9.8.0 (the DDL doesn't change and source_root is
layout-independent, so cache records survive a reshard).
Legacy shards (without the field) return None; reshard tool
populates it on every target shard at migration time.
3. Pre-migration snapshot captured to
bench/results/pre-migration-snapshot.json:
docs 3,468,392 (3,468,226 globally unique)
chunks 6,235,764
edges 90,593,537
audit 3,468,403
This is the reference set post-reshard row counts must match.
4. Audit-event extraction script writes all 3.47M events from
all 4 shards to /tmp/audit-events.ndjson (2.0 GB) for the
Option-A canonical-chain consolidation step. Verifies chain
integrity on extract — all 4 source chains report 0 breaks.
5. Fixed a wrong chunk count in docs/corpus-history.md
(had ~3.54M/shard; actual is ~1.56M/shard) and added the
edge-count column (~22.6M/shard, 90.6M total). 6.24M chunks
total, not 14.12M.
Tests: 29 new pass (22 routing + 7 meta). No existing tests
touched.
The original ingest of Wikipedia 2010 into fox's 4 production shards
was 2026-04-27 21:33-23:13 UTC — ~100 minutes wall, 4-way parallel,
3.47M docs / 14.12M chunks. Captured here because the audit chain is
the only durable record but querying 3.47M rows to recover the
headline number is friction; one line in a doc removes it.
Also informs #000065 reshard planning: ingest rate ceiling on real
XML workload is ~2,350 chunks/sec aggregate (4-way), vs the
~6,400 chunks/sec the M-sweep bench measured on the 2003 cur dump
(which skips XML parsing). The teleport-style reshard should beat
both ceilings because it's just SQLite INSERT throughput, no XML
parse + canonicalize + edge extraction.
New file: docs/corpus-history.md. Indexed in CLAUDE.md docs section.
Append-only convention; future migrations + cold-pack runs add
entries here so the operator log isn't only in the audit chain.
Derivation query (sqlite3 audit_events) embedded in the entry so
future re-derivation is one copy-paste.
Pinned the canonical shard count at M = 4 based on real-Wikipedia
ingest + query benchmark (bench/shard_count_sweep.py). Captured the
"when does SQLite stop being the right substrate" decision tree so
future operators know what bench would justify a fork or replacement.
Bench numbers (Wikipedia 2003 cur dump, 2000 docs, 4 cells of
M ∈ {1, 2, 4, 8}, 50 FTS queries per cell):
M chunks/s q_p50_ms q_p99_ms attach_ms
1 3,648 0.05 0.20 1.61
2 5,649 0.03 0.19 3.24
4 6,405 0.07 0.30 9.00
8 6,959 0.03 0.28 9.65
Key observations:
- M=1→M=2 is the biggest ingest win (+55%). Most gain happens there.
- M=2→M=4 is +13%. M=4→M=8 is only +9% — diminishing returns.
- Real wikitext canonicalization is per-worker Python CPU bound, not
SQLite-writer-lock bound. More shards don't unlock more CPU.
- Query p50/p99 is flat across M within noise (50 queries small).
- ATTACH cost grows linearly: 1.6 / 3.2 / 9.0 / 9.7 ms.
Why M=4 specifically:
- Captures 92% of peak ingest throughput (6,405 / 6,959).
- 6 ATTACH slots free under SQLite's 10 ceiling for aux DBs
(qa.db, snapshots.db, selfmodel-chain.db, crawl_*.db, future
mesh_*.db) — comfortable headroom. M=8 leaves only 2 slots.
- Mobile-tolerable: phone NAND attach is 5-10x slower than NVMe;
M=4 = 45-90 ms cold start (instant), M=8 = 50-100 ms (sluggish
with no headroom).
- Matches fox's current 4-shard layout = cheapest migration.
Decision tree for when SQLite stops being right (full text in
ticket §"When the SQLite-default substrate stops being right"):
A. ATTACH ceiling pressure (auxiliary DBs grow past 5) → bench
forked SQLite with SQLITE_MAX_ATTACHED=125, M ∈ {16, 32, 64};
if attach cost stays linear past M=10, fork viable but pays
permanent "no longer stock sqlite3" tax.
B. Ingest hits >10k chunks/s sustained ceiling → first tune
page_size / WAL checkpoint / mmap_size / synchronous. If
tuning gets 2-5x, stay on SQLite. If still ceiling-limited,
candidates: DuckDB (columnar, MVCC, FTS), libmdbx (B+tree no
FTS; we'd build it). In-house DB rejected without specific
failure of those.
C. Federation needs multi-writer-same-shard → SQLite writer-lock
serializes peers, becomes federation bottleneck. First try
leader-election (single-writer-per-shard with WAL replication
to followers). If true multi-writer required, SQLite is wrong;
candidates: FoundationDB, CRDT-on-KV-store. DuckDB does NOT
solve this — its MVCC is single-process.
Honest verdict: for current arborist workload (single-writer-per-
shard, read-mostly federation), stock python3 sqlite3 is the right
substrate. None of A/B/C are close to firing. The bench discipline
exists to know what to measure when something changes.
bench/results/shard-count-sweep-2026-05-26T16-20-48Z.csv (synthetic
baseline) + 2026-05-26T16-31-34Z.csv (real Wikipedia) committed as
the load-bearing measurement for the M=4 choice.
fox caught the imprecision: "the shards are teleported if we fix this
because we know the shard a doc will end up in."
Re-ingest = re-read source files, re-canonicalize, re-chunk, re-hash.
Hours-to-days from Wikipedia dumps.
What's actually needed = content-addressed rebalance. Every row in
the corpus is already addressed by content (document_root, leaf_hash,
merkle_nodes.hash, audit_event_hash). Migration reads existing rows,
computes new_shard_idx via the routing function, INSERTs into the new
M shards. No source-file work, no canonicalization, no LLM.
Updated the ticket's "Migration story" section with the per-table
teleport recipe:
documents: move by document_root → new shard
chunks (+content): follow doc to its new shard, re-assign chunk_id
chunks_fts: rebuild per new shard from chunks.content
merkle_nodes: move with document_root
edges: move by src_root
concept_relations, move by content-derived parent
derivations,
providence_cache,
citation_aliases,
term_aliases:
audit_events: ALL → shard 000 (canonical), re-numbered + re-hashed
snapshots, mesh_*: canonical shard 000
Audit chain handling: chose Option A (canonical shard 000) over
Option B (split-by-subject_root). Preserves global event ordering,
re-hashing happens once at migration time. The alternative loses
cross-shard event ordering for falsification + replay reasoning.
Estimated wall: ~20-40 min I/O-bound on current corpus
(35 GB across 4 shards → ~35 GB across 8 shards, moves once).
Hard discipline added: move into ~/.arborist/shards.v2/, keep
originals as rollback, atomic dir-swap when verified. Mistake in
routing function = corrupted federation, recoverable only by going
back to the originals.
Surfaced while sizing #000061's federation story. Today shard count
conflates two roles:
- producer ingest parallelism (wants = vCPU count)
- consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10)
On default Python sqlite3, the ATTACH ceiling is 10 and can't be
raised without a custom sqlite3 build (which violates CLAUDE.md's
"python3 + venv + sqlite3 is enough" property). Producer with 16
shards → consumer fails to attach the 11th, federation silently
breaks.
Design: introduce M = canonical shard count (corpus-wide constant,
default 8) decoupled from N = ingest workers. Document → shard
assignment becomes content-deterministic:
shard_idx = int(document_root[:8], 16) % M
Same input → same output across every peer. Today's "spray by ingest
order" is non-deterministic across peers — two peers re-ingesting
the same corpus put the same document_root in different shards. That
weakens federation more than it should.
Migration hard-constraint (fox: "this implies we will need to
reprocess all our data into shards"): re-ingest required. Current
layout is sprayed by ingest order; post-ticket is sprayed by content
hash. Two layouts are incompatible by construction. Captured in
ticket §Migration as the load-bearing operational note.
Phases laid out (0-4: design lock → read path → ingest path →
pack-restore → corpus migration tool). Open audit-chain re-numbering
question (per-shard event_hash chains break when rows rebalance
across shards). Kept as one ticket — do-not-proliferate.
Scaffold only. No code yet — design lock first.
Next ID bumped to 000066.
The 3.15 GB-per-shard index I added in bc7efe4 bloated every live
shard 33% to optimize a transient dump-time operation. Wrong trade —
the shards exist for arborist's query/retrieval/falsification path,
not for cold-pack's convenience. fox: "wtf you added an index to
fucking make the shard bigger?"
Refactor: instead of building a permanent on-disk index to avoid the
sort, configure the dump connection to spill the sort to a temp file:
conn.execute("PRAGMA temp_store = FILE")
conn.execute("PRAGMA cache_size = -10000") # 10 MB
SQLite's sort still happens, but it spills to /tmp instead of growing
the process heap. cache_size caps the page cache the sort works
inside. No permanent disk cost; transient temp disk only during the
dump phase.
DROP INDEX idx_edges_dst_uri_type_anchor + VACUUM ran across the 4
live shards before this commit (out-of-band ops):
000.db: 12.48 → 8.75 GB (~3 GB reclaimed)
001.db: 12.43 → 8.71 GB
002.db: 12.53 → 8.78 GB
003.db: 12.51 → 8.76 GB
total: ~12.6 GB reclaimed across the corpus
For a freshly-restored shard from a metadata pack, the index never
gets created (it was only in _dump_edges_fan_in, not in SCHEMA_SQL),
so consumers get the lean version from day one.
33/33 cold + evict tests pass.
The license_class gate landed in 576cb0e but only push_pack() the
function takes allow_license_class — the CLI couldn't pass it through.
Adds --allow-license-class {public_redistributable,unknown,private}
to `arborist cold pack`. Default stays at public_redistributable;
operators override only after confirming bucket ACL + source license
permit redistribution.
33/33 tests pass.
Dav1d's reviews of #000061 (Response A + Response B/FINAL in
~/Downloads, 2026-05-26) flagged a long list of items — most already
shipped in the SPV-split work. Three were genuine gaps worth folding
into #000061 before close:
Gap 1: manifest/latest pointer for new-peer discovery.
A fresh peer doing `cold list` got a list of metadata-pack hashes
but no obvious "which one is current for shard X." Added
get_latest_pointer + update_latest_pointer to the backend ABC.
push_pack writes manifest/latest.json on every successful metadata
pack push (read-modify-write keyed by snapshot_root). Mutable
pointer; content addressing of the packs themselves preserves the
trust root. Last-writer-wins on contention.
Gap 2: license_class field + producer-side refuse for public buckets.
Maps documents.source_type to a license bucket (wikipedia_cur /
textbook_tex → public_redistributable; html / grok / vcs → unknown;
anything else → unknown). Strictness order: public < unknown <
private. compute_shard_license_class() walks DISTINCT source_type
in documents. push_pack now refuses to upload if the shard's
strictest license is more restrictive than the operator's
allow_license_class (default: public_redistributable). The
metadata pack's manifest carries _license_class so consumers /
auditors can see the producer's classification without inspecting
source documents. ValueError on refusal — the bucket ACL is the
operator's call, but arborist refuses to participate in a
licensing/membership leak unless explicitly opted in.
Gap 3: cold_pending table for resumable uploads.
Killed mid-upload, push_pack left orphan multi-GB tempfiles in
/tmp with no DB trace. Added schema:
CREATE TABLE cold_pending (
tempfile_path TEXT PRIMARY KEY,
pack_hash TEXT NOT NULL,
kind TEXT NOT NULL,
backend_endpoint TEXT NOT NULL,
backend_bucket TEXT NOT NULL,
object_key TEXT NOT NULL,
started_at INTEGER NOT NULL,
state TEXT NOT NULL DEFAULT 'pending'
);
push_pack INSERTs a row before each upload + DELETEs on success.
A killed process leaves the row pointing at the orphan tempfile;
a recovery script (future) reads cold_pending, checks bucket for
the object, either deletes the row + tempfile (success was just
unreported) or re-uploads from the tempfile if it still exists.
Matches the same pattern as the audit chain — explicit state
rows beat inferring from chunks.content IS NULL.
Sibling tickets opened for the larger items the reviews flagged
(scaffold-only, no code; opening them captures the design in the
log without proliferating, per CLAUDE.md):
- #000063 Cold-object private-ciphertext mode (mesh-keyed object
keys for non-public corpora on public-read buckets). Needs mesh
group-key ABI + real non-public corpus before code.
- #000064 Cold-object operations toolkit (verify / diff / doctor /
repair-fts / gc-plan CLI + expanded audit-event taxonomy).
Bundled so the audit-event vocabulary gets one design pass.
5 new tests:
test_gap2_license_gate_refuses_unknown_class_to_public_bucket
test_gap2_license_class_in_metadata_manifest
test_gap1_latest_pointer_resolves_metadata_pack_per_snapshot
test_gap3_cold_pending_clears_on_successful_upload
test_gap3_cold_pending_records_inflight_upload
26 cold-object + 7 evict tests pass (33/33 green incl. boto3 wire).
Next ID bumped to 000065.
Live v3 SPV corpus run (bmq47x6t3) completed cleanly during this work.
Will report sizing + memory profile in the next message.
Three doc-only housekeeping items from Dav1d's de-novo reconciliation
of the architecture stack (no code changes; the active build is
#000061 cold-pack work, which is unaffected by this review):
1. Accepted-error formula. Dav1d corrected p_raw × (1-d)(1-r) to
p_raw × (1 - dr) where r is conditional on detection. Searched
docs/ and arborist/ — the wrong form does NOT appear in this
tree (it lives in the external recursive-truth-maintenance / RCO
manuscripts Dav1d cited). Nothing to fix here; recorded for the
manuscript authors.
2. #000060 H-ABCDEFG split. Folded the M/C/X axis split into the
harness scope: M (mechanism — does the substrate's plumbing work),
C (capability — does it improve task performance), X (external
adversarial — does it generalize outside author-designed fixtures).
ACCEPT requires clearing all three. Without X, the harness risks
self-validating benchmark theology — passing tests its own designers
picked. Tagged at fixture-definition time; aggregator emits per-axis
pass rates + combined ACCEPT verdict. Doesn't change the existing
BatteryResult row schema.
3. #000062 Mechanistic Witness — new scaffold-only ticket. Specifies
a content-addressed MechanisticWitnessRoot over (model_config,
capture_policy, contrastive_prompts, features/neurons, intervention
result, behavioral delta, safety policy) as a DIAGNOSTIC sidecar
feeding SelfModel + benchmark fixtures. Four hard guardrails:
diagnostic-only by default; sandbox intervention only; no production
steering without governance ACCEPT via #000060 M+C+X; feature labels
never become semantic proof. No code until a real falsifier use case
exists + guardrails are CLAUDE.md rules + #000060 harness gates
promotion. The dual-use risk (Pan et al. 2025: 0.1% MLP ablation
breaks refusal in 72B models) makes governance-first framing
load-bearing.
Next ID bumped to 000063.
No code change to arborist/. The in-flight v3 SPV corpus pack
(bmq47x6t3) continues unaffected.
Previously: COLD_PACK_JOBS hardcoded to 4. Caused the v3 corpus run to
fail at fork time on a memory-pressured box (4 GB available, each worker
needed ~5 GB peak → kernel swapped, SQLite executescript exceeded
busy_timeout, errored as "database is locked"). Killing the run + waiting
for memory to free → 2-way run succeeded.
Now: COLD_PACK_JOBS is auto-computed from `free -m` at run time. Reads
available RAM, subtracts headroom for OS + buff/cache, divides by the
observed per-worker peak, caps at COLD_PACK_JOBS_MAX. An explicit
override (COLD_PACK_JOBS=N on the make invocation) still wins for
operators who know better.
Tunables:
COLD_PACK_PER_WORKER_MB 5500 observed peak from real corpus run
COLD_PACK_HEADROOM_MB 2000 keep this RAM for OS / other procs
COLD_PACK_JOBS_MAX 8 ceiling regardless of RAM headroom
Same logic applied to cold-pack-all-dvd (local-dir output for burning).
Both targets print the chosen concurrency + the math behind it before
launching workers. If the auto-computed value is 1, that's a signal to
wait for RAM to free before trying parallelism.
Doesn't fix the underlying fragility of using fork-time RAM availability
to predict peak demand — workers grow over their lifetime as metadata
dumps materialize — but it's a real improvement over a static cap that
ignored system state entirely.
Bidirectional sync: producer always emits both kinds; consumer chooses
how much to pull.
packs/<hash>.metadata.tar.zst — one per shard (~0.6 GB compressed)
packs/<hash>.metadata.manifest.ndjson
packs/<hash>.chunks.tar.zst — N per shard (each ≤ 4.4 GB cap)
packs/<hash>.chunks.manifest.ndjson
Each artifact is independently content-addressed by its own manifest
hash. The metadata pack's manifest carries _chunk_pack_hashes — every
chunk pack covering this shard's content — so a "full" consumer can
iterate them. Chunks packs are anonymous from the consumer side
(reachable only via the metadata pack's reference list).
Consumer sync modes:
arborist cold unpack <metadata_hash> # default: just-enough
arborist cold unpack <metadata_hash> --full # also pulls chunks
just-enough: pull only the metadata pack. Schema fully restored; every
chunks row has content=NULL. Node is immediately queryable for
metadata operations (documents, edges, audit chain, Merkle); chunk-body
queries return null until a future JIT-fetch path fills them on cache
miss.
full: after the metadata pack lands, iterate _chunk_pack_hashes and pull
every chunk pack. Final state: full corpus offline-queryable.
This is fox's original SPV-wallet framing — was right from day one.
Key API changes:
pack_key(hash, *, kind="chunks", manifest=False) — kind in bucket path
stream_packs(chunks, *, max_compressed_bytes, ...) -> Iterator[FilePack]
now emits chunk-only packs (no extra_members)
build_metadata_pack(table_files, *, snapshot_root, chunk_pack_hashes, ...)
single FilePack with v3 metadata manifest
pull_metadata_pack(conn, backend, hash) -> dict
pull_chunk_pack(conn, backend, hash) -> dict
hydrate_from_metadata_pack(conn, backend, metadata_hash, *, mode) -> dict
push_pack orchestrates: dump tables → stream chunk packs (collect hashes)
→ build metadata pack with chunk_pack_hashes → upload all. Returns
{metadata_pack_hash, chunk_pack_hashes, packs: [...]}.
Manifest format v3:
metadata pack:
{"_format_version": 3}
{"_kind": "metadata"}
{"_snapshot_root": "..."}
{"_chunk_pack_hashes": [...]}
{"table_file": "tables/X.jsonl", "hash": "...", "size": N} per table
chunks pack:
{"_format_version": 3}
{"_kind": "chunks"}
{"leaf_hash": "...", "size": N} per chunk
pack_hash for each = hash_leaf(manifest_bytes); content-addressed at
both layers. Two writers with the same shard state produce identical
metadata_pack_hash AND identical chunk_pack_hashes.
Cap-and-split applies only to chunk packs (chunks are bounded N).
Metadata pack is one file per shard; if a single table file is larger
than the cap, that's noted as future row-level split work.
cold list now surfaces kind per artifact (metadata vs chunks), plus
table_count + chunk_pack_hashes (for metadata packs) or chunk_count
(for chunks packs). Operators can quickly find the metadata pack hash
to feed `cold unpack`.
28 cold-object + evict + boto3 tests pass (3 new SPV-shape tests + 1
v3 manifest-shape test + tampered-metadata-pack test).
Caught a real defect on the live v2 corpus run: pack_hash was computed
from the chunk-only manifest, so two packs with the same chunk set but
different table content collided on pack_hash. Observed live: v1 packs
and v2 packs for the same shard produced the SAME pack_hash:
v1 (chunks-only) b3c427baaf9b40e3c33ace8438b4254e3abdc7b38bdcb7b1a8012588cb32848a
v2 (tables + chunks) b3c427baaf9b40e3c33ace8438b4254e3abdc7b38bdcb7b1a8012588cb32848a
↑ identical, different bytes
Consequences if shipped: bucket overwrites swap whose tables you get,
mesh peers A and B disagree on what "pack X" is while both claim to
have it, content-addressing story is broken for the metadata half.
Fix: manifest format v2 prepends a `{"_format_version": 2}` header,
then one row per shipped table:
{"table_file": "tables/audit_events.jsonl", "hash": "...", "size": N}
{"table_file": "tables/chunks.jsonl", "hash": "...", "size": N}
...
{"leaf_hash": "...", "size": N} # chunks come after
{"leaf_hash": "...", "size": N}
pack_hash = hash_leaf(manifest_bytes) now binds the full pack content.
Two writers with the same shard state produce the same pack_hash; same
chunks + different tables produce different pack_hashes.
Implementation:
- hash_file_leaf(path) — streaming sha256 with leaf 0x00 prefix, 1 MB
reads, used to fingerprint multi-GB tables/<name>.jsonl files without
loading them into RAM.
- stream_packs' extra_members now takes (member_name, source_path,
content_hash) tuples. push_pack computes the hash with hash_file_leaf
after the dump finishes.
- _finalize_pack writes the format header + table refs (sorted by name
for determinism) before chunk entries.
- parse_manifest returns a ParsedManifest (format_version, tables[],
chunks[]) instead of just list[PackEntry]. Backward-compatible with
v1 manifests (no header → format_version=1, tables=()).
- pull_pack pulls the manifest aside during tar walk, then after
extraction verifies each tables/<name>.jsonl file via hash_file_leaf
against the manifest's content_hash. Mismatch → ValueError, no rows
reach the live schema.
New tests:
- test_v2_pack_hash_binds_table_contents — asserts manifest format +
table refs + per-table hash & size.
- test_pull_pack_rejects_tampered_table — corrupts one table's bytes
in a real pack, confirms pull_pack raises on hash mismatch.
27 cold-object + evict tests pass.
Live v2 corpus run showed ~5 GB RSS per worker — RssAnon dominant, so
process heap, not mmap. Traced to two unfixed memory pits in the edges
fan-in dump path:
1. SQLite ORDER BY on edges (22M rows, no covering index for the v2
sort order dst_uri+edge_type+anchor) allocates a multi-GB in-memory
sort area before spilling. Adding:
CREATE INDEX IF NOT EXISTS idx_edges_dst_uri_type_anchor
ON edges(dst_uri, edge_type, anchor, dst_root, src_root)
means the ORDER BY walks the index in order — no in-memory sort.
First create takes ~30-60 s on a 22M-row shard; idempotent on
subsequent dumps. Disk cost ~1 GB per shard (4 shards × 1 GB ≈
2-3 % corpus footprint increase). Worth it.
2. Python groupby accumulator: src_roots = [row[4] for row in group]
materializes the entire src_root list per destination. For
en.wikipedia.org/wiki/* destinations with millions of inbound links,
this list is itself ~GB-sized. Switch to bounded batches:
_FAN_IN_BATCH = 10_000 # max src_roots per fan-in JSON row
A destination with N inbound links splits into ceil(N / batch) rows.
Restore path (INSERT OR IGNORE) handles multi-row destinations
correctly because PK includes src_root — accidental duplicates
collapse cleanly.
New regression test test_edges_fan_in_batches_huge_destinations builds
an edges table with FAN_IN_BATCH+137 rows pointing at one dst_uri,
verifies the dump produces the expected number of split rows and the
restore reconstructs all N edges with no loss or duplication.
25 cold-object + evict tests pass.
The extra_members handling in stream_packs did:
with open(source_path, 'rb') as f:
source_bytes = f.read() # loads entire file into RAM
info = tarfile.TarInfo(...)
info.size = len(source_bytes)
state['tar'].addfile(info, io.BytesIO(source_bytes))
For a 4.3 GB edges.jsonl this is 4.3 GB of RAM per worker — exactly the
failure mode the prior streaming-to-disk refactor was supposed to
eliminate (57f8989). 4-way parallel would hit ~17 GB peak RAM, the same
OOM-risk number we saw on the v1 corpus run.
Fix: build the TarInfo by hand (size from .stat(), other fields zeroed),
then pass the open file handle directly to tar.addfile. tarfile reads
the body in chunks (~16 KB at a time) and feeds them into the zstd
stream_writer. Peak RAM per worker stays at ~tens of KB regardless of
metadata file size.
Determinism preserved: mtime/uid/gid default to 0 (NOT taken from the
filesystem via gettarinfo), so two writers' packs converge byte-for-byte
regardless of when they ran or who owns the temp file.
24 cold-object + evict tests pass.
v1 packs (chunks-only) were under-engineered: a new peer landing on
v1 packs would have chunk bodies indexed by leaf_hash but no documents
table, no audit chain, no merkle interior, no edges — couldn't actually
hydrate. fox: "isn't what I wanted you under engineered..."
v2 packs ship every load-bearing shard table alongside chunk bodies in
the same tar.zst:
manifest.jsonl # chunk catalog (unchanged)
tables/documents.jsonl # array-per-line columnar JSONL
tables/chunks.jsonl # without content column
tables/merkle_nodes.jsonl
tables/edges.jsonl # FAN-IN restructured
tables/audit_events.jsonl
tables/derivations.jsonl
tables/concept_relations.jsonl
tables/concept_token_idf.jsonl
tables/providence_cache.jsonl
tables/citation_aliases.jsonl
tables/term_aliases.jsonl
tables/snapshots.jsonl
tables/document_http_meta.jsonl
blobs/<hash[:2]>/<hash[2:]> # raw UTF-8 chunk bodies
Two compression strategies inside the pack:
1. Array-per-line JSONL ({"_columns": [...]} header line + ["v1","v2",...]
data lines) drops ~30% of uncompressed bytes vs object-per-row JSONL.
zstd recovers most of that on its own, but smaller uncompressed
footprint also speeds up stream-restore.
2. Edges fan-in restructure at pack-build time: 22M rows of
(src_root, edge_type, dst_root, dst_uri, anchor) → ~500k unique
(dst_uri, edge_type, anchor, dst_root) groups with src_roots as an
array. ~5-10x compressed savings on the dominant table. Reverses on
unpack into the per-edge live schema. Live queries unchanged.
NOT shipped (per-peer state): mesh_*, selfmodel_*, capital_ledger,
memory_*, controller_events, fork_score_branches, adapter_loss_reports,
falsifications, schema_meta, meta. NOT shipped (rebuildable): chunks_fts*,
documents_fts* — restored from chunks.content + documents.title on
unpack.
push_pack no longer appends `cold_pack_pushed` to the audit chain.
That event leaked into the next push's audit_events.jsonl dump and
broke the "two writers at the same corpus state produce identical
pack_hash" determinism property. The bucket/disc file IS the receipt;
the snapshot_root pinned inside the pack metadata binds it to a corpus
state. No load-bearing consumer of the audit row.
pull_pack restored to handle both v1 (chunks-only) and v2 (tables +
chunks) packs. For v2 it extracts tables/*.jsonl to a temp dir,
calls restore_shard_metadata (which INSERT OR IGNOREs into the live
schema and expands edges back to per-edge rows), then fills chunk
content for every leaf_hash in blobs/. Idempotent against populated
DBs (INSERT OR IGNORE all the way down). Self-cleaning temp dir.
Sizing measured 2026-05-26: ~2.1 GB per shard pack compressed (chunk
content 1.78 GB + metadata ~0.3 GB), ~8.5 GB total across 4 shards.
~20% more than v1 chunks-only for self-sufficient hydration.
24 cold-object + evict tests pass (+1 new test_push_pack_v2_hydrates_fresh_empty_db
that builds a pack from a populated DB and unpacks into a completely
empty DB to verify all tables restored). Full suite: 2558 passed,
28 skipped, 1 xfailed.
The in-memory BytesIO design held a whole 4.4 GB pack in RAM per worker
until finalization. 4-way parallel hit ~17 GB peak RAM — observed live
on the full-corpus run (21 GB used, 0 free, 3 GB swap in use, headed
toward OOM territory). Architectural defect, not just an inconvenience:
parallelism is capped by RAM rather than CPU/network.
stream_packs now:
- Opens a NamedTemporaryFile in work_dir (default tempfile.gettempdir()).
- Wraps it with ZstdCompressor.stream_writer + tarfile mode "w|".
- After each chunk: writer.flush(FLUSH_BLOCK) + file.flush(), check
file.tell() against the cap.
- On finalize: write the manifest tar member, close everything, yield
a FilePack carrying the path (not the bytes).
- On exception: close + unlink the in-flight temp file (no /tmp litter).
ObjectStoreBackend gains put_file(key, path). S3 impl uses boto3
upload_file which streams parts from disk during multipart upload —
memory bounded to max_concurrency × part_size = 10 × 8 MB = 80 MB
regardless of pack size. MemoryBackend reads the file into its dict
(tests only).
push_pack consumes FilePack:
- --push: backend.put_pack_file(path), then unlink the temp
- --local-dir: shutil.move from temp to local_dir/arborist-pack-<hash>
(cheap if same filesystem; one-pass read+write if not)
- both: shutil.move first, copy to bucket via put_pack_file from final
local path? — actually current code moves to local_dir
AFTER the bucket put; bucket put still works from the
temp path before move. Audit row carries pack.body_size
from FilePack (no len(bytes) on a 4 GB body).
Worker memory bound goes from O(pack_size) to O(zstd_buffer + tar_header)
≈ tens of MB. 4-way parallel now uses ~tens of MB total instead of
~17 GB. Disk requirement: work_dir needs (parallelism × cap) of free
space — 4 × 4.4 GB = 18 GB for a default full-cap parallel run.
23 passed in tests/test_cold_object.py + tests/test_evict.py.
Shards are independent SQLite files; the ORDER BY landed in 6f0ceab
makes each shard's chunk-to-pack assignment deterministic. Two new
Makefile targets fan out cold pack across every numbered shard via
xargs -P:
make cold-pack-all # parallel push to S3
make cold-pack-all-dvd LOCAL_DIR= # parallel write to disk (no S3)
Tunables:
SHARDS_DIR defaults to ~/.arborist/shards
SHARDS_PATTERN defaults to [0-9][0-9][0-9].db (numbered shards only,
skipping qa.db / snapshots.db / crawl_*)
COLD_PACK_JOBS defaults to 4
Expected wall-clock improvement: ~4x on full-corpus runs (4 shards
running in parallel saturate either CPU or network before serializing).
Also verified end-to-end pack → list → unpack round trip against the
live DO Spaces NYC3 endpoint (small pack, 20 chunks): pushed, listed,
unpacked into a fresh receiving DB, content restored byte-identical,
hashes matched.
New peers landing on a public bucket need to enumerate pack_hashes
before they can `arborist cold unpack <hash>`. `cold stats` only gives
a count; `cold list` returns the per-pack details:
- pack_hash (taken from the key)
- compressed_bytes (one HEAD round-trip via new object_size method)
- chunk_count (parsed from the small manifest sidecar; skippable
via --no-manifest for huge-bucket fast listing)
`cold stats` also now reports total bucket footprint (sums HEAD sizes).
Backend ABC gains `object_size(key) -> int | None` so both subcommands
get sizes without paying egress for the body. S3 impl uses HEAD;
MemoryBackend reads from the dict.
Verified live against DO Spaces NYC3: list-empty → push tiny pack →
list-with-manifest (pack_hash, size=5311 B, chunk_count=5) → list
--no-manifest → cold stats → cleanup.
23 passed in tests/test_cold_object.py + tests/test_evict.py.
Three improvements after the first DO Spaces smoke + bench:
1. ORDER BY c.leaf_hash on the chunk-selection SQL. Two writers running
cold pack against the same DB at the same snapshot now produce the
same pack_hashes — chunk-to-pack assignment is a function of (chunk
set, cap) and nothing else. Prerequisite for parallel per-shard pack
workers and for two replicas to converge on byte-identical bucket
state. Costs ~25% on build wall (real-bench 31s → 40s on 100k chunks)
due to sort over the leaf_hash index + documents JOIN; worth it.
New test pins the determinism property.
2. boto3 multipart upload via TransferConfig (8 MB threshold + 8 MB
parts + 10-way concurrency) on every put. Required anyway for packs
> 5 GB (DO Spaces single-PUT limit). Measured 5.5 MB/s → 9.0 MB/s
on 121 MB pack to DO Spaces NYC3 (1.6x; ceiling is closer to network
than to boto3 serialization).
3. Stream the SQL cursor in push_pack instead of fetchall(). At 14M
chunks × ~700 bytes/row the prior fetchall materialized ~10 GB of
Python heap before stream_packs ever ran. Cursor iteration bounds
memory by the in-progress pack (~few hundred MB at the 4.4 GB cap).
Full corpus extrapolation revises ~125 min (single-PUT) → ~104 min
(multipart, sequential per-shard). Real wins live in parallel per-shard
pack workers — deferred; the determinism work landed here is the
prerequisite.
22 passed in tests/test_cold_object.py + tests/test_evict.py.
Ship arborist corpus state to new peers and DVD-R archival via
point-in-time tar.zst packs. One artifact serves both channels —
bucket+CDN delivery and physical-media archival.
Bucket holds packs only. Pack key = hash_leaf(manifest_bytes), so same
chunk set on two writers produces the same pack_hash and upload is
idempotent. Each pack pins the corpus snapshot_root it covers in audit
+ result body — packs are delayed snapshots, not live mirrors;
falsifications between repacks produce new pack_hashes.
stream_packs runs streaming zstd over tarfile, peeking compressed-buffer
size after each chunk via FLUSH_BLOCK (preserves dictionary). Default
cap 4_400_000_000 — 4.4 GB DVD-R safe-fit, ~6.5% buffer below the
4.7 GB marketing capacity to absorb ISO9660 overhead, growisofs
lead-in/lead-out, media variance, and drive-edge refusal. Each disc
fills to ~4.4 GB recorded data, not the ~1.5 GB an uncompressed cap
produced.
One backend class (S3CompatibleBackend via boto3 + endpoint_url) covers
AWS S3, DO Spaces, R2, B2, GCS S3-interop, MinIO. Optional dep
[object-store] = boto3>=1.34; dev extras pull moto for the wire test.
Voyeur: credentials via AWS_ACCESS_KEY_ID/_SECRET_ACCESS_KEY env or
~/.aws/credentials, never printed; only endpoint URL + bucket name
surface in logs.
CLI: arborist cold {pack,unpack,stats}. Makefile: cold-pack,
cold-pack-dvd (local-dir output for growisofs), cold-unpack, cold-stats.
Sizing for current shards (14.1M chunks, ~17 GB compressed): ~4 packs
at the default cap, ~\$0.34/mo DO Spaces storage, ~\$0.0001/fresh-peer
hydrate.
Always-on raw-UTF-8 leaf store (per ticket "Hard invariants") deferred
— packs-only for now, backfill later.
2557 passed, 28 skipped, 1 xfailed.
Address Grok's two minor-improvement flags on the docs.
New docs/diagrams/{three-layer-stack,cache-key-8dim,falsification-states}.{dot,svg,png}
embedded into docs/_source/concepts.rst — visual scaffolding for the
3-layer stack, 8-dim cache_key composition, and falsification state
machine (previously prose+tables only).
docs/_source/cookbook.rst gains a "Use arborist as a Python library"
section: open_store + ingest_documents, custom Source subclass,
audit-chain walk + verify, Merkle proof round-trip, programmatic
arborist.qa.query() with OpenAICompatibleClient + StubClient swap.
Every Python recipe smoke-tested against a scratch DB before publish.
make docs-api: 0 new warnings. make test: 2557 passed.
The §12.1 pipeline I added was second-hand from benchmark-matrix.md
and got several things wrong against the code:
- listed 4 verdicts; actual is 5 (missing FABRICATED — the
fabrication-vs-WRONG split that energy-cogs §5.5 leans on for the
qwen-fabricates / hermes-abstains finding)
- "θ=0.85" was right by accident — but it's the code-judge-pinned
_CODE_JUDGE_THETA_CONTRA constant, raised from the manifest 0.5
default after measuring 114 FPs in the 0.5-0.75 band
- omitted the short-answer entity-grounding fast path (which runs
BEFORE NLI per the 2026-05-19 Poland-Tusk smoke)
- omitted the HYBRID rescue ladder (NLI entail / entity rescue /
2026-05-21 verbatim-quote-on-topic rescue)
- conflated WRONG and FABRICATED (the subject-in-gold split is what
distinguishes "source has the topic but a different value" from
"source silent on the topic")
Rewrote §12.1 grounded in `bench/judge_code.py:judge()` (its own
docstring at line 501-528 is the truth on rule order), with file:line
citations and the verdict-mapping in full.
Also fixed a real artifact-vs-doc drift INSIDE the judge: the
module-top docstring still claimed θ_contra default 0.5 and omitted
the short-path and the WRONG/FABRICATED split. Updated to match the
authoritative judge() docstring + current code.
No behavior change — docstring + benchmarks doc only.
A stable façade so another Python app can use arborist as a
content-addressed / Merkle / audit-chained store without the CLI or a
wire protocol. Import from arborist.embed, not internal modules, so
refactors don't break embedders.
Surface: open_store(path), ingest_documents(conn, docs), search(conn, q),
plus re-exported Document/Edge/Source/Hit/IngestStats. Core only
(python+sqlite3) — no extras. _IterableSource adapts a plain doc iterable
into the Source contract.
This is the seam for using arborist as neopig's optional provenance
backend: neopig produces Documents from crawled pages, arborist gives
content-dedup (document_root) + FTS5 + an append-only audit chain
alongside neopig's existing md5/FileVault storage. Docs in
docs/embedding.md. 6 tests pin open/ingest/dedup/idempotence/edges/search.
