Adds bench/cross_model_selfplay.py + `make bench-cross-model` target.
For each question in a fixture, runs `arborist query` once per
configured model (default: Hermes + Qwen) and tabulates:
* audit_mode per model (EVIDENCE-WARRANTED → POINTER-LINKED → UNGROUNDED)
* agreement on primary source URI
* grounding rate per model
* estimated $/grounded-answer (per-call prices configurable)
* cheap-first cascade analysis (try cheapest, escalate on UNGROUNDED)
This is the "ask twice for two options" pattern from the agent
perspective — bakes it in as a benchmark so we can measure whether
the cascade beats always-using-the-stronger-model on $/grounded.
First live run (2 questions × 2 models, $0.41):
- Hermes: 1/1 grounded (1 timeout — operational issue)
- Qwen: 2/2 grounded STRICT
- Cascade: 2/2 grounded for $0.25 — beats always-Qwen ($0.32)
when Hermes succeeds on its first call.
Output: bench/cross_model_results/<utc-iso>.{jsonl,md} (gitignored).
Cloud-query was picking sibling articles (Mona Lisa's Revenge instead
of Mona Lisa, Republics of the Soviet Union instead of Soviet Union,
Mercury 13 instead of Mercury Seven) because:
1. Title-boost counted overlap but not extra title tokens. Both
"Mona Lisa" and "Mona Lisa's Revenge" overlapped query by 2 →
same boost → BM25 favored the shorter movie article.
2. RRF merge squashed per-shard rank-1 hits into a 1/(60+1) tie
across 4 sidecar shards. Tie-breaking was undefined; the right
article was as likely to lose as win.
Two fixes:
sidecar.search title-boost:
+ Filter title tokens through STOPWORDS + len-1 cutoff so 's', 'of',
'the' don't count as extras.
+ Penalty: extras = |title_tokens - query_tokens|; effective bonus
is `max(0, overlap - extras/2) * title_boost`. "Mona Lisa" gets
full bonus; "Mona Lisa's Revenge" gets half.
MultiShardSidecarCorpus.fts_search merge:
+ When all CONTRIBUTING shards have sidecars (their BM25 + boost
scores are directly comparable), merge by max raw score across
shards. RRF was masking score discrimination at the top of the
list.
+ Mixed (sidecar + bucket-direct FTS5) falls back to RRF since
those scales aren't comparable.
Bench (5-question smoke, cloud_vs_local.py):
before fix: 4 regressions / 5
after fix : 1 regression / 5 (and that one is the right source,
only the audit_mode dropped STRICT
→ HYBRID due to LLM-stochastic answer
phrasing)
A query like 'when was final fantasy 7 created?' was scoring every
'Final Fantasy ___' article identically because:
1. '7' got stripped by the length>1 filter, so the query reduced
to {final, fantasy}
2. Even surviving, '7' literally doesn't appear in the corpus when
it spells 'VII'
Two fixes:
* tokenize_text now keeps single-char numeric tokens (so '7' survives
sanitization). Stopword + len filter still applies to letters.
* numeral_expand() pairs 1↔I, 2↔II, ..., 20↔XX. Applied in two places:
- search() expands the query term list before dict lookup, so '7'
finds 'vii' postings (no rebuild needed — chunks tokenized to
'vii' at build time).
- title-boost computes overlap on the expanded set, so 'Final
Fantasy VII' titles win the +24 boost vs +16 for siblings.
Live demo (cloud-query 'when was final fantasy 7 created?' LLM=qwen):
before: UNGROUNDED 0/1 — top sources Final Fantasy X-2, Tactics,
Character design (FF VII article rank 5, never made top-4)
after : EVIDENCE-WARRANTED 2/2 — 'Final Fantasy VII was originally
released on January 31, 1997.' + 'Development began in 1994.'
cited E1 = Final Fantasy VII (#1 in merged top-4)
Single-flag switch to swap the upstream LLM endpoint+model. Default
(LLM unset) leaves --endpoint/--model unspecified so cloud-query
falls back to its built-in default (Hermes-3-8B on ai.unturf.com).
LLM=qwen pins Qwen3.6-27B on qwen.ai.unturf.com (uncloseai). Granular
override still available via LLM_ENDPOINT=… and LLM_MODEL=…
make-vars or the underlying --endpoint/--model CLI flags.
Sample (homer's boss):
make cloud-query Q='who is homer simpsons boss?' LLM=qwen
→ EVIDENCE-WARRANTED · via claim_lattice 2/2 19.4s
(Hermes on the same query landed EVIDENCE-WARRANTED-PARTIAL 1/2)
Two bugs surfaced by 'what are the 3 starter pokemon in pokemon red?'
that returned UNGROUNDED with Russell Ballestrini's blog cited above
every Pokémon Wikipedia article.
Bug 1: ASCII-only word regex split 'Pokémon' on 'é' into ['Pok',
'mon'], so the term 'pokemon' never landed in the sidecar dict and
the title-boost never matched 'Pokémon Red and Blue' against a
'pokemon' query token. Fix: NFKD-fold accents before tokenizing
(both sides — build + query — agree). Mirrored in _to_fts5 too so
the bucket-direct FTS5 path stays consistent (FTS5's unicode61
tokenizer already folds, so the sanitizer was the only place that
needed the fix).
Bug 2: Multi-shard RRF treated a rank-1 hit in a 223-doc personal
blog identically to a rank-1 hit in a 1M-doc shard. Russell
Ballestrini's blog incidentally contains 'red' or 'starter'
somewhere, so its FTS5 returned the root page at rank 1; RRF tied
with Pokémon Red and Blue (also rank 1 in genesis shard 0) and
won by insertion order. Fix: post-merge title-relevance filter
(mirrors local query.py's search.title_filter) — drop hits whose
titles share zero stemmed+folded tokens with the query before RRF
combines them. Russell Ballestrini blog title {russell, ballestrini}
overlaps neither {starter, pokemon, red} → dropped.
Live demo (cloud-query 'what are the 3 starter pokemon in pokemon red?'):
before: UNGROUNDED 0/1 — Russell Ballestrini #1, no useful evidence
after : EVIDENCE-WARRANTED-PARTIAL 3/6 — 'Bulbasaur, Charmander,
and Squirtle' cited to Pokémon Red and Blue article
Mirrors local 'arborist query' instrumentation so the cloud path is
no less observable. Uses the existing arborist.qa.progress.Progress
emitter (auto-on at TTY, override via ARBORIST_PROGRESS=0|1).
Stages emitted:
manifest.start/done URL + shard/sidecar counts
corpus.open.start/done sidecar download + dict parse cost
search.start/done per-shard FTS + RRF merge
context.start/done chunk-content pulls + assembled bytes
llm.start/done model + endpoint + ctx + answer chars
verify.start/done audit_mode + n_verified/n_quotes
Render tail adds (matches local query render):
capacity: prompt N chars (sys N + evidence N + question N) → answer N chars
timings: manifest Xs · corpus_open Xs · search Xs · context Xs ·
llm Xs · verify Xs · **total Xs**
bucket: N HTTP requests · KB · endpoint / model
JSON output (--json / JSON=1) gains 'timings' + 'capacity' fields
with the same shape, so bench harnesses can consume them directly.
Surfaces that sidecar.parse dominates a cold cloud-query (~22s/shard
on the genesis 6M-term sidecar). That's a real future-opt target
(ProcessPool instead of ThreadPool to escape the GIL) but the
mechanism is observable now, which is the prerequisite for tuning.
usage: make cloud-query Q="your question" [JSON=1] [BUCKET_URL=...] [TOP_K=4] [MAX_CONTEXT=24000] [CACHE_MB=64] / — consistent with the
local usage: make query Q="your question" [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N K="extra retrieval keywords" ANSWER_MODE=claim_lattice|claim_lattice_pointer|quote LAYOUT=tail|bookend|per_chunk BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 WITNESS=1 XLANG=1 XLANG_MT=1] / entry points. No alias kept
(per repo convention against backwards-compat hacks for unused
names; this command landed in 65b6fe6 / cb8c1de during a fast-
iteration burst and has no external consumers yet).
CLI: arborist cloud query '...' (was: arborist cloud ask '...')
Make: make cloud-query Q='...' (was: make cloud-ask Q='...')
Pure-BM25 ranking penalizes long main articles ('Homer Simpson'
Wikipedia article) vs short episode articles with the same query
tokens — length-normalization is doing what BM25 was designed to do
but the result mis-ranks the primary source. Local query.py runs a
SEPARATE title-LIKE retrieval route that surfaces title-match docs
unconditionally; sidecar didn't, so the Homer Simpson article landed
rank 4 behind 'Bart vs. the Space Mutants' and similar incidentals.
Fix: at query time, after BM25 scoring, add a fixed bonus per query-
term overlap with the doc title (already stored in the sidecar doc
table — no rebuild needed). Both sides stemmed with the same
possessive+plural normalization as _claim_title_overlap so
'simpsons' (query) overlaps 'simpson' (title). Default
title_boost=8.0 is enough to lift the right primary source above
incidental short-doc matches without overwhelming BM25 elsewhere.
Live demo (cloud-ask 'who is homer simpsons boss?'):
before: 'Hank Scorpio is Homer Simpson's boss' (cited episode article
where Scorpio appears; primary 'Homer Simpson' article ranked
rank 4 in its shard, never reached the merged top-K)
after : 'Homer Simpson's boss is Charles Montgomery Burns.'
(cited E1=Homer Simpson + E3=Homer's Odyssey episode)
EVIDENCE-WARRANTED-PARTIAL · via claim_lattice 1/2 26.6s
_claim_title_overlap's _stem only stripped trailing 's' on words >4
chars. 'homer's' (mid-apostrophe possessive) became 'homer'' after
stem, which didn't match 'homer' from the title — both claims of a
Simpsons answer cited articles that contained the right content but
the title-overlap check failed, demoting audit_mode UNGROUNDED via
the existing 'all resolving claims TITLE_MISMATCH' rule.
Fix: normalize apostrophes (ASCII and Unicode right-single-quote)
to empty before length/suffix check. Localized to _claim_title_overlap's
inline _stem — no other callers of _content_tokens affected.
Repro / regression:
before: _claim_title_overlap("Homer's boss", "Dancin' Homer") → False
after : _claim_title_overlap("Homer's boss", "Dancin' Homer") → True
unrelated still False; plural collapse still True; 115 verifier tests pass.
Live demo (cloud-ask 'who is homer simpsons boss?'):
before: UNGROUNDED 2/2 (false rejection of two correct claims)
after : POINTER-LINKED-PARTIAL 2/2 (Scorpio claim still title-
mismatched against 'You Only Move Twice' — the episode-
naming convention legitimately ducks token overlap there;
that demote is honest)
Was serial — 4 × 745 MB = ~2 min cold for the 4-genesis-shard manifest.
Build a ThreadPoolExecutor with one worker per shard so the wall cost
is max(per-shard download), not sum. Idempotent: SidecarShardClient
re-reads cached files without re-downloading on subsequent runs.
Wires the inverted-index sidecar into the unified `BUCKET_URL` flow:
- BucketManifest gains optional per-shard `sidecar_url`.
- New `SidecarShardClient`: downloads the sidecar once (cached under
~/.arborist/sidecar-cache/<hash>.bin keyed by URL), queries locally
with BM25; uses the bucket's HttpRangeVFS only for chunk content
reads on hit (warm page cache reuse).
- New `MultiShardSidecarCorpus`: mixed-mode — shards with a sidecar
use SidecarShardClient (sub-second FTS); shards without one fall
back to BucketClient (bucket-direct FTS5, slow but functional).
- Cross-shard merge: reciprocal-rank-fusion (RRF, k=60). Sidecar
BM25 and FTS5 BM25 are on incomparable scales; RRF normalizes by
rank-position so a doc that's #1 on the virtback shard beats a doc
that's #1 on the genesis shard regardless of raw score magnitude.
Bucket layout:
clones/full-bench-64k/002.db (7.6 GB)
clones/full-bench-64k/002.sidecar.bin (745 MB — pre-built index)
clones/manifest-sidecar.json (advertises both)
Live demo (BUCKET_URL=…/clones/manifest-sidecar.json):
make cloud-ask Q="who developed virt-back?"
→ EVIDENCE-WARRANTED · via claim_lattice 1/1 6.79s
"Russell Ballestrini developed virt-back."
sources span genesis-Wikipedia + russell.ballestrini.net
28 HTTP RANGEs, 1.8 MB transferred
vs the 4+ minute bucket-direct-only path on the same shard mix.
`load_bucket_manifest` now accepts both bucket-root URLs (appends
`clones/manifest.json` per convention) and explicit `.json` URLs
(uses them as-is), so operators can pin alternate manifests.
`BUCKET_URL` (one env var) → client GETs `clones/manifest.json` →
opens HttpRangeVFS per listed shard → FTS5 across all shards in
parallel (ThreadPoolExecutor; per-thread apsw.Connection) → merge by
BM25 score → pull chunks from the owning shard → LLM + verify.
No per-query --shard-url, no path proliferation.
Two manifests published on s3://arborist/clones/:
manifest.json — default: virtback only (2.5MB, ~5s/query)
manifest-full.json — opt-in: all 5 shards (35GB, prohibitive over
WAN due to FTS5 b-tree walk pattern; needs
smaller shards or co-located query proxy)
HttpRangeVFS read-ahead tuned from per-page (4KB) to 64KB block-aligned
cache. Each cache miss fetches one 64KB block; subsequent reads within
the block are local-fast. Lower miss count, similar bytes-on-wire
(64KB amortizes well over typical 4-16 page b-tree clusters; larger
read-ahead like 4MB over-fetches on random FTS5 reads).
Sample run (default manifest):
make cloud-ask Q="who developed virt-back?"
→ EVIDENCE-WARRANTED · via claim_lattice 1/1 4.71s (bucket-direct)
21 HTTP requests · 1344 KB
ACL: genesis full-bench shards flipped to public-read (CC-BY-SA
Wikipedia content). Reachable now if you want to play with the slow
multi-shard path; not in the default manifest because chat latency
matters more than coverage breadth.
Mirrors `make query` ergonomics:
- default = pretty terminal layout (audit_label, sources w/ roles,
bucket stats footer)
- JSON=1 (or --json) = full machine-readable record
_render_cloud_ask_human shares the audit-label primitive
(_render_audit_label) with the local query renderer so the
HYBRID/STRICT/UNGROUNDED tokens map to the same four-rung ladder
labels in lattice modes (POINTER-LINKED / ANCHOR-WARRANTED / ...).
Bucket-direct path doesn't emit warrant-tail / run-DAG /
retrieval-purity so those sections are trimmed.
Output footer adds 'bucket: N HTTP requests · KB · endpoint / model'
so the operator can see network cost + LLM identity inline.
Both paths grind on the same data (LOCAL_DB defaults to web.db, the
file that was uploaded to the Spaces shard at SHARD_URL). --burn busts
the local QA cache so we measure a fresh inference both sides; the
cloud path has no caching layer so it's always fresh.
Reports answer + audit_mode + verifier_method + sources + wall time
for each path, then prints local-vs-cloud delta + ratio.
Sample run (Q='who developed virt-back?'):
local: 3.8s STRICT (verbatim quote verified)
cloud: 11.8s HYBRID (entity-name verified)
delta: +8.0s (3.10x — the HTTP RANGE * ~99 requests tax)
Cloud audit_mode lands one rung lower because the cloud-ask prompt is
simpler than query.py's full pipeline, so the LLM paraphrases instead
of quoting. Verifier honestly demotes paraphrase to HYBRID. Both
answers are correct; the audit_mode difference is a prompt-shape
artifact, not a cloud-path correctness gap.
`make cloud-search` was retrieval-only; `make cloud-ask Q="..."` runs
the same audited-answer shape as local `make query`, but every byte
read goes through the bucket via HttpRangeVFS — no local DB, no
intermediate arborist server.
Pipeline:
1. FTS5 search bucket-direct → top-k document hits (existing path)
2. SELECT chunks.content for each hit via SQL through the same
apsw conn (reuses the warm page cache from step 1)
3. Assemble context with per-doc budget (max_context_chars / top_k)
4. POST to LLM endpoint (default Hermes, override --endpoint/--model)
5. verify_quotes() locally — same verifier the local path uses
6. Emit {answer_text, audit_mode, verifier_method, n_quotes,
n_verified, sources w/ source_role + n_chunks, stats, timing}
Real-world result on the russell.ballestrini.net Spaces shard:
make cloud-ask Q="who developed virt-back?"
→ "Russell Ballestrini developed virt-back."
audit_mode=HYBRID, verifier_method=entity
98 HTTP RANGE GETs, 388 KB, 10.4 s total
endpoint=hermes.ai.unturf.com/v1
FTS5 chokes on '?' and parses '-' as NOT, so a natural-language
question like 'who developed virt-back?' raised SQLError. BucketClient
now strips non-word characters, filters stopwords (mirroring
qa/query.py:_TITLE_STOPWORDS), and OR's the surviving content tokens.
Examples:
'who developed virt-back?' -> 'virt OR back'
'what is anarchism' -> 'anarchism'
'"virt-back"' -> 'virt OR back' (FTS5-quoted spans lose
their punctuation context;
pass --raw to preserve)
CLI: `arborist cloud search --raw '"phrase"'` passes the query through
untouched for advanced FTS5 expressions (AND/NOT/phrase quotes).
`make cloud-search Q="who developed virt-back?"` now resolves to the
right russell.ballestrini.net article without any escaping.
Drops the "SHARD_URL required" friction. Default points at the
russell.ballestrini.net web-crawl shard now live on DO Spaces
(public-read, ~2.5 MB) — the smallest end-to-end real-data target
for proving bucket-direct queries work against a real S3-compatible
bucket without any local arborist data.
Override SHARD_URL on the command line to point at a different bucket
shard:
make cloud-search Q='"virt-back"' # uses default
make cloud-snapshot-root SHARD_URL=https://other.bucket/.../000.db
Recipe also switched to single-quoted $(Q) so FTS5 phrase quotes
(`'"phrase here"'`) survive shell parsing.
Pure-cloud consumer: client opens an arborist .db file IN PLACE on a
bucket via HTTP RANGE reads, runs FTS5 + SQL locally, fetches chunk
bodies from `blobs/<hash>` on the same bucket. No intermediate server
in the data path. The bucket layout we already produce (Tier A clones
plus --jit-blobs blobs/) is exactly what this consumer needs.
Module `arborist/wallet/bucket.py`:
- HttpRangeFile / HttpRangeVFS: apsw subclasses. xRead → HTTP Range
GET; xFileSize → cached HEAD. xWrite/xTruncate raise (read-only).
IOCAP_IMMUTABLE so SQLite skips locking/journaling. Empty tempfile
backs the apsw VFSFile C-bookkeeping; never actually read.
- _LRUByteCache: thread-safe (offset,length)-keyed LRU; soft byte
budget (default 32 MB). SQLite's own page cache (~8 MB) handles
most hot-path amortization, so our LRU is the second-level safety
net for working sets that overflow SQLite's cache.
- _HttpTransport: stdlib urllib (zero new runtime deps beyond apsw).
- BucketClient: high-level — fts_search / chunks_for_doc /
fetch_chunk_body / snapshot_root + page-cache stats.
CLI (`arborist cloud <sub>`):
- `cloud search Q --shard-url ...`
- `cloud snapshot-root --shard-url ...`
- `cloud fetch-chunk LEAF_HASH --blob-base ...`
Makefile:
- `make bootstrap-bucket` (installs apsw)
- `make cloud-search Q="..." SHARD_URL=https://.../000.db`
- `make cloud-snapshot-root SHARD_URL=...`
- `make cloud-fetch-chunk LEAF_HASH=... BLOB_BASE=...`
- `make cloud-demo` — end-to-end proof on a vanilla laptop: seeds a
tiny bucket layout in tmp, serves it via a Range-aware static
HTTP server, runs all three cloud commands from an isolated HOME
that has no local arborist data. Asserts laptop HOME stays empty
start-to-finish.
Tests (tests/test_wallet_bucket.py, 4 passing):
- bucket-direct FTS5 results == direct sqlite3 results
- chunk fetch round-trip + hash verify
- snapshot_root bucket-direct == snapshot_root local
- second identical query adds 0 HTTP requests (SQLite-cached)
pyproject: new `[bucket]` extra carries apsw>=3.45; folded into [dev].
Three new targets layered on the wallet-demo:
- wallet-serve long-running server pointed at your real corpus
(defaults: SHARDS_DIR=$HOME/.arborist/shards, override
with DB=path); Ctrl-C to stop
- wallet-pin fetch the server's current snapshot_root once and save
to $HOME/.arborist/wallet.anchor. Real SPV trust:
verify the anchor out-of-band before pinning, then
every subsequent ask grounds against the pinned value
- wallet-ask Q="your question" — reads the pinned anchor (warns +
auto-fetches if no pin), submits via wallet client,
prints the verified JSON. Optional ANCHOR= override.
Typical flow:
make wallet-serve & # one terminal
make wallet-pin # one-time bootstrap
make wallet-ask Q="what is X?" # repeat as needed
Self-contained recipe that proves the wallet works from a machine
with zero local arborist data:
1. shows the vanilla-laptop HOME is empty
2. ingests a 2-doc corpus into a separate server HOME
3. starts `arborist serve` (stub LLM, no upstream calls)
4. bootstraps the trust anchor via curl /snapshot_root
5. runs `arborist wallet ask` on the laptop HOME; verifies exit 0
6. confirms laptop HOME is STILL empty after the verified query
7. runs same call with a deadbeef anchor; verifies exit 3
8. ALL CHECKS PASSED message
Two isolated mktemp HOMEs; trap cleans up server + dirs even on
interrupt. Seed logic lives in arborist/wallet/_demo_seed.py so the
Makefile recipe stays one logical command instead of inlining
multiline Python (make recipe lines turn class/def into SyntaxError).
Run: `make wallet-demo` (default port 18780; override with
WALLET_DEMO_PORT=N).
The Merkle bundle gives the wallet authentic chunk bytes, but the
server still decides what audit_mode to claim. Run the existing
verify_quotes() locally on the bundle's chunks so the wallet has an
independent verdict that doesn't trust the server's verifier at all.
`VerifiedAnswer` now carries `local_audit_mode`, `local_n_verified`,
`local_verifier_method` alongside the server's audit_mode. They can
legitimately differ (server's context is larger), but a wallet-side
STRICT against a server-side UNGROUNDED would be a real "server lied
about not finding grounding" signal — exactly what the SPV pattern
exists to catch.
Opt out with `client.ask(q, verify_locally=False)` for pure-stdlib
SPV ports that can't load the verifier.
A wallet client holds only a snapshot_root (trust anchor) and verifies
Merkle proofs on every answer. No SQLite, no FTS, no chunks locally.
Same shape as Bitcoin SPV (Electrum / mobile wallet): server can DOS
but cannot forge content whose hash chains up to the trusted anchor.
New module `arborist/wallet/`:
- proof.py: AnswerBundle + build_answer_bundle (server) +
verify_bundle (client). Two proof legs per chunk:
chunk_body → leaf_hash → document_root via in-doc
Merkle proof, then document_root → snapshot_root via
the corpus-wide sorted-doc-roots tree (mirrors
snapshot.compute_snapshot_root). Single-doc corpus
degenerates to "document_root IS snapshot_root" and is
handled with an explicit `degenerate_single_doc` flag.
- server.py: WalletServer + http.server.ThreadingHTTPServer wrapper.
Pure stdlib. GET /healthz, GET /snapshot_root, POST /ask.
Each request opens its own DB connection so SQLite's
single-writer model never bites.
- client.py: WalletClient: urllib + json + arborist.wallet.proof.
Returns a VerifiedAnswer or raises VerificationError /
WalletError. No corpus dependency.
New CLI:
- `arborist serve` — start the wallet server. ARBORIST_WALLET_STUB=1
swaps the LLM for StubClient (lets ops sanity-check verification
without burning tokens).
- `arborist wallet anchor` — fetch the server's current snapshot_root.
- `arborist wallet ask` — submit a question, verify the AnswerBundle
against --trust-anchor, exit 3 on VerificationError.
Tests (tests/test_wallet_spv.py, 7 cases):
- happy: bundle → verify pass against correct anchor
- dict round-trip via to_dict/from_dict still verifies
- tamper: rewrite a chunk body → body hash check fails
- forged leaf_hash: chunks[i].leaf_hash != chunk_proofs[i].leaf_hash
fails before any hashing
- wrong trust_anchor: bundle.snapshot_root != anchor fails immediately
- /healthz and /snapshot_root over real HTTP
- end-to-end ask: corpus → in-process server → urllib client → verify
Replaces the batched chunk-pack phase with per-chunk content-addressed
blob uploads to `blobs/<hash[:2]>/<hash[2:]>`. The metadata pack still
ships (small, fast to restore), but consumers no longer have to pull
multi-GB chunk packs to get queryable: `cold unpack --mode just-enough`
+ `ARBORIST_JIT_CHUNKS=1` fetches single chunks on cache miss.
Producer (`_stream_jit_blobs` in evict.py):
- ThreadPoolExecutor with bounded queue (workers*4) keeps memory flat
across millions of chunks
- HEAD-checks object_size for idempotent re-upload
- Mutually exclusive with chunk packs — manifest's `chunk_pack_hashes`
is empty in JIT mode (consumer reads that as "JIT-only")
Consumer (`hydrate_doc_jit` in cold_clone.py + `_maybe_jit_hydrate` in
qa/query.py):
- Detects both content shapes that need JIT: NULL (Tier B raw-clone) and
zeroblob placeholders (just-enough pack restore, per #53). Discriminator
is first-byte = NUL — zstd-framed bodies start with 0x28, plain UTF-8
prose never has leading NUL.
- Same placeholder filter applied to chunk-read sites in qa/query.py so
partial hydrate doesn't surface zero-bytes content into the LLM context.
Test (`TestJitBlobsPackMode` in tests/test_cold_unpack_routed.py):
- End-to-end push → just-enough hydrate → JIT-fetch → content matches
original byte-for-byte through `unpack_chunk`.
Docs (cold-object-store.md):
- Hard-invariant #1 updated: bucket holds packs by default; `blobs/`
and `clones/` are opt-in prefixes for the JIT and Tier-A flows.
- New "Three consumer modes" section: full-pack vs JIT-blobs vs raw-clone
comparison table + operator decision tree.
Partial Tier B snapshots (where the producer didn't upload every
chunk, e.g. due to the corpus's BLOB-vs-TEXT-affinity skew) left
some NULL chunks without a corresponding blob in the bucket.
hydrate_doc_jit used to raise on the first NoSuchKey / hash
mismatch, which collapsed the whole JIT query. Now we skip the
offending chunk and let the query path see the same NULL content
it would see on a cold shard. Worst case: partial context, not a
crashed query.
Adds hydrate_doc_jit(conn, doc_root, backend): for every chunk in this
document whose local content IS NULL, fetch blobs/<leaf_hash> from the
configured cold backend, verify hash, cache into the row. _load_doc_text
and _load_doc_chunks call it transparently before reading.
Env-gated: ARBORIST_JIT_CHUNKS=1 opts in (so non-JIT environments stay
a pure no-op); the backend comes from the standard ARBORIST_COLD_*
config. With this wired in, a Tier B consumer can clone metadata-only
shards and the query path transparently fetches answer chunks JIT — no
caller code changes.
Some chunks in real corpora have content stored as decoded text (str)
not bytes, with Unicode codepoints beyond latin-1. Encode as utf-8 so
the conversion always succeeds; if the resulting bytes do not hash to
the row's leaf_hash, skip that chunk (no blob uploaded, content kept
local) rather than aborting the whole shard's snapshot.
Some rows in shards-genesis-v2 came back from the chunks.content BLOB
column as instead of bytes — SQLite's type-affinity rule lets a
BLOB-affinity column hold any storage class. The strip-and-upload path
crashed on bytes(some_str) without an encoding. latin-1 preserves
arbitrary byte values 1:1, so the leaf_hash check still matches.
The previous serial loop uploaded one shard at a time, capping
throughput at single-shard multipart concurrency (~10 parts × 8 MB =
~80 MB outstanding). On the 3090->Spaces NYC3 link that meant ~50+ min
per 12 GB shard. Refactor to one ThreadPoolExecutor worker per shard
(default M-wide); each worker still has 10-way multipart inside, so 4
shards × 10 parts = 40 parts in flight saturates the link far better.
New --workers flag overrides the auto-fanout.
Adds a distribution channel alongside the pack tier that skips
pack/unpack entirely — producer SQLite-Backup-API's each shard to a
raw .db and multipart-uploads; consumer pulls them down in parallel.
Recovery becomes ~download time (the FTS index travels inside the .db,
no rebuild step). Replaces the ~84 min pack/restore measured 2026-05-29
with ~download time for ~35 GB of raw shards.
Two channels live in the same bucket:
- Tier A (full clone): clones/<snap-id>/00N.db. `arborist cold
stream-snapshot` produces, `arborist cold clone` consumes. Targets
capable peers (the 3090 class).
- Tier B (just-enough + JIT): same flow with --just-enough, but the
producer strips chunks.content into per-chunk blobs/<hash> and ships
metadata-only shards. Consumer's local DB is ~a few GB; the
retrieval path can fetch_chunk_jit() from the bucket on demand.
Targets constrained peers (mobile / SPV).
A small clones/CURRENT.json pointer enables atomic-ish discovery;
pinned --snapshot-id works too. New backend.get_file() streams large
objects via boto3 download_file (multipart parallel into a target
file). Round-tripped locally with MemoryBackend on both tiers
(content preserved byte-exact; JIT verified by hash_leaf).
The parallelized _cmd_cold_rebuild_fts calls time.time() but cli.py has
no module-level `import time` (functions import it locally), so the
command crashed with NameError. The command had never actually run via
the CLI before — the full recovery used a standalone script and tests
called _rebuild_fts_on_target directly — so the gap shipped in 1547259.
Now exercised end-to-end: rebuild-fts on the 4-shard corpus runs 4-way
parallel (~294s) and `cold verify` passes.
The FTS pack restore produced a DEAD index — segments present, MATCH=0 —
because the verbatim shadow-table copy used INSERT OR IGNORE, so the
pack's real `_data` rowid-1 "structure" record lost the primary-key
conflict to the empty one `CREATE VIRTUAL TABLE` seeds, leaving a "0
segments" header over orphaned segments. Fix (evict.py): clear the
seeded rows, then copy verbatim (DELETE + INSERT ... SELECT), so each
fts5 shadow table becomes a byte-for-byte copy of the producer's index
and the real structure record survives.
Validated on the 3090: restore one fts pack, NO rebuild -> MATCH
'anarchism'=1189 / 'the'=1.42M (identical to rebuild-from-content);
`_data` id=1 structure record non-empty.
make cold-hydrate: rebuild FTS only when the restored index isn't
already searchable (cold verify-gated). Shipping FTS packs now makes
recovery fast (~24s/shard restore, skip the ~5min rebuild); dropping
them (--no-fts default) keeps the bucket small. Either way cold verify
gates success.
docs/cold-object-store: FTS packs restore correctly now; documented the
restore-vs-rebuild tradeoff, per-consumer guidance, and the fixed bug.
- Hydrating a new peer: M-aware `make cold-hydrate` (serial default,
bulk-tuned restore, parallel FTS rebuild from content, cold verify
self-check) — replaces the stale single-shard `cold unpack` loop.
- New "FTS: rebuild, don't restore" with the measured rebuild-vs-restore
comparison (~5 min rebuild + no extra download vs +4.76 GB for a dead
pack-restored index) and WHY the pack restore is dead: FTS5's _data
rowid-1 "structure" record is left empty because INSERT OR IGNORE
collides with the freshly-created vtable's empty header — a headless
index over orphaned segments (count looks right, MATCH returns 0).
- Invariants: FTS shadow tables are rebuilt on the consumer, not shipped
(cold pack defaults to --no-fts).
- Failure modes: cold verify's zero-filled-content + dead-FTS classes.
Make cold recovery correct-by-default and fail-loud, closing the class
that silently produced a corrupt, unqueryable corpus (2026-05-29):
- cold rebuild-fts now runs in PARALLEL (one process per shard; separate
files, no contention) and clears-first, so it also repairs the dead
index a cold-pack FTS restore leaves. Replaces the serial post-pass.
- new `cold verify`: self-check a hydrated shard set — chunk content
materialized (not zero-filled) AND FTS searchable (MATCH a word taken
from sampled content). Non-zero exit if any shard fails. Validated: it
passes the good recovery and fails the corrupt genesis-test, catching
both the zero-filled-bodies and dead-FTS classes.
- cold pack defaults to --no-fts (FTS is derived from content and the
FTS pack restore is non-functional anyway); --with-fts to opt back in.
- make cold-hydrate: serial by default (M-aware hydrate routes every
pack into all M shared target shards, so parallel workers contend —
#54); always rebuild FTS from content (drop the broken shard-000-only
gate); run `cold verify` at the end so a bad recovery fails loudly.
- make crawl-ingest writes to one central crawl db (CRAWL_DB, default
~/.arborist/crawl/web.db) instead of per-domain shards in the
peer-shared main dir: keeps locally-crawled content out of peer
sharing by default and a growing domain set under SQLite's 10-attach
cap (Makefile, docs/crawler.md).
- arborist query auto-includes the local crawl db (query() gains
extra_shards; CLI --include-shard / --no-crawl-db, default-on when
web.db exists). Fix latent --db single-file query AttributeError
(cli.py). Persist used / used_pointer_ids + retrieval_purity into
merkle_proof so read-only consumers can see which chunks fed the
answer (qa/query.py).
- arborist.read: read-only seam for dashboards / verifiers; on a
multi-source context root surface the real primary source instead of
the opaque corpus://multi-source sentinel (read.py). Backs the
arborist-viz Merkle Command Center (#000069).
- tests for extra_shards, the CLI crawl-db resolver, and the read seam.
Genesis hydrate from cold packs was broken end-to-end (the #46 SPV-wallet
test never passed):
- Phase-1 metadata restore (90M-edge fan-out + zeroblob chunk
placeholders) ran with SQLite's ~2 MB default cache — the 512 MB /
MEMORY-journal tuning was applied AFTER Phase 1, so INSERT OR IGNORE
into the indexed edges table thrashed: ~4 h/pack, never finishing.
Move the bulk PRAGMAs before Phase 1 (evict.py).
- The routed restores held one unbounded transaction; add bounded
incremental commits (edges every 1M rows, routed tables every 200k)
so the txn and in-RAM MEMORY journal stay small (cold_pack_metadata.py).
- FTS packs restore a non-functional index (shadow rows present, MATCH
returns 0). _rebuild_fts_on_target now clears-first so a rebuild from
content overwrites the dead pack-FTS idempotently (migrate.py).
Validated on the 3090: full corpus recovered (3,468,226 docs / 6,235,588
chunks / 90,592,990 edges, exact match to source), search + STRICT Q&A
working. Restore ~4 h/pack -> ~8-20 min/pack; FTS rebuilt from content
in ~5 min (parallel).
Five sections updated to match the post-2026-05-27 substrate state
(the user-payload-layout work, sibling ticket #000068, and the
2026-05-27 bench evidence are all in tree).
Verdict block (top): ADD: companion missed-answer guard -> DONE:
companion guard shipped as #000068 Phase 1+2+3 (default OFF; Phase
4 default flip NO-GO until wider bench + human spot-check).
Companion missed-answer guard section: renamed from "(proposed
sidecar)" to "(shipped 2026-05-27)". Carries the implementation
location (arborist/qa/inspect.py:diagnose_missed_answer), the full
output schema (diagnostic_version / confidence_class /
triggered_clauses / subject_tokens / missed_answer_candidate_spans
with offset_start/end/basis), the Phase 2 bench headline (2/228
fires, both Ballestrini, 100% precision, 0 FPs across 226 non-
Ballestrini), the Phase 3 demote-flag CLI surface
(--demote-on-missed-answer, default OFF), and corrected hash
discipline: Phase 1 sidecar fields fold into governance_policy_hash
only; the Phase 3 demote flag (answerability_demote_enabled) ALSO
folds into verifier_policy_hash because flipping it changes the
rendered audit_mode (a verifier-output property).
Future hardening list: "companion missed-answer falsification
guard" entry now points at #000068 instead of describing a
deterministic-sidecar to be built.
Roadmap Phase 5: "DESIGN OPEN" -> "DONE 2026-05-27" with all four
sub-phases of #000068 named individually (Phase 1 sidecar commit
2ab11d2, Phase 2 bench + Phase 3 demote commit ec55db5, Phase 4
default flip NO-GO per Dav1d 2026-05-27 §3.4). The #000068 phase
numbering is internal to that ticket; this roadmap names the
external-facing milestones.
Related links: past-tense the missed-answer guard ("when it lands"
-> "shipped 2026-05-27"); added cross-reference to
docs/tickets/ticket-000068-*.md; named both Dav1d review files by
path so a re-read can locate the inputs.
AUTOCOUNT tag (76 fixture-rows in bench/qa_questions.txt) still
matches; no test changes.
Phase 2 — bench instrumentation + measurement run
bench/qa_sweep.py picks up the answerability sidecar projection per row
(answerability_fired, answerability_confidence, answerability_denial_
pattern, answerability_answer_type, answerability_candidate_count) and
aggregates per-mode (answerability_fires + S/M/W confidence breakdown)
into a new column in the markdown summary table.
Measurement run on bench/qa_results/phase2-sidecar-on/2026-05-27T14-
16-22Z (76 questions × n=3 × claim_lattice × Hermes-3-8B × tail layout,
228 runs). Headline:
sidecar fires 2/228 (0.88%)
confidence dist 2 strong / 0 medium / 0 weak
precision 100% (2/2 fires were the Ballestrini fixture)
recall on Ballestrini 2/3 across n=3 (third run model extracted
correctly -> sidecar silent,
correct behavior)
false positives 0/226 non-Ballestrini runs
verifier verdict both fires labeled STRICT by the binary
verifier (the verifier-blind class, exactly
as predicted)
Detection rule's three-clause conjunction (denial + extraction-shape +
candidate proximity near cleaned subject tokens) is operating at the
precision floor. The strong-confidence-only firing pattern is what
calibrates Phase 3's demote threshold.
Phase 3 — opt-in demote flag (default OFF per Dav1d Phase 4 NO-GO)
arborist/qa/keys.py: answerability_demote_enabled added to
_VERIFIER_POLICY_FIELDS so flipping the flag partitions cache via
verifier_policy_hash. Justification: when on, the rendered audit_mode
changes (EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL), which IS a
verifier-output property; verifier hash must move accordingly. The
other answerability_* fields stay governance-only (sidecar
diagnostic, no audit_mode mutation).
arborist/cli.py:_render_audit_label extended with answerability +
demote_enabled kwargs. Logic:
demote_triggers = (
demote_enabled
and answerability["answerability_warning"] is True
and answerability["confidence_class"] in ("strong", "medium")
)
lattice modes:
EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL (rung transition)
POINTER-LINKED / ANCHOR-WARRANTED -> "rung · missed-answer"
(tail tag; rung itself already
signals degradation)
non-lattice modes (quote/span/entity/paraphrase):
audit_mode token unchanged + "· missed-answer" tail tag
weak confidence: NEVER demotes (Phase 2 saw zero weak fires on real
failures; reserved for future expanded detection ladder)
CLI flag --demote-on-missed-answer on both `arborist query` and
`arborist ask`, default OFF. Flows into call_policy[
"answerability_demote_enabled"] and through to result[
"answerability_demote_enabled"] so the renderer reads it without
needing the policy dict.
End-to-end verified live: 4 fresh Hermes-3-8B runs with --demote-on-
missed-answer on `songs by veronica ballestrini`, all 4 rendered
EVIDENCE-MISSED-PARTIAL · via claim_lattice (Hermes hit the failure
mode in all 4, sidecar fired strong, demote logic transformed the
label).
Phase 4 (default flip to demote-on) — NO-GO per Dav1d 2026-05-27 §3.4:
"a false sidecar warning is tolerable; a false audit-label demotion
can damage trust in correct abstentions." Phase 2 precision is 100%
but n=2 fires is too few samples to claim precision floor empirically.
Default flip blocks on wider bench + human spot-check of the warnings.
Tests: 47 total (36 Phase 1 + 11 new Phase 3 covering hash partitioning
discipline + render-label projection across all four rung/confidence
matrices). Full suite 2794 passed (delta +22 from prior 2772).
Bench output (bench/qa_results/phase2-sidecar-on/) intentionally not
committed — bench/qa_results/ is gitignored per existing convention;
the ticket carries the headline numbers + path for re-inspection.
Adds a deterministic read-only sidecar to detect a class of failure the
binary verifier is structurally blind to:
Evidence contains the answer.
Model says the evidence does not contain the answer.
Verifier sees no unsupported positive claim -> marks run clean.
User receives a false negative under EVIDENCE-WARRANTED.
The motivating case: "songs by veronica ballestrini" against the 2010
Wikipedia corpus. Hermes-3-8B under user_payload_layout=tail returned
"the specific songs by her are not mentioned in the provided evidence
blocks" when evidence E2 literally contained "Amazing", "Out There
Somewhere", "Fascinated", "What's Up With That", "Don't Say". Verifier
correctly returned EVIDENCE-WARRANTED 2/2 because the existing layered
verifier (quote / span / entity / paraphrase + Rule 8 title-relevance +
Rule 9 subject-tokens-absent + claim-count ceiling) guards unsupported
*presence*, has no hook for unsupported *absence*.
Layout fixes attention placement on the specific instance (the 5/27
n=3x75q bench confirms bookend/per_chunk recover Ballestrini); layout
alone cannot close the class -- adversarial phrasing or a bigger prompt
resurfaces the failure under any layout. The right substrate move is to
falsify "not mentioned" as a testable claim.
Detection rule (three-clause conjunction, all must fire):
A. Denial pattern in answer (sealed v1 phrase list: "not mentioned",
"not provided", "the evidence does not say", "does not mention",
"no specific", "no evidence", "cannot determine from the provided
evidence", "is not stated", "is not specified"). Casefolded +
whitespace-normalized substring match.
B. Question is extraction/list-shaped. Either a surface cue ("songs
by", "works by", "books by", "who wrote", "who composed", "what
year", "list of", "name all", ...) matches, OR the existing
arborist.qa.quantifier classifier returns intensity in {ALL,
COMPREHENSIVE, OPEN_REQUEST, MANY, PLURAL}.
C. Evidence contains candidate spans matching the answer_type within
a proximity window (default 600 chars) of cleaned subject tokens.
Candidate kinds aligned to answer_type:
title_like -> quoted_string, title_case_span, comma_list_item
person -> title_case_span
date -> year, date
Hardenings folded in from the 2026-05-27 Dav1d de-novo review:
1. Subject tokens strip cue/relation/stop words. For "songs by
veronica ballestrini" the cleaned subject is ["veronica",
"ballestrini"], NOT all four tokens. Without this the guard
false-triggers on "Harvard University" or "New York" near
proper-noun subjects.
2. Answer-type alignment. Candidate span kind must match query type
so "songs by John Smith" + evidence about Harvard/NY does not
strong-trigger.
3. Confidence class is deterministic (weak | medium | strong), not
boolean. Strong requires quoted_string near exact subject mention
+ multiple type-matched candidates. Phase 3 demote will gate on
confidence_class.
4. Cap output at 10 candidates (the per_chunk-quote-inflation
lesson). Prevents the guard becoming another claim amplifier.
5. Offsets are offset_start + offset_end + offset_basis=
"evidence_object_text", never an ambiguous single offset.
6. Cache-hit path returns answerability: None. Cached records do not
carry the evidence_map, only the rendered sources summary, so the
sidecar cannot recompute candidate spans without re-running
retrieval. Operators wanting fresh diagnostics use --burn.
7. Phase 1 stays out of verifier_policy_hash. The
answerability_sidecar_enabled / answerability_threshold /
denial_patterns_version / extraction_cues_version fields fold
into governance_policy_hash only. Phase 3 demote flag
(answerability_demote_enabled, default False) will move the
verifier hash WHEN ON because it changes the rendered audit_mode
(EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL).
Sidecar discipline (matches arborist.qa.inspect.diagnose_* sister
functions deflection / coherence / title_relevance):
- no model calls (no LLM-as-judge, no NLI, no translation)
- no audit_events writes
- no providence_cache writes
- no answer text mutation
- no claim promotion -- the trigger conjunction makes promotion
structurally impossible (only fires on denial answers)
- byte-deterministic: same (question, answer, evidence, policy) ->
same output every time
Result-dict integration: result["answerability"] is None when the
guard did not fire, or a structured diagnostic dict when it did
(diagnostic_version, confidence_class, triggered_clauses,
denial_pattern_matched, extraction_cue_matched, extraction_shape,
answer_type, subject_tokens, candidate_count, threshold,
missed_answer_candidate_spans). Read by bench_qa (Phase 2 will add
warning-count aggregate to bench rows) and CLI render.
Three return points carry the key:
- miss-path (full retrieval + verify): computed from evidence_map
- cache-hit: None (Dav1d cache-hit recompute discipline -- evidence
not stored, recompute requires re-retrieval)
- reject-broad early-return: None (no evidence examined)
Tests: 36 new pinning the three-clause logic, positive (Ballestrini)
regression, negative control (John Smith + Harvard/NY), each-clause-
alone silence, schema integrity, byte-determinism, sidecar-disabled
short-circuit, dict-shaped evidence support. All pass; existing
inspect tests (60) all pass.
End-to-end verified live via the CLI on the real corpus (2010 ~/.arborist
/shards): 3 fresh Hermes-3-8B runs on "songs by veronica ballestrini",
run 1 hit the failure (sidecar fired with confidence: strong, 351
candidates, denial: "not mentioned"), runs 2-3 model extracted
correctly and sidecar correctly silent.
Phase 2 (bench + threshold tuning) and Phase 3 (opt-in demote flag)
are open as follow-ups. Per Dav1d: NO-GO on default demote-on until
benchmark + human spot-check confirms low false-positive rate.
Full spec in docs/tickets/ticket-000068-verifier-blind-missed-answer-
guard.md (post-review hardenings section at top names the seven
load-bearing changes from the Dav1d 2026-05-27 review).
format_user_payload helper in arborist/qa/prompts.py becomes the single
source of truth for the user-turn payload. Three layouts:
tail (default) evidence first, question at end (prior behavior)
bookend question repeated before AND after evidence — counters
lost-in-the-middle on small models (≤8B)
per_chunk bookend + a one-line [for: <q>] reminder before each
evidence block; for list/extraction queries
USER_PAYLOAD_LAYOUTS constant exported; unknown layout raises ValueError.
The six _user_payload closures in query.py (3) and runner.py (3) all
delegate to format_user_payload. Quote-mode passes per_chunk_marker=None
to fall back to bookend on flat document/sources context.
Wired through both DEFAULT_QUERY_POLICY and DEFAULT_POLICY. Folds into
governance_policy_hash (the layout changes the user-turn content the
model sees, so the policy hash partitions cleanly per layout); does NOT
fold into verifier_policy_hash (verifier rules unchanged).
Makefile gets LAYOUT_DEFAULT ?= tail and LAYOUT ?= $(LAYOUT_DEFAULT) so
operators can flip per-call (LAYOUT=bookend make query Q="...") or
session-wide (LAYOUT_DEFAULT=bookend make query Q="..."). Recommendation
matrix in the Makefile comment block above the query target encodes the
2026-05-27 bench finding.
Motivating case: the Veronica-Ballestrini "songs by" failure. Hermes-3-
8B under tail layout returned "specific songs by her are not mentioned
in the provided evidence" when evidence E2 literally contained the song
names. Same query under bookend recovered the answer (with conflation
between Veronica Ballestrini and The Veronicas); under per_chunk
recovered AND disambiguated three entities. Qwen-27B unaffected by
layout. The Ballestrini case is added to bench/qa_questions.txt as a
regression fixture under "entity list", with a 4-line comment pointing
to docs/user-payload-layout.md.
2026-05-27 bench (n=3 × 75q, claim_lattice mode, Hermes-3-8B):
tail STRICT 94/225 (0.418) — control
bookend STRICT 95/225 (0.422) — +0.44pp (noise, 5pp floor)
per_chunk STRICT 72/225 (0.320) — -9.78pp (significant regression)
Verdict: tail stays default (cache-preserving and bench-confirmed
neutral). Bookend/per_chunk available as opt-in operator knobs. Per_chunk
regresses in aggregate because the per-chunk reminder over-anchors the
model on every chunk (TOO_MANY_EVIDENCE_IDS violations rose from 20 →
54; mean answer chars in 32-64KB bucket doubled from 720 → 1660). The
Ballestrini-class failure is real but rare across the curated set; a
layout fix that helps the rare case at the cost of 10pp aggregate is a
bad default trade. Documented in full in docs/user-payload-layout.md
along with the Dav1d 2026-05-27 review framing (GO for opt-in, NO-GO
for default promotion, ADD companion missed-answer guard).
CLI changes (--user-payload-layout flag on `query` and `ask`) landed
separately in commit e5ee283 alongside the #54 busy_timeout fix.
2026-05-27 v4-fixed bench on 3090: 3 of 4 parallel workers crashed with
sqlite3.OperationalError: database is locked at PRAGMA journal_mode =
MEMORY. Result: only shard 000 fully hydrated, shards 001/002/003 had
chunks rows landed but content NULL (~99.99% empty), no FTS data.
Peer functionally usable for only ~25% of corpus.
Root cause: parallel hydrate (xargs -P N) spawns N separate processes
that each open connections to all M target shards. Multiple processes
attempting PRAGMA journal_mode change on the same .db file at the same
instant serialize on a brief exclusive lock — without busy_timeout
SQLite throws BUSY immediately and the worker exception-propagates
out of _pull_pack_inner_routed before phase 2 (chunks fill) can run.
Fix: `PRAGMA busy_timeout = 30000` as the FIRST statement on every
hydrate writer connection. SQLite then waits up to 30s for any lock
instead of throwing — slowest worker gets its lock, fastest writes
go through immediately. No measurable cost when there's no
contention (busy_timeout is a wait, not a poll).
Set in TWO places:
arborist/cli.py:_cmd_cold_unpack
Immediately after connect(p), before any other PRAGMA. Covers
the PRAGMA foreign_keys = OFF that runs before the hydrate
pipeline.
arborist/evict.py:_pull_pack_inner_routed
Belt-and-suspenders: in case the function is called with
externally-built connections that didn't set busy_timeout, the
bulk-tuning loop sets it before journal_mode/synchronous/etc.
6 cold-unpack-routed tests pass (no contention in single-process
test fixture; the fix is invisible there). Real validation is the
next 3090 v5 bench against re-packed (#53 pre-sized chunks) bucket.
Real consumer-side bottleneck for cold-pack genesis is the phase-2
chunks-content UPDATE loop: each `UPDATE chunks SET content=? WHERE
chunk_id=?` grows the row from NULL to ~500 bytes, triggering SQLite
page splits, which become ext4 metadata-journal events. With 6M
chunks × 4 parallel writers, those journal events serialize and
dominate consumer wall time (~130 of the 164-min v3-revert run).
Fix: producer dumps a synthetic `_content_size` column in chunks.jsonl
carrying the on-disk byte length of each chunk's content (constant-
time SQLite `length(content)`). Consumer's phase 1 INSERT pre-allocates
the row with `content = bytes(_content_size)` instead of letting it
default to NULL. Phase 2's UPDATE then replaces same-size bytes
in-place — no row growth, no page splits, no per-row journal events.
Implementation:
arborist/cold_pack_metadata.py
_dump_generic_table for `chunks`:
Emit synthetic `_content_size` column = length(content) at the
end of the columnar header. Underscore prefix avoids collision
with any future schema column.
_restore_routed_table:
Detect `_content_size` in the JSONL header; if present (and
table == chunks), build INSERT against [chunks_cols] + ['content']
and substitute a bytes(_content_size) placeholder for the
content position. Phase 2 UPDATE later replaces those bytes.
Forward compatibility:
- Old packs (no _content_size): consumer uses today's NULL-content
INSERT path. No behavior change.
- New packs: consumer auto-detects, uses pre-sized path.
SPV-wallet trade-off:
In just-enough mode the consumer pulls only the metadata pack so
chunks land with the placeholder bytes (NOT NULL anymore). That's
a SEMANTIC CHANGE for SPV — `chunks.content IS NULL` no longer
means "JIT-fetch later." Documented in code; if SPV-mode JIT-fetch
ever ships, it must distinguish placeholder bytes (where every
byte is 0) from real content.
New test (TestPreSizedChunks.test_chunks_content_pre_sized_after_metadata_restore):
Hydrate just-enough → chunks rows have non-NULL bytes content of
correct size. Catches the regression if a future change reverts
the placeholder logic.
Expected impact: ~50-70% reduction in phase-2 wall time. Real number
lands when the next 3090 bench-max iteration runs against re-packed
bucket. 6 cold-unpack-routed tests pass.
v4 bench (2026-05-27 03:09 UTC) measured a corrupt outcome:
chunks_fts=1,561,604 on EVERY target shard regardless of chunks
count. _pull_fts_pack_into_targets had been iterating all M targets
and INSERT'ing each fts pack's shadow tables into every one of them.
Each fts pack's chunks_fts_docsize entries reference chunk_ids that
were independently auto-assigned in its source producer shard (each
source DB has chunk_ids 1..1.56M independently). Inserting all 4
packs into all 4 targets → 4× the per-target FTS rows, pointing at
chunk_ids the target doesn't own. Body searches would return garbage.
Fix: sample one id from the fts pack's chunks_fts_docsize, look it
up in each target's chunks table, INSERT only into the target where
it's found. The other M-1 targets stay untouched and receive their
FTS data from their corresponding fts pack(s) in later iterations.
In post-reshard production topology, each producer source shard's
docs all hash to ONE consumer target, so this 1:1 mapping is exact.
The test fixture is artificial (single-shard producer with docs
hash-distributed across M=4 targets) but still validates the core
property: only one target receives FTS data; the others stay empty.
FTS5 shadow tables aren't subsettable per-row (segment data is
opaque, mixed entries for many docs in one segment) so we can't
filter FTS rows to "only chunks that exist on this target" — we
copy all-or-nothing per pack. That's why the producer's post-reshard
shape (each pack scoped to one target's docs) is the structural
prerequisite for fts packs to make sense.
New regression test
(TestFtsPackRoutingRegression.test_fts_pack_only_on_owning_target):
exactly 1 of M targets has chunks_fts_docsize > 0; the others
must have 0. Pre-fix this asserted on all-4 targets having FTS
data → failed. Post-fix passes.
Returns now include owning_target_idx for forensic visibility into
which target the fts pack landed on.
5 cold-unpack-routed tests pass.
_cmd_cold_list only recognized .metadata.tar.zst and .chunks.tar.zst
key suffixes; .fts.tar.zst keys hit the else-continue and were
invisible. 2026-05-26 producer pushed 4 fts packs and cold list
reported 0 of them.
Trivial extension: add an `elif key.endswith('.fts.tar.zst')` branch
that classifies as kind='fts'. Now cold list shows all 3 kinds.
Also unblocks the make cold-hydrate target's metadata-pack discovery
filter (`kind=='metadata'`) from accidentally matching mis-classified
fts packs.
3090 bench-max v3 run at 13:25 elapsed had shards at 696-760 MB; v2
at 8:46 elapsed (default 4 KB pages) had 549-662 MB. Larger pages
made the chunk-content UPDATE workload SLOWER, not faster.
Reason: chunk-content payloads average ~500 bytes after zstd. With
4 KB pages each UPDATE rewrites a 4 KB page; with 16 KB pages the
same UPDATE rewrites a 16 KB page → 4× write amplification per
row. ext4 journal traffic increases roughly proportionally.
Bigger pages help SCAN-heavy workloads (fewer page reads to walk
a B-tree). For narrow-row-UPDATE-heavy workloads like cold-pack
chunk fill, they hurt. Comment in code captures the measurement
so a future operator doesn't re-attempt this.
The other bench-max measures stay in place:
- synchronous = OFF (per-write fsync removed)
- journal_mode = MEMORY (WAL in RAM)
- temp_store = MEMORY (sort scratch in RAM)
- cache_size = -524288 (512 MB page cache)
- mmap_size = 536870912 (512 MB read mmap)
- foreign_keys = OFF (no FK verify per row)
- deferred FTS5 rebuild post-pass
- batched 5000-row executemany UPDATEs
- bench-max #5 (this commit's revert): page_size stays at 4 KB
Next bench point: v4 against the new bucket packs that include
fts (--include-fts producer run in flight). That eliminates the
FTS rebuild step entirely on the consumer.
push_pack now does a HEAD object_size check against the bucket
before every PUT. If the object exists at the content-addressed
key AND Content-Length matches the local pack body size, skip the
upload. Applies to chunks / fts / metadata pack bodies.
Why content-addressed-key + size is the right check:
pack_hash = hash_leaf(manifest_bytes)
object_key = packs/<pack_hash>.<kind>.tar.zst
The pack_hash IS the cryptographic identity of the body via its
manifest. If the bucket has an object at this exact key, the
content is provably identical (otherwise the producer that wrote
it computed a different pack_hash → different key). Content-Length
match validates against a half-uploaded multipart that a prior
crash might have left.
Why not ETag: S3 ETag for multipart-uploaded objects is the MD5
of per-part MD5s concatenated, then MD5'd, with a "-N" suffix for
part count. Two clients using different multipart_chunksize end
up with different ETags for the same content. Unreliable for
cross-client equivalence. Content-Length is stable.
Result fields gain `skipped_reupload: bool` so callers / the
audit chain can tell whether the bucket PUT happened or was a
no-op.
Real-world impact for tonight's re-pack (--include-fts):
before: 4 chunks PUTs (7 GB re-upload, ~24 min)
+ 4 metadata PUTs (new hashes because manifest now has
_fts_pack_hashes — these MUST upload)
+ 4 fts PUTs (new)
~16 GB uplink, ~50 min wall
after: 4 chunks SKIPPED (same content-addressed key already
in bucket)
+ 4 metadata PUTs (new hashes, MUST upload)
+ 4 fts PUTs (new)
~6 GB uplink, ~20 min wall
cold_pending bookkeeping still fires in the upload path so a
half-uploaded crash recovers cleanly; skip path leaves no trace
because nothing was started.
4 cold-unpack-routed tests pass unchanged.
The 3090 genesis bench at 28-min progress mark showed all 4 workers
sitting in jbd2_log_wait_commit / do_get_write_access — ext4
filesystem journal serializing every page-extension event across
the 4 parallel writers. SQLite synchronous=OFF and journal_mode=
MEMORY skip SQLite's own fsync/WAL costs, but file growth still
goes through ext4's journal layer.
Fix: bump SQLite page_size from default 4 KB to 16 KB on each
hydrate target. Bigger pages mean ~4× fewer page writes during
chunk-fill, which means ~4× fewer file-extension events through
ext4's journal. Direct attack on the measured bottleneck.
page_size can only be set on an EMPTY database; VACUUM finalises
the change. The unpack CLI now opens each target file raw,
PRAGMA page_size = 16384 + PRAGMA secure_delete = OFF + VACUUM,
closes, then connect() applies schema as normal on the
pre-sized empty DB. Idempotent on re-runs since hydrate dirs
are always clean.
Also sets secure_delete = OFF (default is OFF in most builds but
explicit for hydrate writers): no overwrite-with-zeros on row
deletion, less write traffic during the chunk UPDATEs that might
free overflow pages.
Expected impact: ~30-50% reduction in chunk-fill wall time vs the
killed 30-min v2 run. Combined with the prior bench-max round
(journal_mode=MEMORY, deferred FTS, batched UPDATEs) and the
upcoming fts-pack-skip-rebuild path, full genesis should land
under 30 min total instead of the 75-min v2 extrapolation.
4 cold-unpack-routed tests pass unchanged.
