Mirrors `make query` ergonomics:
- default = pretty terminal layout (audit_label, sources w/ roles,
bucket stats footer)
- JSON=1 (or --json) = full machine-readable record
_render_cloud_ask_human shares the audit-label primitive
(_render_audit_label) with the local query renderer so the
HYBRID/STRICT/UNGROUNDED tokens map to the same four-rung ladder
labels in lattice modes (POINTER-LINKED / ANCHOR-WARRANTED / ...).
Bucket-direct path doesn't emit warrant-tail / run-DAG /
retrieval-purity so those sections are trimmed.
Output footer adds 'bucket: N HTTP requests · KB · endpoint / model'
so the operator can see network cost + LLM identity inline.
Both paths grind on the same data (LOCAL_DB defaults to web.db, the
file that was uploaded to the Spaces shard at SHARD_URL). --burn busts
the local QA cache so we measure a fresh inference both sides; the
cloud path has no caching layer so it's always fresh.
Reports answer + audit_mode + verifier_method + sources + wall time
for each path, then prints local-vs-cloud delta + ratio.
Sample run (Q='who developed virt-back?'):
local: 3.8s STRICT (verbatim quote verified)
cloud: 11.8s HYBRID (entity-name verified)
delta: +8.0s (3.10x — the HTTP RANGE * ~99 requests tax)
Cloud audit_mode lands one rung lower because the cloud-ask prompt is
simpler than query.py's full pipeline, so the LLM paraphrases instead
of quoting. Verifier honestly demotes paraphrase to HYBRID. Both
answers are correct; the audit_mode difference is a prompt-shape
artifact, not a cloud-path correctness gap.
`make cloud-search` was retrieval-only; `make cloud-ask Q="..."` runs
the same audited-answer shape as local `make query`, but every byte
read goes through the bucket via HttpRangeVFS — no local DB, no
intermediate arborist server.
Pipeline:
1. FTS5 search bucket-direct → top-k document hits (existing path)
2. SELECT chunks.content for each hit via SQL through the same
apsw conn (reuses the warm page cache from step 1)
3. Assemble context with per-doc budget (max_context_chars / top_k)
4. POST to LLM endpoint (default Hermes, override --endpoint/--model)
5. verify_quotes() locally — same verifier the local path uses
6. Emit {answer_text, audit_mode, verifier_method, n_quotes,
n_verified, sources w/ source_role + n_chunks, stats, timing}
Real-world result on the russell.ballestrini.net Spaces shard:
make cloud-ask Q="who developed virt-back?"
→ "Russell Ballestrini developed virt-back."
audit_mode=HYBRID, verifier_method=entity
98 HTTP RANGE GETs, 388 KB, 10.4 s total
endpoint=hermes.ai.unturf.com/v1
FTS5 chokes on '?' and parses '-' as NOT, so a natural-language
question like 'who developed virt-back?' raised SQLError. BucketClient
now strips non-word characters, filters stopwords (mirroring
qa/query.py:_TITLE_STOPWORDS), and OR's the surviving content tokens.
Examples:
'who developed virt-back?' -> 'virt OR back'
'what is anarchism' -> 'anarchism'
'"virt-back"' -> 'virt OR back' (FTS5-quoted spans lose
their punctuation context;
pass --raw to preserve)
CLI: `arborist cloud search --raw '"phrase"'` passes the query through
untouched for advanced FTS5 expressions (AND/NOT/phrase quotes).
`make cloud-search Q="who developed virt-back?"` now resolves to the
right russell.ballestrini.net article without any escaping.
Drops the "SHARD_URL required" friction. Default points at the
russell.ballestrini.net web-crawl shard now live on DO Spaces
(public-read, ~2.5 MB) — the smallest end-to-end real-data target
for proving bucket-direct queries work against a real S3-compatible
bucket without any local arborist data.
Override SHARD_URL on the command line to point at a different bucket
shard:
make cloud-search Q='"virt-back"' # uses default
make cloud-snapshot-root SHARD_URL=https://other.bucket/.../000.db
Recipe also switched to single-quoted $(Q) so FTS5 phrase quotes
(`'"phrase here"'`) survive shell parsing.
Pure-cloud consumer: client opens an arborist .db file IN PLACE on a
bucket via HTTP RANGE reads, runs FTS5 + SQL locally, fetches chunk
bodies from `blobs/<hash>` on the same bucket. No intermediate server
in the data path. The bucket layout we already produce (Tier A clones
plus --jit-blobs blobs/) is exactly what this consumer needs.
Module `arborist/wallet/bucket.py`:
- HttpRangeFile / HttpRangeVFS: apsw subclasses. xRead → HTTP Range
GET; xFileSize → cached HEAD. xWrite/xTruncate raise (read-only).
IOCAP_IMMUTABLE so SQLite skips locking/journaling. Empty tempfile
backs the apsw VFSFile C-bookkeeping; never actually read.
- _LRUByteCache: thread-safe (offset,length)-keyed LRU; soft byte
budget (default 32 MB). SQLite's own page cache (~8 MB) handles
most hot-path amortization, so our LRU is the second-level safety
net for working sets that overflow SQLite's cache.
- _HttpTransport: stdlib urllib (zero new runtime deps beyond apsw).
- BucketClient: high-level — fts_search / chunks_for_doc /
fetch_chunk_body / snapshot_root + page-cache stats.
CLI (`arborist cloud <sub>`):
- `cloud search Q --shard-url ...`
- `cloud snapshot-root --shard-url ...`
- `cloud fetch-chunk LEAF_HASH --blob-base ...`
Makefile:
- `make bootstrap-bucket` (installs apsw)
- `make cloud-search Q="..." SHARD_URL=https://.../000.db`
- `make cloud-snapshot-root SHARD_URL=...`
- `make cloud-fetch-chunk LEAF_HASH=... BLOB_BASE=...`
- `make cloud-demo` — end-to-end proof on a vanilla laptop: seeds a
tiny bucket layout in tmp, serves it via a Range-aware static
HTTP server, runs all three cloud commands from an isolated HOME
that has no local arborist data. Asserts laptop HOME stays empty
start-to-finish.
Tests (tests/test_wallet_bucket.py, 4 passing):
- bucket-direct FTS5 results == direct sqlite3 results
- chunk fetch round-trip + hash verify
- snapshot_root bucket-direct == snapshot_root local
- second identical query adds 0 HTTP requests (SQLite-cached)
pyproject: new `[bucket]` extra carries apsw>=3.45; folded into [dev].
Three new targets layered on the wallet-demo:
- wallet-serve long-running server pointed at your real corpus
(defaults: SHARDS_DIR=$HOME/.arborist/shards, override
with DB=path); Ctrl-C to stop
- wallet-pin fetch the server's current snapshot_root once and save
to $HOME/.arborist/wallet.anchor. Real SPV trust:
verify the anchor out-of-band before pinning, then
every subsequent ask grounds against the pinned value
- wallet-ask Q="your question" — reads the pinned anchor (warns +
auto-fetches if no pin), submits via wallet client,
prints the verified JSON. Optional ANCHOR= override.
Typical flow:
make wallet-serve & # one terminal
make wallet-pin # one-time bootstrap
make wallet-ask Q="what is X?" # repeat as needed
Self-contained recipe that proves the wallet works from a machine
with zero local arborist data:
1. shows the vanilla-laptop HOME is empty
2. ingests a 2-doc corpus into a separate server HOME
3. starts `arborist serve` (stub LLM, no upstream calls)
4. bootstraps the trust anchor via curl /snapshot_root
5. runs `arborist wallet ask` on the laptop HOME; verifies exit 0
6. confirms laptop HOME is STILL empty after the verified query
7. runs same call with a deadbeef anchor; verifies exit 3
8. ALL CHECKS PASSED message
Two isolated mktemp HOMEs; trap cleans up server + dirs even on
interrupt. Seed logic lives in arborist/wallet/_demo_seed.py so the
Makefile recipe stays one logical command instead of inlining
multiline Python (make recipe lines turn class/def into SyntaxError).
Run: `make wallet-demo` (default port 18780; override with
WALLET_DEMO_PORT=N).
The Merkle bundle gives the wallet authentic chunk bytes, but the
server still decides what audit_mode to claim. Run the existing
verify_quotes() locally on the bundle's chunks so the wallet has an
independent verdict that doesn't trust the server's verifier at all.
`VerifiedAnswer` now carries `local_audit_mode`, `local_n_verified`,
`local_verifier_method` alongside the server's audit_mode. They can
legitimately differ (server's context is larger), but a wallet-side
STRICT against a server-side UNGROUNDED would be a real "server lied
about not finding grounding" signal — exactly what the SPV pattern
exists to catch.
Opt out with `client.ask(q, verify_locally=False)` for pure-stdlib
SPV ports that can't load the verifier.
A wallet client holds only a snapshot_root (trust anchor) and verifies
Merkle proofs on every answer. No SQLite, no FTS, no chunks locally.
Same shape as Bitcoin SPV (Electrum / mobile wallet): server can DOS
but cannot forge content whose hash chains up to the trusted anchor.
New module `arborist/wallet/`:
- proof.py: AnswerBundle + build_answer_bundle (server) +
verify_bundle (client). Two proof legs per chunk:
chunk_body → leaf_hash → document_root via in-doc
Merkle proof, then document_root → snapshot_root via
the corpus-wide sorted-doc-roots tree (mirrors
snapshot.compute_snapshot_root). Single-doc corpus
degenerates to "document_root IS snapshot_root" and is
handled with an explicit `degenerate_single_doc` flag.
- server.py: WalletServer + http.server.ThreadingHTTPServer wrapper.
Pure stdlib. GET /healthz, GET /snapshot_root, POST /ask.
Each request opens its own DB connection so SQLite's
single-writer model never bites.
- client.py: WalletClient: urllib + json + arborist.wallet.proof.
Returns a VerifiedAnswer or raises VerificationError /
WalletError. No corpus dependency.
New CLI:
- `arborist serve` — start the wallet server. ARBORIST_WALLET_STUB=1
swaps the LLM for StubClient (lets ops sanity-check verification
without burning tokens).
- `arborist wallet anchor` — fetch the server's current snapshot_root.
- `arborist wallet ask` — submit a question, verify the AnswerBundle
against --trust-anchor, exit 3 on VerificationError.
Tests (tests/test_wallet_spv.py, 7 cases):
- happy: bundle → verify pass against correct anchor
- dict round-trip via to_dict/from_dict still verifies
- tamper: rewrite a chunk body → body hash check fails
- forged leaf_hash: chunks[i].leaf_hash != chunk_proofs[i].leaf_hash
fails before any hashing
- wrong trust_anchor: bundle.snapshot_root != anchor fails immediately
- /healthz and /snapshot_root over real HTTP
- end-to-end ask: corpus → in-process server → urllib client → verify
Replaces the batched chunk-pack phase with per-chunk content-addressed
blob uploads to `blobs/<hash[:2]>/<hash[2:]>`. The metadata pack still
ships (small, fast to restore), but consumers no longer have to pull
multi-GB chunk packs to get queryable: `cold unpack --mode just-enough`
+ `ARBORIST_JIT_CHUNKS=1` fetches single chunks on cache miss.
Producer (`_stream_jit_blobs` in evict.py):
- ThreadPoolExecutor with bounded queue (workers*4) keeps memory flat
across millions of chunks
- HEAD-checks object_size for idempotent re-upload
- Mutually exclusive with chunk packs — manifest's `chunk_pack_hashes`
is empty in JIT mode (consumer reads that as "JIT-only")
Consumer (`hydrate_doc_jit` in cold_clone.py + `_maybe_jit_hydrate` in
qa/query.py):
- Detects both content shapes that need JIT: NULL (Tier B raw-clone) and
zeroblob placeholders (just-enough pack restore, per #53). Discriminator
is first-byte = NUL — zstd-framed bodies start with 0x28, plain UTF-8
prose never has leading NUL.
- Same placeholder filter applied to chunk-read sites in qa/query.py so
partial hydrate doesn't surface zero-bytes content into the LLM context.
Test (`TestJitBlobsPackMode` in tests/test_cold_unpack_routed.py):
- End-to-end push → just-enough hydrate → JIT-fetch → content matches
original byte-for-byte through `unpack_chunk`.
Docs (cold-object-store.md):
- Hard-invariant #1 updated: bucket holds packs by default; `blobs/`
and `clones/` are opt-in prefixes for the JIT and Tier-A flows.
- New "Three consumer modes" section: full-pack vs JIT-blobs vs raw-clone
comparison table + operator decision tree.
Partial Tier B snapshots (where the producer didn't upload every
chunk, e.g. due to the corpus's BLOB-vs-TEXT-affinity skew) left
some NULL chunks without a corresponding blob in the bucket.
hydrate_doc_jit used to raise on the first NoSuchKey / hash
mismatch, which collapsed the whole JIT query. Now we skip the
offending chunk and let the query path see the same NULL content
it would see on a cold shard. Worst case: partial context, not a
crashed query.
Adds hydrate_doc_jit(conn, doc_root, backend): for every chunk in this
document whose local content IS NULL, fetch blobs/<leaf_hash> from the
configured cold backend, verify hash, cache into the row. _load_doc_text
and _load_doc_chunks call it transparently before reading.
Env-gated: ARBORIST_JIT_CHUNKS=1 opts in (so non-JIT environments stay
a pure no-op); the backend comes from the standard ARBORIST_COLD_*
config. With this wired in, a Tier B consumer can clone metadata-only
shards and the query path transparently fetches answer chunks JIT — no
caller code changes.
Some chunks in real corpora have content stored as decoded text (str)
not bytes, with Unicode codepoints beyond latin-1. Encode as utf-8 so
the conversion always succeeds; if the resulting bytes do not hash to
the row's leaf_hash, skip that chunk (no blob uploaded, content kept
local) rather than aborting the whole shard's snapshot.
Some rows in shards-genesis-v2 came back from the chunks.content BLOB
column as instead of bytes — SQLite's type-affinity rule lets a
BLOB-affinity column hold any storage class. The strip-and-upload path
crashed on bytes(some_str) without an encoding. latin-1 preserves
arbitrary byte values 1:1, so the leaf_hash check still matches.
The previous serial loop uploaded one shard at a time, capping
throughput at single-shard multipart concurrency (~10 parts × 8 MB =
~80 MB outstanding). On the 3090->Spaces NYC3 link that meant ~50+ min
per 12 GB shard. Refactor to one ThreadPoolExecutor worker per shard
(default M-wide); each worker still has 10-way multipart inside, so 4
shards × 10 parts = 40 parts in flight saturates the link far better.
New --workers flag overrides the auto-fanout.
Adds a distribution channel alongside the pack tier that skips
pack/unpack entirely — producer SQLite-Backup-API's each shard to a
raw .db and multipart-uploads; consumer pulls them down in parallel.
Recovery becomes ~download time (the FTS index travels inside the .db,
no rebuild step). Replaces the ~84 min pack/restore measured 2026-05-29
with ~download time for ~35 GB of raw shards.
Two channels live in the same bucket:
- Tier A (full clone): clones/<snap-id>/00N.db. `arborist cold
stream-snapshot` produces, `arborist cold clone` consumes. Targets
capable peers (the 3090 class).
- Tier B (just-enough + JIT): same flow with --just-enough, but the
producer strips chunks.content into per-chunk blobs/<hash> and ships
metadata-only shards. Consumer's local DB is ~a few GB; the
retrieval path can fetch_chunk_jit() from the bucket on demand.
Targets constrained peers (mobile / SPV).
A small clones/CURRENT.json pointer enables atomic-ish discovery;
pinned --snapshot-id works too. New backend.get_file() streams large
objects via boto3 download_file (multipart parallel into a target
file). Round-tripped locally with MemoryBackend on both tiers
(content preserved byte-exact; JIT verified by hash_leaf).
The parallelized _cmd_cold_rebuild_fts calls time.time() but cli.py has
no module-level `import time` (functions import it locally), so the
command crashed with NameError. The command had never actually run via
the CLI before — the full recovery used a standalone script and tests
called _rebuild_fts_on_target directly — so the gap shipped in 1547259.
Now exercised end-to-end: rebuild-fts on the 4-shard corpus runs 4-way
parallel (~294s) and `cold verify` passes.
The FTS pack restore produced a DEAD index — segments present, MATCH=0 —
because the verbatim shadow-table copy used INSERT OR IGNORE, so the
pack's real `_data` rowid-1 "structure" record lost the primary-key
conflict to the empty one `CREATE VIRTUAL TABLE` seeds, leaving a "0
segments" header over orphaned segments. Fix (evict.py): clear the
seeded rows, then copy verbatim (DELETE + INSERT ... SELECT), so each
fts5 shadow table becomes a byte-for-byte copy of the producer's index
and the real structure record survives.
Validated on the 3090: restore one fts pack, NO rebuild -> MATCH
'anarchism'=1189 / 'the'=1.42M (identical to rebuild-from-content);
`_data` id=1 structure record non-empty.
make cold-hydrate: rebuild FTS only when the restored index isn't
already searchable (cold verify-gated). Shipping FTS packs now makes
recovery fast (~24s/shard restore, skip the ~5min rebuild); dropping
them (--no-fts default) keeps the bucket small. Either way cold verify
gates success.
docs/cold-object-store: FTS packs restore correctly now; documented the
restore-vs-rebuild tradeoff, per-consumer guidance, and the fixed bug.
- Hydrating a new peer: M-aware `make cold-hydrate` (serial default,
bulk-tuned restore, parallel FTS rebuild from content, cold verify
self-check) — replaces the stale single-shard `cold unpack` loop.
- New "FTS: rebuild, don't restore" with the measured rebuild-vs-restore
comparison (~5 min rebuild + no extra download vs +4.76 GB for a dead
pack-restored index) and WHY the pack restore is dead: FTS5's _data
rowid-1 "structure" record is left empty because INSERT OR IGNORE
collides with the freshly-created vtable's empty header — a headless
index over orphaned segments (count looks right, MATCH returns 0).
- Invariants: FTS shadow tables are rebuilt on the consumer, not shipped
(cold pack defaults to --no-fts).
- Failure modes: cold verify's zero-filled-content + dead-FTS classes.
Make cold recovery correct-by-default and fail-loud, closing the class
that silently produced a corrupt, unqueryable corpus (2026-05-29):
- cold rebuild-fts now runs in PARALLEL (one process per shard; separate
files, no contention) and clears-first, so it also repairs the dead
index a cold-pack FTS restore leaves. Replaces the serial post-pass.
- new `cold verify`: self-check a hydrated shard set — chunk content
materialized (not zero-filled) AND FTS searchable (MATCH a word taken
from sampled content). Non-zero exit if any shard fails. Validated: it
passes the good recovery and fails the corrupt genesis-test, catching
both the zero-filled-bodies and dead-FTS classes.
- cold pack defaults to --no-fts (FTS is derived from content and the
FTS pack restore is non-functional anyway); --with-fts to opt back in.
- make cold-hydrate: serial by default (M-aware hydrate routes every
pack into all M shared target shards, so parallel workers contend —
#54); always rebuild FTS from content (drop the broken shard-000-only
gate); run `cold verify` at the end so a bad recovery fails loudly.
- make crawl-ingest writes to one central crawl db (CRAWL_DB, default
~/.arborist/crawl/web.db) instead of per-domain shards in the
peer-shared main dir: keeps locally-crawled content out of peer
sharing by default and a growing domain set under SQLite's 10-attach
cap (Makefile, docs/crawler.md).
- arborist query auto-includes the local crawl db (query() gains
extra_shards; CLI --include-shard / --no-crawl-db, default-on when
web.db exists). Fix latent --db single-file query AttributeError
(cli.py). Persist used / used_pointer_ids + retrieval_purity into
merkle_proof so read-only consumers can see which chunks fed the
answer (qa/query.py).
- arborist.read: read-only seam for dashboards / verifiers; on a
multi-source context root surface the real primary source instead of
the opaque corpus://multi-source sentinel (read.py). Backs the
arborist-viz Merkle Command Center (#000069).
- tests for extra_shards, the CLI crawl-db resolver, and the read seam.
Genesis hydrate from cold packs was broken end-to-end (the #46 SPV-wallet
test never passed):
- Phase-1 metadata restore (90M-edge fan-out + zeroblob chunk
placeholders) ran with SQLite's ~2 MB default cache — the 512 MB /
MEMORY-journal tuning was applied AFTER Phase 1, so INSERT OR IGNORE
into the indexed edges table thrashed: ~4 h/pack, never finishing.
Move the bulk PRAGMAs before Phase 1 (evict.py).
- The routed restores held one unbounded transaction; add bounded
incremental commits (edges every 1M rows, routed tables every 200k)
so the txn and in-RAM MEMORY journal stay small (cold_pack_metadata.py).
- FTS packs restore a non-functional index (shadow rows present, MATCH
returns 0). _rebuild_fts_on_target now clears-first so a rebuild from
content overwrites the dead pack-FTS idempotently (migrate.py).
Validated on the 3090: full corpus recovered (3,468,226 docs / 6,235,588
chunks / 90,592,990 edges, exact match to source), search + STRICT Q&A
working. Restore ~4 h/pack -> ~8-20 min/pack; FTS rebuilt from content
in ~5 min (parallel).
Five sections updated to match the post-2026-05-27 substrate state
(the user-payload-layout work, sibling ticket #000068, and the
2026-05-27 bench evidence are all in tree).
Verdict block (top): ADD: companion missed-answer guard -> DONE:
companion guard shipped as #000068 Phase 1+2+3 (default OFF; Phase
4 default flip NO-GO until wider bench + human spot-check).
Companion missed-answer guard section: renamed from "(proposed
sidecar)" to "(shipped 2026-05-27)". Carries the implementation
location (arborist/qa/inspect.py:diagnose_missed_answer), the full
output schema (diagnostic_version / confidence_class /
triggered_clauses / subject_tokens / missed_answer_candidate_spans
with offset_start/end/basis), the Phase 2 bench headline (2/228
fires, both Ballestrini, 100% precision, 0 FPs across 226 non-
Ballestrini), the Phase 3 demote-flag CLI surface
(--demote-on-missed-answer, default OFF), and corrected hash
discipline: Phase 1 sidecar fields fold into governance_policy_hash
only; the Phase 3 demote flag (answerability_demote_enabled) ALSO
folds into verifier_policy_hash because flipping it changes the
rendered audit_mode (a verifier-output property).
Future hardening list: "companion missed-answer falsification
guard" entry now points at #000068 instead of describing a
deterministic-sidecar to be built.
Roadmap Phase 5: "DESIGN OPEN" -> "DONE 2026-05-27" with all four
sub-phases of #000068 named individually (Phase 1 sidecar commit
2ab11d2, Phase 2 bench + Phase 3 demote commit ec55db5, Phase 4
default flip NO-GO per Dav1d 2026-05-27 §3.4). The #000068 phase
numbering is internal to that ticket; this roadmap names the
external-facing milestones.
Related links: past-tense the missed-answer guard ("when it lands"
-> "shipped 2026-05-27"); added cross-reference to
docs/tickets/ticket-000068-*.md; named both Dav1d review files by
path so a re-read can locate the inputs.
AUTOCOUNT tag (76 fixture-rows in bench/qa_questions.txt) still
matches; no test changes.
Phase 2 — bench instrumentation + measurement run
bench/qa_sweep.py picks up the answerability sidecar projection per row
(answerability_fired, answerability_confidence, answerability_denial_
pattern, answerability_answer_type, answerability_candidate_count) and
aggregates per-mode (answerability_fires + S/M/W confidence breakdown)
into a new column in the markdown summary table.
Measurement run on bench/qa_results/phase2-sidecar-on/2026-05-27T14-
16-22Z (76 questions × n=3 × claim_lattice × Hermes-3-8B × tail layout,
228 runs). Headline:
sidecar fires 2/228 (0.88%)
confidence dist 2 strong / 0 medium / 0 weak
precision 100% (2/2 fires were the Ballestrini fixture)
recall on Ballestrini 2/3 across n=3 (third run model extracted
correctly -> sidecar silent,
correct behavior)
false positives 0/226 non-Ballestrini runs
verifier verdict both fires labeled STRICT by the binary
verifier (the verifier-blind class, exactly
as predicted)
Detection rule's three-clause conjunction (denial + extraction-shape +
candidate proximity near cleaned subject tokens) is operating at the
precision floor. The strong-confidence-only firing pattern is what
calibrates Phase 3's demote threshold.
Phase 3 — opt-in demote flag (default OFF per Dav1d Phase 4 NO-GO)
arborist/qa/keys.py: answerability_demote_enabled added to
_VERIFIER_POLICY_FIELDS so flipping the flag partitions cache via
verifier_policy_hash. Justification: when on, the rendered audit_mode
changes (EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL), which IS a
verifier-output property; verifier hash must move accordingly. The
other answerability_* fields stay governance-only (sidecar
diagnostic, no audit_mode mutation).
arborist/cli.py:_render_audit_label extended with answerability +
demote_enabled kwargs. Logic:
demote_triggers = (
demote_enabled
and answerability["answerability_warning"] is True
and answerability["confidence_class"] in ("strong", "medium")
)
lattice modes:
EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL (rung transition)
POINTER-LINKED / ANCHOR-WARRANTED -> "rung · missed-answer"
(tail tag; rung itself already
signals degradation)
non-lattice modes (quote/span/entity/paraphrase):
audit_mode token unchanged + "· missed-answer" tail tag
weak confidence: NEVER demotes (Phase 2 saw zero weak fires on real
failures; reserved for future expanded detection ladder)
CLI flag --demote-on-missed-answer on both `arborist query` and
`arborist ask`, default OFF. Flows into call_policy[
"answerability_demote_enabled"] and through to result[
"answerability_demote_enabled"] so the renderer reads it without
needing the policy dict.
End-to-end verified live: 4 fresh Hermes-3-8B runs with --demote-on-
missed-answer on `songs by veronica ballestrini`, all 4 rendered
EVIDENCE-MISSED-PARTIAL · via claim_lattice (Hermes hit the failure
mode in all 4, sidecar fired strong, demote logic transformed the
label).
Phase 4 (default flip to demote-on) — NO-GO per Dav1d 2026-05-27 §3.4:
"a false sidecar warning is tolerable; a false audit-label demotion
can damage trust in correct abstentions." Phase 2 precision is 100%
but n=2 fires is too few samples to claim precision floor empirically.
Default flip blocks on wider bench + human spot-check of the warnings.
Tests: 47 total (36 Phase 1 + 11 new Phase 3 covering hash partitioning
discipline + render-label projection across all four rung/confidence
matrices). Full suite 2794 passed (delta +22 from prior 2772).
Bench output (bench/qa_results/phase2-sidecar-on/) intentionally not
committed — bench/qa_results/ is gitignored per existing convention;
the ticket carries the headline numbers + path for re-inspection.
Adds a deterministic read-only sidecar to detect a class of failure the
binary verifier is structurally blind to:
Evidence contains the answer.
Model says the evidence does not contain the answer.
Verifier sees no unsupported positive claim -> marks run clean.
User receives a false negative under EVIDENCE-WARRANTED.
The motivating case: "songs by veronica ballestrini" against the 2010
Wikipedia corpus. Hermes-3-8B under user_payload_layout=tail returned
"the specific songs by her are not mentioned in the provided evidence
blocks" when evidence E2 literally contained "Amazing", "Out There
Somewhere", "Fascinated", "What's Up With That", "Don't Say". Verifier
correctly returned EVIDENCE-WARRANTED 2/2 because the existing layered
verifier (quote / span / entity / paraphrase + Rule 8 title-relevance +
Rule 9 subject-tokens-absent + claim-count ceiling) guards unsupported
*presence*, has no hook for unsupported *absence*.
Layout fixes attention placement on the specific instance (the 5/27
n=3x75q bench confirms bookend/per_chunk recover Ballestrini); layout
alone cannot close the class -- adversarial phrasing or a bigger prompt
resurfaces the failure under any layout. The right substrate move is to
falsify "not mentioned" as a testable claim.
Detection rule (three-clause conjunction, all must fire):
A. Denial pattern in answer (sealed v1 phrase list: "not mentioned",
"not provided", "the evidence does not say", "does not mention",
"no specific", "no evidence", "cannot determine from the provided
evidence", "is not stated", "is not specified"). Casefolded +
whitespace-normalized substring match.
B. Question is extraction/list-shaped. Either a surface cue ("songs
by", "works by", "books by", "who wrote", "who composed", "what
year", "list of", "name all", ...) matches, OR the existing
arborist.qa.quantifier classifier returns intensity in {ALL,
COMPREHENSIVE, OPEN_REQUEST, MANY, PLURAL}.
C. Evidence contains candidate spans matching the answer_type within
a proximity window (default 600 chars) of cleaned subject tokens.
Candidate kinds aligned to answer_type:
title_like -> quoted_string, title_case_span, comma_list_item
person -> title_case_span
date -> year, date
Hardenings folded in from the 2026-05-27 Dav1d de-novo review:
1. Subject tokens strip cue/relation/stop words. For "songs by
veronica ballestrini" the cleaned subject is ["veronica",
"ballestrini"], NOT all four tokens. Without this the guard
false-triggers on "Harvard University" or "New York" near
proper-noun subjects.
2. Answer-type alignment. Candidate span kind must match query type
so "songs by John Smith" + evidence about Harvard/NY does not
strong-trigger.
3. Confidence class is deterministic (weak | medium | strong), not
boolean. Strong requires quoted_string near exact subject mention
+ multiple type-matched candidates. Phase 3 demote will gate on
confidence_class.
4. Cap output at 10 candidates (the per_chunk-quote-inflation
lesson). Prevents the guard becoming another claim amplifier.
5. Offsets are offset_start + offset_end + offset_basis=
"evidence_object_text", never an ambiguous single offset.
6. Cache-hit path returns answerability: None. Cached records do not
carry the evidence_map, only the rendered sources summary, so the
sidecar cannot recompute candidate spans without re-running
retrieval. Operators wanting fresh diagnostics use --burn.
7. Phase 1 stays out of verifier_policy_hash. The
answerability_sidecar_enabled / answerability_threshold /
denial_patterns_version / extraction_cues_version fields fold
into governance_policy_hash only. Phase 3 demote flag
(answerability_demote_enabled, default False) will move the
verifier hash WHEN ON because it changes the rendered audit_mode
(EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL).
Sidecar discipline (matches arborist.qa.inspect.diagnose_* sister
functions deflection / coherence / title_relevance):
- no model calls (no LLM-as-judge, no NLI, no translation)
- no audit_events writes
- no providence_cache writes
- no answer text mutation
- no claim promotion -- the trigger conjunction makes promotion
structurally impossible (only fires on denial answers)
- byte-deterministic: same (question, answer, evidence, policy) ->
same output every time
Result-dict integration: result["answerability"] is None when the
guard did not fire, or a structured diagnostic dict when it did
(diagnostic_version, confidence_class, triggered_clauses,
denial_pattern_matched, extraction_cue_matched, extraction_shape,
answer_type, subject_tokens, candidate_count, threshold,
missed_answer_candidate_spans). Read by bench_qa (Phase 2 will add
warning-count aggregate to bench rows) and CLI render.
Three return points carry the key:
- miss-path (full retrieval + verify): computed from evidence_map
- cache-hit: None (Dav1d cache-hit recompute discipline -- evidence
not stored, recompute requires re-retrieval)
- reject-broad early-return: None (no evidence examined)
Tests: 36 new pinning the three-clause logic, positive (Ballestrini)
regression, negative control (John Smith + Harvard/NY), each-clause-
alone silence, schema integrity, byte-determinism, sidecar-disabled
short-circuit, dict-shaped evidence support. All pass; existing
inspect tests (60) all pass.
End-to-end verified live via the CLI on the real corpus (2010 ~/.arborist
/shards): 3 fresh Hermes-3-8B runs on "songs by veronica ballestrini",
run 1 hit the failure (sidecar fired with confidence: strong, 351
candidates, denial: "not mentioned"), runs 2-3 model extracted
correctly and sidecar correctly silent.
Phase 2 (bench + threshold tuning) and Phase 3 (opt-in demote flag)
are open as follow-ups. Per Dav1d: NO-GO on default demote-on until
benchmark + human spot-check confirms low false-positive rate.
Full spec in docs/tickets/ticket-000068-verifier-blind-missed-answer-
guard.md (post-review hardenings section at top names the seven
load-bearing changes from the Dav1d 2026-05-27 review).
format_user_payload helper in arborist/qa/prompts.py becomes the single
source of truth for the user-turn payload. Three layouts:
tail (default) evidence first, question at end (prior behavior)
bookend question repeated before AND after evidence — counters
lost-in-the-middle on small models (≤8B)
per_chunk bookend + a one-line [for: <q>] reminder before each
evidence block; for list/extraction queries
USER_PAYLOAD_LAYOUTS constant exported; unknown layout raises ValueError.
The six _user_payload closures in query.py (3) and runner.py (3) all
delegate to format_user_payload. Quote-mode passes per_chunk_marker=None
to fall back to bookend on flat document/sources context.
Wired through both DEFAULT_QUERY_POLICY and DEFAULT_POLICY. Folds into
governance_policy_hash (the layout changes the user-turn content the
model sees, so the policy hash partitions cleanly per layout); does NOT
fold into verifier_policy_hash (verifier rules unchanged).
Makefile gets LAYOUT_DEFAULT ?= tail and LAYOUT ?= $(LAYOUT_DEFAULT) so
operators can flip per-call (LAYOUT=bookend make query Q="...") or
session-wide (LAYOUT_DEFAULT=bookend make query Q="..."). Recommendation
matrix in the Makefile comment block above the query target encodes the
2026-05-27 bench finding.
Motivating case: the Veronica-Ballestrini "songs by" failure. Hermes-3-
8B under tail layout returned "specific songs by her are not mentioned
in the provided evidence" when evidence E2 literally contained the song
names. Same query under bookend recovered the answer (with conflation
between Veronica Ballestrini and The Veronicas); under per_chunk
recovered AND disambiguated three entities. Qwen-27B unaffected by
layout. The Ballestrini case is added to bench/qa_questions.txt as a
regression fixture under "entity list", with a 4-line comment pointing
to docs/user-payload-layout.md.
2026-05-27 bench (n=3 × 75q, claim_lattice mode, Hermes-3-8B):
tail STRICT 94/225 (0.418) — control
bookend STRICT 95/225 (0.422) — +0.44pp (noise, 5pp floor)
per_chunk STRICT 72/225 (0.320) — -9.78pp (significant regression)
Verdict: tail stays default (cache-preserving and bench-confirmed
neutral). Bookend/per_chunk available as opt-in operator knobs. Per_chunk
regresses in aggregate because the per-chunk reminder over-anchors the
model on every chunk (TOO_MANY_EVIDENCE_IDS violations rose from 20 →
54; mean answer chars in 32-64KB bucket doubled from 720 → 1660). The
Ballestrini-class failure is real but rare across the curated set; a
layout fix that helps the rare case at the cost of 10pp aggregate is a
bad default trade. Documented in full in docs/user-payload-layout.md
along with the Dav1d 2026-05-27 review framing (GO for opt-in, NO-GO
for default promotion, ADD companion missed-answer guard).
CLI changes (--user-payload-layout flag on `query` and `ask`) landed
separately in commit e5ee283 alongside the #54 busy_timeout fix.
2026-05-27 v4-fixed bench on 3090: 3 of 4 parallel workers crashed with
sqlite3.OperationalError: database is locked at PRAGMA journal_mode =
MEMORY. Result: only shard 000 fully hydrated, shards 001/002/003 had
chunks rows landed but content NULL (~99.99% empty), no FTS data.
Peer functionally usable for only ~25% of corpus.
Root cause: parallel hydrate (xargs -P N) spawns N separate processes
that each open connections to all M target shards. Multiple processes
attempting PRAGMA journal_mode change on the same .db file at the same
instant serialize on a brief exclusive lock — without busy_timeout
SQLite throws BUSY immediately and the worker exception-propagates
out of _pull_pack_inner_routed before phase 2 (chunks fill) can run.
Fix: `PRAGMA busy_timeout = 30000` as the FIRST statement on every
hydrate writer connection. SQLite then waits up to 30s for any lock
instead of throwing — slowest worker gets its lock, fastest writes
go through immediately. No measurable cost when there's no
contention (busy_timeout is a wait, not a poll).
Set in TWO places:
arborist/cli.py:_cmd_cold_unpack
Immediately after connect(p), before any other PRAGMA. Covers
the PRAGMA foreign_keys = OFF that runs before the hydrate
pipeline.
arborist/evict.py:_pull_pack_inner_routed
Belt-and-suspenders: in case the function is called with
externally-built connections that didn't set busy_timeout, the
bulk-tuning loop sets it before journal_mode/synchronous/etc.
6 cold-unpack-routed tests pass (no contention in single-process
test fixture; the fix is invisible there). Real validation is the
next 3090 v5 bench against re-packed (#53 pre-sized chunks) bucket.
Real consumer-side bottleneck for cold-pack genesis is the phase-2
chunks-content UPDATE loop: each `UPDATE chunks SET content=? WHERE
chunk_id=?` grows the row from NULL to ~500 bytes, triggering SQLite
page splits, which become ext4 metadata-journal events. With 6M
chunks × 4 parallel writers, those journal events serialize and
dominate consumer wall time (~130 of the 164-min v3-revert run).
Fix: producer dumps a synthetic `_content_size` column in chunks.jsonl
carrying the on-disk byte length of each chunk's content (constant-
time SQLite `length(content)`). Consumer's phase 1 INSERT pre-allocates
the row with `content = bytes(_content_size)` instead of letting it
default to NULL. Phase 2's UPDATE then replaces same-size bytes
in-place — no row growth, no page splits, no per-row journal events.
Implementation:
arborist/cold_pack_metadata.py
_dump_generic_table for `chunks`:
Emit synthetic `_content_size` column = length(content) at the
end of the columnar header. Underscore prefix avoids collision
with any future schema column.
_restore_routed_table:
Detect `_content_size` in the JSONL header; if present (and
table == chunks), build INSERT against [chunks_cols] + ['content']
and substitute a bytes(_content_size) placeholder for the
content position. Phase 2 UPDATE later replaces those bytes.
Forward compatibility:
- Old packs (no _content_size): consumer uses today's NULL-content
INSERT path. No behavior change.
- New packs: consumer auto-detects, uses pre-sized path.
SPV-wallet trade-off:
In just-enough mode the consumer pulls only the metadata pack so
chunks land with the placeholder bytes (NOT NULL anymore). That's
a SEMANTIC CHANGE for SPV — `chunks.content IS NULL` no longer
means "JIT-fetch later." Documented in code; if SPV-mode JIT-fetch
ever ships, it must distinguish placeholder bytes (where every
byte is 0) from real content.
New test (TestPreSizedChunks.test_chunks_content_pre_sized_after_metadata_restore):
Hydrate just-enough → chunks rows have non-NULL bytes content of
correct size. Catches the regression if a future change reverts
the placeholder logic.
Expected impact: ~50-70% reduction in phase-2 wall time. Real number
lands when the next 3090 bench-max iteration runs against re-packed
bucket. 6 cold-unpack-routed tests pass.
v4 bench (2026-05-27 03:09 UTC) measured a corrupt outcome:
chunks_fts=1,561,604 on EVERY target shard regardless of chunks
count. _pull_fts_pack_into_targets had been iterating all M targets
and INSERT'ing each fts pack's shadow tables into every one of them.
Each fts pack's chunks_fts_docsize entries reference chunk_ids that
were independently auto-assigned in its source producer shard (each
source DB has chunk_ids 1..1.56M independently). Inserting all 4
packs into all 4 targets → 4× the per-target FTS rows, pointing at
chunk_ids the target doesn't own. Body searches would return garbage.
Fix: sample one id from the fts pack's chunks_fts_docsize, look it
up in each target's chunks table, INSERT only into the target where
it's found. The other M-1 targets stay untouched and receive their
FTS data from their corresponding fts pack(s) in later iterations.
In post-reshard production topology, each producer source shard's
docs all hash to ONE consumer target, so this 1:1 mapping is exact.
The test fixture is artificial (single-shard producer with docs
hash-distributed across M=4 targets) but still validates the core
property: only one target receives FTS data; the others stay empty.
FTS5 shadow tables aren't subsettable per-row (segment data is
opaque, mixed entries for many docs in one segment) so we can't
filter FTS rows to "only chunks that exist on this target" — we
copy all-or-nothing per pack. That's why the producer's post-reshard
shape (each pack scoped to one target's docs) is the structural
prerequisite for fts packs to make sense.
New regression test
(TestFtsPackRoutingRegression.test_fts_pack_only_on_owning_target):
exactly 1 of M targets has chunks_fts_docsize > 0; the others
must have 0. Pre-fix this asserted on all-4 targets having FTS
data → failed. Post-fix passes.
Returns now include owning_target_idx for forensic visibility into
which target the fts pack landed on.
5 cold-unpack-routed tests pass.
_cmd_cold_list only recognized .metadata.tar.zst and .chunks.tar.zst
key suffixes; .fts.tar.zst keys hit the else-continue and were
invisible. 2026-05-26 producer pushed 4 fts packs and cold list
reported 0 of them.
Trivial extension: add an `elif key.endswith('.fts.tar.zst')` branch
that classifies as kind='fts'. Now cold list shows all 3 kinds.
Also unblocks the make cold-hydrate target's metadata-pack discovery
filter (`kind=='metadata'`) from accidentally matching mis-classified
fts packs.
3090 bench-max v3 run at 13:25 elapsed had shards at 696-760 MB; v2
at 8:46 elapsed (default 4 KB pages) had 549-662 MB. Larger pages
made the chunk-content UPDATE workload SLOWER, not faster.
Reason: chunk-content payloads average ~500 bytes after zstd. With
4 KB pages each UPDATE rewrites a 4 KB page; with 16 KB pages the
same UPDATE rewrites a 16 KB page → 4× write amplification per
row. ext4 journal traffic increases roughly proportionally.
Bigger pages help SCAN-heavy workloads (fewer page reads to walk
a B-tree). For narrow-row-UPDATE-heavy workloads like cold-pack
chunk fill, they hurt. Comment in code captures the measurement
so a future operator doesn't re-attempt this.
The other bench-max measures stay in place:
- synchronous = OFF (per-write fsync removed)
- journal_mode = MEMORY (WAL in RAM)
- temp_store = MEMORY (sort scratch in RAM)
- cache_size = -524288 (512 MB page cache)
- mmap_size = 536870912 (512 MB read mmap)
- foreign_keys = OFF (no FK verify per row)
- deferred FTS5 rebuild post-pass
- batched 5000-row executemany UPDATEs
- bench-max #5 (this commit's revert): page_size stays at 4 KB
Next bench point: v4 against the new bucket packs that include
fts (--include-fts producer run in flight). That eliminates the
FTS rebuild step entirely on the consumer.
push_pack now does a HEAD object_size check against the bucket
before every PUT. If the object exists at the content-addressed
key AND Content-Length matches the local pack body size, skip the
upload. Applies to chunks / fts / metadata pack bodies.
Why content-addressed-key + size is the right check:
pack_hash = hash_leaf(manifest_bytes)
object_key = packs/<pack_hash>.<kind>.tar.zst
The pack_hash IS the cryptographic identity of the body via its
manifest. If the bucket has an object at this exact key, the
content is provably identical (otherwise the producer that wrote
it computed a different pack_hash → different key). Content-Length
match validates against a half-uploaded multipart that a prior
crash might have left.
Why not ETag: S3 ETag for multipart-uploaded objects is the MD5
of per-part MD5s concatenated, then MD5'd, with a "-N" suffix for
part count. Two clients using different multipart_chunksize end
up with different ETags for the same content. Unreliable for
cross-client equivalence. Content-Length is stable.
Result fields gain `skipped_reupload: bool` so callers / the
audit chain can tell whether the bucket PUT happened or was a
no-op.
Real-world impact for tonight's re-pack (--include-fts):
before: 4 chunks PUTs (7 GB re-upload, ~24 min)
+ 4 metadata PUTs (new hashes because manifest now has
_fts_pack_hashes — these MUST upload)
+ 4 fts PUTs (new)
~16 GB uplink, ~50 min wall
after: 4 chunks SKIPPED (same content-addressed key already
in bucket)
+ 4 metadata PUTs (new hashes, MUST upload)
+ 4 fts PUTs (new)
~6 GB uplink, ~20 min wall
cold_pending bookkeeping still fires in the upload path so a
half-uploaded crash recovers cleanly; skip path leaves no trace
because nothing was started.
4 cold-unpack-routed tests pass unchanged.
The 3090 genesis bench at 28-min progress mark showed all 4 workers
sitting in jbd2_log_wait_commit / do_get_write_access — ext4
filesystem journal serializing every page-extension event across
the 4 parallel writers. SQLite synchronous=OFF and journal_mode=
MEMORY skip SQLite's own fsync/WAL costs, but file growth still
goes through ext4's journal layer.
Fix: bump SQLite page_size from default 4 KB to 16 KB on each
hydrate target. Bigger pages mean ~4× fewer page writes during
chunk-fill, which means ~4× fewer file-extension events through
ext4's journal. Direct attack on the measured bottleneck.
page_size can only be set on an EMPTY database; VACUUM finalises
the change. The unpack CLI now opens each target file raw,
PRAGMA page_size = 16384 + PRAGMA secure_delete = OFF + VACUUM,
closes, then connect() applies schema as normal on the
pre-sized empty DB. Idempotent on re-runs since hydrate dirs
are always clean.
Also sets secure_delete = OFF (default is OFF in most builds but
explicit for hydrate writers): no overwrite-with-zeros on row
deletion, less write traffic during the chunk UPDATEs that might
free overflow pages.
Expected impact: ~30-50% reduction in chunk-fill wall time vs the
killed 30-min v2 run. Combined with the prior bench-max round
(journal_mode=MEMORY, deferred FTS, batched UPDATEs) and the
upcoming fts-pack-skip-rebuild path, full genesis should land
under 30 min total instead of the 75-min v2 extrapolation.
4 cold-unpack-routed tests pass unchanged.
Each producer shard now optionally emits a THIRD pack alongside its
metadata and chunks packs: an "fts" pack containing the FTS5 shadow
tables (chunks_fts_data, chunks_fts_idx, chunks_fts_docsize,
chunks_fts_config + the documents_fts_* counterparts) packed as a
fresh SQLite file inside the tar so BLOB columns round-trip natively.
Consumer detects fts_pack_hashes in the metadata pack's manifest,
pulls each fts pack, ATTACHes the embedded sqlite, INSERTs every
shadow-table row into its target's empty shadow tables, and SKIPS
the local FTS rebuild entirely.
Producer side:
arborist/cold_object.py
+ PACK_KIND_FTS = "fts"
+ FTS_SHADOW_TABLES tuple (8 shadow tables)
+ build_fts_pack(src_db_path, ...)
creates a temp sqlite, applies SCHEMA_SQL (so destination
has FTS virtual tables → shadow tables auto-created), copies
every shadow-table row from src via cursor iteration, packs
the sqlite file into tar.zst
+ ParsedManifest.fts_pack_hashes
+ parse_manifest reads _fts_pack_hashes records
+ build_metadata_pack accepts fts_pack_hashes parameter and
writes the new manifest record
arborist/evict.py:push_pack
+ include_fts: bool = True parameter (CLI --no-fts opts out)
+ Phase B.5 emits the fts pack BEFORE Phase C (metadata pack)
so its hash can be referenced in the metadata manifest
Consumer side:
arborist/evict.py
+ _pull_fts_pack_into_targets() — pulls fts pack body, extracts
embedded sqlite, ATTACHes into each target, INSERT OR IGNORE
every shadow-table row. INSERT OR IGNORE protects against
rowid collisions on other targets that don't own these chunks.
+ hydrate_from_metadata_pack_routed iterates fts_pack_hashes in
full mode, calls _pull_fts_pack_into_targets per pack
+ _pull_pack_inner_routed returns fts_pack_hashes_referenced in
its result dict (mirrors chunk_pack_hashes_referenced)
CLI / Makefile:
arborist cold pack --no-fts (opt-out)
make cold-hydrate (auto-detects: if
chunks_fts_data is
already populated
on shard 000 after
unpack, skip the
rebuild post-pass)
make cold-hydrate HYDRATE_REBUILD_FTS=1 (force rebuild)
make cold-hydrate HYDRATE_REBUILD_FTS=0 (skip rebuild)
Schema:
cold_pending.kind CHECK extended to include 'fts'
pack_key() accepts kind="fts" → packs/<hash>.fts.tar.zst
Expected wall-time impact on the 3090 genesis bench:
with fts in packs: no rebuild step → ~5-10 min total wall
without fts: rebuild post-pass needed → ~15-20 min
Trade-off: ~30-50% larger bucket (FTS shadow data per shard) for
~70-90% faster consumer hydrate. Producer flips the trade via
--no-fts. The fts pack is optional in the manifest (empty list →
consumer falls back to rebuild) so old bucket data without fts
packs continues to work unchanged.
34 cold-unpack-routed + migrate + planner tests pass.
Real-measured bottleneck during the 3090 genesis run (2026-05-26):
all 4 parallel workers sitting in jbd2_log_wait_commit /
do_get_write_access — fighting for ext4's single filesystem journal.
The reshard executor was fast because it ran FTS5 rebuild
SEQUENTIALLY (one process per target in turn); the parallel 4-way
unpack collapsed that into journal contention.
Two coupled changes:
1. arborist/evict.py — _pull_pack_inner_routed
* Drop the _rebuild_fts_on_target call from phase 2c entirely.
Parallel inner loop now does: download → metadata route →
chunks fill. NO FTS5 writes during the hot parallel phase.
* Bulk-load PRAGMA tuning on every target connection:
synchronous=OFF no fsync per commit (already had this)
journal_mode=MEMORY WAL in RAM, not on disk (was WAL)
temp_store=MEMORY sort scratch in RAM
cache_size=-524288 512 MB page cache per connection
mmap_size=536870912 512 MB read mmap
Genesis crash recoverability = re-pull from bucket, so
durability of intermediate state has no value — these tunings
trade durability for throughput.
2. arborist/cli.py — new `arborist cold rebuild-fts --shards-dir DIR`
subcommand. Sequentially rebuilds chunks_fts + documents_fts on
every shard, one at a time. Each shard gets the full filesystem
journal in its turn. Total wall = sum of single-shard FTS
rebuild times, NOT 4× contention.
3. Makefile — `cold-hydrate` chains the FTS rebuild automatically
after the parallel unpack (only when HYDRATE_MODE=full, since
just-enough has no chunk bodies to index anyway). Operator can
skip with HYDRATE_REBUILD_FTS=0 for a 2-pass workflow.
Expected wall time on 3090:
parallel hydrate (download + chunks fill): ~5-10 min
serial FTS rebuild × 4 shards: ~5-15 min total
total: ~15-25 min
vs the killed 2.5-hour run.
34 existing cold-unpack-routed + migrate + planner tests still pass
(the tests' FTS check runs against the reshard path, which still
calls _rebuild_fts_on_target inside the executor — that path is one
process, no parallel contention).
Two genesis paths the SPV-wallet design supports, now both exposed
via the same make target:
make cold-hydrate HYDRATE_DIR=… HYDRATE_M=4 HYDRATE_MODE=full
pulls metadata packs + every referenced chunk pack. Full corpus
offline-queryable post-hydrate.
make cold-hydrate HYDRATE_DIR=… HYDRATE_M=4 HYDRATE_MODE=just-enough
pulls only metadata packs (~4 GB on a 37 GB corpus → ~10×
bandwidth reduction). chunks rows land with content=NULL, every
chunk-body query misses local cache and falls through to a
future JIT-fetch path (cache miss → CDN / mesh / re-pull). The
SPV-wallet headline shape.
Defaults to full for backward compatibility. The mode-flag validation
guards against typos at recipe time (invalid mode → exit 2 before
firing parallel boto3 traffic).
Bench target: measure both paths back-to-back on 3090. Just-enough
should bound the network phase only (no chunk-body fill, no FTS5
rebuild); full adds the chunk-restore phase that #000067's bench-
max round just optimized.
Real-world measure 2026-05-26 18:43 UTC: hydrating an 11 GB
post-reshard corpus (3.47M docs / 6.24M chunks) into a fresh 4-shard
peer ran at ~32 MB/min per shard — extrapolating to ~4 hours total,
the same speed as a fresh XML ingest. The whole point of a cold-pack
restore is being MUCH faster than re-ingest; killed mid-flight and
shipped these three optimizations:
(1) Build leaf_hash → (target_idx, chunk_id) map once up front
from each target's chunks table (M scans, total ~6M rows).
Replaces M=4 SELECTs per incoming chunk_body — was 24M
lookups, now 6M scan-once. ~4× win on the dispatch step.
(2) Batched executemany UPDATE per target (BATCH=5000) — replaces
the per-row `with transaction(target): UPDATE; INSERT_FTS`
pattern that opened 6M tiny transactions. ~10× win on disk
write throughput.
(3) Skip chunks_fts during the fill loop entirely; defer to one
bulk rebuild per target after all chunks are in place. Reuses
the existing arborist.migrate._rebuild_fts_on_target primitive
(decompresses via unpack_chunk, streams chunks → FTS via
batched executemany). Each per-row chunks_fts insert costs an
inverted-index update; bulk rebuild is ~10× faster than
incremental.
Plus PRAGMA synchronous=OFF on each target connection: genesis is
end-to-end rebuildable (a crash mid-hydrate leaves empty shards we
re-pull from the bucket), so durability of intermediate WAL pages
is not required. WAL stays bounded by PRAGMA wal_checkpoint(TRUNCATE)
at the end of phase 2b (already done implicitly by _rebuild_fts_on_target
in phase 2c).
Expected speedup: ~30-60× combined. Real number lands when the
re-run on 3090-ai.foxhop.net completes.
All 34 cold-unpack-routed + migrate tests pass unchanged.
Adds two missing pieces to the cold-pack lifecycle:
1. ALLOW_LICENSE_CLASS pass-through on cold-pack and cold-pack-all.
Without this, packs from a corpus containing any `unknown`-class
docs (html crawls / textbooks) refuse to push to a public bucket
per #000061 Gap 2. Real-world bench on 2026-05-26 hit this
because the production corpus has crawl + textbook content.
Usage:
make cold-pack-all ALLOW_LICENSE_CLASS=unknown
make cold-pack DB=~/.arborist/shards/000.db \
ALLOW_LICENSE_CLASS=unknown
2. cold-hydrate — M-aware genesis-from-bucket as a make target.
Discovers every metadata pack in the bucket via `cold list
--no-manifest` + a tiny python json filter, then fires
$(JOBS)-way parallel `arborist cold unpack` workers (one per
metadata pack) into the same target shards directory. Each
worker writes to one disjoint target shard because the bucket's
packs come from the post-reshard #000065 source layout (each
producer shard's docs all hash to one consumer target) — so
zero write contention across the parallel workers.
Usage:
make cold-hydrate \
HYDRATE_DIR=~/.arborist/shards-genesis-test \
HYDRATE_M=4
Why these belong here (not in /tmp wrapper scripts that get tossed):
fox 2026-05-26 — "make sure the pack and upload and the pull and
ingest all happen with makefile targets". The cold-pack lifecycle
is a first-class operations surface; throwaway tmp scripts hide
the real interface from future operators.
Both targets are listed in `make help`. The .PHONY line covers
cold-hydrate so a directory named cold-hydrate can't shadow it.
Sequence to repeat today's run end-to-end:
producer: make cold-pack-all ALLOW_LICENSE_CLASS=unknown
consumer: make cold-hydrate HYDRATE_DIR=~/.arborist/shards \
HYDRATE_M=4
Today's in-flight 3090 hydrate (started before this commit) used
the equivalent shell loop; from now on every operator uses the
make target.
Self-contained benchmark for the SPV-wallet validation. Records:
PRODUCER bucket state + pack count + compressed bytes
CONSUMER wall time, exit status, post-hydrate shard sizes,
per-shard documents/chunks/edges counts
Driver runs against the real DO Spaces bucket + the real fresh peer
on 3090-ai.foxhop.net. Writes one JSON artifact per run to
bench/results/cold-pack-roundtrip-<ISO>.json so a future operator
can diff hydrate times across pack-format changes (#000061 v3 →
graft mode #000066 → mesh-pull future).
Consumer command uses /usr/bin/time -v wrapped around
`arborist cold unpack --hydrate-shards-dir … --hydrate-M 4 --full`.
Hydrates into ~/.arborist/shards-genesis-test/ so it doesn't
clobber anything on the 3090.
No new ticket — this is task #45/#46 instrumentation. Existing
tests untouched.
Today's hydrate_from_metadata_pack writes every row into one shard.
With the corpus in M=4 hash-routed topology (#000065), a fresh peer
pulling packs must land each doc on shard_for_document(root, M) —
same routing function as the producer — or the consumer's M=4
ATTACH-and-route assumption is just decoration over a single-shard
reality.
Code (new entry points alongside the existing single-conn ones):
arborist/cold_pack_metadata.py
+ restore_shard_metadata_routed(targets, M, table_dir)
+ _restore_routed_table (per-document tables)
+ _restore_edges_fan_out_routed (edges by src_root)
+ _shard_idx_for_root helper (mirrors arborist.document)
+ routing rules: _ROUTED_BY_COL / _CONSOLIDATED_TO_SHARD_0
(mirror migrate.py's ROUTED_BY_DOCUMENT_ROOT / CONSOLIDATED_TABLES)
arborist/evict.py
+ hydrate_from_metadata_pack_routed(targets, backend, hash, M=, mode=)
+ _pull_pack_inner_routed (mirrors _pull_pack_inner; phase 2
chunk-body fill iterates every target shard — leaf_hash lookup
naturally hits at most one since each chunk's metadata row
landed on exactly one target during phase 1)
arborist/cli.py
arborist cold unpack
+ --hydrate-shards-dir DIR (M-aware genesis path)
+ --hydrate-M N (default 4 matches #000065)
legacy --db / --global-shards-dir path unchanged
Behaviour notes:
- FK enforcement off on target writes (cross-shard refs are valid
under hash routing, same fix as #000065 reshard executor)
- audit_events lands on target 0 (Option A consolidation)
- corpus-wide tables (snapshots / concepts / aliases / providence)
consolidate to target 0
- per-document tables route by document_root (documents,
document_http_meta, chunks, merkle_nodes)
- edges route by src_root (matches migrate.py)
- derivations route by core_root (matches migrate.py)
- existing single-conn API unchanged — callers that didn't pass a
shards-dir get the legacy single-shard behaviour
4 new tests:
test_pack_then_hydrate_routed — end-to-end pack → hydrate → assert
every doc on its hash-routed target, no doc on the wrong shard
test_corpus_shard_count_set_on_routed_hydrate — meta plumbing
test_M_mismatch_rejected
test_empty_targets_rejected
All 59 prior tests still pass.
Refactor opportunity (not taken): _ROUTED_BY_COL duplicates
migrate.py's ROUTED_BY_DOCUMENT_ROOT. A shared arborist/multi_shard.py
module would serve both reshard and graft (#000066). Left as
follow-up since the duplication is small and graft is still scaffold.
Unblocks #46 genesis on 3090: that's now a single arborist cold unpack
--hydrate-shards-dir ~/.arborist/shards --hydrate-M 4 invocation
instead of the α two-step kludge (hydrate-then-reshard).
Open ticket. Today's hydrate_from_metadata_pack takes one conn and
writes every incoming row into one shard — fine when the corpus
was a single shard, broken now that #000065 put the producer in
M=4 hash-routed topology. A fresh peer pulling packs must land each
doc on `shard_for_document(document_root, M)` — same routing
function as the producer — or the consumer's M=4 ATTACH-and-route
assumption is just decoration over a single-shard reality.
Plan:
1. Add corpus_shard_count to pack manifest (read from source meta
during dump_shard_metadata) — pack carries the M it was built
against.
2. restore_shard_metadata_routed(targets, M, table_dir) in
cold_pack_metadata.py — mirrors _route_per_doc_table from
migrate.py (per-document tables route by document_root /
src_root / core_root; consolidated tables all go to target 0).
3. hydrate_from_metadata_pack gains a targets / shards_dir param.
4. arborist cold unpack --shards-dir DIR initialises M target
shards from the manifest's corpus_shard_count and routes.
5. Regression test: pack 2 shards → hydrate into fresh 4 shards
→ assert every doc on its hash-routed target.
Refactor question (raised, not decided): the routing rules
(ROUTED_BY_DOCUMENT_ROOT, CONSOLIDATED_TABLES) currently live in
migrate.py. Either duplicate them in cold_pack_metadata.py (fast)
or factor into arborist/multi_shard.py (cleaner, also serves
#000066 graft mode). Shared module is more honest.
Prerequisite for #46 (genesis on 3090 from cloud). Without this,
genesis is a 2-step α-kludge (hydrate-then-reshard) that wastes
~30 min and treats packed shards as if from an arbitrary topology.
Index entry bumped; next-id 67→68. Per-ticket spec doc to follow
when the implementation gates open.
2026-05-26 19:47 UTC. ~94 min wall. 3,468,226 globally-unique docs +
6,235,588 chunks + 90,592,990 edges + 3,468,403 audit events re-routed
from non-deterministic spray-by-ingest-order layout to canonical
content-hash M=4 layout (shard_idx = int(document_root[:8], 16) % 4).
Final state:
per-shard doc uniformity within ±0.04% (theoretical max ±0.05% for
first-32-bit SHA-256 prefix)
audit chain consolidated to shard 000 via Option A (re-sorted by ts,
re-chained; bodies preserved unchanged; tail event type=reshard
carries plan+result body, hash 8da3aa19…)
on-disk sizes: 000=11.0 / 001=8.8 / 002=8.8 / 003=8.8 GB
validation: chunks delta 176 (0.003%) + edges delta 547 (0.0006%)
are cross-shard dupes from re-ingest history, collapsed by
INSERT OR IGNORE; within the 1% tolerance gate
smoke queries: Barack Obama / YouTube / Albert Einstein all
returned proper evidence from correct (hash-routed) shards
chain-check-shards: 0 breaks on every shard
Two mid-flight defect fixes (also committed):
04edff7: derivations.src_root FK guard fired on legitimately
cross-shard refs → writer connection PRAGMA foreign_keys = OFF
c86d5ac: WAL accumulated ~37 GB across passes (SQLite auto-checkpoint
blocked by open reader cursors) → _checkpoint_truncate between
phases. Production migration was rescued mid-flight by manual
sibling-connection wal_checkpoint(TRUNCATE) freeing 27.7 GB.
Closes#000065 in both the index and the per-ticket file. The
per-ticket design doc stays open as a design-log artifact (its
content is still the right reference for the next reshard / for
graft mode #000066).
Follow-on tracked as tasks #44–#47:
#44 re-pack post-reshard shards into DO Spaces (current bucket
packs are stale, still in pre-reshard topology)
#45 verify bucket pack hydration is deterministic against the new
content-hash layout
#46 genesis a fresh peer on 3090-ai.foxhop.net from cloud — first
real SPV-wallet end-to-end test
#47 retire stale pre-reshard bucket packs after #46 confirms
Add PRAGMA wal_checkpoint(TRUNCATE) at two points in
_execute_all_at_once so committed WAL pages don't pin disk through
subsequent passes. Production migration on 2026-05-26 hit 7 GB free
disk (down from 89) because SQLite's auto-checkpoint can't run while
a reader cursor is open, and FTS rebuild keeps a SELECT cursor open
through 1.5M chunks per target. Across 4 targets the FTS rebuild
plus audit consolidate held ~37 GB of committed-but-unreclaimed WAL.
Manual sibling-connection wal_checkpoint(TRUNCATE) freed 27 GB
mid-migration.
Checkpoints land at:
* end of _rebuild_fts_on_target (after the SELECT cursor is
explicitly cur.close()'d so the TRUNCATE checkpoint can actually
fire — TRUNCATE/RESTART block on active readers)
* end of _consolidate_audit_chain (after the 3.47M-row giant
transaction commits, before the next phase touches the same
connection)
VACUUM is already implicitly a checkpoint, so the existing per-
target VACUUM pass continues to handle the final checkpoint
naturally.
Helper _checkpoint_truncate(conn) returns the (busy, log_frames,
checkpointed) tuple SQLite emits; for the serial executor, busy=1
is improbable since each phase finishes before moving on.
Regression test
(TestWalCheckpointing.test_no_large_wal_after_migration) asserts
no WAL file exceeds 4 MB after migration completes. Without the
checkpoint calls this would fail on real-sized corpora; with them
the test passes deterministically.
Doesn't affect the running migration (it loaded the module from
memory before this commit). Future reshards run with bounded WAL —
no near-ENOSPC scares.
The 2026-05-26 cutover crashed mid-build with sqlite3.IntegrityError
"FOREIGN KEY constraint failed" inside _route_per_doc_table on the
derivations table.
Root cause: derivations.src_root carries a FK to
documents.document_root, but under content-hash routing a derivation
row's src_root can legitimately reference a surface doc that hashes
to a DIFFERENT target shard than the derivation's core_root. The FK
is a single-shard-era guard; it must stay live for the runtime
write path (to catch typo'd inserts into the wrong shard) but must
be OFF for the migration writer which legitimately produces
cross-shard refs.
Fix: arborist/migrate.py _connect_target now applies
`PRAGMA foreign_keys = OFF` after SCHEMA_SQL executescript runs.
Schema's own `PRAGMA foreign_keys = ON` still applies to the schema
DDL pass (and runtime connect() / connect_query() still get FK=ON
since they don't touch this helper). Only the migration writer is
relaxed. Documented inline.
Regression test
(TestCrossShardForeignKeys.test_cross_shard_derivation_succeeds)
synthesizes a derivation row whose core_root and src_root hash to
different M=4 target shards, runs the migration, asserts the row
lands on core_root's target with src_root pointing cross-shard. Pre-
fix this raised IntegrityError; post-fix it passes.
Originals untouched on the production host — the executor crashed
BEFORE the atomic-promote step, so .db files are intact;
~/.arborist/shards/00X.db.new files from the failed run will be
cleared by the next attempt's "stale .new before opening" cleanup
hook (already in _execute_all_at_once).
Scaffold-only ticket. Captures the architecture for taking the
#000061 cold-pack format and adding a second mode beside hydrate:
overlay an existing pack onto a populated shard set ("graft").
Surfaced while running the #000065 reshard cutover and fox extended
the design: each pack carries a `corpus_name` field
(wikipedia-2010, wikipedia-current, arxiv-cs, ...), making
`arborist cold graft wikipedia-current` feel like `apt install`.
Three concerns analysed:
doc/chunk/edge overlay trivial (INSERT OR IGNORE on content-
addressed PKs collapses dupes)
FTS5 overlay trivial (new chunk_ids → new fts rows)
audit chain overlay the only hard part — three approaches:
A graft receipt (chosen): one event in
host chain carrying pack_hash +
event_count + first/last hashes; pack
file is the durable witness; zero
schema cost; aligned with v8/v9
witness pattern
B re-chain everything: rejected — graft
is frequent so invalidating external
refs is wrong tradeoff (different
story from the one-time reshard)
C chain forest with chain_id col: right
answer when graft dominates lifecycle
but premature now
Long-game payoff: mesh-peer-corpus-merge. Two peers diverge over a
partition, each carries packs the other lacks, reconciliation =
exchange + graft what's missing. Makes "mesh of arborists"
coherent rather than "fleet of arborists."
Scaffold gated on (a) #000065 lands+stabilises, (b) a second
corpus exists to graft, (c) at least two peers want to exchange.
No code until then; the design lock is what the ticket buys.
Index bumped Next ID 000066→000067.
Reshard no longer creates a parallel "shards.v2/" directory. Builds
write to "<target_dir>/00X.db.new" alongside originals; when the
executor's validation passes, each .db.new is atomically renamed to
its final 00X.db name via os.replace (POSIX-atomic per file).
For an in-place migration (target_dir == source_dir, which is the
canonical use case), the rename REPLACES the original shard files.
~/.arborist/shards/ never contains a parallel "v2" or "next" or
"backup" directory — it always holds exactly one corpus, just with
the new topology after promote completes.
Validation gate (refuses to promote on mismatch):
* --expected-row-counts <snapshot.json> threads the pre-migration
snapshot's documents/chunks/edges totals through to the executor.
* Tolerance: ±1% to absorb the dupe-collapse from INSERT OR IGNORE
on cross-shard duplicate document_roots (166 dupes measured
pre-migration; <0.005% drift).
* Mismatch → RuntimeError, .db.new files left in place for
inspection, no rename performed.
CLI:
arborist corpus reshard --to 4 \
--source-dir ~/.arborist/shards \
--target-dir ~/.arborist/shards \
--audit-events-ndjson /tmp/audit-events.ndjson \
--expected-row-counts bench/results/pre-migration-snapshot.json
3 new tests:
* test_target_dir_only_has_db_files_after_completion — no .db.new
sidecars survive a successful run
* test_inplace_reshard_overwrites_originals — target_dir ==
source_dir works end-to-end; final files are the new hash-routed
shards
* test_validation_failure_leaves_new_files — bad expected counts
trip the validation guard; .db.new files survive; no .db files
promoted
Stale .db.new files from a prior failed run are unlinked before
opening fresh targets, so a partial-fail-and-retry is idempotent.
WAL/SHM sidecars removed at promote time so the new live .db
produces fresh sidecars on next open.
Build the 'all_at_once' executor and wire it into the CLI as
`arborist corpus reshard`. Plan-only preview against fox's host
confirms the planner picks all_at_once at 95.5 GB free, 64.2 GB
peak draw, 31.3 GB free at peak (well above the 4 GB safety).
Executor (arborist/migrate.py):
* ROUTED_BY_DOCUMENT_ROOT — documents, document_http_meta, chunks,
merkle_nodes, edges (by src_root), derivations (by core_root).
Each row hashed to shard_for_document(root, M) and INSERT'd into
the chosen target.
* CONSOLIDATED_TABLES — snapshots, concept_relations,
concept_token_idf, citation_aliases, term_aliases,
providence_cache, falsifications all land on canonical
target shard 000.
* REBUILT_ON_TARGET — chunks_fts + documents_fts rebuilt from the
materialized data after content moves. chunks.content is
zstd-packed at rest so the rebuilder decompresses via
arborist.compress.unpack_chunk before inserting plaintext into
the FTS5 index.
* Audit chain consolidation (Option A) — all rows from
/tmp/audit-events.ndjson are sorted by (ts, src_shard, src_seq),
re-chained with fresh event_hash values, and INSERT'd into
target shard 000. Bodies preserved unchanged for forensic
fidelity.
* One 'reshard' audit event appended at the tail, carrying the
plan + result as the body. An operator months later can answer
"where did this corpus topology come from" from this one row.
* corpus_shard_count meta stamped on every target shard.
* VACUUM each target at end to reclaim INSERT-pattern fragmentation.
CLI: `arborist corpus reshard --to M --source-dir DIR --target-dir DIR
[--plan-only | --force-strategy X
| --allow-in-place | --dry-run]`
11 integration tests build a tiny 2-shard corpus, run the migration,
verify per-doc routing, chunk-follows-doc, audit chain integrity,
reshard event at tail, corpus_shard_count meta on every target,
chunks_fts searchability, doc count preservation, dry-run no-op,
error handling.
Also: get_meta() now indexes by position so callers without
row_factory=sqlite3.Row don't trip. No behavior change for callers
that DO use Row factory.
Strategy B (per_source_shard) and C (streaming_row) are stubbed in
the planner (peak-draw estimators wired) but execute_plan raises
NotImplementedError for them. Not needed at fox's current disk
(88+ GB free); skipping the build until that's ever the constrained
path.
Disk-aware strategy picker for the corpus reshard. Pure-compute API:
feed it (corpus facts, free bytes) and it returns a HydrationPlan
naming one of three strategies + a peak-draw estimate.
Strategies (preference order):
all_at_once - direct source→target with overlap; fastest;
peak draw = corpus + WAL × M_target + VACUUM
per_source_shard - pack→delete→hydrate per round; safest;
peak draw = pack + WAL + 1 target VACUUM
streaming_row - in-place mutation, --allow-in-place opt-in;
peak draw = WAL only
Safety budget: 4 GB margin above the strategy's peak draw — the
buffer that survives one bad sort + one badly-sized WAL grow.
Production-host preview at 95 GB free / 38 GB corpus:
strategy: all_at_once
peak draw est: 64.2 GB
free at peak: 31.2 GB (well above safety)
rationale: all_at_once: peak draw 64.2 GB + 4.3 GB safety ≤ free 95.4 GB
The plan's full readout (strategy, free bytes, peak draw, rationale)
is JSON-serializable so it goes into the migration audit event as a
single forensic record.
14 tests cover: strategy pick by free-disk level, --force_strategy
override, --allow-in-place gating, skewed shard sizes, json-
serializable audit body, target_M validation.
No execution yet — this is just the planner. Next: the strategy A
executor (direct source→target read+write).
