Sibling to #000070 on the second axis of Joseph (@TrudoJo)'s spec.
Joseph posted the bridge-framework extension 6h after the 6-dim
spatial-ontology: chain A has its own world rules, chain B has its
own world rules, a bridge exists only where a deterministic
agreement layer can be proven. Three kinds: Agreement, Translation,
Embassy.
Phase 1 is doc-only -- the primary contribution is a retroactive-
validation appendix naming three already-shipped arborist primitives
as Joseph-bridge-shape patterns one-for-one:
#000066 graft-receipt mode = Agreement bridge
#000056 Operation Sandwich = Translation bridge (opus-mt hash
IS the translator_id)
#000059 graveyard (proposed) = Embassy bridge (POINTER-LINKED,
banner-labelled, never promotes)
The framework is not a new requirement on arborist. It is a unifying
spec that names what we already do in 3+ places, plus surfaces a
fourth pattern (bridge atlas for multi-instance federation with
potentially-different governance_policy_hash peers) as genuinely new
work for arborist/mesh/.
Bundle:
- docs/tickets/ticket-000071-world-bridge-grammar.md (~440 lines)
- docs/TICKETS.md index row + Next ID 000071 -> 000072
- docs/tickets/ticket-000070-spatial-anchor-pi-w-object.md §6 gains
one-line cross-reference to #000071 as adjacent axis
- Joseph memory extended to cover both frameworks
Phase 2/3/4 deferred to separate tickets. Hard constraints: A1-A3,
no audit_mode token, no SQL change at Phase 1, sovereignty rule.
Status: open, awaiting dav1d Phase 1 review + fox go/no-go.
Replaces the hand-rolled BM25 sidecar (arborist/wallet/sidecar.py, ~690
LOC) with a slim SQLite file that just COPIES the source shard's FTS5
shadow tables verbatim + minimal doc/chunk metadata. Cloud retrieval
then runs SQLite FTS5 bm25() on the same bytes the local shard uses —
bit-for-bit parity by construction. 5/5 source + audit_mode agreement
on the smoke fixture between local corpus-query and cloud-query against
the new manifest-fts.json.
Why
---
Custom binary sidecar was per-document BM25; main encyclopedia articles
got length-normalized so hard that on "why did the dinosaurs go extinct?"
"Edwina, the Dinosaur Who Didn't Know She Was Extinct" beat "Dinosaur"
(measured cloud-vs-local divergence). Local FTS5 indexes per-chunk so
each chunk is a moderate-length doc and the main article wins multiply.
Different granularity, not a tuning knob — fix is to use the same
indexer cloud-side.
What ships
----------
- arborist/wallet/fts_sidecar_build.py — builder. ATTACH source shard,
copy documents (root/uri/title only), copy chunks (id/root/idx/leaf
only, NO content), CREATE VIRTUAL TABLE chunks_fts/documents_fts with
same DDL as source, bulk-copy the four shadow tables verbatim,
VACUUM. 8.78 GB shard → 2.15 GB sidecar (24.5%) in ~45 s; full
4-shard wiki corpus 37.4 GB → 8.1 GB (21.7%) in ~3 min.
- arborist/wallet/bucket.py: FtsSidecarShardClient — downloads slim
sidecar once into ~/.arborist/sidecar-fts-cache/<hash>.idx.db, opens
read-only sqlite3 (check_same_thread=False for parallel shard fan-
out), runs FTS5 MATCH locally. Chunk content fetches via blobs/<hash>
with HTTP-range big-shard fallback when blobs aren't published.
MultiShardSidecarCorpus simplified to fts_sidecar_url ∨ bucket-direct
(both are FTS5 backends; merge by raw bm25 MIN ascending).
- arborist/qa/corpus.py: SidecarBucketCorpus.higher_is_better=False
(FTS5 bm25 is negative, lower=better). chunks_for_doc dispatches on
fetch_chunk_body attr for the slim-FTS5 client. apply_title_boost
imports tokenizer helpers from new arborist/qa/_text_norm.py.
- arborist/qa/_text_norm.py — fold_accents, numeral_expand,
tokenize_text, STOPWORDS — extracted from the deleted sidecar.py so
apply_title_boost keeps its lexical shape.
- arborist/cli.py: `arborist sidecar build-fts` subcommand; old
`sidecar build`/`sidecar search` removed. cloud_query recognizes
fts_sidecar_url + sidecar_url alike.
- Makefile: `sidecar-build-fts` + `sidecar-build-fts-all` targets;
`sidecar-build` + `sidecar-search` removed.
- scripts/upload_fts_sidecars.py — boto3 producer: uploads slim
sidecars to clones/sidecars-fts/<n>.idx.db, publishes
clones/manifest-fts.json (4 wikipedia shards inherit existing
shard_url for content fallback; ACL public-read; idempotent on
size match). Existing manifest-sidecar.json untouched.
- tests/test_qa_corpus_functional.py + test_qa_corpus_integration.py
converted from build_sidecar → build_fts_sidecar; 6 fixtures pass.
- bench/slim_fts_parity_bench.py — local 3-way bench
(legacy/corpus/slim_fts) over the smoke fixture.
Validation
----------
- Per-shard FTS5 parity: slim sidecar returns IDENTICAL rowids + bm25
scores to the source shard for top-10 of "dinosaurs extinct".
- 3-way bench (legacy local / corpus local / slim-FTS5 over real
bucket fallback): 5/5 source agreement AND 5/5 audit_mode agreement
between corpus and slim_fts. Q5 legacy disagreement (Edwina vs
Dinosaur) is the pre-existing 2000-line query() retrieval quirk,
unrelated.
- End-to-end cloud query against published manifest-fts.json (cold-
start, ~149 s sidecar download once): STRICT · Dinosaur, every
quote verified (2/2).
- Full pytest suite: 2737 passed, 28 skipped, 1 xfailed. One pre-
existing failure (tests/test_doc_counts.py — claim_pack docs row-
count drift) and one pre-existing cold_object failure, both reproduce
on main HEAD.
Bucket state
------------
- s3://arborist/clones/sidecars-fts/00[0-3].idx.db (8.1 GB) — new
- s3://arborist/clones/manifest-fts.json — new
- s3://arborist/clones/manifest-sidecar.json — kept live (deprecated
but still readable; downstream callers should switch to
manifest-fts.json)
Updates the spatial-anchor ticket so the pre-review empirical bench
(landed in 55b651f) reads as a self-contained dav1d review packet:
- Status line: notes bench shipped in 55b651f
- §7a Pre-review empirical evidence: expanded with reproducibility
notes (RNG seed 0xa8c90e551fd34427, wall ~2s, stdlib-only), the
five bench purposes mapped to open questions, full headline-results
table, and per-headline "what each one buys dav1d" prose for Q1
through Q9.
- §3 Land sequence: step 1 marked done in 55b651f with explicit
pre-review-bench scope; step 2 annotates that 5 of 10 open
questions now resolve by measurement.
- docs/TICKETS.md index row: leading status cell now opens with the
bench headline + per-section numbers so the index reads as the
dav1d handoff summary, not just "open · awaiting review".
No new code, no new commits to the bundle itself -- 55b651f remains
the bench's shipping commit. Pure documentation refresh.
New ticket for Joseph (@TrudoJo)'s 6-dim procedural spatial-anchor
framework as the first verifier kernel under the #000013 v7-W reserved
namespace. A single committed 32-byte SHA-256 hash deterministically
expands into six 32-byte regions H1..H6 via the HMAC-SHA-512 KDF already
shipped in arborist/substrate/anchor_prg.py (#000035); each region drives
one quantized object dimension under a fixed canonical mapper. Domain
separation from #000035 by dedicated spatial_anchor_seed published in
the v7-W manifest -- preserves #000035's KAT freeze + dav1d 2026-05-11
final review intact.
Bundle:
- docs/tickets/ticket-000070-spatial-anchor-pi-w-object.md (718 lines):
full spec with 8 design-choice subsections, working Python sketch,
12-NOT scope boundaries, 8-row cross-references, five-step deletions,
10 open questions for dav1d.
- bench/spatial_anchor_validation.py: pure-stdlib pre-review evidence
(~2s, RNG-seed-pinned, reproducible). Five benches: avalanche,
cell-distribution uniformity, collision vs birthday-bound, cross-region
independence, domain separation.
- bench/spatial_anchor_validation_results.md: report from first run.
- Makefile: 'make bench-spatial-anchor [SPATIAL_N=N]' target + PHONY.
- docs/TICKETS.md: index row + Next ID 000070 -> 000071.
Bench headlines (N=10000):
- Avalanche mean 767.85 bits (PRF null 768, z=-0.49) -> PASS
- Cell-distribution chi^2 |z|<1 at L=2,3,4 -> PASS
- Birthday-bound ratio obs/exp 0.989/1.038 at populated L -> PASS
- Cross-region Pearson all 15 pairs < 2sigma -> PASS
- Domain separation Arm A 767.91 / Arm B exact 0 collision -> PASS
Five of dav1d's ten open questions (Q1 seed source, Q2 segmentation,
Q3 position mapper, Q8 endianness, Q9 KAT adversarial vectors) now
resolve with measurements rather than appeals to PRF authority.
Q4/Q5/Q6/Q7/Q10 remain non-empirical design decisions.
Status: open, awaiting dav1d review + fox go/no-go. No registry slot
booked, no substrate-paper amendment landed, no kernel module created.
The Hank-Scorpio-vs-Mr.-Burns problem: body BM25 alone outranks the
canonical primary-source article ("Homer Simpson") below sibling
articles that incidentally mention the same tokens. The sidecar
pipeline already had title-boost-with-extras-penalty; lift it into
a shared utility that the orchestrator applies to ANY adapter's
fts_body output.
arborist/qa/corpus.py:apply_title_boost(hits, query, *,
higher_is_better)
Stems both sides (possessive + plural collapse), numeral-expands
(7↔VII), accent-folds (é→e). Effective bonus per hit:
max(0, overlap - extras/2) * boost
where extras = title tokens NOT in query. Score direction honors
each adapter's convention (BM25 negative → subtract; sidecar BM25
positive → add).
arborist/qa/corpus_query.py:run_query
Now oversamples fts_body by 4× and reranks via apply_title_boost
before slicing to top_k. Without oversampling the right primary
can sit at rank 7-15 in the body BM25 output and get cut before
the rerank sees it.
bench/three_way_bench.py: NEW — drives same fixture through
- `arborist query` (legacy 2000-line pipeline)
- `arborist corpus-query` (new, local shards via Corpus + run_query)
- `arborist cloud query` (new, sidecar via Corpus + run_query)
prints 3-column table + flags primary-source disagreements.
Smoke fixture (5 questions, all paths):
4/5 all-paths agree on primary source (was 2/5 pre-fix)
1/5 disagrees: dinosaur-extinction
legacy: UNGROUNDED · Edwina (children's book) ← wrong
corpus: STRICT · Dinosaur (main article) ← right
cloud: HYBRID · Edwina (children's book) ← wrong
The new title-boost lifted corpus-query above the legacy here;
cloud still picks Edwina because SidecarReader.search applies
title-boost INTERNALLY (it predates the shared util), so the
extras-penalty stacks weirdly when run_query applies it again.
Next fix: disable internal boost in SidecarReader once shared
util is the single source of truth.
26 corpus/wallet/sidecar tests still green.
SqliteShardCorpus.fts_body uses `chunks_fts JOIN chunks ON rowid`
which doesn't bridge per-shard rowid namespaces under a single
connect_query(ATTACH+UNION) connection. The fix is per-shard
connections: one SqliteShardCorpus per shard, fts_body merged by
raw BM25 score (FTS5's score is comparable across same-tokenizer
shards).
MultiShardSqliteCorpus
- opens each shard with connect(path), wraps in SqliteShardCorpus
- fts_body: sequential per-shard, merge by ascending score
(FTS5 BM25 is 'lower is better'), top-K
- chunks_for_doc: walk shards until one returns rows
- snapshot_root: Merkle over union-deduped document_roots
- Hit.shard_id annotates which shard surfaced each hit so
future routing decisions have it
- Sequential (not threaded): sqlite3.Connection enforces
single-thread access by default and the silent threadpool
exception-swallow was returning [] — local SSD FTS5 is fast
enough (~10ms/shard) that serial is fine
CLI wiring (_cmd_corpus_query):
- Prefer shards_dir over single db when both are set (args.db has
a non-None default which was always winning)
- --shards-dir → MultiShardSqliteCorpus, --db → SqliteShardCorpus
- Skip system shards (qa.db, snapshots.db, selfmodel-chain.db)
Live demo:
make corpus-query Q="who is homer simpsons boss?" LLM=qwen
→ EVIDENCE-WARRANTED · via claim_lattice 2/2 3.97s
(search 0.82s across 5 shards + LLM 3.14s + verify 0.01s)
Tests added (2): per-shard merge, chunks_for_doc routing.
20 corpus tests + 16 wallet/bucket tests still green (36 total).
Retrieval-quality note: the answer still surfaces "Hank Scorpio"
(the You Only Move Twice episode title boost) over Mr. Burns
because MultiShardSqliteCorpus doesn't yet have the title-boost
extras-penalty that SidecarBucketCorpus gained earlier. That's the
next fix to migrate — when it lands in SqliteShardCorpus too, both
backends pick the Homer Simpson primary article.
Parallel to `arborist cloud query`: same corpus_query.run_query
orchestrator, only the Corpus adapter differs (SqliteShardCorpus
instead of SidecarBucketCorpus). Proves the protocol works against
both backends with identical pipeline code:
arborist query "Q" legacy 2000-line query() (untouched)
arborist cloud query "Q" corpus_query.run_query + SidecarBucketCorpus
arborist corpus-query "Q" corpus_query.run_query + SqliteShardCorpus ← NEW
Same render layer (_render_cloud_query_human), same audit_mode +
sources + capacity + timings shape across both new paths.
Caveat: single-DB mode only for now. SqliteShardCorpus.fts_body uses
`chunks_fts JOIN chunks ON rowid`, which doesn't bridge per-shard
rowid namespaces under connect_query() ATTACH-and-UNION. Multi-shard
search needs a per-shard query + merge — the next protocol method
(fts_body_per_shard) to add. Until then, --db <path> or auto-picks
the first non-system .db under --shards-dir.
Live demo (homer + virt-back queries):
make corpus-query Q="who developed virt-back?" LLM=qwen
→ EVIDENCE-WARRANTED · via claim_lattice 1/1 2.29s
(search 0.01s · llm 2.26s · verify 0.01s · total 2.29s)
Same E1 citation + render as `make cloud-query` on the same
question; only "0 HTTP requests" footer betrays local vs cloud.
Makefile: `make corpus-query Q="..." [LLM=qwen]`.
`arborist/qa/corpus_query.py:run_query(corpus, question, chat_client, ...)`
is now the single retrieval + evidence + LLM + verify + annotate
pipeline. Takes any Corpus adapter (SqliteShardCorpus | SidecarBucketCorpus
| future edge-proxy), returns the same result-dict shape today's
cloud-query emits (audit_mode, sources w/ used + pointer-ids, capacity,
timings, raw_answer, rendered answer).
`_cmd_cloud_query` reduced from ~310 lines of inline pipeline to ~80
lines of corpus construction + run_query call + progress emission +
render. Behavior identical: same audit_mode, same sources, same
capacity/timings tail.
Before: cli._cmd_cloud_query owned chunk-pull, evidence-build, prompt
construction, LLM call, verifier call, source annotation,
spotlight render — 310 lines of duplication with the local
query() pipeline.
After : cli._cmd_cloud_query owns ONLY corpus construction +
progress emission + render layer. The pipeline lives in
corpus_query.run_query and will be the single source of
truth once query.py refactor moves local onto the protocol.
Tests (corpus_query unit, 4 passing):
* verbatim quote → STRICT (n_verified ≥ 1, used annotation correct)
* capacity + timings dict populated with expected keys
* empty retrieval → UNGROUNDED + zero LLM calls (StubClient.calls == [])
* corpus.name leaks into result for bench attribution
Bench parity preserved (cloud_vs_local.py smoke, 5 questions):
before refactor: 1 regression (Mercury Seven STRICT → HYBRID)
after refactor : 1 regression (same — LLM-stochastic, primary source
identical both sides)
Behavior-preserving. Foundation laid for the local query() refactor:
when query.py learns to take a Corpus parameter, it'll delegate to
the same run_query() and every quality fix lands once for both.
32 prior wallet + corpus tests still green.
First step toward DRYing the local-vs-cloud retrieval diff. Defines a
minimal Corpus Protocol (Hit, ChunkRow, fts_body, fts_title, fts_phrase,
chunks_for_doc, snapshot_root) and ships two adapters:
SqliteShardCorpus wraps a connect_query() connection. Implements
fts_body (via _to_fts5 sanitizer + chunks_fts
MATCH) and chunks_for_doc + snapshot_root.
fts_title / fts_phrase raise NotSupportedError
(those routes will move out of query.py inline
SQL in the next refactor pass).
SidecarBucketCorpus wraps a MultiShardSidecarCorpus. fts_body
delegates to the sidecar BM25 + title-boost +
extras-penalty + title-relevance pipeline.
chunks_for_doc walks the shard's apsw conn via
the bucket VFS, reusing warm page cache.
Title / phrase routes raise NotSupportedError
until the sidecar format ships those indexes.
Together: query() future-refactor takes a Corpus parameter, runs the
routes the adapter supports, skips NotSupportedError, falls through.
Every quality fix (today: possessive stem, accent fold, numeral fold,
extras-penalty title-boost, title-relevance filter) lands once and
both backends consume it through the protocol.
Tests (20 passing, three layers):
tests/test_qa_corpus.py (unit) — Hit/ChunkRow invariants,
Protocol conformance (runtime_checkable isinstance), NotSupportedError
on unimplemented routes, fts_body sanitization (no FTS5 syntax leak),
stopword-only query → [], chunks_for_doc idx-ascending shape.
tests/test_qa_corpus_integration.py (integration) — real corpus →
both adapters → assert recall (target doc in top-K of both) and
byte-identical chunks_for_doc decoding. Catches tokenizer/stem/fold
drift between FTS5's unicode61 and the sidecar's NFKD+ASCII path.
tests/test_qa_corpus_functional.py (functional) — full retrieval +
StubClient LLM + verify_claim_lattice pipeline routed via the
protocol for both adapters; asserts shared audit_mode + primary
source agreement. Prototype of what the post-refactor query() does.
Side fix: MultiShardSidecarCorpus title-relevance filter now falls
open when the title tokenizes to zero content tokens (single-char or
all-stopword titles like "A" or "I" shouldn't be dropped just because
the filter side has nothing to match on).
Real-world: shared endpoints can queue (hermes was queued behind
opencompletion this morning, single calls pushed past 180s but
completed in 121s on retry). 600s tolerates ~10× the typical
non-queued latency before declaring timeout.
Renamed DEFAULT_PRICE_CENTS (per-call) → DEFAULT_DOLLARS_PER_1K_GROUNDED.
Per fox 2026-05-31: at owned-hardware scale, COGS is naturally
expressed per-1000-grounded-answers (hermes $0.09, qwen $0.16),
not per-call cents. The baseline already amortizes ungrounded calls.
Report now shows: per-model implied spend, effective COGS in
$/1k grounded, and cascade delta as % vs always-most-expensive.
20-question live bench (real run): cascade beats always-qwen by
-39% COGS AND grounds more answers (19/20 vs 17/20). Hermes and
qwen are equally reliable (17/20 each) but catch *different*
questions — additive coverage.
Adds bench/cross_model_selfplay.py + `make bench-cross-model` target.
For each question in a fixture, runs `arborist query` once per
configured model (default: Hermes + Qwen) and tabulates:
* audit_mode per model (EVIDENCE-WARRANTED → POINTER-LINKED → UNGROUNDED)
* agreement on primary source URI
* grounding rate per model
* estimated $/grounded-answer (per-call prices configurable)
* cheap-first cascade analysis (try cheapest, escalate on UNGROUNDED)
This is the "ask twice for two options" pattern from the agent
perspective — bakes it in as a benchmark so we can measure whether
the cascade beats always-using-the-stronger-model on $/grounded.
First live run (2 questions × 2 models, $0.41):
- Hermes: 1/1 grounded (1 timeout — operational issue)
- Qwen: 2/2 grounded STRICT
- Cascade: 2/2 grounded for $0.25 — beats always-Qwen ($0.32)
when Hermes succeeds on its first call.
Output: bench/cross_model_results/<utc-iso>.{jsonl,md} (gitignored).
Cloud-query was picking sibling articles (Mona Lisa's Revenge instead
of Mona Lisa, Republics of the Soviet Union instead of Soviet Union,
Mercury 13 instead of Mercury Seven) because:
1. Title-boost counted overlap but not extra title tokens. Both
"Mona Lisa" and "Mona Lisa's Revenge" overlapped query by 2 →
same boost → BM25 favored the shorter movie article.
2. RRF merge squashed per-shard rank-1 hits into a 1/(60+1) tie
across 4 sidecar shards. Tie-breaking was undefined; the right
article was as likely to lose as win.
Two fixes:
sidecar.search title-boost:
+ Filter title tokens through STOPWORDS + len-1 cutoff so 's', 'of',
'the' don't count as extras.
+ Penalty: extras = |title_tokens - query_tokens|; effective bonus
is `max(0, overlap - extras/2) * title_boost`. "Mona Lisa" gets
full bonus; "Mona Lisa's Revenge" gets half.
MultiShardSidecarCorpus.fts_search merge:
+ When all CONTRIBUTING shards have sidecars (their BM25 + boost
scores are directly comparable), merge by max raw score across
shards. RRF was masking score discrimination at the top of the
list.
+ Mixed (sidecar + bucket-direct FTS5) falls back to RRF since
those scales aren't comparable.
Bench (5-question smoke, cloud_vs_local.py):
before fix: 4 regressions / 5
after fix : 1 regression / 5 (and that one is the right source,
only the audit_mode dropped STRICT
→ HYBRID due to LLM-stochastic answer
phrasing)
A query like 'when was final fantasy 7 created?' was scoring every
'Final Fantasy ___' article identically because:
1. '7' got stripped by the length>1 filter, so the query reduced
to {final, fantasy}
2. Even surviving, '7' literally doesn't appear in the corpus when
it spells 'VII'
Two fixes:
* tokenize_text now keeps single-char numeric tokens (so '7' survives
sanitization). Stopword + len filter still applies to letters.
* numeral_expand() pairs 1↔I, 2↔II, ..., 20↔XX. Applied in two places:
- search() expands the query term list before dict lookup, so '7'
finds 'vii' postings (no rebuild needed — chunks tokenized to
'vii' at build time).
- title-boost computes overlap on the expanded set, so 'Final
Fantasy VII' titles win the +24 boost vs +16 for siblings.
Live demo (cloud-query 'when was final fantasy 7 created?' LLM=qwen):
before: UNGROUNDED 0/1 — top sources Final Fantasy X-2, Tactics,
Character design (FF VII article rank 5, never made top-4)
after : EVIDENCE-WARRANTED 2/2 — 'Final Fantasy VII was originally
released on January 31, 1997.' + 'Development began in 1994.'
cited E1 = Final Fantasy VII (#1 in merged top-4)
Single-flag switch to swap the upstream LLM endpoint+model. Default
(LLM unset) leaves --endpoint/--model unspecified so cloud-query
falls back to its built-in default (Hermes-3-8B on ai.unturf.com).
LLM=qwen pins Qwen3.6-27B on qwen.ai.unturf.com (uncloseai). Granular
override still available via LLM_ENDPOINT=… and LLM_MODEL=…
make-vars or the underlying --endpoint/--model CLI flags.
Sample (homer's boss):
make cloud-query Q='who is homer simpsons boss?' LLM=qwen
→ EVIDENCE-WARRANTED · via claim_lattice 2/2 19.4s
(Hermes on the same query landed EVIDENCE-WARRANTED-PARTIAL 1/2)
Two bugs surfaced by 'what are the 3 starter pokemon in pokemon red?'
that returned UNGROUNDED with Russell Ballestrini's blog cited above
every Pokémon Wikipedia article.
Bug 1: ASCII-only word regex split 'Pokémon' on 'é' into ['Pok',
'mon'], so the term 'pokemon' never landed in the sidecar dict and
the title-boost never matched 'Pokémon Red and Blue' against a
'pokemon' query token. Fix: NFKD-fold accents before tokenizing
(both sides — build + query — agree). Mirrored in _to_fts5 too so
the bucket-direct FTS5 path stays consistent (FTS5's unicode61
tokenizer already folds, so the sanitizer was the only place that
needed the fix).
Bug 2: Multi-shard RRF treated a rank-1 hit in a 223-doc personal
blog identically to a rank-1 hit in a 1M-doc shard. Russell
Ballestrini's blog incidentally contains 'red' or 'starter'
somewhere, so its FTS5 returned the root page at rank 1; RRF tied
with Pokémon Red and Blue (also rank 1 in genesis shard 0) and
won by insertion order. Fix: post-merge title-relevance filter
(mirrors local query.py's search.title_filter) — drop hits whose
titles share zero stemmed+folded tokens with the query before RRF
combines them. Russell Ballestrini blog title {russell, ballestrini}
overlaps neither {starter, pokemon, red} → dropped.
Live demo (cloud-query 'what are the 3 starter pokemon in pokemon red?'):
before: UNGROUNDED 0/1 — Russell Ballestrini #1, no useful evidence
after : EVIDENCE-WARRANTED-PARTIAL 3/6 — 'Bulbasaur, Charmander,
and Squirtle' cited to Pokémon Red and Blue article
Mirrors local 'arborist query' instrumentation so the cloud path is
no less observable. Uses the existing arborist.qa.progress.Progress
emitter (auto-on at TTY, override via ARBORIST_PROGRESS=0|1).
Stages emitted:
manifest.start/done URL + shard/sidecar counts
corpus.open.start/done sidecar download + dict parse cost
search.start/done per-shard FTS + RRF merge
context.start/done chunk-content pulls + assembled bytes
llm.start/done model + endpoint + ctx + answer chars
verify.start/done audit_mode + n_verified/n_quotes
Render tail adds (matches local query render):
capacity: prompt N chars (sys N + evidence N + question N) → answer N chars
timings: manifest Xs · corpus_open Xs · search Xs · context Xs ·
llm Xs · verify Xs · **total Xs**
bucket: N HTTP requests · KB · endpoint / model
JSON output (--json / JSON=1) gains 'timings' + 'capacity' fields
with the same shape, so bench harnesses can consume them directly.
Surfaces that sidecar.parse dominates a cold cloud-query (~22s/shard
on the genesis 6M-term sidecar). That's a real future-opt target
(ProcessPool instead of ThreadPool to escape the GIL) but the
mechanism is observable now, which is the prerequisite for tuning.
usage: make cloud-query Q="your question" [JSON=1] [BUCKET_URL=...] [TOP_K=4] [MAX_CONTEXT=24000] [CACHE_MB=64] / — consistent with the
local usage: make query Q="your question" [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N K="extra retrieval keywords" ANSWER_MODE=claim_lattice|claim_lattice_pointer|quote LAYOUT=tail|bookend|per_chunk BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 WITNESS=1 XLANG=1 XLANG_MT=1] / entry points. No alias kept
(per repo convention against backwards-compat hacks for unused
names; this command landed in 65b6fe6 / cb8c1de during a fast-
iteration burst and has no external consumers yet).
CLI: arborist cloud query '...' (was: arborist cloud ask '...')
Make: make cloud-query Q='...' (was: make cloud-ask Q='...')
Pure-BM25 ranking penalizes long main articles ('Homer Simpson'
Wikipedia article) vs short episode articles with the same query
tokens — length-normalization is doing what BM25 was designed to do
but the result mis-ranks the primary source. Local query.py runs a
SEPARATE title-LIKE retrieval route that surfaces title-match docs
unconditionally; sidecar didn't, so the Homer Simpson article landed
rank 4 behind 'Bart vs. the Space Mutants' and similar incidentals.
Fix: at query time, after BM25 scoring, add a fixed bonus per query-
term overlap with the doc title (already stored in the sidecar doc
table — no rebuild needed). Both sides stemmed with the same
possessive+plural normalization as _claim_title_overlap so
'simpsons' (query) overlaps 'simpson' (title). Default
title_boost=8.0 is enough to lift the right primary source above
incidental short-doc matches without overwhelming BM25 elsewhere.
Live demo (cloud-ask 'who is homer simpsons boss?'):
before: 'Hank Scorpio is Homer Simpson's boss' (cited episode article
where Scorpio appears; primary 'Homer Simpson' article ranked
rank 4 in its shard, never reached the merged top-K)
after : 'Homer Simpson's boss is Charles Montgomery Burns.'
(cited E1=Homer Simpson + E3=Homer's Odyssey episode)
EVIDENCE-WARRANTED-PARTIAL · via claim_lattice 1/2 26.6s
_claim_title_overlap's _stem only stripped trailing 's' on words >4
chars. 'homer's' (mid-apostrophe possessive) became 'homer'' after
stem, which didn't match 'homer' from the title — both claims of a
Simpsons answer cited articles that contained the right content but
the title-overlap check failed, demoting audit_mode UNGROUNDED via
the existing 'all resolving claims TITLE_MISMATCH' rule.
Fix: normalize apostrophes (ASCII and Unicode right-single-quote)
to empty before length/suffix check. Localized to _claim_title_overlap's
inline _stem — no other callers of _content_tokens affected.
Repro / regression:
before: _claim_title_overlap("Homer's boss", "Dancin' Homer") → False
after : _claim_title_overlap("Homer's boss", "Dancin' Homer") → True
unrelated still False; plural collapse still True; 115 verifier tests pass.
Live demo (cloud-ask 'who is homer simpsons boss?'):
before: UNGROUNDED 2/2 (false rejection of two correct claims)
after : POINTER-LINKED-PARTIAL 2/2 (Scorpio claim still title-
mismatched against 'You Only Move Twice' — the episode-
naming convention legitimately ducks token overlap there;
that demote is honest)
Was serial — 4 × 745 MB = ~2 min cold for the 4-genesis-shard manifest.
Build a ThreadPoolExecutor with one worker per shard so the wall cost
is max(per-shard download), not sum. Idempotent: SidecarShardClient
re-reads cached files without re-downloading on subsequent runs.
Wires the inverted-index sidecar into the unified `BUCKET_URL` flow:
- BucketManifest gains optional per-shard `sidecar_url`.
- New `SidecarShardClient`: downloads the sidecar once (cached under
~/.arborist/sidecar-cache/<hash>.bin keyed by URL), queries locally
with BM25; uses the bucket's HttpRangeVFS only for chunk content
reads on hit (warm page cache reuse).
- New `MultiShardSidecarCorpus`: mixed-mode — shards with a sidecar
use SidecarShardClient (sub-second FTS); shards without one fall
back to BucketClient (bucket-direct FTS5, slow but functional).
- Cross-shard merge: reciprocal-rank-fusion (RRF, k=60). Sidecar
BM25 and FTS5 BM25 are on incomparable scales; RRF normalizes by
rank-position so a doc that's #1 on the virtback shard beats a doc
that's #1 on the genesis shard regardless of raw score magnitude.
Bucket layout:
clones/full-bench-64k/002.db (7.6 GB)
clones/full-bench-64k/002.sidecar.bin (745 MB — pre-built index)
clones/manifest-sidecar.json (advertises both)
Live demo (BUCKET_URL=…/clones/manifest-sidecar.json):
make cloud-ask Q="who developed virt-back?"
→ EVIDENCE-WARRANTED · via claim_lattice 1/1 6.79s
"Russell Ballestrini developed virt-back."
sources span genesis-Wikipedia + russell.ballestrini.net
28 HTTP RANGEs, 1.8 MB transferred
vs the 4+ minute bucket-direct-only path on the same shard mix.
`load_bucket_manifest` now accepts both bucket-root URLs (appends
`clones/manifest.json` per convention) and explicit `.json` URLs
(uses them as-is), so operators can pin alternate manifests.
`BUCKET_URL` (one env var) → client GETs `clones/manifest.json` →
opens HttpRangeVFS per listed shard → FTS5 across all shards in
parallel (ThreadPoolExecutor; per-thread apsw.Connection) → merge by
BM25 score → pull chunks from the owning shard → LLM + verify.
No per-query --shard-url, no path proliferation.
Two manifests published on s3://arborist/clones/:
manifest.json — default: virtback only (2.5MB, ~5s/query)
manifest-full.json — opt-in: all 5 shards (35GB, prohibitive over
WAN due to FTS5 b-tree walk pattern; needs
smaller shards or co-located query proxy)
HttpRangeVFS read-ahead tuned from per-page (4KB) to 64KB block-aligned
cache. Each cache miss fetches one 64KB block; subsequent reads within
the block are local-fast. Lower miss count, similar bytes-on-wire
(64KB amortizes well over typical 4-16 page b-tree clusters; larger
read-ahead like 4MB over-fetches on random FTS5 reads).
Sample run (default manifest):
make cloud-ask Q="who developed virt-back?"
→ EVIDENCE-WARRANTED · via claim_lattice 1/1 4.71s (bucket-direct)
21 HTTP requests · 1344 KB
ACL: genesis full-bench shards flipped to public-read (CC-BY-SA
Wikipedia content). Reachable now if you want to play with the slow
multi-shard path; not in the default manifest because chat latency
matters more than coverage breadth.
Mirrors `make query` ergonomics:
- default = pretty terminal layout (audit_label, sources w/ roles,
bucket stats footer)
- JSON=1 (or --json) = full machine-readable record
_render_cloud_ask_human shares the audit-label primitive
(_render_audit_label) with the local query renderer so the
HYBRID/STRICT/UNGROUNDED tokens map to the same four-rung ladder
labels in lattice modes (POINTER-LINKED / ANCHOR-WARRANTED / ...).
Bucket-direct path doesn't emit warrant-tail / run-DAG /
retrieval-purity so those sections are trimmed.
Output footer adds 'bucket: N HTTP requests · KB · endpoint / model'
so the operator can see network cost + LLM identity inline.
Both paths grind on the same data (LOCAL_DB defaults to web.db, the
file that was uploaded to the Spaces shard at SHARD_URL). --burn busts
the local QA cache so we measure a fresh inference both sides; the
cloud path has no caching layer so it's always fresh.
Reports answer + audit_mode + verifier_method + sources + wall time
for each path, then prints local-vs-cloud delta + ratio.
Sample run (Q='who developed virt-back?'):
local: 3.8s STRICT (verbatim quote verified)
cloud: 11.8s HYBRID (entity-name verified)
delta: +8.0s (3.10x — the HTTP RANGE * ~99 requests tax)
Cloud audit_mode lands one rung lower because the cloud-ask prompt is
simpler than query.py's full pipeline, so the LLM paraphrases instead
of quoting. Verifier honestly demotes paraphrase to HYBRID. Both
answers are correct; the audit_mode difference is a prompt-shape
artifact, not a cloud-path correctness gap.
`make cloud-search` was retrieval-only; `make cloud-ask Q="..."` runs
the same audited-answer shape as local `make query`, but every byte
read goes through the bucket via HttpRangeVFS — no local DB, no
intermediate arborist server.
Pipeline:
1. FTS5 search bucket-direct → top-k document hits (existing path)
2. SELECT chunks.content for each hit via SQL through the same
apsw conn (reuses the warm page cache from step 1)
3. Assemble context with per-doc budget (max_context_chars / top_k)
4. POST to LLM endpoint (default Hermes, override --endpoint/--model)
5. verify_quotes() locally — same verifier the local path uses
6. Emit {answer_text, audit_mode, verifier_method, n_quotes,
n_verified, sources w/ source_role + n_chunks, stats, timing}
Real-world result on the russell.ballestrini.net Spaces shard:
make cloud-ask Q="who developed virt-back?"
→ "Russell Ballestrini developed virt-back."
audit_mode=HYBRID, verifier_method=entity
98 HTTP RANGE GETs, 388 KB, 10.4 s total
endpoint=hermes.ai.unturf.com/v1
FTS5 chokes on '?' and parses '-' as NOT, so a natural-language
question like 'who developed virt-back?' raised SQLError. BucketClient
now strips non-word characters, filters stopwords (mirroring
qa/query.py:_TITLE_STOPWORDS), and OR's the surviving content tokens.
Examples:
'who developed virt-back?' -> 'virt OR back'
'what is anarchism' -> 'anarchism'
'"virt-back"' -> 'virt OR back' (FTS5-quoted spans lose
their punctuation context;
pass --raw to preserve)
CLI: `arborist cloud search --raw '"phrase"'` passes the query through
untouched for advanced FTS5 expressions (AND/NOT/phrase quotes).
`make cloud-search Q="who developed virt-back?"` now resolves to the
right russell.ballestrini.net article without any escaping.
Drops the "SHARD_URL required" friction. Default points at the
russell.ballestrini.net web-crawl shard now live on DO Spaces
(public-read, ~2.5 MB) — the smallest end-to-end real-data target
for proving bucket-direct queries work against a real S3-compatible
bucket without any local arborist data.
Override SHARD_URL on the command line to point at a different bucket
shard:
make cloud-search Q='"virt-back"' # uses default
make cloud-snapshot-root SHARD_URL=https://other.bucket/.../000.db
Recipe also switched to single-quoted $(Q) so FTS5 phrase quotes
(`'"phrase here"'`) survive shell parsing.
Pure-cloud consumer: client opens an arborist .db file IN PLACE on a
bucket via HTTP RANGE reads, runs FTS5 + SQL locally, fetches chunk
bodies from `blobs/<hash>` on the same bucket. No intermediate server
in the data path. The bucket layout we already produce (Tier A clones
plus --jit-blobs blobs/) is exactly what this consumer needs.
Module `arborist/wallet/bucket.py`:
- HttpRangeFile / HttpRangeVFS: apsw subclasses. xRead → HTTP Range
GET; xFileSize → cached HEAD. xWrite/xTruncate raise (read-only).
IOCAP_IMMUTABLE so SQLite skips locking/journaling. Empty tempfile
backs the apsw VFSFile C-bookkeeping; never actually read.
- _LRUByteCache: thread-safe (offset,length)-keyed LRU; soft byte
budget (default 32 MB). SQLite's own page cache (~8 MB) handles
most hot-path amortization, so our LRU is the second-level safety
net for working sets that overflow SQLite's cache.
- _HttpTransport: stdlib urllib (zero new runtime deps beyond apsw).
- BucketClient: high-level — fts_search / chunks_for_doc /
fetch_chunk_body / snapshot_root + page-cache stats.
CLI (`arborist cloud <sub>`):
- `cloud search Q --shard-url ...`
- `cloud snapshot-root --shard-url ...`
- `cloud fetch-chunk LEAF_HASH --blob-base ...`
Makefile:
- `make bootstrap-bucket` (installs apsw)
- `make cloud-search Q="..." SHARD_URL=https://.../000.db`
- `make cloud-snapshot-root SHARD_URL=...`
- `make cloud-fetch-chunk LEAF_HASH=... BLOB_BASE=...`
- `make cloud-demo` — end-to-end proof on a vanilla laptop: seeds a
tiny bucket layout in tmp, serves it via a Range-aware static
HTTP server, runs all three cloud commands from an isolated HOME
that has no local arborist data. Asserts laptop HOME stays empty
start-to-finish.
Tests (tests/test_wallet_bucket.py, 4 passing):
- bucket-direct FTS5 results == direct sqlite3 results
- chunk fetch round-trip + hash verify
- snapshot_root bucket-direct == snapshot_root local
- second identical query adds 0 HTTP requests (SQLite-cached)
pyproject: new `[bucket]` extra carries apsw>=3.45; folded into [dev].
Three new targets layered on the wallet-demo:
- wallet-serve long-running server pointed at your real corpus
(defaults: SHARDS_DIR=$HOME/.arborist/shards, override
with DB=path); Ctrl-C to stop
- wallet-pin fetch the server's current snapshot_root once and save
to $HOME/.arborist/wallet.anchor. Real SPV trust:
verify the anchor out-of-band before pinning, then
every subsequent ask grounds against the pinned value
- wallet-ask Q="your question" — reads the pinned anchor (warns +
auto-fetches if no pin), submits via wallet client,
prints the verified JSON. Optional ANCHOR= override.
Typical flow:
make wallet-serve & # one terminal
make wallet-pin # one-time bootstrap
make wallet-ask Q="what is X?" # repeat as needed
Self-contained recipe that proves the wallet works from a machine
with zero local arborist data:
1. shows the vanilla-laptop HOME is empty
2. ingests a 2-doc corpus into a separate server HOME
3. starts `arborist serve` (stub LLM, no upstream calls)
4. bootstraps the trust anchor via curl /snapshot_root
5. runs `arborist wallet ask` on the laptop HOME; verifies exit 0
6. confirms laptop HOME is STILL empty after the verified query
7. runs same call with a deadbeef anchor; verifies exit 3
8. ALL CHECKS PASSED message
Two isolated mktemp HOMEs; trap cleans up server + dirs even on
interrupt. Seed logic lives in arborist/wallet/_demo_seed.py so the
Makefile recipe stays one logical command instead of inlining
multiline Python (make recipe lines turn class/def into SyntaxError).
Run: `make wallet-demo` (default port 18780; override with
WALLET_DEMO_PORT=N).
The Merkle bundle gives the wallet authentic chunk bytes, but the
server still decides what audit_mode to claim. Run the existing
verify_quotes() locally on the bundle's chunks so the wallet has an
independent verdict that doesn't trust the server's verifier at all.
`VerifiedAnswer` now carries `local_audit_mode`, `local_n_verified`,
`local_verifier_method` alongside the server's audit_mode. They can
legitimately differ (server's context is larger), but a wallet-side
STRICT against a server-side UNGROUNDED would be a real "server lied
about not finding grounding" signal — exactly what the SPV pattern
exists to catch.
Opt out with `client.ask(q, verify_locally=False)` for pure-stdlib
SPV ports that can't load the verifier.
A wallet client holds only a snapshot_root (trust anchor) and verifies
Merkle proofs on every answer. No SQLite, no FTS, no chunks locally.
Same shape as Bitcoin SPV (Electrum / mobile wallet): server can DOS
but cannot forge content whose hash chains up to the trusted anchor.
New module `arborist/wallet/`:
- proof.py: AnswerBundle + build_answer_bundle (server) +
verify_bundle (client). Two proof legs per chunk:
chunk_body → leaf_hash → document_root via in-doc
Merkle proof, then document_root → snapshot_root via
the corpus-wide sorted-doc-roots tree (mirrors
snapshot.compute_snapshot_root). Single-doc corpus
degenerates to "document_root IS snapshot_root" and is
handled with an explicit `degenerate_single_doc` flag.
- server.py: WalletServer + http.server.ThreadingHTTPServer wrapper.
Pure stdlib. GET /healthz, GET /snapshot_root, POST /ask.
Each request opens its own DB connection so SQLite's
single-writer model never bites.
- client.py: WalletClient: urllib + json + arborist.wallet.proof.
Returns a VerifiedAnswer or raises VerificationError /
WalletError. No corpus dependency.
New CLI:
- `arborist serve` — start the wallet server. ARBORIST_WALLET_STUB=1
swaps the LLM for StubClient (lets ops sanity-check verification
without burning tokens).
- `arborist wallet anchor` — fetch the server's current snapshot_root.
- `arborist wallet ask` — submit a question, verify the AnswerBundle
against --trust-anchor, exit 3 on VerificationError.
Tests (tests/test_wallet_spv.py, 7 cases):
- happy: bundle → verify pass against correct anchor
- dict round-trip via to_dict/from_dict still verifies
- tamper: rewrite a chunk body → body hash check fails
- forged leaf_hash: chunks[i].leaf_hash != chunk_proofs[i].leaf_hash
fails before any hashing
- wrong trust_anchor: bundle.snapshot_root != anchor fails immediately
- /healthz and /snapshot_root over real HTTP
- end-to-end ask: corpus → in-process server → urllib client → verify
Replaces the batched chunk-pack phase with per-chunk content-addressed
blob uploads to `blobs/<hash[:2]>/<hash[2:]>`. The metadata pack still
ships (small, fast to restore), but consumers no longer have to pull
multi-GB chunk packs to get queryable: `cold unpack --mode just-enough`
+ `ARBORIST_JIT_CHUNKS=1` fetches single chunks on cache miss.
Producer (`_stream_jit_blobs` in evict.py):
- ThreadPoolExecutor with bounded queue (workers*4) keeps memory flat
across millions of chunks
- HEAD-checks object_size for idempotent re-upload
- Mutually exclusive with chunk packs — manifest's `chunk_pack_hashes`
is empty in JIT mode (consumer reads that as "JIT-only")
Consumer (`hydrate_doc_jit` in cold_clone.py + `_maybe_jit_hydrate` in
qa/query.py):
- Detects both content shapes that need JIT: NULL (Tier B raw-clone) and
zeroblob placeholders (just-enough pack restore, per #53). Discriminator
is first-byte = NUL — zstd-framed bodies start with 0x28, plain UTF-8
prose never has leading NUL.
- Same placeholder filter applied to chunk-read sites in qa/query.py so
partial hydrate doesn't surface zero-bytes content into the LLM context.
Test (`TestJitBlobsPackMode` in tests/test_cold_unpack_routed.py):
- End-to-end push → just-enough hydrate → JIT-fetch → content matches
original byte-for-byte through `unpack_chunk`.
Docs (cold-object-store.md):
- Hard-invariant #1 updated: bucket holds packs by default; `blobs/`
and `clones/` are opt-in prefixes for the JIT and Tier-A flows.
- New "Three consumer modes" section: full-pack vs JIT-blobs vs raw-clone
comparison table + operator decision tree.
Partial Tier B snapshots (where the producer didn't upload every
chunk, e.g. due to the corpus's BLOB-vs-TEXT-affinity skew) left
some NULL chunks without a corresponding blob in the bucket.
hydrate_doc_jit used to raise on the first NoSuchKey / hash
mismatch, which collapsed the whole JIT query. Now we skip the
offending chunk and let the query path see the same NULL content
it would see on a cold shard. Worst case: partial context, not a
crashed query.
Adds hydrate_doc_jit(conn, doc_root, backend): for every chunk in this
document whose local content IS NULL, fetch blobs/<leaf_hash> from the
configured cold backend, verify hash, cache into the row. _load_doc_text
and _load_doc_chunks call it transparently before reading.
Env-gated: ARBORIST_JIT_CHUNKS=1 opts in (so non-JIT environments stay
a pure no-op); the backend comes from the standard ARBORIST_COLD_*
config. With this wired in, a Tier B consumer can clone metadata-only
shards and the query path transparently fetches answer chunks JIT — no
caller code changes.
Some chunks in real corpora have content stored as decoded text (str)
not bytes, with Unicode codepoints beyond latin-1. Encode as utf-8 so
the conversion always succeeds; if the resulting bytes do not hash to
the row's leaf_hash, skip that chunk (no blob uploaded, content kept
local) rather than aborting the whole shard's snapshot.
Some rows in shards-genesis-v2 came back from the chunks.content BLOB
column as instead of bytes — SQLite's type-affinity rule lets a
BLOB-affinity column hold any storage class. The strip-and-upload path
crashed on bytes(some_str) without an encoding. latin-1 preserves
arbitrary byte values 1:1, so the leaf_hash check still matches.
The previous serial loop uploaded one shard at a time, capping
throughput at single-shard multipart concurrency (~10 parts × 8 MB =
~80 MB outstanding). On the 3090->Spaces NYC3 link that meant ~50+ min
per 12 GB shard. Refactor to one ThreadPoolExecutor worker per shard
(default M-wide); each worker still has 10-way multipart inside, so 4
shards × 10 parts = 40 parts in flight saturates the link far better.
New --workers flag overrides the auto-fanout.
Adds a distribution channel alongside the pack tier that skips
pack/unpack entirely — producer SQLite-Backup-API's each shard to a
raw .db and multipart-uploads; consumer pulls them down in parallel.
Recovery becomes ~download time (the FTS index travels inside the .db,
no rebuild step). Replaces the ~84 min pack/restore measured 2026-05-29
with ~download time for ~35 GB of raw shards.
Two channels live in the same bucket:
- Tier A (full clone): clones/<snap-id>/00N.db. `arborist cold
stream-snapshot` produces, `arborist cold clone` consumes. Targets
capable peers (the 3090 class).
- Tier B (just-enough + JIT): same flow with --just-enough, but the
producer strips chunks.content into per-chunk blobs/<hash> and ships
metadata-only shards. Consumer's local DB is ~a few GB; the
retrieval path can fetch_chunk_jit() from the bucket on demand.
Targets constrained peers (mobile / SPV).
A small clones/CURRENT.json pointer enables atomic-ish discovery;
pinned --snapshot-id works too. New backend.get_file() streams large
objects via boto3 download_file (multipart parallel into a target
file). Round-tripped locally with MemoryBackend on both tiers
(content preserved byte-exact; JIT verified by hash_leaf).
The parallelized _cmd_cold_rebuild_fts calls time.time() but cli.py has
no module-level `import time` (functions import it locally), so the
command crashed with NameError. The command had never actually run via
the CLI before — the full recovery used a standalone script and tests
called _rebuild_fts_on_target directly — so the gap shipped in 1547259.
Now exercised end-to-end: rebuild-fts on the 4-shard corpus runs 4-way
parallel (~294s) and `cold verify` passes.
The FTS pack restore produced a DEAD index — segments present, MATCH=0 —
because the verbatim shadow-table copy used INSERT OR IGNORE, so the
pack's real `_data` rowid-1 "structure" record lost the primary-key
conflict to the empty one `CREATE VIRTUAL TABLE` seeds, leaving a "0
segments" header over orphaned segments. Fix (evict.py): clear the
seeded rows, then copy verbatim (DELETE + INSERT ... SELECT), so each
fts5 shadow table becomes a byte-for-byte copy of the producer's index
and the real structure record survives.
Validated on the 3090: restore one fts pack, NO rebuild -> MATCH
'anarchism'=1189 / 'the'=1.42M (identical to rebuild-from-content);
`_data` id=1 structure record non-empty.
make cold-hydrate: rebuild FTS only when the restored index isn't
already searchable (cold verify-gated). Shipping FTS packs now makes
recovery fast (~24s/shard restore, skip the ~5min rebuild); dropping
them (--no-fts default) keeps the bucket small. Either way cold verify
gates success.
docs/cold-object-store: FTS packs restore correctly now; documented the
restore-vs-rebuild tradeoff, per-consumer guidance, and the fixed bug.
- Hydrating a new peer: M-aware `make cold-hydrate` (serial default,
bulk-tuned restore, parallel FTS rebuild from content, cold verify
self-check) — replaces the stale single-shard `cold unpack` loop.
- New "FTS: rebuild, don't restore" with the measured rebuild-vs-restore
comparison (~5 min rebuild + no extra download vs +4.76 GB for a dead
pack-restored index) and WHY the pack restore is dead: FTS5's _data
rowid-1 "structure" record is left empty because INSERT OR IGNORE
collides with the freshly-created vtable's empty header — a headless
index over orphaned segments (count looks right, MATCH returns 0).
- Invariants: FTS shadow tables are rebuilt on the consumer, not shipped
(cold pack defaults to --no-fts).
- Failure modes: cold verify's zero-filled-content + dead-FTS classes.
Make cold recovery correct-by-default and fail-loud, closing the class
that silently produced a corrupt, unqueryable corpus (2026-05-29):
- cold rebuild-fts now runs in PARALLEL (one process per shard; separate
files, no contention) and clears-first, so it also repairs the dead
index a cold-pack FTS restore leaves. Replaces the serial post-pass.
- new `cold verify`: self-check a hydrated shard set — chunk content
materialized (not zero-filled) AND FTS searchable (MATCH a word taken
from sampled content). Non-zero exit if any shard fails. Validated: it
passes the good recovery and fails the corrupt genesis-test, catching
both the zero-filled-bodies and dead-FTS classes.
- cold pack defaults to --no-fts (FTS is derived from content and the
FTS pack restore is non-functional anyway); --with-fts to opt back in.
- make cold-hydrate: serial by default (M-aware hydrate routes every
pack into all M shared target shards, so parallel workers contend —
#54); always rebuild FTS from content (drop the broken shard-000-only
gate); run `cold verify` at the end so a bad recovery fails loudly.
- make crawl-ingest writes to one central crawl db (CRAWL_DB, default
~/.arborist/crawl/web.db) instead of per-domain shards in the
peer-shared main dir: keeps locally-crawled content out of peer
sharing by default and a growing domain set under SQLite's 10-attach
cap (Makefile, docs/crawler.md).
- arborist query auto-includes the local crawl db (query() gains
extra_shards; CLI --include-shard / --no-crawl-db, default-on when
web.db exists). Fix latent --db single-file query AttributeError
(cli.py). Persist used / used_pointer_ids + retrieval_purity into
merkle_proof so read-only consumers can see which chunks fed the
answer (qa/query.py).
- arborist.read: read-only seam for dashboards / verifiers; on a
multi-source context root surface the real primary source instead of
the opaque corpus://multi-source sentinel (read.py). Backs the
arborist-viz Merkle Command Center (#000069).
- tests for extra_shards, the CLI crawl-db resolver, and the read seam.
Genesis hydrate from cold packs was broken end-to-end (the #46 SPV-wallet
test never passed):
- Phase-1 metadata restore (90M-edge fan-out + zeroblob chunk
placeholders) ran with SQLite's ~2 MB default cache — the 512 MB /
MEMORY-journal tuning was applied AFTER Phase 1, so INSERT OR IGNORE
into the indexed edges table thrashed: ~4 h/pack, never finishing.
Move the bulk PRAGMAs before Phase 1 (evict.py).
- The routed restores held one unbounded transaction; add bounded
incremental commits (edges every 1M rows, routed tables every 200k)
so the txn and in-RAM MEMORY journal stay small (cold_pack_metadata.py).
- FTS packs restore a non-functional index (shadow rows present, MATCH
returns 0). _rebuild_fts_on_target now clears-first so a rebuild from
content overwrites the dead pack-FTS idempotently (migrate.py).
Validated on the 3090: full corpus recovered (3,468,226 docs / 6,235,588
chunks / 90,592,990 edges, exact match to source), search + STRICT Q&A
working. Restore ~4 h/pack -> ~8-20 min/pack; FTS rebuilt from content
in ~5 min (parallel).
