Refactor from per-session sqlite files to one shared shard at
~/.arborist/sessions.db. Three things that didn't work before now do:
1. Queries are first-class members of the tree.
nodes_fts (FTS5 over question + answer_text + cited_titles) lets
/find <query> walk every prior turn across every session. Cached
answers and threads become findable, surface in the REPL as
`[<bates>] <audit> <question>` lines.
2. Forking from history works the same as forking from a sibling.
parent_bates can cross sids. After /find returns a hit from
last week's session, /cd <bates> + ask = your next question
lands as a child under that historical turn. The cross-session
parent's subtree_hash ripples up its session_root.
3. One global audit chain instead of per-file.
audit_events.event_hash = sha256(prev || canonical body), one
chain over every state change in the shard. `make
session-chain-check` is now a single pass; tampering anywhere
in the operator's history breaks the chain.
Wire:
- arborist/qa/session.py — drop file-per-session SessionStore class;
Session becomes a viewport on SessionStore. cited_titles_json +
n_cited_sources materialized at insert time so FTS5 doesn't need
a join into providence_cache.
- arborist/cli.py — `session` subcommand swaps --gc for --find;
REPL adds /find. Ancestor-keyword extraction now reads
nodes.cited_titles_json directly (no qa.db roundtrip).
- Makefile — `make session-find Q="..." [LIMIT=N JSON=1]`; `make
session-gc` retired (no per-session files to GC).
- docs/sessions.md — rewritten for the single-shard shape.
- tests/test_session.py — 17 tests: create, add, fork (incl.
cross-session), find (FTS5 + by_cache_key), path_to_root crossing
sessions, audit chain (intact + tampered), cited-title extraction,
subtree_hash ripple across sessions.
Migration: pre-existing per-session dbs at ~/.arborist/sessions/*.db
become orphaned. None lost data — test sessions only. Operator can
rm -rf ~/.arborist/sessions/ (or rename to sessions-old/) at leisure.
126 session+providence+verify+inspect tests pass.
Three fixes from interactive feedback:
1. **readline in the REPL** — input() now has up-arrow line history
and emacs/vi editing. Persists to ~/.arborist/sessions/.history
across runs (2000-entry cap). Without this, every typo required
retyping the whole line.
2. **π* canonical projection preflight in providence_query** —
bare arithmetic ("33/66", "0.1+0.2") was tokenized by FTS5 as
["33", "66"] and matched the year-article titles "33" and "66"
(low-DF integer tokens dominate bm25); the LLM then "interpreted"
the bare fraction as years 33 AD / 66 AD. The legacy query()
pipeline has the same preflight at the top of its pipeline;
providence_query was missing it (drift from the #000015/#000030
π* registry work). Now firing the arithmetic@v1 / algebra@v1 /
logic-kernel@v1 kernel short-circuits retrieval + LLM entirely
and returns audit_mode=CANONICAL_PROJECTION with the exact
rational answer.
Live: `make query Q="33/66"` → "1/2" (was HYBRID citing year
articles).
3. **Qwen as default LLM for `make session`** — interactive multi-
turn use rewards a smarter model over Hermes-3-8B's fast-sweep
strengths. LLM=hermes still toggles back to 8B; SESSION_LLM_*
env vars override.
66 verify+session+providence tests pass.
`arborist session` is an interactive multi-turn Q&A REPL where every
turn (or fork) mints one node in a per-session SQLite-backed tree.
Each node carries a stable Bates id (`<sid>-<6-digit>`) and folds into
a Merkle subtree-hash chain; the root node's subtree_hash is the
session_root.
Tree shape lets:
- **Forks** happen implicitly: `/cd <bates>` to a prior node, ask
again → sibling under that parent. Branch points (≥2 children)
surfaced by `/branches`.
- **Page-refresh caching** stay cheap: a client tracking
(bates → subtree_hash, body) only refetches subtrees whose hash
changed. Sibling subtrees that didn't change are byte-identical
→ cache-equivalent. Same property git pack-protocol and IPFS MFS
use.
- **Audit-chain verification** be per-session and independent: each
session db has its own session_audit_events with event_hash =
sha256(prev_hash || canonical_body). `make session-chain-check`
walks all sessions; 0 breaks each = intact.
Wire:
- arborist/qa/session.py — Session class, Bates minting, Merkle
recompute on O(depth) insert, audit chain, helpers (list, render,
resolve <bates|seq|label>).
- arborist/cli.py — `session` subcommand: REPL + --list / --tree
/ --chain-check / --gc / --json flags. Ancestor-titles → retrieval
keywords (parsed from cited-pointer lines in answer_text) flow down
the branch via policy["retrieval_keywords"].
- arborist/qa/providence_query.py — honor policy["retrieval_keywords"]:
augment FTS5 retrieval query without touching cache_key (mirrors
legacy --retrieval-keywords discipline, #000001).
- Makefile — `make session [SID=...]`, `make session-list`,
`make session-tree SID=...`, `make session-chain-check`,
`make session-gc SESSION_KEEP=N`.
- docs/sessions.md — schema, Merkle conventions (portability for
non-Python consumers), REPL command reference.
- tests/test_session.py — 15 tests: create, resume, add_node, fork
via cd, branches, root determinism, audit chain (intact + tampered),
resolve, list, render, sibling-invariance of subtree_hash.
Storage: ~/.arborist/sessions/<sid>.db (self-contained — no FK into
main store). Answers live in providence_cache keyed by cache_key;
session only carries conversation shape. Cache hits stay live across
sessions. Bounded growth via --gc.
Phase 1 scope: tree + Merkle + Bates + retrieval-keyword flow.
NOT in Phase 1: LLM-side conversation_history (threading prior Q&A
into the LLM prompt + conversation_hash). A bare-pronoun follow-up
("who created him?") gets the right retrieval today but the LLM may
still UNGROUNDED because it sees only the new question as user
message. Folding conversation_history into the prompt + cache_key's
conversation_hash dimension is the natural Phase 2.
197 tests pass.
Two perf fixes after the snapshot_root() removal in 9f4152e
unblocked the real bottlenecks on multi-token natural-language
queries against the 5-shard genesis corpus.
1. MultiShardSqliteCorpus.fts_body + _fanout now parallel
============================================================
Previously sequential — 5 shards × ~6s per shard on a query like
"when did aliens film come out?" = 25s wall time per fts_body call.
ThreadPoolExecutor over the per-shard sqlite3 connections drops
that to roughly max(per_shard) instead of sum.
Connections are now opened with `check_same_thread=False`
(`arborist.store.connect()` gains a kwarg, default True preserves
the existing behavior; MultiShardSqliteCorpus passes False so
read-only fan-out works across threads). Read-only FTS5 queries
serialize under SQLite's internal locks; the per-Connection thread
check is what was blocking cross-thread use.
Measured (in-process, 5 shards, "when did aliens film come out?"):
before: 25.5 s sequential fan-out
after: 7.6 s parallel fan-out (warm)
12.3 s parallel fan-out (cold)
End-to-end `arborist query` on the same fixture:
before (post-snapshot_root fix): 65-110 s
after: 14-18 s (cache hit)
16-18 s (cache miss + persist)
legacy reference (--legacy): 10-13 s
Default path is now within 30-40% of legacy on this workload (down
from the 5-6× slowdown fox saw before). Headroom remains because
providence_query still re-runs retrieval inside run_query on miss
(redundant ~7 s); a future refactor can thread precomputed_hits
through.
2. Makefile: `make query LLM=qwen` actually routes to Qwen
============================================================
The `ifeq ($(LLM),qwen)` block setting LLM_ENDPOINT + LLM_MODEL
lived AT LINE 1633, AFTER the `query:` target at line 191.
Make evaluates top-down, so by the time `query:` ran the LLM
variables weren't set — and the recipe didn't reference them
anyway. Result: `make query LLM=qwen Q="..."` silently used
Hermes, the CLI default.
Fix: moved the ifeq block to line 192 (just above the `query:`
target so both `query:` AND the later `cloud-query:` see the
same definitions), and added
`$(if $(LLM_ENDPOINT),--endpoint $(LLM_ENDPOINT) --model $(LLM_MODEL),)`
to the query recipe. cloud-query unchanged.
Smoke:
$ LLM=qwen Q="..." make query
# llm: https://qwen.ai.unturf.com/v1 / Qwen3.6-27B-UD-Q4_K_XL.gguf
.venv/bin/arborist ... query ... --endpoint https://qwen.ai.unturf.com/v1
--model Qwen3.6-27B-UD-Q4_K_XL.gguf ...
176 tests pass.
Replaces the hand-rolled BM25 sidecar (arborist/wallet/sidecar.py, ~690
LOC) with a slim SQLite file that just COPIES the source shard's FTS5
shadow tables verbatim + minimal doc/chunk metadata. Cloud retrieval
then runs SQLite FTS5 bm25() on the same bytes the local shard uses —
bit-for-bit parity by construction. 5/5 source + audit_mode agreement
on the smoke fixture between local corpus-query and cloud-query against
the new manifest-fts.json.
Why
---
Custom binary sidecar was per-document BM25; main encyclopedia articles
got length-normalized so hard that on "why did the dinosaurs go extinct?"
"Edwina, the Dinosaur Who Didn't Know She Was Extinct" beat "Dinosaur"
(measured cloud-vs-local divergence). Local FTS5 indexes per-chunk so
each chunk is a moderate-length doc and the main article wins multiply.
Different granularity, not a tuning knob — fix is to use the same
indexer cloud-side.
What ships
----------
- arborist/wallet/fts_sidecar_build.py — builder. ATTACH source shard,
copy documents (root/uri/title only), copy chunks (id/root/idx/leaf
only, NO content), CREATE VIRTUAL TABLE chunks_fts/documents_fts with
same DDL as source, bulk-copy the four shadow tables verbatim,
VACUUM. 8.78 GB shard → 2.15 GB sidecar (24.5%) in ~45 s; full
4-shard wiki corpus 37.4 GB → 8.1 GB (21.7%) in ~3 min.
- arborist/wallet/bucket.py: FtsSidecarShardClient — downloads slim
sidecar once into ~/.arborist/sidecar-fts-cache/<hash>.idx.db, opens
read-only sqlite3 (check_same_thread=False for parallel shard fan-
out), runs FTS5 MATCH locally. Chunk content fetches via blobs/<hash>
with HTTP-range big-shard fallback when blobs aren't published.
MultiShardSidecarCorpus simplified to fts_sidecar_url ∨ bucket-direct
(both are FTS5 backends; merge by raw bm25 MIN ascending).
- arborist/qa/corpus.py: SidecarBucketCorpus.higher_is_better=False
(FTS5 bm25 is negative, lower=better). chunks_for_doc dispatches on
fetch_chunk_body attr for the slim-FTS5 client. apply_title_boost
imports tokenizer helpers from new arborist/qa/_text_norm.py.
- arborist/qa/_text_norm.py — fold_accents, numeral_expand,
tokenize_text, STOPWORDS — extracted from the deleted sidecar.py so
apply_title_boost keeps its lexical shape.
- arborist/cli.py: `arborist sidecar build-fts` subcommand; old
`sidecar build`/`sidecar search` removed. cloud_query recognizes
fts_sidecar_url + sidecar_url alike.
- Makefile: `sidecar-build-fts` + `sidecar-build-fts-all` targets;
`sidecar-build` + `sidecar-search` removed.
- scripts/upload_fts_sidecars.py — boto3 producer: uploads slim
sidecars to clones/sidecars-fts/<n>.idx.db, publishes
clones/manifest-fts.json (4 wikipedia shards inherit existing
shard_url for content fallback; ACL public-read; idempotent on
size match). Existing manifest-sidecar.json untouched.
- tests/test_qa_corpus_functional.py + test_qa_corpus_integration.py
converted from build_sidecar → build_fts_sidecar; 6 fixtures pass.
- bench/slim_fts_parity_bench.py — local 3-way bench
(legacy/corpus/slim_fts) over the smoke fixture.
Validation
----------
- Per-shard FTS5 parity: slim sidecar returns IDENTICAL rowids + bm25
scores to the source shard for top-10 of "dinosaurs extinct".
- 3-way bench (legacy local / corpus local / slim-FTS5 over real
bucket fallback): 5/5 source agreement AND 5/5 audit_mode agreement
between corpus and slim_fts. Q5 legacy disagreement (Edwina vs
Dinosaur) is the pre-existing 2000-line query() retrieval quirk,
unrelated.
- End-to-end cloud query against published manifest-fts.json (cold-
start, ~149 s sidecar download once): STRICT · Dinosaur, every
quote verified (2/2).
- Full pytest suite: 2737 passed, 28 skipped, 1 xfailed. One pre-
existing failure (tests/test_doc_counts.py — claim_pack docs row-
count drift) and one pre-existing cold_object failure, both reproduce
on main HEAD.
Bucket state
------------
- s3://arborist/clones/sidecars-fts/00[0-3].idx.db (8.1 GB) — new
- s3://arborist/clones/manifest-fts.json — new
- s3://arborist/clones/manifest-sidecar.json — kept live (deprecated
but still readable; downstream callers should switch to
manifest-fts.json)
New ticket for Joseph (@TrudoJo)'s 6-dim procedural spatial-anchor
framework as the first verifier kernel under the #000013 v7-W reserved
namespace. A single committed 32-byte SHA-256 hash deterministically
expands into six 32-byte regions H1..H6 via the HMAC-SHA-512 KDF already
shipped in arborist/substrate/anchor_prg.py (#000035); each region drives
one quantized object dimension under a fixed canonical mapper. Domain
separation from #000035 by dedicated spatial_anchor_seed published in
the v7-W manifest -- preserves #000035's KAT freeze + dav1d 2026-05-11
final review intact.
Bundle:
- docs/tickets/ticket-000070-spatial-anchor-pi-w-object.md (718 lines):
full spec with 8 design-choice subsections, working Python sketch,
12-NOT scope boundaries, 8-row cross-references, five-step deletions,
10 open questions for dav1d.
- bench/spatial_anchor_validation.py: pure-stdlib pre-review evidence
(~2s, RNG-seed-pinned, reproducible). Five benches: avalanche,
cell-distribution uniformity, collision vs birthday-bound, cross-region
independence, domain separation.
- bench/spatial_anchor_validation_results.md: report from first run.
- Makefile: 'make bench-spatial-anchor [SPATIAL_N=N]' target + PHONY.
- docs/TICKETS.md: index row + Next ID 000070 -> 000071.
Bench headlines (N=10000):
- Avalanche mean 767.85 bits (PRF null 768, z=-0.49) -> PASS
- Cell-distribution chi^2 |z|<1 at L=2,3,4 -> PASS
- Birthday-bound ratio obs/exp 0.989/1.038 at populated L -> PASS
- Cross-region Pearson all 15 pairs < 2sigma -> PASS
- Domain separation Arm A 767.91 / Arm B exact 0 collision -> PASS
Five of dav1d's ten open questions (Q1 seed source, Q2 segmentation,
Q3 position mapper, Q8 endianness, Q9 KAT adversarial vectors) now
resolve with measurements rather than appeals to PRF authority.
Q4/Q5/Q6/Q7/Q10 remain non-empirical design decisions.
Status: open, awaiting dav1d review + fox go/no-go. No registry slot
booked, no substrate-paper amendment landed, no kernel module created.
Parallel to `arborist cloud query`: same corpus_query.run_query
orchestrator, only the Corpus adapter differs (SqliteShardCorpus
instead of SidecarBucketCorpus). Proves the protocol works against
both backends with identical pipeline code:
arborist query "Q" legacy 2000-line query() (untouched)
arborist cloud query "Q" corpus_query.run_query + SidecarBucketCorpus
arborist corpus-query "Q" corpus_query.run_query + SqliteShardCorpus ← NEW
Same render layer (_render_cloud_query_human), same audit_mode +
sources + capacity + timings shape across both new paths.
Caveat: single-DB mode only for now. SqliteShardCorpus.fts_body uses
`chunks_fts JOIN chunks ON rowid`, which doesn't bridge per-shard
rowid namespaces under connect_query() ATTACH-and-UNION. Multi-shard
search needs a per-shard query + merge — the next protocol method
(fts_body_per_shard) to add. Until then, --db <path> or auto-picks
the first non-system .db under --shards-dir.
Live demo (homer + virt-back queries):
make corpus-query Q="who developed virt-back?" LLM=qwen
→ EVIDENCE-WARRANTED · via claim_lattice 1/1 2.29s
(search 0.01s · llm 2.26s · verify 0.01s · total 2.29s)
Same E1 citation + render as `make cloud-query` on the same
question; only "0 HTTP requests" footer betrays local vs cloud.
Makefile: `make corpus-query Q="..." [LLM=qwen]`.
Real-world: shared endpoints can queue (hermes was queued behind
opencompletion this morning, single calls pushed past 180s but
completed in 121s on retry). 600s tolerates ~10× the typical
non-queued latency before declaring timeout.
Adds bench/cross_model_selfplay.py + `make bench-cross-model` target.
For each question in a fixture, runs `arborist query` once per
configured model (default: Hermes + Qwen) and tabulates:
* audit_mode per model (EVIDENCE-WARRANTED → POINTER-LINKED → UNGROUNDED)
* agreement on primary source URI
* grounding rate per model
* estimated $/grounded-answer (per-call prices configurable)
* cheap-first cascade analysis (try cheapest, escalate on UNGROUNDED)
This is the "ask twice for two options" pattern from the agent
perspective — bakes it in as a benchmark so we can measure whether
the cascade beats always-using-the-stronger-model on $/grounded.
First live run (2 questions × 2 models, $0.41):
- Hermes: 1/1 grounded (1 timeout — operational issue)
- Qwen: 2/2 grounded STRICT
- Cascade: 2/2 grounded for $0.25 — beats always-Qwen ($0.32)
when Hermes succeeds on its first call.
Output: bench/cross_model_results/<utc-iso>.{jsonl,md} (gitignored).
Single-flag switch to swap the upstream LLM endpoint+model. Default
(LLM unset) leaves --endpoint/--model unspecified so cloud-query
falls back to its built-in default (Hermes-3-8B on ai.unturf.com).
LLM=qwen pins Qwen3.6-27B on qwen.ai.unturf.com (uncloseai). Granular
override still available via LLM_ENDPOINT=… and LLM_MODEL=…
make-vars or the underlying --endpoint/--model CLI flags.
Sample (homer's boss):
make cloud-query Q='who is homer simpsons boss?' LLM=qwen
→ EVIDENCE-WARRANTED · via claim_lattice 2/2 19.4s
(Hermes on the same query landed EVIDENCE-WARRANTED-PARTIAL 1/2)
usage: make cloud-query Q="your question" [JSON=1] [BUCKET_URL=...] [TOP_K=4] [MAX_CONTEXT=24000] [CACHE_MB=64] / — consistent with the
local usage: make query Q="your question" [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N K="extra retrieval keywords" ANSWER_MODE=claim_lattice|claim_lattice_pointer|quote LAYOUT=tail|bookend|per_chunk BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 WITNESS=1 XLANG=1 XLANG_MT=1] / entry points. No alias kept
(per repo convention against backwards-compat hacks for unused
names; this command landed in 65b6fe6 / cb8c1de during a fast-
iteration burst and has no external consumers yet).
CLI: arborist cloud query '...' (was: arborist cloud ask '...')
Make: make cloud-query Q='...' (was: make cloud-ask Q='...')
`BUCKET_URL` (one env var) → client GETs `clones/manifest.json` →
opens HttpRangeVFS per listed shard → FTS5 across all shards in
parallel (ThreadPoolExecutor; per-thread apsw.Connection) → merge by
BM25 score → pull chunks from the owning shard → LLM + verify.
No per-query --shard-url, no path proliferation.
Two manifests published on s3://arborist/clones/:
manifest.json — default: virtback only (2.5MB, ~5s/query)
manifest-full.json — opt-in: all 5 shards (35GB, prohibitive over
WAN due to FTS5 b-tree walk pattern; needs
smaller shards or co-located query proxy)
HttpRangeVFS read-ahead tuned from per-page (4KB) to 64KB block-aligned
cache. Each cache miss fetches one 64KB block; subsequent reads within
the block are local-fast. Lower miss count, similar bytes-on-wire
(64KB amortizes well over typical 4-16 page b-tree clusters; larger
read-ahead like 4MB over-fetches on random FTS5 reads).
Sample run (default manifest):
make cloud-ask Q="who developed virt-back?"
→ EVIDENCE-WARRANTED · via claim_lattice 1/1 4.71s (bucket-direct)
21 HTTP requests · 1344 KB
ACL: genesis full-bench shards flipped to public-read (CC-BY-SA
Wikipedia content). Reachable now if you want to play with the slow
multi-shard path; not in the default manifest because chat latency
matters more than coverage breadth.
Mirrors `make query` ergonomics:
- default = pretty terminal layout (audit_label, sources w/ roles,
bucket stats footer)
- JSON=1 (or --json) = full machine-readable record
_render_cloud_ask_human shares the audit-label primitive
(_render_audit_label) with the local query renderer so the
HYBRID/STRICT/UNGROUNDED tokens map to the same four-rung ladder
labels in lattice modes (POINTER-LINKED / ANCHOR-WARRANTED / ...).
Bucket-direct path doesn't emit warrant-tail / run-DAG /
retrieval-purity so those sections are trimmed.
Output footer adds 'bucket: N HTTP requests · KB · endpoint / model'
so the operator can see network cost + LLM identity inline.
Both paths grind on the same data (LOCAL_DB defaults to web.db, the
file that was uploaded to the Spaces shard at SHARD_URL). --burn busts
the local QA cache so we measure a fresh inference both sides; the
cloud path has no caching layer so it's always fresh.
Reports answer + audit_mode + verifier_method + sources + wall time
for each path, then prints local-vs-cloud delta + ratio.
Sample run (Q='who developed virt-back?'):
local: 3.8s STRICT (verbatim quote verified)
cloud: 11.8s HYBRID (entity-name verified)
delta: +8.0s (3.10x — the HTTP RANGE * ~99 requests tax)
Cloud audit_mode lands one rung lower because the cloud-ask prompt is
simpler than query.py's full pipeline, so the LLM paraphrases instead
of quoting. Verifier honestly demotes paraphrase to HYBRID. Both
answers are correct; the audit_mode difference is a prompt-shape
artifact, not a cloud-path correctness gap.
`make cloud-search` was retrieval-only; `make cloud-ask Q="..."` runs
the same audited-answer shape as local `make query`, but every byte
read goes through the bucket via HttpRangeVFS — no local DB, no
intermediate arborist server.
Pipeline:
1. FTS5 search bucket-direct → top-k document hits (existing path)
2. SELECT chunks.content for each hit via SQL through the same
apsw conn (reuses the warm page cache from step 1)
3. Assemble context with per-doc budget (max_context_chars / top_k)
4. POST to LLM endpoint (default Hermes, override --endpoint/--model)
5. verify_quotes() locally — same verifier the local path uses
6. Emit {answer_text, audit_mode, verifier_method, n_quotes,
n_verified, sources w/ source_role + n_chunks, stats, timing}
Real-world result on the russell.ballestrini.net Spaces shard:
make cloud-ask Q="who developed virt-back?"
→ "Russell Ballestrini developed virt-back."
audit_mode=HYBRID, verifier_method=entity
98 HTTP RANGE GETs, 388 KB, 10.4 s total
endpoint=hermes.ai.unturf.com/v1
Drops the "SHARD_URL required" friction. Default points at the
russell.ballestrini.net web-crawl shard now live on DO Spaces
(public-read, ~2.5 MB) — the smallest end-to-end real-data target
for proving bucket-direct queries work against a real S3-compatible
bucket without any local arborist data.
Override SHARD_URL on the command line to point at a different bucket
shard:
make cloud-search Q='"virt-back"' # uses default
make cloud-snapshot-root SHARD_URL=https://other.bucket/.../000.db
Recipe also switched to single-quoted $(Q) so FTS5 phrase quotes
(`'"phrase here"'`) survive shell parsing.
Pure-cloud consumer: client opens an arborist .db file IN PLACE on a
bucket via HTTP RANGE reads, runs FTS5 + SQL locally, fetches chunk
bodies from `blobs/<hash>` on the same bucket. No intermediate server
in the data path. The bucket layout we already produce (Tier A clones
plus --jit-blobs blobs/) is exactly what this consumer needs.
Module `arborist/wallet/bucket.py`:
- HttpRangeFile / HttpRangeVFS: apsw subclasses. xRead → HTTP Range
GET; xFileSize → cached HEAD. xWrite/xTruncate raise (read-only).
IOCAP_IMMUTABLE so SQLite skips locking/journaling. Empty tempfile
backs the apsw VFSFile C-bookkeeping; never actually read.
- _LRUByteCache: thread-safe (offset,length)-keyed LRU; soft byte
budget (default 32 MB). SQLite's own page cache (~8 MB) handles
most hot-path amortization, so our LRU is the second-level safety
net for working sets that overflow SQLite's cache.
- _HttpTransport: stdlib urllib (zero new runtime deps beyond apsw).
- BucketClient: high-level — fts_search / chunks_for_doc /
fetch_chunk_body / snapshot_root + page-cache stats.
CLI (`arborist cloud <sub>`):
- `cloud search Q --shard-url ...`
- `cloud snapshot-root --shard-url ...`
- `cloud fetch-chunk LEAF_HASH --blob-base ...`
Makefile:
- `make bootstrap-bucket` (installs apsw)
- `make cloud-search Q="..." SHARD_URL=https://.../000.db`
- `make cloud-snapshot-root SHARD_URL=...`
- `make cloud-fetch-chunk LEAF_HASH=... BLOB_BASE=...`
- `make cloud-demo` — end-to-end proof on a vanilla laptop: seeds a
tiny bucket layout in tmp, serves it via a Range-aware static
HTTP server, runs all three cloud commands from an isolated HOME
that has no local arborist data. Asserts laptop HOME stays empty
start-to-finish.
Tests (tests/test_wallet_bucket.py, 4 passing):
- bucket-direct FTS5 results == direct sqlite3 results
- chunk fetch round-trip + hash verify
- snapshot_root bucket-direct == snapshot_root local
- second identical query adds 0 HTTP requests (SQLite-cached)
pyproject: new `[bucket]` extra carries apsw>=3.45; folded into [dev].
Three new targets layered on the wallet-demo:
- wallet-serve long-running server pointed at your real corpus
(defaults: SHARDS_DIR=$HOME/.arborist/shards, override
with DB=path); Ctrl-C to stop
- wallet-pin fetch the server's current snapshot_root once and save
to $HOME/.arborist/wallet.anchor. Real SPV trust:
verify the anchor out-of-band before pinning, then
every subsequent ask grounds against the pinned value
- wallet-ask Q="your question" — reads the pinned anchor (warns +
auto-fetches if no pin), submits via wallet client,
prints the verified JSON. Optional ANCHOR= override.
Typical flow:
make wallet-serve & # one terminal
make wallet-pin # one-time bootstrap
make wallet-ask Q="what is X?" # repeat as needed
Self-contained recipe that proves the wallet works from a machine
with zero local arborist data:
1. shows the vanilla-laptop HOME is empty
2. ingests a 2-doc corpus into a separate server HOME
3. starts `arborist serve` (stub LLM, no upstream calls)
4. bootstraps the trust anchor via curl /snapshot_root
5. runs `arborist wallet ask` on the laptop HOME; verifies exit 0
6. confirms laptop HOME is STILL empty after the verified query
7. runs same call with a deadbeef anchor; verifies exit 3
8. ALL CHECKS PASSED message
Two isolated mktemp HOMEs; trap cleans up server + dirs even on
interrupt. Seed logic lives in arborist/wallet/_demo_seed.py so the
Makefile recipe stays one logical command instead of inlining
multiline Python (make recipe lines turn class/def into SyntaxError).
Run: `make wallet-demo` (default port 18780; override with
WALLET_DEMO_PORT=N).
Adds a distribution channel alongside the pack tier that skips
pack/unpack entirely — producer SQLite-Backup-API's each shard to a
raw .db and multipart-uploads; consumer pulls them down in parallel.
Recovery becomes ~download time (the FTS index travels inside the .db,
no rebuild step). Replaces the ~84 min pack/restore measured 2026-05-29
with ~download time for ~35 GB of raw shards.
Two channels live in the same bucket:
- Tier A (full clone): clones/<snap-id>/00N.db. `arborist cold
stream-snapshot` produces, `arborist cold clone` consumes. Targets
capable peers (the 3090 class).
- Tier B (just-enough + JIT): same flow with --just-enough, but the
producer strips chunks.content into per-chunk blobs/<hash> and ships
metadata-only shards. Consumer's local DB is ~a few GB; the
retrieval path can fetch_chunk_jit() from the bucket on demand.
Targets constrained peers (mobile / SPV).
A small clones/CURRENT.json pointer enables atomic-ish discovery;
pinned --snapshot-id works too. New backend.get_file() streams large
objects via boto3 download_file (multipart parallel into a target
file). Round-tripped locally with MemoryBackend on both tiers
(content preserved byte-exact; JIT verified by hash_leaf).
The FTS pack restore produced a DEAD index — segments present, MATCH=0 —
because the verbatim shadow-table copy used INSERT OR IGNORE, so the
pack's real `_data` rowid-1 "structure" record lost the primary-key
conflict to the empty one `CREATE VIRTUAL TABLE` seeds, leaving a "0
segments" header over orphaned segments. Fix (evict.py): clear the
seeded rows, then copy verbatim (DELETE + INSERT ... SELECT), so each
fts5 shadow table becomes a byte-for-byte copy of the producer's index
and the real structure record survives.
Validated on the 3090: restore one fts pack, NO rebuild -> MATCH
'anarchism'=1189 / 'the'=1.42M (identical to rebuild-from-content);
`_data` id=1 structure record non-empty.
make cold-hydrate: rebuild FTS only when the restored index isn't
already searchable (cold verify-gated). Shipping FTS packs now makes
recovery fast (~24s/shard restore, skip the ~5min rebuild); dropping
them (--no-fts default) keeps the bucket small. Either way cold verify
gates success.
docs/cold-object-store: FTS packs restore correctly now; documented the
restore-vs-rebuild tradeoff, per-consumer guidance, and the fixed bug.
Make cold recovery correct-by-default and fail-loud, closing the class
that silently produced a corrupt, unqueryable corpus (2026-05-29):
- cold rebuild-fts now runs in PARALLEL (one process per shard; separate
files, no contention) and clears-first, so it also repairs the dead
index a cold-pack FTS restore leaves. Replaces the serial post-pass.
- new `cold verify`: self-check a hydrated shard set — chunk content
materialized (not zero-filled) AND FTS searchable (MATCH a word taken
from sampled content). Non-zero exit if any shard fails. Validated: it
passes the good recovery and fails the corrupt genesis-test, catching
both the zero-filled-bodies and dead-FTS classes.
- cold pack defaults to --no-fts (FTS is derived from content and the
FTS pack restore is non-functional anyway); --with-fts to opt back in.
- make cold-hydrate: serial by default (M-aware hydrate routes every
pack into all M shared target shards, so parallel workers contend —
#54); always rebuild FTS from content (drop the broken shard-000-only
gate); run `cold verify` at the end so a bad recovery fails loudly.
- make crawl-ingest writes to one central crawl db (CRAWL_DB, default
~/.arborist/crawl/web.db) instead of per-domain shards in the
peer-shared main dir: keeps locally-crawled content out of peer
sharing by default and a growing domain set under SQLite's 10-attach
cap (Makefile, docs/crawler.md).
- arborist query auto-includes the local crawl db (query() gains
extra_shards; CLI --include-shard / --no-crawl-db, default-on when
web.db exists). Fix latent --db single-file query AttributeError
(cli.py). Persist used / used_pointer_ids + retrieval_purity into
merkle_proof so read-only consumers can see which chunks fed the
answer (qa/query.py).
- arborist.read: read-only seam for dashboards / verifiers; on a
multi-source context root surface the real primary source instead of
the opaque corpus://multi-source sentinel (read.py). Backs the
arborist-viz Merkle Command Center (#000069).
- tests for extra_shards, the CLI crawl-db resolver, and the read seam.
format_user_payload helper in arborist/qa/prompts.py becomes the single
source of truth for the user-turn payload. Three layouts:
tail (default) evidence first, question at end (prior behavior)
bookend question repeated before AND after evidence — counters
lost-in-the-middle on small models (≤8B)
per_chunk bookend + a one-line [for: <q>] reminder before each
evidence block; for list/extraction queries
USER_PAYLOAD_LAYOUTS constant exported; unknown layout raises ValueError.
The six _user_payload closures in query.py (3) and runner.py (3) all
delegate to format_user_payload. Quote-mode passes per_chunk_marker=None
to fall back to bookend on flat document/sources context.
Wired through both DEFAULT_QUERY_POLICY and DEFAULT_POLICY. Folds into
governance_policy_hash (the layout changes the user-turn content the
model sees, so the policy hash partitions cleanly per layout); does NOT
fold into verifier_policy_hash (verifier rules unchanged).
Makefile gets LAYOUT_DEFAULT ?= tail and LAYOUT ?= $(LAYOUT_DEFAULT) so
operators can flip per-call (LAYOUT=bookend make query Q="...") or
session-wide (LAYOUT_DEFAULT=bookend make query Q="..."). Recommendation
matrix in the Makefile comment block above the query target encodes the
2026-05-27 bench finding.
Motivating case: the Veronica-Ballestrini "songs by" failure. Hermes-3-
8B under tail layout returned "specific songs by her are not mentioned
in the provided evidence" when evidence E2 literally contained the song
names. Same query under bookend recovered the answer (with conflation
between Veronica Ballestrini and The Veronicas); under per_chunk
recovered AND disambiguated three entities. Qwen-27B unaffected by
layout. The Ballestrini case is added to bench/qa_questions.txt as a
regression fixture under "entity list", with a 4-line comment pointing
to docs/user-payload-layout.md.
2026-05-27 bench (n=3 × 75q, claim_lattice mode, Hermes-3-8B):
tail STRICT 94/225 (0.418) — control
bookend STRICT 95/225 (0.422) — +0.44pp (noise, 5pp floor)
per_chunk STRICT 72/225 (0.320) — -9.78pp (significant regression)
Verdict: tail stays default (cache-preserving and bench-confirmed
neutral). Bookend/per_chunk available as opt-in operator knobs. Per_chunk
regresses in aggregate because the per-chunk reminder over-anchors the
model on every chunk (TOO_MANY_EVIDENCE_IDS violations rose from 20 →
54; mean answer chars in 32-64KB bucket doubled from 720 → 1660). The
Ballestrini-class failure is real but rare across the curated set; a
layout fix that helps the rare case at the cost of 10pp aggregate is a
bad default trade. Documented in full in docs/user-payload-layout.md
along with the Dav1d 2026-05-27 review framing (GO for opt-in, NO-GO
for default promotion, ADD companion missed-answer guard).
CLI changes (--user-payload-layout flag on `query` and `ask`) landed
separately in commit e5ee283 alongside the #54 busy_timeout fix.
Each producer shard now optionally emits a THIRD pack alongside its
metadata and chunks packs: an "fts" pack containing the FTS5 shadow
tables (chunks_fts_data, chunks_fts_idx, chunks_fts_docsize,
chunks_fts_config + the documents_fts_* counterparts) packed as a
fresh SQLite file inside the tar so BLOB columns round-trip natively.
Consumer detects fts_pack_hashes in the metadata pack's manifest,
pulls each fts pack, ATTACHes the embedded sqlite, INSERTs every
shadow-table row into its target's empty shadow tables, and SKIPS
the local FTS rebuild entirely.
Producer side:
arborist/cold_object.py
+ PACK_KIND_FTS = "fts"
+ FTS_SHADOW_TABLES tuple (8 shadow tables)
+ build_fts_pack(src_db_path, ...)
creates a temp sqlite, applies SCHEMA_SQL (so destination
has FTS virtual tables → shadow tables auto-created), copies
every shadow-table row from src via cursor iteration, packs
the sqlite file into tar.zst
+ ParsedManifest.fts_pack_hashes
+ parse_manifest reads _fts_pack_hashes records
+ build_metadata_pack accepts fts_pack_hashes parameter and
writes the new manifest record
arborist/evict.py:push_pack
+ include_fts: bool = True parameter (CLI --no-fts opts out)
+ Phase B.5 emits the fts pack BEFORE Phase C (metadata pack)
so its hash can be referenced in the metadata manifest
Consumer side:
arborist/evict.py
+ _pull_fts_pack_into_targets() — pulls fts pack body, extracts
embedded sqlite, ATTACHes into each target, INSERT OR IGNORE
every shadow-table row. INSERT OR IGNORE protects against
rowid collisions on other targets that don't own these chunks.
+ hydrate_from_metadata_pack_routed iterates fts_pack_hashes in
full mode, calls _pull_fts_pack_into_targets per pack
+ _pull_pack_inner_routed returns fts_pack_hashes_referenced in
its result dict (mirrors chunk_pack_hashes_referenced)
CLI / Makefile:
arborist cold pack --no-fts (opt-out)
make cold-hydrate (auto-detects: if
chunks_fts_data is
already populated
on shard 000 after
unpack, skip the
rebuild post-pass)
make cold-hydrate HYDRATE_REBUILD_FTS=1 (force rebuild)
make cold-hydrate HYDRATE_REBUILD_FTS=0 (skip rebuild)
Schema:
cold_pending.kind CHECK extended to include 'fts'
pack_key() accepts kind="fts" → packs/<hash>.fts.tar.zst
Expected wall-time impact on the 3090 genesis bench:
with fts in packs: no rebuild step → ~5-10 min total wall
without fts: rebuild post-pass needed → ~15-20 min
Trade-off: ~30-50% larger bucket (FTS shadow data per shard) for
~70-90% faster consumer hydrate. Producer flips the trade via
--no-fts. The fts pack is optional in the manifest (empty list →
consumer falls back to rebuild) so old bucket data without fts
packs continues to work unchanged.
34 cold-unpack-routed + migrate + planner tests pass.
Real-measured bottleneck during the 3090 genesis run (2026-05-26):
all 4 parallel workers sitting in jbd2_log_wait_commit /
do_get_write_access — fighting for ext4's single filesystem journal.
The reshard executor was fast because it ran FTS5 rebuild
SEQUENTIALLY (one process per target in turn); the parallel 4-way
unpack collapsed that into journal contention.
Two coupled changes:
1. arborist/evict.py — _pull_pack_inner_routed
* Drop the _rebuild_fts_on_target call from phase 2c entirely.
Parallel inner loop now does: download → metadata route →
chunks fill. NO FTS5 writes during the hot parallel phase.
* Bulk-load PRAGMA tuning on every target connection:
synchronous=OFF no fsync per commit (already had this)
journal_mode=MEMORY WAL in RAM, not on disk (was WAL)
temp_store=MEMORY sort scratch in RAM
cache_size=-524288 512 MB page cache per connection
mmap_size=536870912 512 MB read mmap
Genesis crash recoverability = re-pull from bucket, so
durability of intermediate state has no value — these tunings
trade durability for throughput.
2. arborist/cli.py — new `arborist cold rebuild-fts --shards-dir DIR`
subcommand. Sequentially rebuilds chunks_fts + documents_fts on
every shard, one at a time. Each shard gets the full filesystem
journal in its turn. Total wall = sum of single-shard FTS
rebuild times, NOT 4× contention.
3. Makefile — `cold-hydrate` chains the FTS rebuild automatically
after the parallel unpack (only when HYDRATE_MODE=full, since
just-enough has no chunk bodies to index anyway). Operator can
skip with HYDRATE_REBUILD_FTS=0 for a 2-pass workflow.
Expected wall time on 3090:
parallel hydrate (download + chunks fill): ~5-10 min
serial FTS rebuild × 4 shards: ~5-15 min total
total: ~15-25 min
vs the killed 2.5-hour run.
34 existing cold-unpack-routed + migrate + planner tests still pass
(the tests' FTS check runs against the reshard path, which still
calls _rebuild_fts_on_target inside the executor — that path is one
process, no parallel contention).
Two genesis paths the SPV-wallet design supports, now both exposed
via the same make target:
make cold-hydrate HYDRATE_DIR=… HYDRATE_M=4 HYDRATE_MODE=full
pulls metadata packs + every referenced chunk pack. Full corpus
offline-queryable post-hydrate.
make cold-hydrate HYDRATE_DIR=… HYDRATE_M=4 HYDRATE_MODE=just-enough
pulls only metadata packs (~4 GB on a 37 GB corpus → ~10×
bandwidth reduction). chunks rows land with content=NULL, every
chunk-body query misses local cache and falls through to a
future JIT-fetch path (cache miss → CDN / mesh / re-pull). The
SPV-wallet headline shape.
Defaults to full for backward compatibility. The mode-flag validation
guards against typos at recipe time (invalid mode → exit 2 before
firing parallel boto3 traffic).
Bench target: measure both paths back-to-back on 3090. Just-enough
should bound the network phase only (no chunk-body fill, no FTS5
rebuild); full adds the chunk-restore phase that #000067's bench-
max round just optimized.
Adds two missing pieces to the cold-pack lifecycle:
1. ALLOW_LICENSE_CLASS pass-through on cold-pack and cold-pack-all.
Without this, packs from a corpus containing any `unknown`-class
docs (html crawls / textbooks) refuse to push to a public bucket
per #000061 Gap 2. Real-world bench on 2026-05-26 hit this
because the production corpus has crawl + textbook content.
Usage:
make cold-pack-all ALLOW_LICENSE_CLASS=unknown
make cold-pack DB=~/.arborist/shards/000.db \
ALLOW_LICENSE_CLASS=unknown
2. cold-hydrate — M-aware genesis-from-bucket as a make target.
Discovers every metadata pack in the bucket via `cold list
--no-manifest` + a tiny python json filter, then fires
$(JOBS)-way parallel `arborist cold unpack` workers (one per
metadata pack) into the same target shards directory. Each
worker writes to one disjoint target shard because the bucket's
packs come from the post-reshard #000065 source layout (each
producer shard's docs all hash to one consumer target) — so
zero write contention across the parallel workers.
Usage:
make cold-hydrate \
HYDRATE_DIR=~/.arborist/shards-genesis-test \
HYDRATE_M=4
Why these belong here (not in /tmp wrapper scripts that get tossed):
fox 2026-05-26 — "make sure the pack and upload and the pull and
ingest all happen with makefile targets". The cold-pack lifecycle
is a first-class operations surface; throwaway tmp scripts hide
the real interface from future operators.
Both targets are listed in `make help`. The .PHONY line covers
cold-hydrate so a directory named cold-hydrate can't shadow it.
Sequence to repeat today's run end-to-end:
producer: make cold-pack-all ALLOW_LICENSE_CLASS=unknown
consumer: make cold-hydrate HYDRATE_DIR=~/.arborist/shards \
HYDRATE_M=4
Today's in-flight 3090 hydrate (started before this commit) used
the equivalent shell loop; from now on every operator uses the
make target.
Previously: COLD_PACK_JOBS hardcoded to 4. Caused the v3 corpus run to
fail at fork time on a memory-pressured box (4 GB available, each worker
needed ~5 GB peak → kernel swapped, SQLite executescript exceeded
busy_timeout, errored as "database is locked"). Killing the run + waiting
for memory to free → 2-way run succeeded.
Now: COLD_PACK_JOBS is auto-computed from `free -m` at run time. Reads
available RAM, subtracts headroom for OS + buff/cache, divides by the
observed per-worker peak, caps at COLD_PACK_JOBS_MAX. An explicit
override (COLD_PACK_JOBS=N on the make invocation) still wins for
operators who know better.
Tunables:
COLD_PACK_PER_WORKER_MB 5500 observed peak from real corpus run
COLD_PACK_HEADROOM_MB 2000 keep this RAM for OS / other procs
COLD_PACK_JOBS_MAX 8 ceiling regardless of RAM headroom
Same logic applied to cold-pack-all-dvd (local-dir output for burning).
Both targets print the chosen concurrency + the math behind it before
launching workers. If the auto-computed value is 1, that's a signal to
wait for RAM to free before trying parallelism.
Doesn't fix the underlying fragility of using fork-time RAM availability
to predict peak demand — workers grow over their lifetime as metadata
dumps materialize — but it's a real improvement over a static cap that
ignored system state entirely.
Shards are independent SQLite files; the ORDER BY landed in 6f0ceab
makes each shard's chunk-to-pack assignment deterministic. Two new
Makefile targets fan out cold pack across every numbered shard via
xargs -P:
make cold-pack-all # parallel push to S3
make cold-pack-all-dvd LOCAL_DIR= # parallel write to disk (no S3)
Tunables:
SHARDS_DIR defaults to ~/.arborist/shards
SHARDS_PATTERN defaults to [0-9][0-9][0-9].db (numbered shards only,
skipping qa.db / snapshots.db / crawl_*)
COLD_PACK_JOBS defaults to 4
Expected wall-clock improvement: ~4x on full-corpus runs (4 shards
running in parallel saturate either CPU or network before serializing).
Also verified end-to-end pack → list → unpack round trip against the
live DO Spaces NYC3 endpoint (small pack, 20 chunks): pushed, listed,
unpacked into a fresh receiving DB, content restored byte-identical,
hashes matched.
New peers landing on a public bucket need to enumerate pack_hashes
before they can `arborist cold unpack <hash>`. `cold stats` only gives
a count; `cold list` returns the per-pack details:
- pack_hash (taken from the key)
- compressed_bytes (one HEAD round-trip via new object_size method)
- chunk_count (parsed from the small manifest sidecar; skippable
via --no-manifest for huge-bucket fast listing)
`cold stats` also now reports total bucket footprint (sums HEAD sizes).
Backend ABC gains `object_size(key) -> int | None` so both subcommands
get sizes without paying egress for the body. S3 impl uses HEAD;
MemoryBackend reads from the dict.
Verified live against DO Spaces NYC3: list-empty → push tiny pack →
list-with-manifest (pack_hash, size=5311 B, chunk_count=5) → list
--no-manifest → cold stats → cleanup.
23 passed in tests/test_cold_object.py + tests/test_evict.py.
Ship arborist corpus state to new peers and DVD-R archival via
point-in-time tar.zst packs. One artifact serves both channels —
bucket+CDN delivery and physical-media archival.
Bucket holds packs only. Pack key = hash_leaf(manifest_bytes), so same
chunk set on two writers produces the same pack_hash and upload is
idempotent. Each pack pins the corpus snapshot_root it covers in audit
+ result body — packs are delayed snapshots, not live mirrors;
falsifications between repacks produce new pack_hashes.
stream_packs runs streaming zstd over tarfile, peeking compressed-buffer
size after each chunk via FLUSH_BLOCK (preserves dictionary). Default
cap 4_400_000_000 — 4.4 GB DVD-R safe-fit, ~6.5% buffer below the
4.7 GB marketing capacity to absorb ISO9660 overhead, growisofs
lead-in/lead-out, media variance, and drive-edge refusal. Each disc
fills to ~4.4 GB recorded data, not the ~1.5 GB an uncompressed cap
produced.
One backend class (S3CompatibleBackend via boto3 + endpoint_url) covers
AWS S3, DO Spaces, R2, B2, GCS S3-interop, MinIO. Optional dep
[object-store] = boto3>=1.34; dev extras pull moto for the wire test.
Voyeur: credentials via AWS_ACCESS_KEY_ID/_SECRET_ACCESS_KEY env or
~/.aws/credentials, never printed; only endpoint URL + bucket name
surface in logs.
CLI: arborist cold {pack,unpack,stats}. Makefile: cold-pack,
cold-pack-dvd (local-dir output for growisofs), cold-unpack, cold-stats.
Sizing for current shards (14.1M chunks, ~17 GB compressed): ~4 packs
at the default cap, ~\$0.34/mo DO Spaces storage, ~\$0.0001/fresh-peer
hydrate.
Always-on raw-UTF-8 leaf store (per ticket "Hard invariants") deferred
— packs-only for now, backfill later.
2557 passed, 28 skipped, 1 xfailed.
Answers fox's 2026-05-21 question — 'are we being swamped because we're
open to internet?' — before the multi-day GPU bench, where uncontrolled
internet traffic on the single-slot hermes (3090/vLLM) and qwen
(4090/llama.cpp) endpoints would contaminate wattage + throughput.
Three stdlib subcommands (urllib + sqlite3 + hand-rolled SVG, nothing to
install):
poll — scrape each endpoint's Prometheus /metrics on an interval into
SQLite; queue depth (num_requests_waiting / requests_deferred)
is the swamp signal a single GPU slot exposes. Prunes past
--retention-days each cycle (bounded store, no cancer growth).
graph — multi-panel SVG: queue depth, running, req/s, tok/s, e2e latency.
access — parse Caddy/nginx access log for the real client IPs the backend
can't see behind the proxy hop; top talkers + per-IP rate SVG.
Backend /metrics = HOW MUCH; proxy log = WHO. make monitor-poll /
monitor-graph / monitor-access.
v1 of the same-model substrate-delta harness's non-jagged metric.
For one corpus title, surface-perturb its question (numeral / accent /
hyphen / honorific / amp / brit) preserving the referent, then ask
whether retrieval surfaces the SAME target for canonical vs perturbed
phrasing. J_norm = XOR disagreement rate @k (lower = less jagged);
graded mean |Δrank| catches rank instability the binary metric misses.
Pure query --dry-run: no LLM, no verifier, no judge, no n=3 noise, no
5pp floor — the recall_at_k discipline. Reuses recall_at_k.probe +
mine_questions._surface_variant. Feeds #000012 ForkScore
ΔJaggednessReduction. A-vs-C answer-quality arm already exists under
#000057 (control_ab/control_sweep) — not rebuilt. Curvature + LLM-arm
jaggedness delta remain open (ticket §8).
make bench-jaggedness JAGGED_LIMIT=40 JAGGED_K=8
fox's call: capture the RAPL permission as a Makefile target run with
sudo, not an ad-hoc chmod. `sudo make rapl-access` on each GPU box
installs a udev rule (/etc/udev/rules.d/99-rapl-readable.rules) that
makes intel-rapl energy_uj world-readable on every powercap add event
(survives reboot), and applies chmod immediately so no reboot is
needed. Idempotent; reversible via `sudo make rapl-access-revoke`.
energy_uj is root-only by default (PLATYPUS side-channel mitigation,
CVE-2020-8694) — that's why bench/watt_bench.py's CpuSampler read null
CPU watts as fox during recon. After this target runs, watt_bench
reads CPU package energy directly (no --cpu-energy-cmd needed). GPU
watts via nvidia-smi never needed special perm.
Non-root guard + help entries verified; Makefile parses clean.
fox: 'not 11 hours it shouldn't take that long'. Probed endpoints —
Qwen-27B absorbs 8 concurrent with 0 errors (0.5->2.1s); the
bottleneck is the serialized claude -p Opus judge, so more workers
≈ near-linear speedup. control_sweep.py gains --resume PATH: appends
to an existing JSONL, skips items already COMPLETE (full
models×variants for solo + variants for arborist if i<=arborist-n),
re-runs partial items; _aggregate now dedupes (i,arm,model,variant)
last-wins so a killed-mid-unit restart never double-counts, and
_load_recs tolerates a truncated trailing line from the kill.
Makefile control-sweep gains CONTROL_SWEEP_WORKERS / _RESUME / _ARB_N
so make stays the interface. 6-worker run killed cleanly (specific
pids, no pkill), relaunched resume @ 12 workers — 12 done items
preserved, 374 to run, ~5h -> ~2-2.5h.
fox ruling: 'we both do not know which framing is right, measure all
benchmarks and bring results forward for review'; 'we have qwen with
and without reasoning to use'.
bench/control_sweep.py: sweeps the CONTROL (solo) arm across
{Hermes-3-8B, Qwen3.6-27B reasoning, Qwen3.6-27B no-reasoning} ×
{plain, source-relative, as-of-corpus-era} questions, judged vs the
fixed corpus-vintage gold; Arborist-Hermes treatment reference run
alongside at the same N. Presents the SAME judge verdicts under both
the accuracy framing (the naive read fox flagged as unfair-as-truth,
shown for contrast) and the grounding-fidelity framing (the
defensible read), plus an explicit note on the faithfulness-ablation
framing + Arborist×Qwen — both deliberately NOT run (proof-path
surgery / different instrument, not a relabelling).
Answers fox's open question empirically: does a 27B *reasoning*
control honestly ABSTAIN where the 8B fabricates (gap = weak-small-
model artefact) or still confidently assert post-corpus (gap = real,
scale-independent)? as_of_corpus separates 'can't recall the era'
from 'won't constrain to a source'.
Qwen toggles probed live: reasoning answer in message.content (CoT in
separate reasoning_content, not surfaced — logged limitation);
no-reasoning via chat_template_kwargs{enable_thinking:false}. Gate =
in-script judge self_test (aborts on fail) — stronger than a make
edge, no double self-test spend.
Opt-in third NLI inference backend (ARBORIST_NLI_BACKEND=tinygrad) in
qa/nli/shadow.py, parallel to torch/onnx-int8, behind the #000049
cage: shadow-only, never an audit_mode input, never auto-preempts the
proven path (guarded so it cannot regress torch/onnx). Loads the ONNX
export through tinygrad's frontend wrapped to the existing
model(**enc).logits contract so _nli_batch is byte-unchanged.
bench/scripts/nli_backend_ab.py + make bench-nli-backends: deterministic
A/B, gate is numerical agreement with the torch reference first,
latency second (a divergent engine = a different shadow signal = a
different nli_policy_hash). Instrument is honest — reports a requested
backend as unavailable rather than relabelling a fallback's numbers.
First CPU-smoke run already quantified that the deployed §7 #22 int8
export diverges Δmax≈0.42 from torch — the immunity property made
measurable, not a defect. Real tinygrad numbers pending a producer-box
run (tinygrad not an arborist dep; frontend op-coverage for the large
MNLI checkpoints unverified by design).
docs/onnx-vendor-capture-immunity.md: why the model-in-proof-path cage
makes the inference engine an interchangeable sidecar, never a trust
dependency — public-domain positioning capital. Indexed in CLAUDE.md.
Full suite 2498 passed (identical to baseline); 24/24 NLI tests green.
bench/control_ab.py + `make control-ab` (gated on judge-self-test
as a make dependency — instrument gate cannot be skipped). Same
model both arms; gold = target-article text by target_root; Arborist
[E…] scaffolding stripped (blinding — format can't betray the arm);
Arborist UNGROUNDED credited as honest abstention; hermetic Opus
judge; deterministic aggregate; self-auditing JSONL; threats-to-
validity printed in the report.
N=2 smoke: clean end-to-end, 0 JUDGE_ERROR — and already surfaced a
case AGAINST the treatment (solo correctly ABSTAINED; Arborist
HYBRID-WRONG). The instrument can falsify the Arborist value claim;
that is the point. n=2 proves nothing (report says so) — verdict
needs a real N.
fox ruled judge = Opus via `claude -p`. bench/judge.py:
hermetic (`env -u CLAUDECODE claude -p`, fresh process, context =
only (Q, answer, gold) — no arm label, no Arborist context, no
session), blinded-by-caller, reference-grounded against the fixed
gold (ignore parametric knowledge), structured via FINAL_VERDICT=
sentinel parsed LAST-match.
Instrument-before-experiment gate worked: first cut parsed
first-match over the model's chain-of-thought → 0/3 self-test. The
judge REASONED correctly; the parser was the defect (+ two bad test
fixtures, my error). Hardened (sentinel contract + fixed fixtures),
re-verified: `make judge-self-test` = 4/4 on known-verdict triples
via real claude -p. The make target is the precondition gate; no
control run trusts the judge until it passes.
Threat to validity recorded, not hidden: same model family judging;
mitigated (blind + no-stake + reference-grounded) not eliminated —
different-family SOTA cross-check is the only full removal.
Next: bench/control_ab.py + `make control-ab` (Hermes-solo vs
Arborist, gold=target-article text, blinded, judged) — NOT yet
built; no broken make target shipped for it.
Three workstreams, full suite 2482 passed, experimental paths default-OFF.
#000055 — Windows quickstart without make
tasks.py (pure-stdlib runner) + make.bat shim + .gitattributes;
README Windows section rewritten. Quickstart needs only Python
3.10+ (no make/bzip2/curl/bash). Mirrors the Makefile quickstart
subset; drift-pinned by tests/test_tasks_runner.py.
#000001 §7 Phase 0 — deterministic cross-language guard
arborist/qa/crosslang.py: non-English signal (¿/¡/non-ASCII) + an
es function-word stoppack. Fail-closed to UNGROUNDED before
retrieval/LLM (mirrors the quantifier reject-DAG) when no content
token survives, else strips es stopwords from the retrieval query
only. English path byte-identical by construction. Default OFF
(crosslang_guard_enabled). Measured: the anarcocapitalismo field
case 10.4s -> 1.6s.
#000056 — Operation Sandwich (cross-language grounding)
arborist/qa/mt/: opus-mt es/fr/ru<->en, lazy per-pair memoised
singleton (fixes the 88%-engine-error concurrency defect),
manifest-pinned, [mt] extra; entity_mask wrapper. Sandwich =
translate query in (retrieval + LLM prompt) -> English answer ->
UNTOUCHED verifier grounds English-vs-English -> translate the
verified answer out as display-only (banner-labelled, zero
grounding). question_hash + verifier_policy_hash invariant; MT
engine identity binds into RetrievalPlan, not governance. CLI
--crosslang-translate / make XLANG_MT=1. Default OFF; entity_mask
default OFF (measured net-negative at bench scale). Fan-out bench
(bench/*.py): Spanish ~0% -> 71% grounded vs the real no-support
baseline; the round-trip predictor was tried and refuted; the
entity-mask lever failed at scale (corpus-title anchoring untried).
CLAUDE.md: cross-language bright-line convention + module map.
Pre-existing modified diagram files are intentionally excluded.
1-pager (docs/_source/arborist-one-pager.rst, 1 page) for AI-literate readers: the trichotomy, the 8-dim cache key, CTI synthetic-elision-impossible, soft-channel separation, real-traffic bench numbers (mis-cite 100% @ 0% FP, warrant 92/92, quote 0.54 STRICT-rate).
2-pager (docs/_source/arborist-two-pager.rst, 3 pages = 2 body + 1 appendix) for technical reviewers: letterhead, Permacomputer Preamble license box, six numbered sections, plus appendix figures (pager-arch-stack 3-layer architecture, pager-verifier-flow question→pointer→verifier→trichotomy).
Both pages live under docs/_source/ so the same RST renders into the Sphinx readthedocs site (toctree caption "Summary pages" added to docs/_source/index.rst) AND into standalone PDFs via rst2pdf (docs/pager.style, lazy install into .venv).
Makefile targets: docs-one-pager, docs-two-pager, docs-pagers, docs-pagers-clean. Diagrams render through the existing DOT pipeline.
bootstrap-nli-only: a venv + [nli] only — no [dev] extras (no crawler/
hessian/vec/sympy). On a CUDA host PyPI's torch wheel is the CUDA build,
so ShadowNLI auto-detects cuda and bench-nli-shadow / export-nli-onnx /
the candidate bench all run on the GPU with no further wiring. Verified
on the ai box (RTX 4090): torch 2.11+cu130, cuda True; 82M cross-encoder
batched ≈ 0.09 ms/pair (512 pairs in 0.047s — ~350x onnx-int8-cpu,
~1300x torch-cpu-batch1); 62-record synthetic shadow sweep in 2.4s wall;
24/24 tests pass. The GPU only accelerates the NLI half — make bench-qa
(Hermes LLM) still runs wherever the shard corpus is.
Speedup (§3 plan): ShadowNLI._nli_batch batches forwards
(ARBORIST_NLI_BATCH=64); device auto-detect (ARBORIST_NLI_DEVICE, else
cuda-if-available); auto-prefer an ONNX export — bench/scripts/export_nli_onnx.py
/ make export-nli-onnx exports + int8-dynamic-quantizes the pinned
checkpoint into ~/.arborist/models/nli/<ver>/onnx/ (operator state, NOT
committed), _ensure_loaded loads model_quantized.onnx via
optimum.onnxruntime (backend onnx-int8), falls back to torch silently.
torch-cpu-batch1 ~120ms/pair → onnx-int8-cpu-batched ~32ms/pair (~4x);
seconds on a 4090. optimum[onnxruntime] added to the [nli] extra; 24
tests.
Gate-item-4 verdict at proper n: ARBORIST_NLI_SHADOW=1 make bench-qa
BENCH_QA_N=1 → 223 cells (89 STRICT / 90 HYBRID / 44 UNGROUNDED; also
surfaced + fixed a lone-surrogate bug). Shadow sweep over those: NLI-as-
runtime-veto on STRICT has ~26% FP at θc 0.5, ~8% at θc 0.90, ~0% only
at θc 0.99 — and θc 0.99 gives up most recombination recall (hard
synthetic recombinations bottom out ~0.76). FAILS the §7 #12 gate on
this design. Only untried path that might pass: a Phase-3 runtime hook
running NLI on the verifier's actual matched clauses (1-3), not
top-6-by-overlap. Until then: runtime NLI demotion stays off; the 2
fixtures stay permanent boundary markers; θc stays 0.5. Production
verifier unchanged; falsification-hard stays 10/12.
arborist/qa/nli/ — SHADOW ONLY (never an audit_mode input; manifest not
yet in governance_policy_hash per §7 #2). manifest.json pins
cross-encoder/nli-MiniLM2-L6-H768 @ a fixed HF revision + the
bench-validated θc 0.5/θe 0.9 + 2 alternates + the Phase-3 TODO;
shadow.py = ShadowNLI/shadow_check (lazy transformers+torch behind a new
[nli] extra, clauses() segmenter, the §7 #5 clause-level Demote()
decision, degrades to available=False when [nli] absent);
bench/scripts/nli_shadow_sweep.py + make bootstrap-nli / bench-nli-shadow
(the gate-item-4 instrument); 16 tests.
First sweep (116 records — 5f-falsification packs + the arborist-nli-bench
eval sets): 28/28 synth recombination demoted, 0/26 FP on legit summaries,
0/9 fires on already-STRICT_SPAN records, 25/50 on UNGROUNDED (the
contradiction half; quiet on non-sequiturs). Gate items 1/2/3/5/6 clear
on available data; item 4 — shadow FP rate on a real live-bench-qa
sample — remains the open measurement. Production verifier unchanged;
falsification-hard stays 10/12.
Closes the 8 mis-segments #000046 left in formulate-hard-v1.jsonl.
The parser was line/bullet-only — one line ⇒ one claim — so a line
that crammed several pointered claims onto one row ("Water is wet
[E1]; fire is hot [E2]", "X happened [E1]. Y followed [E2]") became
one monolithic claim with all the pointers, and a wrapped bullet
became two.
arborist/qa/parse_claims.py: _SEGMENT_SEP_RE splits a line on ';',
sentence boundaries ('. '/'! '/'? ' then a Capital), spaced dashes
(' - '/' — '/' – '), ' and '/' or '/' because '/' although '/' since
'/' while ', inline '(N)' enumeration markers, and commas — with
'(?![^\[]*\])' so a comma inside a [E1, E2] bracket never splits it.
_segment_line keeps the split ONLY IF every resulting non-empty
segment is a well-pointered claim — a legit single claim ("The cat
is black and white [E1].", "The cast: A, B, C [E1].") is never
broken because splitting it would manufacture pointer-less prose
fragments → guard rejects; a leading colon-terminated header with no
pointer ("Two facts:", "Key points:") is allowed and dropped. Plus a
wrapped-bullet join: a continuation line (leading whitespace then a
lowercase letter, no bullet glyph) folds its text + pointers into the
previous claim.
Effect: formulate-hard rate 4/12 → 12/12 (the pack is now at ceiling
— a harder Formulate tier would re-open below-ceiling headroom; a
#000046 follow-up). Remaining #000048 headroom: 2 STRICT_PARAPHRASE
recombinations in falsification-hard (Mercury, Einstein — step 2.2).
Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells; parse_pointer_claims
feeds the 450 claim_lattice_pointer + claim_lattice cells) after
(bench/qa_results/2026-05-11T20-26-37Z) vs the pre-step-2.4 baseline
(...T17-12-41Z = HEAD's parse_claims.py). STRICT-rate quote 0.54→0.55,
pointer 0.22→0.22, lattice 0.43→0.45 — all within the 5-pp noise
floor. Per-row diff: the segmenter changed the parsed-claim count on
the SAME answer text for 7 of the 450 lattice cells (0 in
claim_lattice, 7 in claim_lattice_pointer); of those, 2 caused an
audit_mode change — both correct: a wrap-join recovered an answer's
intended structure (4 claims, 2 pointer-less wrap-fragments → HYBRID)
into 2 well-pointered claims → STRICT; and a crammed-one-line blob (1
monolithic claim, all pointers → STRICT) split into 8 claims, some
not individually verifying → HYBRID (the honest verdict — false-
positive STRICT was the corruption). Every other lattice/quote delta
is LLM re-answer variance. No regression — the segmenter's only
visible effects on real traffic are honest improvements. Summarized
in qa-modes-bench.md Addendum 7 + ticket-000048 §5 step 2.4.
Tests: 8 new in test_claim_lattice.py (semicolon/sentence/conjunction
splits; pointerless-fragment + cast-list guards; leading-colon-header
drop; wrapped-bullet join; pointer-order/multi-pointer); existing
parse_pointer_claims tests pass untouched; test_5f_formulate_hard_pack
re-pinned 4/12 → 12/12. make test 2358 passed, 28 skipped.
#000048 → steps 2.1 + 2.4 landed; #000046 / #000012 §8 / TICKETS.md /
Makefile / fixture _meta + notes updated.
Closes the 4 HYBRID_ENTITY over-grounds #000046 left in
falsification-hard-v1.jsonl. The entity strategy grants HYBRID when a
multi-word proper noun matches the source — but "Insulin was
discovered by Alexander Fleming" against "Penicillin was discovered by
Alexander Fleming" matches on the shared "Alexander Fleming" while the
swapped subject "Insulin" (the falsehood) is ignored.
arborist/qa/verify.py: _entity_salient_disagrees(answer_text, norm_ctx)
flags a >4-char Capitalized content token (stopword-filtered) or a
digit-number in the answer absent from the source.
_is_single_sentence(text) — no internal '. '/'! '/'? ' break. Gated in
verify_quotes' entity branch (proximity policy) in the weakest-grounding
slot only: not cluster AND len(verified) <= 1 AND _is_single_sentence
AND _entity_salient_disagrees → UNGROUNDED. The narrow caller-gate is
what keeps a structured multi-claim summary untouched — the Matrix cast
list (many entities, a tight cluster) and the TMNT answer (a numbered
list with parenthetical nicknames the source omits): model-added
accurate detail in a real summary isn't a contradiction, only the
single-sentence-one-weak-match shape is. The Matrix/TMNT/hybrid
entity-path regression tests still pass, pinned untouched.
Effect: falsification-hard rate 6/12 → 10/12 = 0.833 (Insulin / Berlin
/ 1889 / Pacific now correctly UNGROUNDED). The 2 live-pack fixtures it
newly demotes — 5f-fal-live-003 (the exact gap #000046 built its hard
pack around) and 5f-fal-live-028 — had expected_reason updated
HYBRID_ENTITY → UNGROUNDED (the live pack records what verify_quotes
actually does). Remaining hard-pack headroom: 2 STRICT_PARAPHRASE
recombinations (Mercury, Einstein — step 2.2) + 8 Formulate
mis-segments (step 2.4).
Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells) after
(bench/qa_results/2026-05-11T17-12-41Z) vs the pre-step-2.1 baseline
(...T14-19-51Z = HEAD's verify.py). STRICT-rate quote 0.50→0.54,
pointer 0.25→0.22, lattice 0.45→0.43 — all within the 5-pp noise
floor. Per-row diff (675 common cells, 30 quote-mode rows changed
audit_mode): 0 quote-mode rows demoted to UNGROUNDED from the entity
path — the gate fired on 0 legitimate QA answers in the whole bench.
Every transition was LLM re-answer variance (verifier quote→quote with
the verdict flipping); pointer/lattice deltas are noise too (the gate
is in verify_quotes / quote mode, not the claim-lattice verifier). No
regression — the gate is provably narrow on real traffic. Summarized in
qa-modes-bench.md Addendum 6 + ticket-000048 §5 step 2.1.
Tests: 4 new in test_verify.py (_is_single_sentence helper,
_entity_salient_disagrees helper, swapped-subject → UNGROUNDED,
gate-narrow-on-multi-claim); test_5f_falsification_hard_pack_below_ceiling
re-pinned 6/12 → 10/12; test_fork_score_positive_gamma_5f_... updated
(positive γ·Δ5f on the real lift — possibly MARGINAL given the ÷5
dilution; ACCEPT via a degraded-parent sub-scenario).
make test 2343 passed, 28 skipped.
#000048 → step 2.1 landed; #000046 / #000012 §8 / TICKETS.md /
Makefile / fixture _meta + notes / baseline JSON updated.
`make chain-check-shards` / `chain-check` previously only counted dangling
prev_event_hash references — it missed forks (two events chained off the
same head; qa.db seq 7724/7725 was that, and the cheap check reported 0).
CHAIN_CHECK_SQL now reports the sum of: dangling prev refs + forked
parents (a prev_event_hash claimed by >1 row) + extra genesis rows
(>1 row with prev_event_hash IS NULL). 0 = a single linear chain.
(Working-tree Makefile also carried in-flight bench-5f-* target additions
from a concurrent session; they ride along in this commit.)
