`BUCKET_URL` (one env var) → client GETs `clones/manifest.json` →
opens HttpRangeVFS per listed shard → FTS5 across all shards in
parallel (ThreadPoolExecutor; per-thread apsw.Connection) → merge by
BM25 score → pull chunks from the owning shard → LLM + verify.
No per-query --shard-url, no path proliferation.
Two manifests published on s3://arborist/clones/:
manifest.json — default: virtback only (2.5MB, ~5s/query)
manifest-full.json — opt-in: all 5 shards (35GB, prohibitive over
WAN due to FTS5 b-tree walk pattern; needs
smaller shards or co-located query proxy)
HttpRangeVFS read-ahead tuned from per-page (4KB) to 64KB block-aligned
cache. Each cache miss fetches one 64KB block; subsequent reads within
the block are local-fast. Lower miss count, similar bytes-on-wire
(64KB amortizes well over typical 4-16 page b-tree clusters; larger
read-ahead like 4MB over-fetches on random FTS5 reads).
Sample run (default manifest):
make cloud-ask Q="who developed virt-back?"
→ EVIDENCE-WARRANTED · via claim_lattice 1/1 4.71s (bucket-direct)
21 HTTP requests · 1344 KB
ACL: genesis full-bench shards flipped to public-read (CC-BY-SA
Wikipedia content). Reachable now if you want to play with the slow
multi-shard path; not in the default manifest because chat latency
matters more than coverage breadth.
Pure-cloud consumer: client opens an arborist .db file IN PLACE on a
bucket via HTTP RANGE reads, runs FTS5 + SQL locally, fetches chunk
bodies from `blobs/<hash>` on the same bucket. No intermediate server
in the data path. The bucket layout we already produce (Tier A clones
plus --jit-blobs blobs/) is exactly what this consumer needs.
Module `arborist/wallet/bucket.py`:
- HttpRangeFile / HttpRangeVFS: apsw subclasses. xRead → HTTP Range
GET; xFileSize → cached HEAD. xWrite/xTruncate raise (read-only).
IOCAP_IMMUTABLE so SQLite skips locking/journaling. Empty tempfile
backs the apsw VFSFile C-bookkeeping; never actually read.
- _LRUByteCache: thread-safe (offset,length)-keyed LRU; soft byte
budget (default 32 MB). SQLite's own page cache (~8 MB) handles
most hot-path amortization, so our LRU is the second-level safety
net for working sets that overflow SQLite's cache.
- _HttpTransport: stdlib urllib (zero new runtime deps beyond apsw).
- BucketClient: high-level — fts_search / chunks_for_doc /
fetch_chunk_body / snapshot_root + page-cache stats.
CLI (`arborist cloud <sub>`):
- `cloud search Q --shard-url ...`
- `cloud snapshot-root --shard-url ...`
- `cloud fetch-chunk LEAF_HASH --blob-base ...`
Makefile:
- `make bootstrap-bucket` (installs apsw)
- `make cloud-search Q="..." SHARD_URL=https://.../000.db`
- `make cloud-snapshot-root SHARD_URL=...`
- `make cloud-fetch-chunk LEAF_HASH=... BLOB_BASE=...`
- `make cloud-demo` — end-to-end proof on a vanilla laptop: seeds a
tiny bucket layout in tmp, serves it via a Range-aware static
HTTP server, runs all three cloud commands from an isolated HOME
that has no local arborist data. Asserts laptop HOME stays empty
start-to-finish.
Tests (tests/test_wallet_bucket.py, 4 passing):
- bucket-direct FTS5 results == direct sqlite3 results
- chunk fetch round-trip + hash verify
- snapshot_root bucket-direct == snapshot_root local
- second identical query adds 0 HTTP requests (SQLite-cached)
pyproject: new `[bucket]` extra carries apsw>=3.45; folded into [dev].
The Merkle bundle gives the wallet authentic chunk bytes, but the
server still decides what audit_mode to claim. Run the existing
verify_quotes() locally on the bundle's chunks so the wallet has an
independent verdict that doesn't trust the server's verifier at all.
`VerifiedAnswer` now carries `local_audit_mode`, `local_n_verified`,
`local_verifier_method` alongside the server's audit_mode. They can
legitimately differ (server's context is larger), but a wallet-side
STRICT against a server-side UNGROUNDED would be a real "server lied
about not finding grounding" signal — exactly what the SPV pattern
exists to catch.
Opt out with `client.ask(q, verify_locally=False)` for pure-stdlib
SPV ports that can't load the verifier.
A wallet client holds only a snapshot_root (trust anchor) and verifies
Merkle proofs on every answer. No SQLite, no FTS, no chunks locally.
Same shape as Bitcoin SPV (Electrum / mobile wallet): server can DOS
but cannot forge content whose hash chains up to the trusted anchor.
New module `arborist/wallet/`:
- proof.py: AnswerBundle + build_answer_bundle (server) +
verify_bundle (client). Two proof legs per chunk:
chunk_body → leaf_hash → document_root via in-doc
Merkle proof, then document_root → snapshot_root via
the corpus-wide sorted-doc-roots tree (mirrors
snapshot.compute_snapshot_root). Single-doc corpus
degenerates to "document_root IS snapshot_root" and is
handled with an explicit `degenerate_single_doc` flag.
- server.py: WalletServer + http.server.ThreadingHTTPServer wrapper.
Pure stdlib. GET /healthz, GET /snapshot_root, POST /ask.
Each request opens its own DB connection so SQLite's
single-writer model never bites.
- client.py: WalletClient: urllib + json + arborist.wallet.proof.
Returns a VerifiedAnswer or raises VerificationError /
WalletError. No corpus dependency.
New CLI:
- `arborist serve` — start the wallet server. ARBORIST_WALLET_STUB=1
swaps the LLM for StubClient (lets ops sanity-check verification
without burning tokens).
- `arborist wallet anchor` — fetch the server's current snapshot_root.
- `arborist wallet ask` — submit a question, verify the AnswerBundle
against --trust-anchor, exit 3 on VerificationError.
Tests (tests/test_wallet_spv.py, 7 cases):
- happy: bundle → verify pass against correct anchor
- dict round-trip via to_dict/from_dict still verifies
- tamper: rewrite a chunk body → body hash check fails
- forged leaf_hash: chunks[i].leaf_hash != chunk_proofs[i].leaf_hash
fails before any hashing
- wrong trust_anchor: bundle.snapshot_root != anchor fails immediately
- /healthz and /snapshot_root over real HTTP
- end-to-end ask: corpus → in-process server → urllib client → verify
Replaces the batched chunk-pack phase with per-chunk content-addressed
blob uploads to `blobs/<hash[:2]>/<hash[2:]>`. The metadata pack still
ships (small, fast to restore), but consumers no longer have to pull
multi-GB chunk packs to get queryable: `cold unpack --mode just-enough`
+ `ARBORIST_JIT_CHUNKS=1` fetches single chunks on cache miss.
Producer (`_stream_jit_blobs` in evict.py):
- ThreadPoolExecutor with bounded queue (workers*4) keeps memory flat
across millions of chunks
- HEAD-checks object_size for idempotent re-upload
- Mutually exclusive with chunk packs — manifest's `chunk_pack_hashes`
is empty in JIT mode (consumer reads that as "JIT-only")
Consumer (`hydrate_doc_jit` in cold_clone.py + `_maybe_jit_hydrate` in
qa/query.py):
- Detects both content shapes that need JIT: NULL (Tier B raw-clone) and
zeroblob placeholders (just-enough pack restore, per #53). Discriminator
is first-byte = NUL — zstd-framed bodies start with 0x28, plain UTF-8
prose never has leading NUL.
- Same placeholder filter applied to chunk-read sites in qa/query.py so
partial hydrate doesn't surface zero-bytes content into the LLM context.
Test (`TestJitBlobsPackMode` in tests/test_cold_unpack_routed.py):
- End-to-end push → just-enough hydrate → JIT-fetch → content matches
original byte-for-byte through `unpack_chunk`.
Docs (cold-object-store.md):
- Hard-invariant #1 updated: bucket holds packs by default; `blobs/`
and `clones/` are opt-in prefixes for the JIT and Tier-A flows.
- New "Three consumer modes" section: full-pack vs JIT-blobs vs raw-clone
comparison table + operator decision tree.
- make crawl-ingest writes to one central crawl db (CRAWL_DB, default
~/.arborist/crawl/web.db) instead of per-domain shards in the
peer-shared main dir: keeps locally-crawled content out of peer
sharing by default and a growing domain set under SQLite's 10-attach
cap (Makefile, docs/crawler.md).
- arborist query auto-includes the local crawl db (query() gains
extra_shards; CLI --include-shard / --no-crawl-db, default-on when
web.db exists). Fix latent --db single-file query AttributeError
(cli.py). Persist used / used_pointer_ids + retrieval_purity into
merkle_proof so read-only consumers can see which chunks fed the
answer (qa/query.py).
- arborist.read: read-only seam for dashboards / verifiers; on a
multi-source context root surface the real primary source instead of
the opaque corpus://multi-source sentinel (read.py). Backs the
arborist-viz Merkle Command Center (#000069).
- tests for extra_shards, the CLI crawl-db resolver, and the read seam.
Phase 2 — bench instrumentation + measurement run
bench/qa_sweep.py picks up the answerability sidecar projection per row
(answerability_fired, answerability_confidence, answerability_denial_
pattern, answerability_answer_type, answerability_candidate_count) and
aggregates per-mode (answerability_fires + S/M/W confidence breakdown)
into a new column in the markdown summary table.
Measurement run on bench/qa_results/phase2-sidecar-on/2026-05-27T14-
16-22Z (76 questions × n=3 × claim_lattice × Hermes-3-8B × tail layout,
228 runs). Headline:
sidecar fires 2/228 (0.88%)
confidence dist 2 strong / 0 medium / 0 weak
precision 100% (2/2 fires were the Ballestrini fixture)
recall on Ballestrini 2/3 across n=3 (third run model extracted
correctly -> sidecar silent,
correct behavior)
false positives 0/226 non-Ballestrini runs
verifier verdict both fires labeled STRICT by the binary
verifier (the verifier-blind class, exactly
as predicted)
Detection rule's three-clause conjunction (denial + extraction-shape +
candidate proximity near cleaned subject tokens) is operating at the
precision floor. The strong-confidence-only firing pattern is what
calibrates Phase 3's demote threshold.
Phase 3 — opt-in demote flag (default OFF per Dav1d Phase 4 NO-GO)
arborist/qa/keys.py: answerability_demote_enabled added to
_VERIFIER_POLICY_FIELDS so flipping the flag partitions cache via
verifier_policy_hash. Justification: when on, the rendered audit_mode
changes (EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL), which IS a
verifier-output property; verifier hash must move accordingly. The
other answerability_* fields stay governance-only (sidecar
diagnostic, no audit_mode mutation).
arborist/cli.py:_render_audit_label extended with answerability +
demote_enabled kwargs. Logic:
demote_triggers = (
demote_enabled
and answerability["answerability_warning"] is True
and answerability["confidence_class"] in ("strong", "medium")
)
lattice modes:
EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL (rung transition)
POINTER-LINKED / ANCHOR-WARRANTED -> "rung · missed-answer"
(tail tag; rung itself already
signals degradation)
non-lattice modes (quote/span/entity/paraphrase):
audit_mode token unchanged + "· missed-answer" tail tag
weak confidence: NEVER demotes (Phase 2 saw zero weak fires on real
failures; reserved for future expanded detection ladder)
CLI flag --demote-on-missed-answer on both `arborist query` and
`arborist ask`, default OFF. Flows into call_policy[
"answerability_demote_enabled"] and through to result[
"answerability_demote_enabled"] so the renderer reads it without
needing the policy dict.
End-to-end verified live: 4 fresh Hermes-3-8B runs with --demote-on-
missed-answer on `songs by veronica ballestrini`, all 4 rendered
EVIDENCE-MISSED-PARTIAL · via claim_lattice (Hermes hit the failure
mode in all 4, sidecar fired strong, demote logic transformed the
label).
Phase 4 (default flip to demote-on) — NO-GO per Dav1d 2026-05-27 §3.4:
"a false sidecar warning is tolerable; a false audit-label demotion
can damage trust in correct abstentions." Phase 2 precision is 100%
but n=2 fires is too few samples to claim precision floor empirically.
Default flip blocks on wider bench + human spot-check of the warnings.
Tests: 47 total (36 Phase 1 + 11 new Phase 3 covering hash partitioning
discipline + render-label projection across all four rung/confidence
matrices). Full suite 2794 passed (delta +22 from prior 2772).
Bench output (bench/qa_results/phase2-sidecar-on/) intentionally not
committed — bench/qa_results/ is gitignored per existing convention;
the ticket carries the headline numbers + path for re-inspection.
Adds a deterministic read-only sidecar to detect a class of failure the
binary verifier is structurally blind to:
Evidence contains the answer.
Model says the evidence does not contain the answer.
Verifier sees no unsupported positive claim -> marks run clean.
User receives a false negative under EVIDENCE-WARRANTED.
The motivating case: "songs by veronica ballestrini" against the 2010
Wikipedia corpus. Hermes-3-8B under user_payload_layout=tail returned
"the specific songs by her are not mentioned in the provided evidence
blocks" when evidence E2 literally contained "Amazing", "Out There
Somewhere", "Fascinated", "What's Up With That", "Don't Say". Verifier
correctly returned EVIDENCE-WARRANTED 2/2 because the existing layered
verifier (quote / span / entity / paraphrase + Rule 8 title-relevance +
Rule 9 subject-tokens-absent + claim-count ceiling) guards unsupported
*presence*, has no hook for unsupported *absence*.
Layout fixes attention placement on the specific instance (the 5/27
n=3x75q bench confirms bookend/per_chunk recover Ballestrini); layout
alone cannot close the class -- adversarial phrasing or a bigger prompt
resurfaces the failure under any layout. The right substrate move is to
falsify "not mentioned" as a testable claim.
Detection rule (three-clause conjunction, all must fire):
A. Denial pattern in answer (sealed v1 phrase list: "not mentioned",
"not provided", "the evidence does not say", "does not mention",
"no specific", "no evidence", "cannot determine from the provided
evidence", "is not stated", "is not specified"). Casefolded +
whitespace-normalized substring match.
B. Question is extraction/list-shaped. Either a surface cue ("songs
by", "works by", "books by", "who wrote", "who composed", "what
year", "list of", "name all", ...) matches, OR the existing
arborist.qa.quantifier classifier returns intensity in {ALL,
COMPREHENSIVE, OPEN_REQUEST, MANY, PLURAL}.
C. Evidence contains candidate spans matching the answer_type within
a proximity window (default 600 chars) of cleaned subject tokens.
Candidate kinds aligned to answer_type:
title_like -> quoted_string, title_case_span, comma_list_item
person -> title_case_span
date -> year, date
Hardenings folded in from the 2026-05-27 Dav1d de-novo review:
1. Subject tokens strip cue/relation/stop words. For "songs by
veronica ballestrini" the cleaned subject is ["veronica",
"ballestrini"], NOT all four tokens. Without this the guard
false-triggers on "Harvard University" or "New York" near
proper-noun subjects.
2. Answer-type alignment. Candidate span kind must match query type
so "songs by John Smith" + evidence about Harvard/NY does not
strong-trigger.
3. Confidence class is deterministic (weak | medium | strong), not
boolean. Strong requires quoted_string near exact subject mention
+ multiple type-matched candidates. Phase 3 demote will gate on
confidence_class.
4. Cap output at 10 candidates (the per_chunk-quote-inflation
lesson). Prevents the guard becoming another claim amplifier.
5. Offsets are offset_start + offset_end + offset_basis=
"evidence_object_text", never an ambiguous single offset.
6. Cache-hit path returns answerability: None. Cached records do not
carry the evidence_map, only the rendered sources summary, so the
sidecar cannot recompute candidate spans without re-running
retrieval. Operators wanting fresh diagnostics use --burn.
7. Phase 1 stays out of verifier_policy_hash. The
answerability_sidecar_enabled / answerability_threshold /
denial_patterns_version / extraction_cues_version fields fold
into governance_policy_hash only. Phase 3 demote flag
(answerability_demote_enabled, default False) will move the
verifier hash WHEN ON because it changes the rendered audit_mode
(EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL).
Sidecar discipline (matches arborist.qa.inspect.diagnose_* sister
functions deflection / coherence / title_relevance):
- no model calls (no LLM-as-judge, no NLI, no translation)
- no audit_events writes
- no providence_cache writes
- no answer text mutation
- no claim promotion -- the trigger conjunction makes promotion
structurally impossible (only fires on denial answers)
- byte-deterministic: same (question, answer, evidence, policy) ->
same output every time
Result-dict integration: result["answerability"] is None when the
guard did not fire, or a structured diagnostic dict when it did
(diagnostic_version, confidence_class, triggered_clauses,
denial_pattern_matched, extraction_cue_matched, extraction_shape,
answer_type, subject_tokens, candidate_count, threshold,
missed_answer_candidate_spans). Read by bench_qa (Phase 2 will add
warning-count aggregate to bench rows) and CLI render.
Three return points carry the key:
- miss-path (full retrieval + verify): computed from evidence_map
- cache-hit: None (Dav1d cache-hit recompute discipline -- evidence
not stored, recompute requires re-retrieval)
- reject-broad early-return: None (no evidence examined)
Tests: 36 new pinning the three-clause logic, positive (Ballestrini)
regression, negative control (John Smith + Harvard/NY), each-clause-
alone silence, schema integrity, byte-determinism, sidecar-disabled
short-circuit, dict-shaped evidence support. All pass; existing
inspect tests (60) all pass.
End-to-end verified live via the CLI on the real corpus (2010 ~/.arborist
/shards): 3 fresh Hermes-3-8B runs on "songs by veronica ballestrini",
run 1 hit the failure (sidecar fired with confidence: strong, 351
candidates, denial: "not mentioned"), runs 2-3 model extracted
correctly and sidecar correctly silent.
Phase 2 (bench + threshold tuning) and Phase 3 (opt-in demote flag)
are open as follow-ups. Per Dav1d: NO-GO on default demote-on until
benchmark + human spot-check confirms low false-positive rate.
Full spec in docs/tickets/ticket-000068-verifier-blind-missed-answer-
guard.md (post-review hardenings section at top names the seven
load-bearing changes from the Dav1d 2026-05-27 review).
Real consumer-side bottleneck for cold-pack genesis is the phase-2
chunks-content UPDATE loop: each `UPDATE chunks SET content=? WHERE
chunk_id=?` grows the row from NULL to ~500 bytes, triggering SQLite
page splits, which become ext4 metadata-journal events. With 6M
chunks × 4 parallel writers, those journal events serialize and
dominate consumer wall time (~130 of the 164-min v3-revert run).
Fix: producer dumps a synthetic `_content_size` column in chunks.jsonl
carrying the on-disk byte length of each chunk's content (constant-
time SQLite `length(content)`). Consumer's phase 1 INSERT pre-allocates
the row with `content = bytes(_content_size)` instead of letting it
default to NULL. Phase 2's UPDATE then replaces same-size bytes
in-place — no row growth, no page splits, no per-row journal events.
Implementation:
arborist/cold_pack_metadata.py
_dump_generic_table for `chunks`:
Emit synthetic `_content_size` column = length(content) at the
end of the columnar header. Underscore prefix avoids collision
with any future schema column.
_restore_routed_table:
Detect `_content_size` in the JSONL header; if present (and
table == chunks), build INSERT against [chunks_cols] + ['content']
and substitute a bytes(_content_size) placeholder for the
content position. Phase 2 UPDATE later replaces those bytes.
Forward compatibility:
- Old packs (no _content_size): consumer uses today's NULL-content
INSERT path. No behavior change.
- New packs: consumer auto-detects, uses pre-sized path.
SPV-wallet trade-off:
In just-enough mode the consumer pulls only the metadata pack so
chunks land with the placeholder bytes (NOT NULL anymore). That's
a SEMANTIC CHANGE for SPV — `chunks.content IS NULL` no longer
means "JIT-fetch later." Documented in code; if SPV-mode JIT-fetch
ever ships, it must distinguish placeholder bytes (where every
byte is 0) from real content.
New test (TestPreSizedChunks.test_chunks_content_pre_sized_after_metadata_restore):
Hydrate just-enough → chunks rows have non-NULL bytes content of
correct size. Catches the regression if a future change reverts
the placeholder logic.
Expected impact: ~50-70% reduction in phase-2 wall time. Real number
lands when the next 3090 bench-max iteration runs against re-packed
bucket. 6 cold-unpack-routed tests pass.
v4 bench (2026-05-27 03:09 UTC) measured a corrupt outcome:
chunks_fts=1,561,604 on EVERY target shard regardless of chunks
count. _pull_fts_pack_into_targets had been iterating all M targets
and INSERT'ing each fts pack's shadow tables into every one of them.
Each fts pack's chunks_fts_docsize entries reference chunk_ids that
were independently auto-assigned in its source producer shard (each
source DB has chunk_ids 1..1.56M independently). Inserting all 4
packs into all 4 targets → 4× the per-target FTS rows, pointing at
chunk_ids the target doesn't own. Body searches would return garbage.
Fix: sample one id from the fts pack's chunks_fts_docsize, look it
up in each target's chunks table, INSERT only into the target where
it's found. The other M-1 targets stay untouched and receive their
FTS data from their corresponding fts pack(s) in later iterations.
In post-reshard production topology, each producer source shard's
docs all hash to ONE consumer target, so this 1:1 mapping is exact.
The test fixture is artificial (single-shard producer with docs
hash-distributed across M=4 targets) but still validates the core
property: only one target receives FTS data; the others stay empty.
FTS5 shadow tables aren't subsettable per-row (segment data is
opaque, mixed entries for many docs in one segment) so we can't
filter FTS rows to "only chunks that exist on this target" — we
copy all-or-nothing per pack. That's why the producer's post-reshard
shape (each pack scoped to one target's docs) is the structural
prerequisite for fts packs to make sense.
New regression test
(TestFtsPackRoutingRegression.test_fts_pack_only_on_owning_target):
exactly 1 of M targets has chunks_fts_docsize > 0; the others
must have 0. Pre-fix this asserted on all-4 targets having FTS
data → failed. Post-fix passes.
Returns now include owning_target_idx for forensic visibility into
which target the fts pack landed on.
5 cold-unpack-routed tests pass.
Today's hydrate_from_metadata_pack writes every row into one shard.
With the corpus in M=4 hash-routed topology (#000065), a fresh peer
pulling packs must land each doc on shard_for_document(root, M) —
same routing function as the producer — or the consumer's M=4
ATTACH-and-route assumption is just decoration over a single-shard
reality.
Code (new entry points alongside the existing single-conn ones):
arborist/cold_pack_metadata.py
+ restore_shard_metadata_routed(targets, M, table_dir)
+ _restore_routed_table (per-document tables)
+ _restore_edges_fan_out_routed (edges by src_root)
+ _shard_idx_for_root helper (mirrors arborist.document)
+ routing rules: _ROUTED_BY_COL / _CONSOLIDATED_TO_SHARD_0
(mirror migrate.py's ROUTED_BY_DOCUMENT_ROOT / CONSOLIDATED_TABLES)
arborist/evict.py
+ hydrate_from_metadata_pack_routed(targets, backend, hash, M=, mode=)
+ _pull_pack_inner_routed (mirrors _pull_pack_inner; phase 2
chunk-body fill iterates every target shard — leaf_hash lookup
naturally hits at most one since each chunk's metadata row
landed on exactly one target during phase 1)
arborist/cli.py
arborist cold unpack
+ --hydrate-shards-dir DIR (M-aware genesis path)
+ --hydrate-M N (default 4 matches #000065)
legacy --db / --global-shards-dir path unchanged
Behaviour notes:
- FK enforcement off on target writes (cross-shard refs are valid
under hash routing, same fix as #000065 reshard executor)
- audit_events lands on target 0 (Option A consolidation)
- corpus-wide tables (snapshots / concepts / aliases / providence)
consolidate to target 0
- per-document tables route by document_root (documents,
document_http_meta, chunks, merkle_nodes)
- edges route by src_root (matches migrate.py)
- derivations route by core_root (matches migrate.py)
- existing single-conn API unchanged — callers that didn't pass a
shards-dir get the legacy single-shard behaviour
4 new tests:
test_pack_then_hydrate_routed — end-to-end pack → hydrate → assert
every doc on its hash-routed target, no doc on the wrong shard
test_corpus_shard_count_set_on_routed_hydrate — meta plumbing
test_M_mismatch_rejected
test_empty_targets_rejected
All 59 prior tests still pass.
Refactor opportunity (not taken): _ROUTED_BY_COL duplicates
migrate.py's ROUTED_BY_DOCUMENT_ROOT. A shared arborist/multi_shard.py
module would serve both reshard and graft (#000066). Left as
follow-up since the duplication is small and graft is still scaffold.
Unblocks #46 genesis on 3090: that's now a single arborist cold unpack
--hydrate-shards-dir ~/.arborist/shards --hydrate-M 4 invocation
instead of the α two-step kludge (hydrate-then-reshard).
Add PRAGMA wal_checkpoint(TRUNCATE) at two points in
_execute_all_at_once so committed WAL pages don't pin disk through
subsequent passes. Production migration on 2026-05-26 hit 7 GB free
disk (down from 89) because SQLite's auto-checkpoint can't run while
a reader cursor is open, and FTS rebuild keeps a SELECT cursor open
through 1.5M chunks per target. Across 4 targets the FTS rebuild
plus audit consolidate held ~37 GB of committed-but-unreclaimed WAL.
Manual sibling-connection wal_checkpoint(TRUNCATE) freed 27 GB
mid-migration.
Checkpoints land at:
* end of _rebuild_fts_on_target (after the SELECT cursor is
explicitly cur.close()'d so the TRUNCATE checkpoint can actually
fire — TRUNCATE/RESTART block on active readers)
* end of _consolidate_audit_chain (after the 3.47M-row giant
transaction commits, before the next phase touches the same
connection)
VACUUM is already implicitly a checkpoint, so the existing per-
target VACUUM pass continues to handle the final checkpoint
naturally.
Helper _checkpoint_truncate(conn) returns the (busy, log_frames,
checkpointed) tuple SQLite emits; for the serial executor, busy=1
is improbable since each phase finishes before moving on.
Regression test
(TestWalCheckpointing.test_no_large_wal_after_migration) asserts
no WAL file exceeds 4 MB after migration completes. Without the
checkpoint calls this would fail on real-sized corpora; with them
the test passes deterministically.
Doesn't affect the running migration (it loaded the module from
memory before this commit). Future reshards run with bounded WAL —
no near-ENOSPC scares.
The 2026-05-26 cutover crashed mid-build with sqlite3.IntegrityError
"FOREIGN KEY constraint failed" inside _route_per_doc_table on the
derivations table.
Root cause: derivations.src_root carries a FK to
documents.document_root, but under content-hash routing a derivation
row's src_root can legitimately reference a surface doc that hashes
to a DIFFERENT target shard than the derivation's core_root. The FK
is a single-shard-era guard; it must stay live for the runtime
write path (to catch typo'd inserts into the wrong shard) but must
be OFF for the migration writer which legitimately produces
cross-shard refs.
Fix: arborist/migrate.py _connect_target now applies
`PRAGMA foreign_keys = OFF` after SCHEMA_SQL executescript runs.
Schema's own `PRAGMA foreign_keys = ON` still applies to the schema
DDL pass (and runtime connect() / connect_query() still get FK=ON
since they don't touch this helper). Only the migration writer is
relaxed. Documented inline.
Regression test
(TestCrossShardForeignKeys.test_cross_shard_derivation_succeeds)
synthesizes a derivation row whose core_root and src_root hash to
different M=4 target shards, runs the migration, asserts the row
lands on core_root's target with src_root pointing cross-shard. Pre-
fix this raised IntegrityError; post-fix it passes.
Originals untouched on the production host — the executor crashed
BEFORE the atomic-promote step, so .db files are intact;
~/.arborist/shards/00X.db.new files from the failed run will be
cleared by the next attempt's "stale .new before opening" cleanup
hook (already in _execute_all_at_once).
Reshard no longer creates a parallel "shards.v2/" directory. Builds
write to "<target_dir>/00X.db.new" alongside originals; when the
executor's validation passes, each .db.new is atomically renamed to
its final 00X.db name via os.replace (POSIX-atomic per file).
For an in-place migration (target_dir == source_dir, which is the
canonical use case), the rename REPLACES the original shard files.
~/.arborist/shards/ never contains a parallel "v2" or "next" or
"backup" directory — it always holds exactly one corpus, just with
the new topology after promote completes.
Validation gate (refuses to promote on mismatch):
* --expected-row-counts <snapshot.json> threads the pre-migration
snapshot's documents/chunks/edges totals through to the executor.
* Tolerance: ±1% to absorb the dupe-collapse from INSERT OR IGNORE
on cross-shard duplicate document_roots (166 dupes measured
pre-migration; <0.005% drift).
* Mismatch → RuntimeError, .db.new files left in place for
inspection, no rename performed.
CLI:
arborist corpus reshard --to 4 \
--source-dir ~/.arborist/shards \
--target-dir ~/.arborist/shards \
--audit-events-ndjson /tmp/audit-events.ndjson \
--expected-row-counts bench/results/pre-migration-snapshot.json
3 new tests:
* test_target_dir_only_has_db_files_after_completion — no .db.new
sidecars survive a successful run
* test_inplace_reshard_overwrites_originals — target_dir ==
source_dir works end-to-end; final files are the new hash-routed
shards
* test_validation_failure_leaves_new_files — bad expected counts
trip the validation guard; .db.new files survive; no .db files
promoted
Stale .db.new files from a prior failed run are unlinked before
opening fresh targets, so a partial-fail-and-retry is idempotent.
WAL/SHM sidecars removed at promote time so the new live .db
produces fresh sidecars on next open.
Build the 'all_at_once' executor and wire it into the CLI as
`arborist corpus reshard`. Plan-only preview against fox's host
confirms the planner picks all_at_once at 95.5 GB free, 64.2 GB
peak draw, 31.3 GB free at peak (well above the 4 GB safety).
Executor (arborist/migrate.py):
* ROUTED_BY_DOCUMENT_ROOT — documents, document_http_meta, chunks,
merkle_nodes, edges (by src_root), derivations (by core_root).
Each row hashed to shard_for_document(root, M) and INSERT'd into
the chosen target.
* CONSOLIDATED_TABLES — snapshots, concept_relations,
concept_token_idf, citation_aliases, term_aliases,
providence_cache, falsifications all land on canonical
target shard 000.
* REBUILT_ON_TARGET — chunks_fts + documents_fts rebuilt from the
materialized data after content moves. chunks.content is
zstd-packed at rest so the rebuilder decompresses via
arborist.compress.unpack_chunk before inserting plaintext into
the FTS5 index.
* Audit chain consolidation (Option A) — all rows from
/tmp/audit-events.ndjson are sorted by (ts, src_shard, src_seq),
re-chained with fresh event_hash values, and INSERT'd into
target shard 000. Bodies preserved unchanged for forensic
fidelity.
* One 'reshard' audit event appended at the tail, carrying the
plan + result as the body. An operator months later can answer
"where did this corpus topology come from" from this one row.
* corpus_shard_count meta stamped on every target shard.
* VACUUM each target at end to reclaim INSERT-pattern fragmentation.
CLI: `arborist corpus reshard --to M --source-dir DIR --target-dir DIR
[--plan-only | --force-strategy X
| --allow-in-place | --dry-run]`
11 integration tests build a tiny 2-shard corpus, run the migration,
verify per-doc routing, chunk-follows-doc, audit chain integrity,
reshard event at tail, corpus_shard_count meta on every target,
chunks_fts searchability, doc count preservation, dry-run no-op,
error handling.
Also: get_meta() now indexes by position so callers without
row_factory=sqlite3.Row don't trip. No behavior change for callers
that DO use Row factory.
Strategy B (per_source_shard) and C (streaming_row) are stubbed in
the planner (peak-draw estimators wired) but execute_plan raises
NotImplementedError for them. Not needed at fox's current disk
(88+ GB free); skipping the build until that's ever the constrained
path.
Disk-aware strategy picker for the corpus reshard. Pure-compute API:
feed it (corpus facts, free bytes) and it returns a HydrationPlan
naming one of three strategies + a peak-draw estimate.
Strategies (preference order):
all_at_once - direct source→target with overlap; fastest;
peak draw = corpus + WAL × M_target + VACUUM
per_source_shard - pack→delete→hydrate per round; safest;
peak draw = pack + WAL + 1 target VACUUM
streaming_row - in-place mutation, --allow-in-place opt-in;
peak draw = WAL only
Safety budget: 4 GB margin above the strategy's peak draw — the
buffer that survives one bad sort + one badly-sized WAL grow.
Production-host preview at 95 GB free / 38 GB corpus:
strategy: all_at_once
peak draw est: 64.2 GB
free at peak: 31.2 GB (well above safety)
rationale: all_at_once: peak draw 64.2 GB + 4.3 GB safety ≤ free 95.4 GB
The plan's full readout (strategy, free bytes, peak draw, rationale)
is JSON-serializable so it goes into the migration audit event as a
single forensic record.
14 tests cover: strategy pick by free-disk level, --force_strategy
override, --allow-in-place gating, skewed shard sizes, json-
serializable audit body, target_M validation.
No execution yet — this is just the planner. Next: the strategy A
executor (direct source→target read+write).
Three pieces, all read-only or additive — no shard mutation, no
schema-version bump:
1. shard_for_document(document_root, M) in arborist/document.py.
Pure function: int(document_root[:8], 16) % M. 22 tests cover
determinism, range-bounds, near-uniform distribution (±5pp at
N=20k), and seven lock-in fixtures so peers will disagree
loudly if anyone changes the formula.
2. corpus_shard_count meta field + get/set helpers in store.py.
Lives in the existing key/value meta table; SCHEMA_VERSION
stays at v9.8.0 (the DDL doesn't change and source_root is
layout-independent, so cache records survive a reshard).
Legacy shards (without the field) return None; reshard tool
populates it on every target shard at migration time.
3. Pre-migration snapshot captured to
bench/results/pre-migration-snapshot.json:
docs 3,468,392 (3,468,226 globally unique)
chunks 6,235,764
edges 90,593,537
audit 3,468,403
This is the reference set post-reshard row counts must match.
4. Audit-event extraction script writes all 3.47M events from
all 4 shards to /tmp/audit-events.ndjson (2.0 GB) for the
Option-A canonical-chain consolidation step. Verifies chain
integrity on extract — all 4 source chains report 0 breaks.
5. Fixed a wrong chunk count in docs/corpus-history.md
(had ~3.54M/shard; actual is ~1.56M/shard) and added the
edge-count column (~22.6M/shard, 90.6M total). 6.24M chunks
total, not 14.12M.
Tests: 29 new pass (22 routing + 7 meta). No existing tests
touched.
Dav1d's reviews of #000061 (Response A + Response B/FINAL in
~/Downloads, 2026-05-26) flagged a long list of items — most already
shipped in the SPV-split work. Three were genuine gaps worth folding
into #000061 before close:
Gap 1: manifest/latest pointer for new-peer discovery.
A fresh peer doing `cold list` got a list of metadata-pack hashes
but no obvious "which one is current for shard X." Added
get_latest_pointer + update_latest_pointer to the backend ABC.
push_pack writes manifest/latest.json on every successful metadata
pack push (read-modify-write keyed by snapshot_root). Mutable
pointer; content addressing of the packs themselves preserves the
trust root. Last-writer-wins on contention.
Gap 2: license_class field + producer-side refuse for public buckets.
Maps documents.source_type to a license bucket (wikipedia_cur /
textbook_tex → public_redistributable; html / grok / vcs → unknown;
anything else → unknown). Strictness order: public < unknown <
private. compute_shard_license_class() walks DISTINCT source_type
in documents. push_pack now refuses to upload if the shard's
strictest license is more restrictive than the operator's
allow_license_class (default: public_redistributable). The
metadata pack's manifest carries _license_class so consumers /
auditors can see the producer's classification without inspecting
source documents. ValueError on refusal — the bucket ACL is the
operator's call, but arborist refuses to participate in a
licensing/membership leak unless explicitly opted in.
Gap 3: cold_pending table for resumable uploads.
Killed mid-upload, push_pack left orphan multi-GB tempfiles in
/tmp with no DB trace. Added schema:
CREATE TABLE cold_pending (
tempfile_path TEXT PRIMARY KEY,
pack_hash TEXT NOT NULL,
kind TEXT NOT NULL,
backend_endpoint TEXT NOT NULL,
backend_bucket TEXT NOT NULL,
object_key TEXT NOT NULL,
started_at INTEGER NOT NULL,
state TEXT NOT NULL DEFAULT 'pending'
);
push_pack INSERTs a row before each upload + DELETEs on success.
A killed process leaves the row pointing at the orphan tempfile;
a recovery script (future) reads cold_pending, checks bucket for
the object, either deletes the row + tempfile (success was just
unreported) or re-uploads from the tempfile if it still exists.
Matches the same pattern as the audit chain — explicit state
rows beat inferring from chunks.content IS NULL.
Sibling tickets opened for the larger items the reviews flagged
(scaffold-only, no code; opening them captures the design in the
log without proliferating, per CLAUDE.md):
- #000063 Cold-object private-ciphertext mode (mesh-keyed object
keys for non-public corpora on public-read buckets). Needs mesh
group-key ABI + real non-public corpus before code.
- #000064 Cold-object operations toolkit (verify / diff / doctor /
repair-fts / gc-plan CLI + expanded audit-event taxonomy).
Bundled so the audit-event vocabulary gets one design pass.
5 new tests:
test_gap2_license_gate_refuses_unknown_class_to_public_bucket
test_gap2_license_class_in_metadata_manifest
test_gap1_latest_pointer_resolves_metadata_pack_per_snapshot
test_gap3_cold_pending_clears_on_successful_upload
test_gap3_cold_pending_records_inflight_upload
26 cold-object + 7 evict tests pass (33/33 green incl. boto3 wire).
Next ID bumped to 000065.
Live v3 SPV corpus run (bmq47x6t3) completed cleanly during this work.
Will report sizing + memory profile in the next message.
Bidirectional sync: producer always emits both kinds; consumer chooses
how much to pull.
packs/<hash>.metadata.tar.zst — one per shard (~0.6 GB compressed)
packs/<hash>.metadata.manifest.ndjson
packs/<hash>.chunks.tar.zst — N per shard (each ≤ 4.4 GB cap)
packs/<hash>.chunks.manifest.ndjson
Each artifact is independently content-addressed by its own manifest
hash. The metadata pack's manifest carries _chunk_pack_hashes — every
chunk pack covering this shard's content — so a "full" consumer can
iterate them. Chunks packs are anonymous from the consumer side
(reachable only via the metadata pack's reference list).
Consumer sync modes:
arborist cold unpack <metadata_hash> # default: just-enough
arborist cold unpack <metadata_hash> --full # also pulls chunks
just-enough: pull only the metadata pack. Schema fully restored; every
chunks row has content=NULL. Node is immediately queryable for
metadata operations (documents, edges, audit chain, Merkle); chunk-body
queries return null until a future JIT-fetch path fills them on cache
miss.
full: after the metadata pack lands, iterate _chunk_pack_hashes and pull
every chunk pack. Final state: full corpus offline-queryable.
This is fox's original SPV-wallet framing — was right from day one.
Key API changes:
pack_key(hash, *, kind="chunks", manifest=False) — kind in bucket path
stream_packs(chunks, *, max_compressed_bytes, ...) -> Iterator[FilePack]
now emits chunk-only packs (no extra_members)
build_metadata_pack(table_files, *, snapshot_root, chunk_pack_hashes, ...)
single FilePack with v3 metadata manifest
pull_metadata_pack(conn, backend, hash) -> dict
pull_chunk_pack(conn, backend, hash) -> dict
hydrate_from_metadata_pack(conn, backend, metadata_hash, *, mode) -> dict
push_pack orchestrates: dump tables → stream chunk packs (collect hashes)
→ build metadata pack with chunk_pack_hashes → upload all. Returns
{metadata_pack_hash, chunk_pack_hashes, packs: [...]}.
Manifest format v3:
metadata pack:
{"_format_version": 3}
{"_kind": "metadata"}
{"_snapshot_root": "..."}
{"_chunk_pack_hashes": [...]}
{"table_file": "tables/X.jsonl", "hash": "...", "size": N} per table
chunks pack:
{"_format_version": 3}
{"_kind": "chunks"}
{"leaf_hash": "...", "size": N} per chunk
pack_hash for each = hash_leaf(manifest_bytes); content-addressed at
both layers. Two writers with the same shard state produce identical
metadata_pack_hash AND identical chunk_pack_hashes.
Cap-and-split applies only to chunk packs (chunks are bounded N).
Metadata pack is one file per shard; if a single table file is larger
than the cap, that's noted as future row-level split work.
cold list now surfaces kind per artifact (metadata vs chunks), plus
table_count + chunk_pack_hashes (for metadata packs) or chunk_count
(for chunks packs). Operators can quickly find the metadata pack hash
to feed `cold unpack`.
28 cold-object + evict + boto3 tests pass (3 new SPV-shape tests + 1
v3 manifest-shape test + tampered-metadata-pack test).
Caught a real defect on the live v2 corpus run: pack_hash was computed
from the chunk-only manifest, so two packs with the same chunk set but
different table content collided on pack_hash. Observed live: v1 packs
and v2 packs for the same shard produced the SAME pack_hash:
v1 (chunks-only) b3c427baaf9b40e3c33ace8438b4254e3abdc7b38bdcb7b1a8012588cb32848a
v2 (tables + chunks) b3c427baaf9b40e3c33ace8438b4254e3abdc7b38bdcb7b1a8012588cb32848a
↑ identical, different bytes
Consequences if shipped: bucket overwrites swap whose tables you get,
mesh peers A and B disagree on what "pack X" is while both claim to
have it, content-addressing story is broken for the metadata half.
Fix: manifest format v2 prepends a `{"_format_version": 2}` header,
then one row per shipped table:
{"table_file": "tables/audit_events.jsonl", "hash": "...", "size": N}
{"table_file": "tables/chunks.jsonl", "hash": "...", "size": N}
...
{"leaf_hash": "...", "size": N} # chunks come after
{"leaf_hash": "...", "size": N}
pack_hash = hash_leaf(manifest_bytes) now binds the full pack content.
Two writers with the same shard state produce the same pack_hash; same
chunks + different tables produce different pack_hashes.
Implementation:
- hash_file_leaf(path) — streaming sha256 with leaf 0x00 prefix, 1 MB
reads, used to fingerprint multi-GB tables/<name>.jsonl files without
loading them into RAM.
- stream_packs' extra_members now takes (member_name, source_path,
content_hash) tuples. push_pack computes the hash with hash_file_leaf
after the dump finishes.
- _finalize_pack writes the format header + table refs (sorted by name
for determinism) before chunk entries.
- parse_manifest returns a ParsedManifest (format_version, tables[],
chunks[]) instead of just list[PackEntry]. Backward-compatible with
v1 manifests (no header → format_version=1, tables=()).
- pull_pack pulls the manifest aside during tar walk, then after
extraction verifies each tables/<name>.jsonl file via hash_file_leaf
against the manifest's content_hash. Mismatch → ValueError, no rows
reach the live schema.
New tests:
- test_v2_pack_hash_binds_table_contents — asserts manifest format +
table refs + per-table hash & size.
- test_pull_pack_rejects_tampered_table — corrupts one table's bytes
in a real pack, confirms pull_pack raises on hash mismatch.
27 cold-object + evict tests pass.
Live v2 corpus run showed ~5 GB RSS per worker — RssAnon dominant, so
process heap, not mmap. Traced to two unfixed memory pits in the edges
fan-in dump path:
1. SQLite ORDER BY on edges (22M rows, no covering index for the v2
sort order dst_uri+edge_type+anchor) allocates a multi-GB in-memory
sort area before spilling. Adding:
CREATE INDEX IF NOT EXISTS idx_edges_dst_uri_type_anchor
ON edges(dst_uri, edge_type, anchor, dst_root, src_root)
means the ORDER BY walks the index in order — no in-memory sort.
First create takes ~30-60 s on a 22M-row shard; idempotent on
subsequent dumps. Disk cost ~1 GB per shard (4 shards × 1 GB ≈
2-3 % corpus footprint increase). Worth it.
2. Python groupby accumulator: src_roots = [row[4] for row in group]
materializes the entire src_root list per destination. For
en.wikipedia.org/wiki/* destinations with millions of inbound links,
this list is itself ~GB-sized. Switch to bounded batches:
_FAN_IN_BATCH = 10_000 # max src_roots per fan-in JSON row
A destination with N inbound links splits into ceil(N / batch) rows.
Restore path (INSERT OR IGNORE) handles multi-row destinations
correctly because PK includes src_root — accidental duplicates
collapse cleanly.
New regression test test_edges_fan_in_batches_huge_destinations builds
an edges table with FAN_IN_BATCH+137 rows pointing at one dst_uri,
verifies the dump produces the expected number of split rows and the
restore reconstructs all N edges with no loss or duplication.
25 cold-object + evict tests pass.
v1 packs (chunks-only) were under-engineered: a new peer landing on
v1 packs would have chunk bodies indexed by leaf_hash but no documents
table, no audit chain, no merkle interior, no edges — couldn't actually
hydrate. fox: "isn't what I wanted you under engineered..."
v2 packs ship every load-bearing shard table alongside chunk bodies in
the same tar.zst:
manifest.jsonl # chunk catalog (unchanged)
tables/documents.jsonl # array-per-line columnar JSONL
tables/chunks.jsonl # without content column
tables/merkle_nodes.jsonl
tables/edges.jsonl # FAN-IN restructured
tables/audit_events.jsonl
tables/derivations.jsonl
tables/concept_relations.jsonl
tables/concept_token_idf.jsonl
tables/providence_cache.jsonl
tables/citation_aliases.jsonl
tables/term_aliases.jsonl
tables/snapshots.jsonl
tables/document_http_meta.jsonl
blobs/<hash[:2]>/<hash[2:]> # raw UTF-8 chunk bodies
Two compression strategies inside the pack:
1. Array-per-line JSONL ({"_columns": [...]} header line + ["v1","v2",...]
data lines) drops ~30% of uncompressed bytes vs object-per-row JSONL.
zstd recovers most of that on its own, but smaller uncompressed
footprint also speeds up stream-restore.
2. Edges fan-in restructure at pack-build time: 22M rows of
(src_root, edge_type, dst_root, dst_uri, anchor) → ~500k unique
(dst_uri, edge_type, anchor, dst_root) groups with src_roots as an
array. ~5-10x compressed savings on the dominant table. Reverses on
unpack into the per-edge live schema. Live queries unchanged.
NOT shipped (per-peer state): mesh_*, selfmodel_*, capital_ledger,
memory_*, controller_events, fork_score_branches, adapter_loss_reports,
falsifications, schema_meta, meta. NOT shipped (rebuildable): chunks_fts*,
documents_fts* — restored from chunks.content + documents.title on
unpack.
push_pack no longer appends `cold_pack_pushed` to the audit chain.
That event leaked into the next push's audit_events.jsonl dump and
broke the "two writers at the same corpus state produce identical
pack_hash" determinism property. The bucket/disc file IS the receipt;
the snapshot_root pinned inside the pack metadata binds it to a corpus
state. No load-bearing consumer of the audit row.
pull_pack restored to handle both v1 (chunks-only) and v2 (tables +
chunks) packs. For v2 it extracts tables/*.jsonl to a temp dir,
calls restore_shard_metadata (which INSERT OR IGNOREs into the live
schema and expands edges back to per-edge rows), then fills chunk
content for every leaf_hash in blobs/. Idempotent against populated
DBs (INSERT OR IGNORE all the way down). Self-cleaning temp dir.
Sizing measured 2026-05-26: ~2.1 GB per shard pack compressed (chunk
content 1.78 GB + metadata ~0.3 GB), ~8.5 GB total across 4 shards.
~20% more than v1 chunks-only for self-sufficient hydration.
24 cold-object + evict tests pass (+1 new test_push_pack_v2_hydrates_fresh_empty_db
that builds a pack from a populated DB and unpacks into a completely
empty DB to verify all tables restored). Full suite: 2558 passed,
28 skipped, 1 xfailed.
New peers landing on a public bucket need to enumerate pack_hashes
before they can `arborist cold unpack <hash>`. `cold stats` only gives
a count; `cold list` returns the per-pack details:
- pack_hash (taken from the key)
- compressed_bytes (one HEAD round-trip via new object_size method)
- chunk_count (parsed from the small manifest sidecar; skippable
via --no-manifest for huge-bucket fast listing)
`cold stats` also now reports total bucket footprint (sums HEAD sizes).
Backend ABC gains `object_size(key) -> int | None` so both subcommands
get sizes without paying egress for the body. S3 impl uses HEAD;
MemoryBackend reads from the dict.
Verified live against DO Spaces NYC3: list-empty → push tiny pack →
list-with-manifest (pack_hash, size=5311 B, chunk_count=5) → list
--no-manifest → cold stats → cleanup.
23 passed in tests/test_cold_object.py + tests/test_evict.py.
Three improvements after the first DO Spaces smoke + bench:
1. ORDER BY c.leaf_hash on the chunk-selection SQL. Two writers running
cold pack against the same DB at the same snapshot now produce the
same pack_hashes — chunk-to-pack assignment is a function of (chunk
set, cap) and nothing else. Prerequisite for parallel per-shard pack
workers and for two replicas to converge on byte-identical bucket
state. Costs ~25% on build wall (real-bench 31s → 40s on 100k chunks)
due to sort over the leaf_hash index + documents JOIN; worth it.
New test pins the determinism property.
2. boto3 multipart upload via TransferConfig (8 MB threshold + 8 MB
parts + 10-way concurrency) on every put. Required anyway for packs
> 5 GB (DO Spaces single-PUT limit). Measured 5.5 MB/s → 9.0 MB/s
on 121 MB pack to DO Spaces NYC3 (1.6x; ceiling is closer to network
than to boto3 serialization).
3. Stream the SQL cursor in push_pack instead of fetchall(). At 14M
chunks × ~700 bytes/row the prior fetchall materialized ~10 GB of
Python heap before stream_packs ever ran. Cursor iteration bounds
memory by the in-progress pack (~few hundred MB at the 4.4 GB cap).
Full corpus extrapolation revises ~125 min (single-PUT) → ~104 min
(multipart, sequential per-shard). Real wins live in parallel per-shard
pack workers — deferred; the determinism work landed here is the
prerequisite.
22 passed in tests/test_cold_object.py + tests/test_evict.py.
Ship arborist corpus state to new peers and DVD-R archival via
point-in-time tar.zst packs. One artifact serves both channels —
bucket+CDN delivery and physical-media archival.
Bucket holds packs only. Pack key = hash_leaf(manifest_bytes), so same
chunk set on two writers produces the same pack_hash and upload is
idempotent. Each pack pins the corpus snapshot_root it covers in audit
+ result body — packs are delayed snapshots, not live mirrors;
falsifications between repacks produce new pack_hashes.
stream_packs runs streaming zstd over tarfile, peeking compressed-buffer
size after each chunk via FLUSH_BLOCK (preserves dictionary). Default
cap 4_400_000_000 — 4.4 GB DVD-R safe-fit, ~6.5% buffer below the
4.7 GB marketing capacity to absorb ISO9660 overhead, growisofs
lead-in/lead-out, media variance, and drive-edge refusal. Each disc
fills to ~4.4 GB recorded data, not the ~1.5 GB an uncompressed cap
produced.
One backend class (S3CompatibleBackend via boto3 + endpoint_url) covers
AWS S3, DO Spaces, R2, B2, GCS S3-interop, MinIO. Optional dep
[object-store] = boto3>=1.34; dev extras pull moto for the wire test.
Voyeur: credentials via AWS_ACCESS_KEY_ID/_SECRET_ACCESS_KEY env or
~/.aws/credentials, never printed; only endpoint URL + bucket name
surface in logs.
CLI: arborist cold {pack,unpack,stats}. Makefile: cold-pack,
cold-pack-dvd (local-dir output for growisofs), cold-unpack, cold-stats.
Sizing for current shards (14.1M chunks, ~17 GB compressed): ~4 packs
at the default cap, ~\$0.34/mo DO Spaces storage, ~\$0.0001/fresh-peer
hydrate.
Always-on raw-UTF-8 leaf store (per ticket "Hard invariants") deferred
— packs-only for now, backfill later.
2557 passed, 28 skipped, 1 xfailed.
A stable façade so another Python app can use arborist as a
content-addressed / Merkle / audit-chained store without the CLI or a
wire protocol. Import from arborist.embed, not internal modules, so
refactors don't break embedders.
Surface: open_store(path), ingest_documents(conn, docs), search(conn, q),
plus re-exported Document/Edge/Source/Hit/IngestStats. Core only
(python+sqlite3) — no extras. _IterableSource adapts a plain doc iterable
into the Source contract.
This is the seam for using arborist as neopig's optional provenance
backend: neopig produces Documents from crawled pages, arborist gives
content-dedup (document_root) + FTS5 + an append-only audit chain
alongside neopig's existing md5/FileVault storage. Docs in
docs/embedding.md. 6 tests pin open/ingest/dedup/idempotence/edges/search.
Full --fast crawl of russell.ballestrini.net (242 URIs): 26s -> ~5s.
Three changes, biggest first:
1. fast_mode now actually ignores crawl-delay (the ~5x). The delay was
only zeroed on the robots-200 path; a site with no robots.txt (404)
or a robots fetch error fell back to default_crawl_delay (2s). Under
--fast that made every concurrent fetch wave sleep ~2s — ~10 waves
x 2s dominated the wall time. _enforce_crawl_delay now short-circuits
when fast_mode, matching the documented "ignore crawl-delay"
contract regardless of robots status. Disallow is still honored
(separate path).
2. One shared ClientSession for the fetcher's lifetime (keepalive TCP
connector sized to page-worker width) instead of a fresh session per
fetch — ~3x on a 24-page wave. Lazily built in-loop via _get_session;
the bridge closes it in a finally (guarded on owning the fetcher).
3. Drop the per-page preflight HEAD. aiohttp exposes response headers
before the body is read, so the existing content-type binary guard
skips images/video/audio without downloading them — the HEAD was a
redundant round trip that doubled per-page latency.
Diverges arborist's AsyncWebFetcher from the agents.ai.unturf.com/core
verbatim lift (fox-approved); candidate to upstream. Regression tests
pin fast=no-delay / polite=delay, shared-session lifecycle, and bridge
session teardown (owned vs injected).
Two crawler-discovery changes surfaced while chasing fast-crawl wall
time on russell.ballestrini.net:
1. Feed-skip in BFS discovery: the bridge fetched feed/sitemap URLs
(a multi-MB atom.xml among them) only for ingest_crawled to discard
them. Gate enqueue on the existing _looks_like_feed_url so we never
fetch crawl-infrastructure URLs — less wasted work and one fewer
slow wave straggler.
2. lxml link extraction, DRY'd: the three duplicated BeautifulSoup
html.parser closures (fresh fetch + 2 cache paths) collapse into one
module-level extract_page_links() backed by lxml.html (C parser,
releases the GIL so to_thread actually parallelises) with a BS4
fallback for markup lxml rejects. Parse on a 24-page wave 3.5s->2.5s.
Honest scope: neither moves full-crawl wall time much — measurement
showed the dominant cost is the per-page HEAD+GET double round-trip on
a per-call ClientSession, not parsing. These are correct-and-cleaner
on their own; the wall-time lever (shared session + drop redundant
HEAD) is a separate change. lxml extraction is regression-pinned
against the BS4 fallback for parity.
The bridge BFS fetched pages one-at-a-time, so --fast only dropped the
crawl-delay (sequential, zero-wait). Fast_mode's CPU*3 page-worker
budget never reached the path operators actually run.
Replace the popleft loop with a wave loop: each iteration pulls up to
`fetcher.max_page_workers` URLs off the queue front and fetches them
with asyncio.gather. Width is CPU*3 under fast_mode, 1 otherwise, so
the polite path stays byte-for-byte sequential and the per-page
crawl-delay still serialises same-domain fetches. Wave size is capped
to the remaining max_pages budget; dedup moves from pop-time to
enqueue-time so a URL linked from two parents in one wave is fetched
exactly once.
Measured on russell.ballestrini.net (own host, robots 404): same
12-page work 23.1s polite -> 4.0s fast (5.7x); full 243-page crawl
~25s vs the ~486s polite floor (19x). Disallow still honored; only
the rate limit is lifted.
Tests: peak-in-flight pins (>1 fast, ==1 polite) plus all existing
BFS bound / dedup / depth / max-pages cases on the width=1 path.
When a re-crawl detects a real content delta (a just-ingested root that
supersedes a prior version — content hash changed, not redeploy/ETag
noise the idempotent ingest already no-op'd), the pipeline now surfaces
the page's document chain over time instead of just 'something changed'.
bridge.py: version_chain(conn, uri) walks a URI's documents by ingest_ts
(each content change = new content-addressed doc + supersedes edge);
delta_report() adds the word-level similarity of the latest change;
render_delta_report() prints it. ingest_crawled() detects superseding
roots, emits the lineage report to stderr per changed page, and returns
'deltas' in its summary. Validated on the live russell.ballestrini.net
re-crawl: 223 pages, full redeploy, exactly 1 content change (/about/),
rendered as a 2-version chain (90% similar to prior). 2 tests; suite
2551 passed.
The code judge bailed to JUDGE_ERROR on 40% of in-corpus answers: HYBRID
(partial grounding) with low NLI entail, where the entity-grounding
rescue needs ZERO unsourced specifics. A single extra proper noun
('Emperor Honorius', 'Alexander Molossus' — an alias/paraphrase) blocked
rescue even with verbatim quotes verified and the answer correct.
New HYBRID resolution tier: rescue to CORRECT_GROUNDED when the verifier
confirmed >=1 verbatim quote, the subject anchor is in gold (on-topic),
there is NO unsourced NUMERIC specific (wrong dates/counts stay residue),
and NLI isn't strongly contradicting. Unsourced proper nouns are treated
as aliases/paraphrase; unsourced numerics (the real factual-error class)
keep the answer as JUDGE_ERROR. Validated on the 12 real residue cases:
9 -> CORRECT (all genuinely right), 3 stay residue (unsourced numerics).
JUDGE_ERROR 40% -> ~10%. self-test 4/4; 2 new tier tests; suite 2549.
Root cause of 'arborist abstains on everything with qwen' (fox 2026-05-21):
Qwen3 thinking-on default burns the entire token budget on hidden <think>
reasoning over a 20K RAG context and returns EMPTY message.content
(measured: 768/768 completion tokens, content '') -> every arborist answer
UNGROUNDED. Bench harnesses passed enable_thinking=False via the MODELS
dict, but the CLI + control_ab did not, so the quality bench was measuring
a thinking-budget-exhaustion artifact, not abstention.
OpenAICompatibleClient now defaults Qwen3 to enable_thinking=False unless a
caller set it explicitly (reasoning-variant path passes True, preserved).
Verified: same France query goes empty/UNGROUNDED -> STRICT 'Nicolas
Sarkozy' with the flag. Fixes every caller (CLI, control_ab). 5 tests;
full suite 2547 passed. Today's qwen QUALITY numbers are void and need
re-running; energy numbers stand (real inference happened regardless).
fox 2026-05-21: account wattage for input and output separately. Prefill
(process all prompt tokens, parallel/compute-bound) and decode (generate
output, autoregressive/bandwidth-bound) are different GPU ops with
different J/token — a single per-token number can't represent both.
Slope calibration (no sub-request power alignment): sweep prompt length
at tiny max_tokens -> prefill J/input-tok (fixed overhead cancels in the
slope); fix a tiny prompt and sweep forced output length (ignore_eos) ->
decode J/output-tok. Prefill kept COLD (unique filler so cached_tokens=0).
Reuses watt_bench probes. Bad points (context overflow) skip, not abort.
Measured qwen-nothink/4090 @$0.33/kWh: prefill 0.175 J/tok
($0.016/M-input-tok), decode 6.16 J/tok ($0.564/M-output-tok) — decode
35x dearer per token. Predicts measured substrate J/q within ~5%. 14
tests (+ slope). Validated live.
fox 2026-05-21: (1) use REAL API token usage, not len//4; (2) the
substrate prefills a large retrieved CONTEXT as INPUT while solo feeds
~nothing, so per-completion-token over-charges the substrate — and per-
TOTAL-token UNDER-charges it (its mix is ~98% cheap prefill tokens).
Measured n=30 qwen-nothink/4090: substrate prefills ~6.6k input tok/query
(claim_lattice) vs solo ~52 — ~127x. Neither single per-token denominator
is honest; prefill (parallel, cheap/tok) and decode (autoregressive,
dear/tok) must be costed separately.
- OpenAICompatibleClient stashes data['usage'] as .last_usage (non-
invasive; return type unchanged).
- watt_bench captures real prompt_tokens + completion_tokens per call
(both arms), aggregates per cell, and energy_cogs reports gross +
marginal per BOTH 1k-total-tok and 1k-completion-tok plus the context
size. Prints the prompt/completion split.
- 12 tests incl. the prompt-context artifact (per-total cheap, per-
completion dear). Full suite 2540 passed.
The clean per-input-tok / per-output-tok split rides bench/watt_calibrate
(slope calibration; separate commit once validated live).
fox 2026-05-21: 'gen 200W' was a bug — joules/window blends the ~400W
generation bursts with the sub-100W gaps (retrieval/verify/network) into
a power state the card never sits at. A card occupies DISTINCT states
(idle / middle-idle = resident-between-requests / generation), differing
per card×model×server.
watt_probe.classify_power_bands(): largest-gap split of the window
samples into a low band (serving floor) and high band (generation draw)
+ duty cycle. Data-derived, never hardcoded — tested at two scales. The
worker emits the decomposition + raw samples; RemoteProbe/LocalProbe
expose band_stats() uniformly.
energy_cogs: marginal now taken against the measured SERVING FLOOR (the
standing cost of being ready), not deep idle; the blend is kept but
labelled window_mean_w. Reports idle/serving-floor/gen-draw/duty.
Cache-miss certainty (fox's question): the arborist arm runs
burn_existing=True (force-deletes any live providence row before
inference) and asserts cache_hits==0 with a loud warning + real_inference
flag — so we time real generation, never a SQLite lookup. Solo has no
cache path. 11 tests (energy math + band split). Validated live on the
isolated 4090: solo gen 308W/70%-duty vs substrate 396W/8.6%-duty —
substrate marginal/tok is LOWER, gross/tok higher (it holds the card
longer for retrieval).
fox 2026-05-21: compute cost-of-goods-sold by kWh vs tokens, with the
three power states (idle / warm-idle / generation) MEASURED per
card×model×server — never hardcoded (his 40/127/380 W were illustrative
of one 3090). The only operator input is --price-per-kwh (default 0.33
USD/kWh, a configurable site rate).
energy_cogs() (pure, unit-tested) decomposes measured generation energy
against the measured warm-idle baseline:
* gross — all measured joules over the window (all-in, includes the
warm-idle cost of keeping the model hot, amortized).
* marginal — joules ABOVE warm-idle: what one more request's burst
actually costs (clamped >=0).
kWh = J/3.6e6; $/1k-tok is the unit that compares to API pricing. Both
surface per cell + a COGS print line.
watt_bench's arborist arm now loads the frozen bench.stock_v1 policy
(--answer-mode, drift-guarded on non-reasoning) so cost is measured for
the SAME substrate the campaign grades. Cells record
window_start/end_unix so a post-hoc load_monitor queue-depth cross-ref
can flag organic-traffic contamination on the non-isolated single-slot
endpoints. 6 COGS tests; full suite 2534 passed.
Two surgical fixes unblock 'arborist with synthesis LLM = Qwen-on-
llama.cpp' as a viable arm in the control sweep. Pre-existing
docstring said 'Arborist×Qwen needs proof-path guided_json+extra_body
surgery — coupled follow-up'; this is that follow-up.
Fix 1 — multi-engine structured-output extras
The runner / query JSON-mode paths previously sent only vLLM's
'guided_json' key for the claim_lattice schema. llama.cpp silently
drops it, leaving Qwen un-enforced (the parse-tolerant fallback did
all the work). Helper
claim_lattice_structured_output_extras() in arborist/qa/verify.py
now returns a dict carrying the schema under all three engine
conventions:
- guided_json (vLLM grammar-constrained sampling)
- json_schema (llama.cpp native shorthand)
- response_format (OpenAI-spec, honoured by llama.cpp and newer vLLM)
Each engine recognises its own key and silently drops the others.
Used at both inference call sites (runner.py:740, query.py:3324).
Hermes/vLLM path is unchanged — it picks up 'guided_json' and
ignores the other two.
Fix 2 — query() accepts user-supplied extra_body, merges with defaults
query() grew a keyword-only extra_body parameter (default None).
Per-model knobs (Qwen's {'chat_template_kwargs': {'enable_thinking':
False}} toggle, future template knobs) can flow from the caller to
the synthesis chat-completion call. Schema-enforcement extras are
added inside query() and merge under user keys — common case is
disjoint namespaces, but if a caller wants to override 'guided_json'
they can.
bench/control_sweep.py now passes MODELS[arborist_ref]['extra']
through to query() in the arborist branch, so --arborist-ref
qwen-nothink runs with reasoning disabled and --arborist-ref
qwen-think runs with reasoning enabled. Phase 1's arborist arm with
--arborist-ref=hermes is unaffected (MODELS['hermes']['extra'] is
None, merges to no-op).
Tests
+ 3 new in tests/test_verify_json.py covering helper default shape,
alternate-schema reuse, and query()'s new extra_body parameter
220 affected tests still green (verify / claim_lattice / judge /
runner suite)
pytest test_verify_json: 27/27
Next: small smoke run --arborist-ref qwen-nothink against 4-8 items
to confirm end-to-end before any full sweep. Phase 2 (qwen-think solo)
still running in background, unaffected — it doesn't touch the
arborist arm.
The Arborist arm runs answer_mode='claim_lattice' (per control_sweep.py
:179, control_ab.py:155) so its answers arrive as the JSON envelope
{"claims":[{"text":"...","evidence_ids":["E1"]},...]}.
_descaffold strips the [E1] evidence-pointer markup but the JSON
braces + key syntax remain. The verifier's strategy-2 (span) and
strategy-3 (proper-noun) extractors see brace noise instead of the
inner claim prose — every Arborist record degraded to UNGROUNDED.
The 2026-05-19T17-01-17Z sweep, re-graded with the freshly calibrated
judge (5a17f61), surfaced this: Arborist arm reported 0 CG across all
three variants in the live phase 1 output (the live run was pre-
calibration), and 29/120 CG (24%) under the calibrated rescore — clear
improvement just from theta_contra=0.85, but the JSON envelope was
still hobbling the verifier paths.
Fix: _unwrap_claim_lattice_json runs BEFORE all downstream rules.
Detection is conservative (three independent signals: starts-with-
brace AND "claims" key AND "text" key) so plain-prose answers
pass through unchanged. Multi-claim envelopes concatenate as discrete
sentences (extract_claim_spans treats each as its own span).
Malformed JSON falls back to the original answer — no silent
rewriting on broken input.
Smoke result on the Iceland Arborist case
ans: {"claims":[{"text":"The current president of Iceland is
Ólafur Ragnar Grímsson.","evidence_ids":["E1"]}]}
gold: {{Infobox Political post |post = President |body = Iceland
|incumbent = [[Ólafur Ragnar Grímsson]] ...}}
before: UNGROUNDED → FABRICATED (then WRONG after calibration)
after: short_entity_grounded → CORRECT_GROUNDED
pytest: 27/27 (added 7 unwrap-coverage tests covering single-claim
envelopes, multi-claim concatenation, plain-prose passthrough,
malformed-JSON tolerance, unrelated-JSON passthrough, and the
end-to-end Arborist-envelope CG flow). Self-test 4/4 unchanged.
Re-rescores of 17:01 sweep + phase 1 sweep run after this commit
to measure final Arborist scorecard improvement.
bench/judge_code.py — drop-in alternative to bench/judge.py with the
same Verdict shape & closed verdict vocabulary (CG/W/F/A/JE) but zero
quota cost: composes verifier + NLI + abstention + specificity into a
fixed-order pipeline. fox 2026-05-19: 'data first, judging later' —
this is the data-collection arm; LLM-based judging (Opus batched
needle-haystack, or Grok credit-card) is a separate downstream
concern that operates on the residue this judge cannot classify
deterministically.
Pipeline (first hit decides):
1. empty / no-gold guards
2. explicit abstention phrases (lexical regex)
3. NLI contradiction (arborist.qa.nli.shadow_check) — strongest
signal: gold contradicts the claim → WRONG
4. lexical verifier (arborist.qa.verify.verify_quotes) →
STRICT → CORRECT_GROUNDED
HYBRID + NLI entail >= 0.55 → CORRECT_GROUNDED
UNGROUNDED + specifics-not-in-gold → FABRICATED
UNGROUNDED + no specifics → ABSTAINED
HYBRID without NLI corroboration → JUDGE_ERROR (residue
for an LLM judge)
Threshold note: _CODE_JUDGE_THETA_ENTAIL_CORROBORATE=0.55 is distinct
from the NLI manifest's entailment_block_veto=0.9. The manifest's
threshold is calibrated for OVERRIDING a STRICT lexical signal with
negative evidence — high bar. The corroboration use here is the
opposite direction: additive positive evidence on an already-positive
anchor — moderate bar appropriate. Self-test case 1 measures NLI
entail=0.769 (clearly entailed, clear margin above 0.55).
Specificity for FABRICATED layers three scanners:
- verifier's multi-word proper-noun extractor (Higgs Boson, ...)
- local single-word capitalised-token scanner (Napoleon, Mars, ...)
deliberately separate because the verifier's gate is conservative
by design (multi-word only)
- numerics (years, dates, large counts, money)
Self-test: same 4 fixtures as bench/judge.py:self_test() so the two
instruments can be cross-checked when fox re-fires the Opus judge on
the residue later. Result: 4/4 INSTRUMENT TRUSTWORTHY.
tests/test_judge_code.py — pulls the contract into make test
(18 cases): module identifiers pinned, dataclass shape parity,
empty / no-gold guards, parametrised abstention phrases, specificity
layer behaviour, the canonical 4-case self-test, batch helper, and
graceful NLI-unavailable degradation. 18/18 pass.
Pre-existing known limitation, documented in the docstring: terse
correct answers ('In 1945.' against gold containing '1945') route to
ABSTAINED because the verifier's span extractor needs prose shape;
NLI sees no clause-level overlap at very short claims. The conservative
ABSTAINED label is correct deferral; tuning this is a calibration
question for real bench data, not the instrument's contract.
No callers touched yet — control_sweep.py & control_ab.py still
import the disabled Opus judge. Wiring this in is a separate ticket
move per fox's data-first sequencing.
The review's central, correct finding: _title_query_tokens is
hot-path (every query AND title) and its fold set (hyphen #000007 +
numeral/accent/honorific/brit) changes which documents retrieve, but
that normalization's version was bound nowhere → a replay cannot
identify which token-normalization produced an old providence
record's sources. Same provenance class as the #000001 keyword gap.
Severity is honest: replay-provenance gap, NOT cache corruption —
different folds → different sources → different context_root →
different cache_key, so no false answer-cache aliasing or false
STRICT. Verifier/proof path unchanged.
Fix follows the repo's OWN #000001 §5/§6 decision (bind retrieval
transforms into the run-DAG RetrievalPlan/retrieval_plan_hash, NOT
governance_policy_hash). The review suggested governance "Option A";
repo precedent is run-DAG binding (same status as retrieval_keywords
and #000056 MT-engine identity) — the discrepancy is surfaced for
fox as an explicit call, not silently overridden.
- RetrievalPlan.title_token_policy (empty default → omitted from
canonical() → every prior retrieval_plan_hash byte-identical; the
§5 zero-churn discipline, same as the #000056 MT fields).
- _TITLE_TOKEN_POLICY single source of truth in query.py, bound at
the plan construction site; bump on any fold change.
- Plus the review's edge cases: Roman-substring-in-word not folded,
out-of-range not folded, Unicode Roman explicitly unsupported,
hyphen∘numeral composition. Full suite 2498, 0 regressions.
Declined (not engineering, per don't-proliferate): the review's
SelfModel/MemoryRoot/5S-5T-5F/capital-ledger ceremony — the ticket
design log is the single source of truth; scope recorded there.
Two more MEASURED fold-search wins on mined ground-truth fixtures
(deterministic recall, no LLM), both lifting at @1/@3/@8 (not
coarse-k artifacts):
honorific (Mt/St/Dr <-> Mount/Saint/Doctor): recall@1 45% -> 75%
(+30pp), @8 62% -> 85%, misses 15 -> 6
brit (British <-> American spelling): recall@1 50% -> 70%
(+20pp), @8 70% -> 85%, misses 12 -> 6
Both _*_fold_variants are additive+symmetric, strict closed sets
(no English-word collision), no-op outside their class (verified
independent: brit no-ops on honorific titles & vice versa), unioned
into _title_query_tokens beside hyphen(#000007)/numeral/accent.
Full suite 2488 passed, 0 regressions (hot-path); real-path tests
(FakeSource->ingest->query()->real _Hit).
Fold-search FINAL across the survey backlog, ranked by MEASURED @1
headroom (not prevalence — the instrument's job):
SHIPPED: numeral (a3ac653) accent (b573c59) honorific brit (here)
NO-BUILD: hyphen — existing #000007 already delivers 90%@1
(the measure-the-unmeasured-thing check pays off)
NO-BUILD: amp — 82%@1 with no fold (prevalence-overranked;
instrument killed it cheaply, like digit-ordinal pre-build)
Net: 4 deterministic retrieval wins + a reusable mined-recall
instrument + the discipline codified in CLAUDE.md, from a goal that
4 prior hypotheses died on because the bench couldn't measure them.
Second MEASURED fold-search win, and the instrument correcting my own
premature call. accent-fold ON vs OFF on the mined accent fixture:
recall@1 55% -> 85% (+30pp), rank-1 22/40 -> 34/40. recall@8 was
flat (95->98) — a too-lenient k nearly got a real lever wrongly
reverted; @1/@3 is the resolution that drives primary-source
selection. _accent_fold_variants: ASCII-fold then re-tokenise so a
diacritic title ("Béla Bartók", which _TITLE_TOKEN_RE otherwise
fragments to junk) matches the ASCII form a user types. Additive+
symmetric, no-op on pure-ASCII (zero effect on non-accent
queries/titles), mirrors _hyphen_fold_variants (#000007).
Also fixes a defect I shipped in a3ac653: an orphaned duplicate
body left as dead code after `return base` in _title_query_tokens
(unreachable — numeral-fold behaviour/measurement were valid — but
cruft; removed).
Fold-search factory, fanned out across the full survey backlog
(deterministic recall, no LLM, parallel — serial-by-caution was
halting in disguise):
- recall_at_k.py: returns rank -> recall@1/@3/@k from one retrieval
(verified offline). A coarse k hides rank-only lifts.
- mine_questions.py: numeral/accent/hyphen/honorific/amp/brit
ground-truth classes; fixtures committed.
- Measured @1 headroom verdicts: accent SHIP (this commit);
honorific 45% / brit 50% = real headroom (build next); hyphen
90% = existing #000007 already delivers, NOTHING to build (the
measure-the-unmeasured-thing check pays off); amp 82% = no fold
needed (prevalence-overranked, instrument kills it cheaply).
CLAUDE.md bench-maxing: two measured lessons codified — report
recall@1/@3/@k (a lenient k hides rank lifts; prevalence != miss-
rate), and fan out independent measurements (serial-by-caution is
halting). Full suite 2488 passed, 0 regressions (accent-fold is
hot-path in _title_query_tokens); real-path test (FakeSource->
ingest->query()->real _Hit).
The first MEASURED, above-noise retrieval win this thread. The 75-q
n=3 audit_mode bench couldn't resolve any single lever (every failure
class <=3-5 q, sub the 5pp floor — four hypotheses died there). Fix
the instrument, not just the lever:
- bench/mine_questions.py + bench/recall_at_k.py: mine questions from
corpus titles (ground-truth target known by construction), grade by
deterministic retrieval recall@k via `query --dry-run` — no LLM, no
verifier, no n=3 noise, scalable to the 22K-deep numeral pool. The
curated qa_questions.txt stays the separate verifier-honesty/trap
gate; mined fixtures measure the answerable long tail per class.
- _numeral_fold_variants in query.py: ordinal-word ("Alexander the
second") <-> multi-char Roman ("Alexander II"), additive+symmetric,
unioned into _title_query_tokens exactly like _hyphen_fold_variants
(#000007). Strict 2..40 Roman set → no English-word collision;
single-char Romans (I/V/X) intentionally out of scope (universal
len>1 token filter — stated before building, ~4 of 10 residual
misses).
Measured on the mined numeral fixture: recall@8 22/40 (55%) -> 30/40
(75%), +20pp; 20 hits now rank-1. Discipline applied end to end:
measured-first, mirrored precedent, full-suite regression run (2482
passed, 0 regressions — numeral-fold is hot-path in
_title_query_tokens), real-path test (FakeSource->ingest->query()->
real _Hit, not a hand-built object), measured-after on a noise-free
instrument. The ~6 multi-char residual misses are a different
downstream cause the instrument now exposes for future iteration.
Three workstreams, full suite 2482 passed, experimental paths default-OFF.
#000055 — Windows quickstart without make
tasks.py (pure-stdlib runner) + make.bat shim + .gitattributes;
README Windows section rewritten. Quickstart needs only Python
3.10+ (no make/bzip2/curl/bash). Mirrors the Makefile quickstart
subset; drift-pinned by tests/test_tasks_runner.py.
#000001 §7 Phase 0 — deterministic cross-language guard
arborist/qa/crosslang.py: non-English signal (¿/¡/non-ASCII) + an
es function-word stoppack. Fail-closed to UNGROUNDED before
retrieval/LLM (mirrors the quantifier reject-DAG) when no content
token survives, else strips es stopwords from the retrieval query
only. English path byte-identical by construction. Default OFF
(crosslang_guard_enabled). Measured: the anarcocapitalismo field
case 10.4s -> 1.6s.
#000056 — Operation Sandwich (cross-language grounding)
arborist/qa/mt/: opus-mt es/fr/ru<->en, lazy per-pair memoised
singleton (fixes the 88%-engine-error concurrency defect),
manifest-pinned, [mt] extra; entity_mask wrapper. Sandwich =
translate query in (retrieval + LLM prompt) -> English answer ->
UNTOUCHED verifier grounds English-vs-English -> translate the
verified answer out as display-only (banner-labelled, zero
grounding). question_hash + verifier_policy_hash invariant; MT
engine identity binds into RetrievalPlan, not governance. CLI
--crosslang-translate / make XLANG_MT=1. Default OFF; entity_mask
default OFF (measured net-negative at bench scale). Fan-out bench
(bench/*.py): Spanish ~0% -> 71% grounded vs the real no-support
baseline; the round-trip predictor was tried and refuted; the
entity-mask lever failed at scale (corpus-title anchoring untried).
CLAUDE.md: cross-language bright-line convention + module map.
Pre-existing modified diagram files are intentionally excluded.
Five rule tightenings, each targeting a specific bench-qa STRICT
false-positive shape (the xfail regressions from the previous commit):
1. claim-lattice bracket-artifact skip — sentences matching
`\[E\d+\s*\|` (pointer markup) or `..."\]` (truncation tail)
are no longer parsed as natural-language assertions; `Such a
thesis was..."]` no longer fires vacuous.
2. Circular-rule differentia cap — circular now requires the
predicate to be (a) entirely vacuous OR (b) leads with a subject
token AND has ≤ 2 non-subject non-filler differentia tokens.
"Michael Jordan's Restaurant was a restaurant in Chicago,
Illinois, named after the basketball player Michael Jordan"
has 6 differentia → no longer fires. Pre-existing positive test
"The entity is the entity referring to the State of Israel"
has 2 differentia (state, israel) → still fires (under threshold).
3. phrase_component_reuse translation-chain exception — when the
predicate ALSO contains a quoted phrase (translation /
definition / etymology context), token reuse with the subject's
quoted phrase is legitimate, not circular. "The name 'Rosebud
River' is a translation … 'the river of the roses'" no longer
fires.
4. Vacuous-rule short-acronym escape — `_coherence_predicate_has_short_acronym_content`
recognizes title-cased element-symbols (Au, Fe, Pb…) and all-caps
2-5-char acronyms (DNA, FBI, USB, NASA…) as content even though
they're below the ≥3-char content-token filter. "The chemical
symbol for gold is Au." no longer fires; "Iron has the chemical
symbol Fe." also clean; tautology "DNA stands for DNA." still
correctly flagged circular.
5. (the 'term <X>' idiom xfail stays xfail — borderline, no clean
lexical fix.)
5 xfail → passing (the regression coverage is now executable proof
of fix); 1 xfail remains. Pooled bench-qa STRICT FP rate test ceiling
tightened from 7% to 2%. All 7 pre-existing positive coherence tests
still fire correctly. 110 total tests pass; 1 xfailed; no regressions
on doc-counts / nli / relevance.
§3.1 diagnose_coherence (now 19 tests, +10 from the parallel session's 9):
- 3 more positive shapes (multi-sentence vacuous, named-entity circular,
grammar-term phrase_component_reuse).
- 5 xfail regression tests for SHAPES THAT FALSE-FIRE on real bench-qa
STRICT data (44/808 = 5.4% FP rate measured on the pooled n=1+3+5
STRICT answers). Each xfail names the exact shape + why it should
ideally be 'ok' + which rule needs tightening:
* 'The chemical symbol for gold is Au.' → vacuous (short predicate)
* 'Michael Jordan's Restaurant was a restaurant ... named after
Michael Jordan.' → circular (named-after re-use)
* 'The Western X was the western half of the X' → circular
* 'The name <Phrase> is a translation ... of the <derivative>' →
phrase_component_reuse (translation/etymology)
* claim-lattice [E1 | … …"] tails → vacuous (truncated bracket
fragment)
* 'The term <X>' → phrase_component_reuse (idiomatic English)
- 1 load-bearing real-traffic test: FP rate on 808-cell pooled STRICT
must stay ≤ 7% (current 5.4%) — fires loud if a future change
regresses it. Skips on fresh-checkout (bench/qa_results/ gitignored).
§3.2 ShadowRelevance (now 20 tests, +7 from the round-1 scaffold):
- Manifest tests for round-2 primary (bge-reranker-large), the size
spectrum coverage (50-560MB), the candidate-bench findings block
(biggest-within-family / not-across-families / deeper-not-better /
capacity-floor).
- Pair-kind distinction (question_answer vs claim_source recorded
separately for downstream telemetry / governance hashing).
- Batch-order preservation (_score_batch must return scores in input
order — load-bearing for downstream zip-back).
- Empty-input handling (Q empty, D empty, whitespace-only).
- Zionist-entity discriminator sanity (on-topic > off-topic logit).
- demote_below_score-stays-null invariant (the §7 #18→#27 discipline:
no hardcoded threshold; must come from a real-traffic shadow sweep).
Total: 101 passed + 6 xfailed (5 §3.1 regressions documented + 1 from
parallel session). The 5 xfails are the bench-maxing receipts — they
document EXACTLY which shapes §3.1 false-fires on, with the rule that
needs tightening named in each reason.
arborist/qa/relevance/ — manifest pins cross-encoder/ms-marco-MiniLM-L-6-v2
(~80MB, Apache) as primary; alternates: L-12, BAAI/bge-reranker-base,
ms-marco-electra-base. demote_below_score=null on purpose — the
#000049 §7 #18→#27 discipline (proved 6× that clean-eval thresholds
don't transfer to bench-qa data) requires the threshold to be set by a
shadow sweep against pooled real STRICT, not by a literature number.
ShadowRelevance class mirrors ShadowNLI (lazy [nli]-extra import, cuda
auto-detect via ARBORIST_RELEVANCE_DEVICE or ARBORIST_NLI_DEVICE,
batched _score_batch, graceful degrade-to-available=False). Two surface
methods: check_question_answer (deflection / Q-A drift) and
check_claim_source (topic-collision mis-cite). 13 tests.
Sanity on the motivating field case (Zionist entity): ON-topic +9.96
vs OFF-topic -9.04 → 18-pt margin. Mona Lisa Q→A deflection: on +10.45
vs deflect +3.56 → ~7-pt margin. The model CLEANLY discriminates the
failure modes #000052 §1 named. It does NOT catch the
recombination-where-the-different-entity-clause-also-mentions-the-target
case (Kilimanjaro/Mount Kenya) — and that's the right architectural
split: aboutness (#000052 §3.2) and entailment (#000049 NLI) are
orthogonal axes; the Kilimanjaro recombination case needs the semantic
candidate selector (#000050/#000051 vec hybrid).
Remaining: build candidate-bench eval (~20-30 deflection + mis-cite
fixtures), shadow-sweep θ over pooled bench-qa STRICT (expect another
walk-back per the #000049 lesson), recall-side realism check, then
fox+dav1d sign-off. Still SHADOW; production verifier unchanged.
`arborist/concepts/extract.py:acronym_parens_synonym` — new
corpus-agnostic extractor. Scans each doc's lead chunk (first 4000
chars) for `<Multi-Word Phrase> (ACRO)` where the all-caps acronym's
letters strictly match the content-word initials of the phrase, in
order, after function-word filtering. Emits bidirectional synonym
edges between the lowercased acronym and each ≥3-char content token
of the phrase, evidence_kind="acronym_parens", anchored to that doc's
document_root. Idempotent like link_reciprocity_synonym.
Why this complements link_reciprocity: Wikipedia represents
abbreviation→expansion as a one-way *redirect* (CPU →
Central processing unit), which the ingest does not record as an
edge — so the existing reciprocal-link extractor never learned the
relation. The relation IS in body text by near-universal convention
("Central processing unit (CPU) is..."), which this extractor reads.
Corpus-agnostic: HTML, blogs, textbooks benefit equally.
Conservative: strict 1:1 acronym-to-atom match (rejects HTTP-shape,
where letters land mid-word), function words filtered, repeated
definitions deduped per doc, ≥3-char target floor. 8 new tests
covering CPU bidirectional emit, RAM idempotency, FBI function-word
filter, HTTP length-mismatch reject, XYZ initial-mismatch reject,
ROM hyphenated-word handling, per-doc dedupe, registry presence.
Retrieval-side only — synonym edges reshape FTS5 candidate selection
via synonym_expand at query time, never enter audit_mode / cache_key
/ audit_event_hash. No governance hash bump, no cache invalidation.
Closes#000050 §2a's CPU/GPU abbreviation rows *upstream* of vec;
the Orwell-shape conceptual-allusion row remains the genuine #000050
justification. Operational follow-up (not code): run on each shard
via `arborist concepts derive --extractor acronym_parens` (CLI
surface itself is aspirational in docstrings; extractors are called
programmatically today). Next ID 000054 -> 000055.
`arborist.qa.evidence._content_tokens` dropped every token under 4
chars, so a short all-caps acronym (CPU, GPU, DNA, FBI, USB…) never
registered as a content token — which defeated Rule 8
(_claim_title_overlap / TITLE_MISMATCH), the subject-tokens-absent
check (Rule 9), the bare-name-claim guard, and spotlight-excerpt token
selection whenever a question/claim's topic IS an acronym. The field
case: `what is a CPU?` cited to the "CPU design" article tripped
TITLE_MISMATCH even though claim and title both contain "CPU".
Fix: keep a token if it's an all-caps 2-3-char alpha run in the source
text; everything else unchanged. The change only ever ADDS tokens, so
TITLE_MISMATCH / SUBJECT_TOKENS_ABSENT / BARE_NAME_CLAIM can only stop
firing, never start — monotone toward fewer spurious demotes; no
STRICT→non-STRICT transition is possible from it.
Versioned: `content_token_rules: "v2-acronym-aware"` added to
runner.DEFAULT_POLICY + query.DEFAULT_QUERY_POLICY +
keys._VERIFIER_POLICY_FIELDS → folds into verifier_policy_hash, prior
cache records orphan on lookup (by design; same discipline as
base_version / hyphen_fold_v1). Does NOT touch the retrieval
abbreviation→expansion gap (CPU→Central processing unit — #000050 vec
hybrid / concepts/ synonym edges; the root cause of the satellite
retrieval). 8 new tests; full suite green (2502); bench-qa-smoke clean.
Next ID 000053 -> 000054.
