Documents the work shipped this session (Phase 1 foundation, 10
commits 9ba6317..20faae0) and the blocker that stops Phase 2: the
multi_route pipeline regresses on 2/5 smoke fixture questions
(Mercury Seven → Sam T. Beddingfield; dinosaurs extinct →
Paul Austin Kelly) because Phase 1 hasn't ported the 5 downstream
rerank stages legacy query() uses to suppress noisy phrase-route
hits — body-density check, body-coverage sqrt rerank, source-role
rerank, title-purity rerank, ordered-token-match rerank.
Path B chosen 2026-05-31: stop here, leave multi_route off by
default (which IS default — policy=None preserves body-only
behavior). Phase 1 foundation stays in tree as future-ready
infrastructure; user surface unchanged.
Path A (port the 5 stages, multi-day effort) reserved for a
future focused session. Until then, legacy query() keeps
producing the same answers it always has, INCLUDING the wrong
"Edwina" pick on dinosaur Q5. The slim-FTS5 cloud path already
fixes that bug for `arborist cloud query` / `arborist
corpus-query` callers (proven 5/5 source parity in d9fb6a9).
Bumps Next ID 000072 → 000073.
The three "92 claim_pack docs" tags were drifting against shard 000.db's
21 rows because the harness only counted one shard, but the doc prose
("#000031 closed at 92") meant the corpus total (21+16+38+17 across
genesis shards 000-003).
Two-line fix path: extend the harness to sum across all ???.db shards
via a `*:` prefix (e.g. `*:documents?source_type=claim_pack`), then
prefix the three drifted tags. Aligns the harness scope with the
semantic scope of the claim instead of forcing the claim to shrink to
one shard.
The `*:` glob:
- Matches `[0-9][0-9][0-9].db` basenames only (operator sidecars
qa.db / snapshots.db / selfmodel-chain.db skipped)
- Skips shards lacking the named table (schema-version tolerance)
- Returns _DB_MISSING when no genesis shard exists (CI / fresh-
checkout skip semantic preserved)
- Returns _TABLE_MISSING when no contributing shard has the table
Documented in ticket-000044 §3.4 + a third example showing the new
syntax. Diagnosis credit to a sub-agent investigation that confirmed
zero eviction/falsification audit events on claim_packs — the data is
intact; the harness was just single-shard.
Sibling to #000070 on the second axis of Joseph (@TrudoJo)'s spec.
Joseph posted the bridge-framework extension 6h after the 6-dim
spatial-ontology: chain A has its own world rules, chain B has its
own world rules, a bridge exists only where a deterministic
agreement layer can be proven. Three kinds: Agreement, Translation,
Embassy.
Phase 1 is doc-only -- the primary contribution is a retroactive-
validation appendix naming three already-shipped arborist primitives
as Joseph-bridge-shape patterns one-for-one:
#000066 graft-receipt mode = Agreement bridge
#000056 Operation Sandwich = Translation bridge (opus-mt hash
IS the translator_id)
#000059 graveyard (proposed) = Embassy bridge (POINTER-LINKED,
banner-labelled, never promotes)
The framework is not a new requirement on arborist. It is a unifying
spec that names what we already do in 3+ places, plus surfaces a
fourth pattern (bridge atlas for multi-instance federation with
potentially-different governance_policy_hash peers) as genuinely new
work for arborist/mesh/.
Bundle:
- docs/tickets/ticket-000071-world-bridge-grammar.md (~440 lines)
- docs/TICKETS.md index row + Next ID 000071 -> 000072
- docs/tickets/ticket-000070-spatial-anchor-pi-w-object.md §6 gains
one-line cross-reference to #000071 as adjacent axis
- Joseph memory extended to cover both frameworks
Phase 2/3/4 deferred to separate tickets. Hard constraints: A1-A3,
no audit_mode token, no SQL change at Phase 1, sovereignty rule.
Status: open, awaiting dav1d Phase 1 review + fox go/no-go.
Updates the spatial-anchor ticket so the pre-review empirical bench
(landed in 55b651f) reads as a self-contained dav1d review packet:
- Status line: notes bench shipped in 55b651f
- §7a Pre-review empirical evidence: expanded with reproducibility
notes (RNG seed 0xa8c90e551fd34427, wall ~2s, stdlib-only), the
five bench purposes mapped to open questions, full headline-results
table, and per-headline "what each one buys dav1d" prose for Q1
through Q9.
- §3 Land sequence: step 1 marked done in 55b651f with explicit
pre-review-bench scope; step 2 annotates that 5 of 10 open
questions now resolve by measurement.
- docs/TICKETS.md index row: leading status cell now opens with the
bench headline + per-section numbers so the index reads as the
dav1d handoff summary, not just "open · awaiting review".
No new code, no new commits to the bundle itself -- 55b651f remains
the bench's shipping commit. Pure documentation refresh.
New ticket for Joseph (@TrudoJo)'s 6-dim procedural spatial-anchor
framework as the first verifier kernel under the #000013 v7-W reserved
namespace. A single committed 32-byte SHA-256 hash deterministically
expands into six 32-byte regions H1..H6 via the HMAC-SHA-512 KDF already
shipped in arborist/substrate/anchor_prg.py (#000035); each region drives
one quantized object dimension under a fixed canonical mapper. Domain
separation from #000035 by dedicated spatial_anchor_seed published in
the v7-W manifest -- preserves #000035's KAT freeze + dav1d 2026-05-11
final review intact.
Bundle:
- docs/tickets/ticket-000070-spatial-anchor-pi-w-object.md (718 lines):
full spec with 8 design-choice subsections, working Python sketch,
12-NOT scope boundaries, 8-row cross-references, five-step deletions,
10 open questions for dav1d.
- bench/spatial_anchor_validation.py: pure-stdlib pre-review evidence
(~2s, RNG-seed-pinned, reproducible). Five benches: avalanche,
cell-distribution uniformity, collision vs birthday-bound, cross-region
independence, domain separation.
- bench/spatial_anchor_validation_results.md: report from first run.
- Makefile: 'make bench-spatial-anchor [SPATIAL_N=N]' target + PHONY.
- docs/TICKETS.md: index row + Next ID 000070 -> 000071.
Bench headlines (N=10000):
- Avalanche mean 767.85 bits (PRF null 768, z=-0.49) -> PASS
- Cell-distribution chi^2 |z|<1 at L=2,3,4 -> PASS
- Birthday-bound ratio obs/exp 0.989/1.038 at populated L -> PASS
- Cross-region Pearson all 15 pairs < 2sigma -> PASS
- Domain separation Arm A 767.91 / Arm B exact 0 collision -> PASS
Five of dav1d's ten open questions (Q1 seed source, Q2 segmentation,
Q3 position mapper, Q8 endianness, Q9 KAT adversarial vectors) now
resolve with measurements rather than appeals to PRF authority.
Q4/Q5/Q6/Q7/Q10 remain non-empirical design decisions.
Status: open, awaiting dav1d review + fox go/no-go. No registry slot
booked, no substrate-paper amendment landed, no kernel module created.
Replaces the batched chunk-pack phase with per-chunk content-addressed
blob uploads to `blobs/<hash[:2]>/<hash[2:]>`. The metadata pack still
ships (small, fast to restore), but consumers no longer have to pull
multi-GB chunk packs to get queryable: `cold unpack --mode just-enough`
+ `ARBORIST_JIT_CHUNKS=1` fetches single chunks on cache miss.
Producer (`_stream_jit_blobs` in evict.py):
- ThreadPoolExecutor with bounded queue (workers*4) keeps memory flat
across millions of chunks
- HEAD-checks object_size for idempotent re-upload
- Mutually exclusive with chunk packs — manifest's `chunk_pack_hashes`
is empty in JIT mode (consumer reads that as "JIT-only")
Consumer (`hydrate_doc_jit` in cold_clone.py + `_maybe_jit_hydrate` in
qa/query.py):
- Detects both content shapes that need JIT: NULL (Tier B raw-clone) and
zeroblob placeholders (just-enough pack restore, per #53). Discriminator
is first-byte = NUL — zstd-framed bodies start with 0x28, plain UTF-8
prose never has leading NUL.
- Same placeholder filter applied to chunk-read sites in qa/query.py so
partial hydrate doesn't surface zero-bytes content into the LLM context.
Test (`TestJitBlobsPackMode` in tests/test_cold_unpack_routed.py):
- End-to-end push → just-enough hydrate → JIT-fetch → content matches
original byte-for-byte through `unpack_chunk`.
Docs (cold-object-store.md):
- Hard-invariant #1 updated: bucket holds packs by default; `blobs/`
and `clones/` are opt-in prefixes for the JIT and Tier-A flows.
- New "Three consumer modes" section: full-pack vs JIT-blobs vs raw-clone
comparison table + operator decision tree.
The FTS pack restore produced a DEAD index — segments present, MATCH=0 —
because the verbatim shadow-table copy used INSERT OR IGNORE, so the
pack's real `_data` rowid-1 "structure" record lost the primary-key
conflict to the empty one `CREATE VIRTUAL TABLE` seeds, leaving a "0
segments" header over orphaned segments. Fix (evict.py): clear the
seeded rows, then copy verbatim (DELETE + INSERT ... SELECT), so each
fts5 shadow table becomes a byte-for-byte copy of the producer's index
and the real structure record survives.
Validated on the 3090: restore one fts pack, NO rebuild -> MATCH
'anarchism'=1189 / 'the'=1.42M (identical to rebuild-from-content);
`_data` id=1 structure record non-empty.
make cold-hydrate: rebuild FTS only when the restored index isn't
already searchable (cold verify-gated). Shipping FTS packs now makes
recovery fast (~24s/shard restore, skip the ~5min rebuild); dropping
them (--no-fts default) keeps the bucket small. Either way cold verify
gates success.
docs/cold-object-store: FTS packs restore correctly now; documented the
restore-vs-rebuild tradeoff, per-consumer guidance, and the fixed bug.
- Hydrating a new peer: M-aware `make cold-hydrate` (serial default,
bulk-tuned restore, parallel FTS rebuild from content, cold verify
self-check) — replaces the stale single-shard `cold unpack` loop.
- New "FTS: rebuild, don't restore" with the measured rebuild-vs-restore
comparison (~5 min rebuild + no extra download vs +4.76 GB for a dead
pack-restored index) and WHY the pack restore is dead: FTS5's _data
rowid-1 "structure" record is left empty because INSERT OR IGNORE
collides with the freshly-created vtable's empty header — a headless
index over orphaned segments (count looks right, MATCH returns 0).
- Invariants: FTS shadow tables are rebuilt on the consumer, not shipped
(cold pack defaults to --no-fts).
- Failure modes: cold verify's zero-filled-content + dead-FTS classes.
- make crawl-ingest writes to one central crawl db (CRAWL_DB, default
~/.arborist/crawl/web.db) instead of per-domain shards in the
peer-shared main dir: keeps locally-crawled content out of peer
sharing by default and a growing domain set under SQLite's 10-attach
cap (Makefile, docs/crawler.md).
- arborist query auto-includes the local crawl db (query() gains
extra_shards; CLI --include-shard / --no-crawl-db, default-on when
web.db exists). Fix latent --db single-file query AttributeError
(cli.py). Persist used / used_pointer_ids + retrieval_purity into
merkle_proof so read-only consumers can see which chunks fed the
answer (qa/query.py).
- arborist.read: read-only seam for dashboards / verifiers; on a
multi-source context root surface the real primary source instead of
the opaque corpus://multi-source sentinel (read.py). Backs the
arborist-viz Merkle Command Center (#000069).
- tests for extra_shards, the CLI crawl-db resolver, and the read seam.
Five sections updated to match the post-2026-05-27 substrate state
(the user-payload-layout work, sibling ticket #000068, and the
2026-05-27 bench evidence are all in tree).
Verdict block (top): ADD: companion missed-answer guard -> DONE:
companion guard shipped as #000068 Phase 1+2+3 (default OFF; Phase
4 default flip NO-GO until wider bench + human spot-check).
Companion missed-answer guard section: renamed from "(proposed
sidecar)" to "(shipped 2026-05-27)". Carries the implementation
location (arborist/qa/inspect.py:diagnose_missed_answer), the full
output schema (diagnostic_version / confidence_class /
triggered_clauses / subject_tokens / missed_answer_candidate_spans
with offset_start/end/basis), the Phase 2 bench headline (2/228
fires, both Ballestrini, 100% precision, 0 FPs across 226 non-
Ballestrini), the Phase 3 demote-flag CLI surface
(--demote-on-missed-answer, default OFF), and corrected hash
discipline: Phase 1 sidecar fields fold into governance_policy_hash
only; the Phase 3 demote flag (answerability_demote_enabled) ALSO
folds into verifier_policy_hash because flipping it changes the
rendered audit_mode (a verifier-output property).
Future hardening list: "companion missed-answer falsification
guard" entry now points at #000068 instead of describing a
deterministic-sidecar to be built.
Roadmap Phase 5: "DESIGN OPEN" -> "DONE 2026-05-27" with all four
sub-phases of #000068 named individually (Phase 1 sidecar commit
2ab11d2, Phase 2 bench + Phase 3 demote commit ec55db5, Phase 4
default flip NO-GO per Dav1d 2026-05-27 §3.4). The #000068 phase
numbering is internal to that ticket; this roadmap names the
external-facing milestones.
Related links: past-tense the missed-answer guard ("when it lands"
-> "shipped 2026-05-27"); added cross-reference to
docs/tickets/ticket-000068-*.md; named both Dav1d review files by
path so a re-read can locate the inputs.
AUTOCOUNT tag (76 fixture-rows in bench/qa_questions.txt) still
matches; no test changes.
Phase 2 — bench instrumentation + measurement run
bench/qa_sweep.py picks up the answerability sidecar projection per row
(answerability_fired, answerability_confidence, answerability_denial_
pattern, answerability_answer_type, answerability_candidate_count) and
aggregates per-mode (answerability_fires + S/M/W confidence breakdown)
into a new column in the markdown summary table.
Measurement run on bench/qa_results/phase2-sidecar-on/2026-05-27T14-
16-22Z (76 questions × n=3 × claim_lattice × Hermes-3-8B × tail layout,
228 runs). Headline:
sidecar fires 2/228 (0.88%)
confidence dist 2 strong / 0 medium / 0 weak
precision 100% (2/2 fires were the Ballestrini fixture)
recall on Ballestrini 2/3 across n=3 (third run model extracted
correctly -> sidecar silent,
correct behavior)
false positives 0/226 non-Ballestrini runs
verifier verdict both fires labeled STRICT by the binary
verifier (the verifier-blind class, exactly
as predicted)
Detection rule's three-clause conjunction (denial + extraction-shape +
candidate proximity near cleaned subject tokens) is operating at the
precision floor. The strong-confidence-only firing pattern is what
calibrates Phase 3's demote threshold.
Phase 3 — opt-in demote flag (default OFF per Dav1d Phase 4 NO-GO)
arborist/qa/keys.py: answerability_demote_enabled added to
_VERIFIER_POLICY_FIELDS so flipping the flag partitions cache via
verifier_policy_hash. Justification: when on, the rendered audit_mode
changes (EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL), which IS a
verifier-output property; verifier hash must move accordingly. The
other answerability_* fields stay governance-only (sidecar
diagnostic, no audit_mode mutation).
arborist/cli.py:_render_audit_label extended with answerability +
demote_enabled kwargs. Logic:
demote_triggers = (
demote_enabled
and answerability["answerability_warning"] is True
and answerability["confidence_class"] in ("strong", "medium")
)
lattice modes:
EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL (rung transition)
POINTER-LINKED / ANCHOR-WARRANTED -> "rung · missed-answer"
(tail tag; rung itself already
signals degradation)
non-lattice modes (quote/span/entity/paraphrase):
audit_mode token unchanged + "· missed-answer" tail tag
weak confidence: NEVER demotes (Phase 2 saw zero weak fires on real
failures; reserved for future expanded detection ladder)
CLI flag --demote-on-missed-answer on both `arborist query` and
`arborist ask`, default OFF. Flows into call_policy[
"answerability_demote_enabled"] and through to result[
"answerability_demote_enabled"] so the renderer reads it without
needing the policy dict.
End-to-end verified live: 4 fresh Hermes-3-8B runs with --demote-on-
missed-answer on `songs by veronica ballestrini`, all 4 rendered
EVIDENCE-MISSED-PARTIAL · via claim_lattice (Hermes hit the failure
mode in all 4, sidecar fired strong, demote logic transformed the
label).
Phase 4 (default flip to demote-on) — NO-GO per Dav1d 2026-05-27 §3.4:
"a false sidecar warning is tolerable; a false audit-label demotion
can damage trust in correct abstentions." Phase 2 precision is 100%
but n=2 fires is too few samples to claim precision floor empirically.
Default flip blocks on wider bench + human spot-check of the warnings.
Tests: 47 total (36 Phase 1 + 11 new Phase 3 covering hash partitioning
discipline + render-label projection across all four rung/confidence
matrices). Full suite 2794 passed (delta +22 from prior 2772).
Bench output (bench/qa_results/phase2-sidecar-on/) intentionally not
committed — bench/qa_results/ is gitignored per existing convention;
the ticket carries the headline numbers + path for re-inspection.
Adds a deterministic read-only sidecar to detect a class of failure the
binary verifier is structurally blind to:
Evidence contains the answer.
Model says the evidence does not contain the answer.
Verifier sees no unsupported positive claim -> marks run clean.
User receives a false negative under EVIDENCE-WARRANTED.
The motivating case: "songs by veronica ballestrini" against the 2010
Wikipedia corpus. Hermes-3-8B under user_payload_layout=tail returned
"the specific songs by her are not mentioned in the provided evidence
blocks" when evidence E2 literally contained "Amazing", "Out There
Somewhere", "Fascinated", "What's Up With That", "Don't Say". Verifier
correctly returned EVIDENCE-WARRANTED 2/2 because the existing layered
verifier (quote / span / entity / paraphrase + Rule 8 title-relevance +
Rule 9 subject-tokens-absent + claim-count ceiling) guards unsupported
*presence*, has no hook for unsupported *absence*.
Layout fixes attention placement on the specific instance (the 5/27
n=3x75q bench confirms bookend/per_chunk recover Ballestrini); layout
alone cannot close the class -- adversarial phrasing or a bigger prompt
resurfaces the failure under any layout. The right substrate move is to
falsify "not mentioned" as a testable claim.
Detection rule (three-clause conjunction, all must fire):
A. Denial pattern in answer (sealed v1 phrase list: "not mentioned",
"not provided", "the evidence does not say", "does not mention",
"no specific", "no evidence", "cannot determine from the provided
evidence", "is not stated", "is not specified"). Casefolded +
whitespace-normalized substring match.
B. Question is extraction/list-shaped. Either a surface cue ("songs
by", "works by", "books by", "who wrote", "who composed", "what
year", "list of", "name all", ...) matches, OR the existing
arborist.qa.quantifier classifier returns intensity in {ALL,
COMPREHENSIVE, OPEN_REQUEST, MANY, PLURAL}.
C. Evidence contains candidate spans matching the answer_type within
a proximity window (default 600 chars) of cleaned subject tokens.
Candidate kinds aligned to answer_type:
title_like -> quoted_string, title_case_span, comma_list_item
person -> title_case_span
date -> year, date
Hardenings folded in from the 2026-05-27 Dav1d de-novo review:
1. Subject tokens strip cue/relation/stop words. For "songs by
veronica ballestrini" the cleaned subject is ["veronica",
"ballestrini"], NOT all four tokens. Without this the guard
false-triggers on "Harvard University" or "New York" near
proper-noun subjects.
2. Answer-type alignment. Candidate span kind must match query type
so "songs by John Smith" + evidence about Harvard/NY does not
strong-trigger.
3. Confidence class is deterministic (weak | medium | strong), not
boolean. Strong requires quoted_string near exact subject mention
+ multiple type-matched candidates. Phase 3 demote will gate on
confidence_class.
4. Cap output at 10 candidates (the per_chunk-quote-inflation
lesson). Prevents the guard becoming another claim amplifier.
5. Offsets are offset_start + offset_end + offset_basis=
"evidence_object_text", never an ambiguous single offset.
6. Cache-hit path returns answerability: None. Cached records do not
carry the evidence_map, only the rendered sources summary, so the
sidecar cannot recompute candidate spans without re-running
retrieval. Operators wanting fresh diagnostics use --burn.
7. Phase 1 stays out of verifier_policy_hash. The
answerability_sidecar_enabled / answerability_threshold /
denial_patterns_version / extraction_cues_version fields fold
into governance_policy_hash only. Phase 3 demote flag
(answerability_demote_enabled, default False) will move the
verifier hash WHEN ON because it changes the rendered audit_mode
(EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL).
Sidecar discipline (matches arborist.qa.inspect.diagnose_* sister
functions deflection / coherence / title_relevance):
- no model calls (no LLM-as-judge, no NLI, no translation)
- no audit_events writes
- no providence_cache writes
- no answer text mutation
- no claim promotion -- the trigger conjunction makes promotion
structurally impossible (only fires on denial answers)
- byte-deterministic: same (question, answer, evidence, policy) ->
same output every time
Result-dict integration: result["answerability"] is None when the
guard did not fire, or a structured diagnostic dict when it did
(diagnostic_version, confidence_class, triggered_clauses,
denial_pattern_matched, extraction_cue_matched, extraction_shape,
answer_type, subject_tokens, candidate_count, threshold,
missed_answer_candidate_spans). Read by bench_qa (Phase 2 will add
warning-count aggregate to bench rows) and CLI render.
Three return points carry the key:
- miss-path (full retrieval + verify): computed from evidence_map
- cache-hit: None (Dav1d cache-hit recompute discipline -- evidence
not stored, recompute requires re-retrieval)
- reject-broad early-return: None (no evidence examined)
Tests: 36 new pinning the three-clause logic, positive (Ballestrini)
regression, negative control (John Smith + Harvard/NY), each-clause-
alone silence, schema integrity, byte-determinism, sidecar-disabled
short-circuit, dict-shaped evidence support. All pass; existing
inspect tests (60) all pass.
End-to-end verified live via the CLI on the real corpus (2010 ~/.arborist
/shards): 3 fresh Hermes-3-8B runs on "songs by veronica ballestrini",
run 1 hit the failure (sidecar fired with confidence: strong, 351
candidates, denial: "not mentioned"), runs 2-3 model extracted
correctly and sidecar correctly silent.
Phase 2 (bench + threshold tuning) and Phase 3 (opt-in demote flag)
are open as follow-ups. Per Dav1d: NO-GO on default demote-on until
benchmark + human spot-check confirms low false-positive rate.
Full spec in docs/tickets/ticket-000068-verifier-blind-missed-answer-
guard.md (post-review hardenings section at top names the seven
load-bearing changes from the Dav1d 2026-05-27 review).
format_user_payload helper in arborist/qa/prompts.py becomes the single
source of truth for the user-turn payload. Three layouts:
tail (default) evidence first, question at end (prior behavior)
bookend question repeated before AND after evidence — counters
lost-in-the-middle on small models (≤8B)
per_chunk bookend + a one-line [for: <q>] reminder before each
evidence block; for list/extraction queries
USER_PAYLOAD_LAYOUTS constant exported; unknown layout raises ValueError.
The six _user_payload closures in query.py (3) and runner.py (3) all
delegate to format_user_payload. Quote-mode passes per_chunk_marker=None
to fall back to bookend on flat document/sources context.
Wired through both DEFAULT_QUERY_POLICY and DEFAULT_POLICY. Folds into
governance_policy_hash (the layout changes the user-turn content the
model sees, so the policy hash partitions cleanly per layout); does NOT
fold into verifier_policy_hash (verifier rules unchanged).
Makefile gets LAYOUT_DEFAULT ?= tail and LAYOUT ?= $(LAYOUT_DEFAULT) so
operators can flip per-call (LAYOUT=bookend make query Q="...") or
session-wide (LAYOUT_DEFAULT=bookend make query Q="..."). Recommendation
matrix in the Makefile comment block above the query target encodes the
2026-05-27 bench finding.
Motivating case: the Veronica-Ballestrini "songs by" failure. Hermes-3-
8B under tail layout returned "specific songs by her are not mentioned
in the provided evidence" when evidence E2 literally contained the song
names. Same query under bookend recovered the answer (with conflation
between Veronica Ballestrini and The Veronicas); under per_chunk
recovered AND disambiguated three entities. Qwen-27B unaffected by
layout. The Ballestrini case is added to bench/qa_questions.txt as a
regression fixture under "entity list", with a 4-line comment pointing
to docs/user-payload-layout.md.
2026-05-27 bench (n=3 × 75q, claim_lattice mode, Hermes-3-8B):
tail STRICT 94/225 (0.418) — control
bookend STRICT 95/225 (0.422) — +0.44pp (noise, 5pp floor)
per_chunk STRICT 72/225 (0.320) — -9.78pp (significant regression)
Verdict: tail stays default (cache-preserving and bench-confirmed
neutral). Bookend/per_chunk available as opt-in operator knobs. Per_chunk
regresses in aggregate because the per-chunk reminder over-anchors the
model on every chunk (TOO_MANY_EVIDENCE_IDS violations rose from 20 →
54; mean answer chars in 32-64KB bucket doubled from 720 → 1660). The
Ballestrini-class failure is real but rare across the curated set; a
layout fix that helps the rare case at the cost of 10pp aggregate is a
bad default trade. Documented in full in docs/user-payload-layout.md
along with the Dav1d 2026-05-27 review framing (GO for opt-in, NO-GO
for default promotion, ADD companion missed-answer guard).
CLI changes (--user-payload-layout flag on `query` and `ask`) landed
separately in commit e5ee283 alongside the #54 busy_timeout fix.
Each producer shard now optionally emits a THIRD pack alongside its
metadata and chunks packs: an "fts" pack containing the FTS5 shadow
tables (chunks_fts_data, chunks_fts_idx, chunks_fts_docsize,
chunks_fts_config + the documents_fts_* counterparts) packed as a
fresh SQLite file inside the tar so BLOB columns round-trip natively.
Consumer detects fts_pack_hashes in the metadata pack's manifest,
pulls each fts pack, ATTACHes the embedded sqlite, INSERTs every
shadow-table row into its target's empty shadow tables, and SKIPS
the local FTS rebuild entirely.
Producer side:
arborist/cold_object.py
+ PACK_KIND_FTS = "fts"
+ FTS_SHADOW_TABLES tuple (8 shadow tables)
+ build_fts_pack(src_db_path, ...)
creates a temp sqlite, applies SCHEMA_SQL (so destination
has FTS virtual tables → shadow tables auto-created), copies
every shadow-table row from src via cursor iteration, packs
the sqlite file into tar.zst
+ ParsedManifest.fts_pack_hashes
+ parse_manifest reads _fts_pack_hashes records
+ build_metadata_pack accepts fts_pack_hashes parameter and
writes the new manifest record
arborist/evict.py:push_pack
+ include_fts: bool = True parameter (CLI --no-fts opts out)
+ Phase B.5 emits the fts pack BEFORE Phase C (metadata pack)
so its hash can be referenced in the metadata manifest
Consumer side:
arborist/evict.py
+ _pull_fts_pack_into_targets() — pulls fts pack body, extracts
embedded sqlite, ATTACHes into each target, INSERT OR IGNORE
every shadow-table row. INSERT OR IGNORE protects against
rowid collisions on other targets that don't own these chunks.
+ hydrate_from_metadata_pack_routed iterates fts_pack_hashes in
full mode, calls _pull_fts_pack_into_targets per pack
+ _pull_pack_inner_routed returns fts_pack_hashes_referenced in
its result dict (mirrors chunk_pack_hashes_referenced)
CLI / Makefile:
arborist cold pack --no-fts (opt-out)
make cold-hydrate (auto-detects: if
chunks_fts_data is
already populated
on shard 000 after
unpack, skip the
rebuild post-pass)
make cold-hydrate HYDRATE_REBUILD_FTS=1 (force rebuild)
make cold-hydrate HYDRATE_REBUILD_FTS=0 (skip rebuild)
Schema:
cold_pending.kind CHECK extended to include 'fts'
pack_key() accepts kind="fts" → packs/<hash>.fts.tar.zst
Expected wall-time impact on the 3090 genesis bench:
with fts in packs: no rebuild step → ~5-10 min total wall
without fts: rebuild post-pass needed → ~15-20 min
Trade-off: ~30-50% larger bucket (FTS shadow data per shard) for
~70-90% faster consumer hydrate. Producer flips the trade via
--no-fts. The fts pack is optional in the manifest (empty list →
consumer falls back to rebuild) so old bucket data without fts
packs continues to work unchanged.
34 cold-unpack-routed + migrate + planner tests pass.
Open ticket. Today's hydrate_from_metadata_pack takes one conn and
writes every incoming row into one shard — fine when the corpus
was a single shard, broken now that #000065 put the producer in
M=4 hash-routed topology. A fresh peer pulling packs must land each
doc on `shard_for_document(document_root, M)` — same routing
function as the producer — or the consumer's M=4 ATTACH-and-route
assumption is just decoration over a single-shard reality.
Plan:
1. Add corpus_shard_count to pack manifest (read from source meta
during dump_shard_metadata) — pack carries the M it was built
against.
2. restore_shard_metadata_routed(targets, M, table_dir) in
cold_pack_metadata.py — mirrors _route_per_doc_table from
migrate.py (per-document tables route by document_root /
src_root / core_root; consolidated tables all go to target 0).
3. hydrate_from_metadata_pack gains a targets / shards_dir param.
4. arborist cold unpack --shards-dir DIR initialises M target
shards from the manifest's corpus_shard_count and routes.
5. Regression test: pack 2 shards → hydrate into fresh 4 shards
→ assert every doc on its hash-routed target.
Refactor question (raised, not decided): the routing rules
(ROUTED_BY_DOCUMENT_ROOT, CONSOLIDATED_TABLES) currently live in
migrate.py. Either duplicate them in cold_pack_metadata.py (fast)
or factor into arborist/multi_shard.py (cleaner, also serves
#000066 graft mode). Shared module is more honest.
Prerequisite for #46 (genesis on 3090 from cloud). Without this,
genesis is a 2-step α-kludge (hydrate-then-reshard) that wastes
~30 min and treats packed shards as if from an arbitrary topology.
Index entry bumped; next-id 67→68. Per-ticket spec doc to follow
when the implementation gates open.
2026-05-26 19:47 UTC. ~94 min wall. 3,468,226 globally-unique docs +
6,235,588 chunks + 90,592,990 edges + 3,468,403 audit events re-routed
from non-deterministic spray-by-ingest-order layout to canonical
content-hash M=4 layout (shard_idx = int(document_root[:8], 16) % 4).
Final state:
per-shard doc uniformity within ±0.04% (theoretical max ±0.05% for
first-32-bit SHA-256 prefix)
audit chain consolidated to shard 000 via Option A (re-sorted by ts,
re-chained; bodies preserved unchanged; tail event type=reshard
carries plan+result body, hash 8da3aa19…)
on-disk sizes: 000=11.0 / 001=8.8 / 002=8.8 / 003=8.8 GB
validation: chunks delta 176 (0.003%) + edges delta 547 (0.0006%)
are cross-shard dupes from re-ingest history, collapsed by
INSERT OR IGNORE; within the 1% tolerance gate
smoke queries: Barack Obama / YouTube / Albert Einstein all
returned proper evidence from correct (hash-routed) shards
chain-check-shards: 0 breaks on every shard
Two mid-flight defect fixes (also committed):
04edff7: derivations.src_root FK guard fired on legitimately
cross-shard refs → writer connection PRAGMA foreign_keys = OFF
c86d5ac: WAL accumulated ~37 GB across passes (SQLite auto-checkpoint
blocked by open reader cursors) → _checkpoint_truncate between
phases. Production migration was rescued mid-flight by manual
sibling-connection wal_checkpoint(TRUNCATE) freeing 27.7 GB.
Closes#000065 in both the index and the per-ticket file. The
per-ticket design doc stays open as a design-log artifact (its
content is still the right reference for the next reshard / for
graft mode #000066).
Follow-on tracked as tasks #44–#47:
#44 re-pack post-reshard shards into DO Spaces (current bucket
packs are stale, still in pre-reshard topology)
#45 verify bucket pack hydration is deterministic against the new
content-hash layout
#46 genesis a fresh peer on 3090-ai.foxhop.net from cloud — first
real SPV-wallet end-to-end test
#47 retire stale pre-reshard bucket packs after #46 confirms
Scaffold-only ticket. Captures the architecture for taking the
#000061 cold-pack format and adding a second mode beside hydrate:
overlay an existing pack onto a populated shard set ("graft").
Surfaced while running the #000065 reshard cutover and fox extended
the design: each pack carries a `corpus_name` field
(wikipedia-2010, wikipedia-current, arxiv-cs, ...), making
`arborist cold graft wikipedia-current` feel like `apt install`.
Three concerns analysed:
doc/chunk/edge overlay trivial (INSERT OR IGNORE on content-
addressed PKs collapses dupes)
FTS5 overlay trivial (new chunk_ids → new fts rows)
audit chain overlay the only hard part — three approaches:
A graft receipt (chosen): one event in
host chain carrying pack_hash +
event_count + first/last hashes; pack
file is the durable witness; zero
schema cost; aligned with v8/v9
witness pattern
B re-chain everything: rejected — graft
is frequent so invalidating external
refs is wrong tradeoff (different
story from the one-time reshard)
C chain forest with chain_id col: right
answer when graft dominates lifecycle
but premature now
Long-game payoff: mesh-peer-corpus-merge. Two peers diverge over a
partition, each carries packs the other lacks, reconciliation =
exchange + graft what's missing. Makes "mesh of arborists"
coherent rather than "fleet of arborists."
Scaffold gated on (a) #000065 lands+stabilises, (b) a second
corpus exists to graft, (c) at least two peers want to exchange.
No code until then; the design lock is what the ticket buys.
Index bumped Next ID 000066→000067.
Three pieces, all read-only or additive — no shard mutation, no
schema-version bump:
1. shard_for_document(document_root, M) in arborist/document.py.
Pure function: int(document_root[:8], 16) % M. 22 tests cover
determinism, range-bounds, near-uniform distribution (±5pp at
N=20k), and seven lock-in fixtures so peers will disagree
loudly if anyone changes the formula.
2. corpus_shard_count meta field + get/set helpers in store.py.
Lives in the existing key/value meta table; SCHEMA_VERSION
stays at v9.8.0 (the DDL doesn't change and source_root is
layout-independent, so cache records survive a reshard).
Legacy shards (without the field) return None; reshard tool
populates it on every target shard at migration time.
3. Pre-migration snapshot captured to
bench/results/pre-migration-snapshot.json:
docs 3,468,392 (3,468,226 globally unique)
chunks 6,235,764
edges 90,593,537
audit 3,468,403
This is the reference set post-reshard row counts must match.
4. Audit-event extraction script writes all 3.47M events from
all 4 shards to /tmp/audit-events.ndjson (2.0 GB) for the
Option-A canonical-chain consolidation step. Verifies chain
integrity on extract — all 4 source chains report 0 breaks.
5. Fixed a wrong chunk count in docs/corpus-history.md
(had ~3.54M/shard; actual is ~1.56M/shard) and added the
edge-count column (~22.6M/shard, 90.6M total). 6.24M chunks
total, not 14.12M.
Tests: 29 new pass (22 routing + 7 meta). No existing tests
touched.
The original ingest of Wikipedia 2010 into fox's 4 production shards
was 2026-04-27 21:33-23:13 UTC — ~100 minutes wall, 4-way parallel,
3.47M docs / 14.12M chunks. Captured here because the audit chain is
the only durable record but querying 3.47M rows to recover the
headline number is friction; one line in a doc removes it.
Also informs #000065 reshard planning: ingest rate ceiling on real
XML workload is ~2,350 chunks/sec aggregate (4-way), vs the
~6,400 chunks/sec the M-sweep bench measured on the 2003 cur dump
(which skips XML parsing). The teleport-style reshard should beat
both ceilings because it's just SQLite INSERT throughput, no XML
parse + canonicalize + edge extraction.
New file: docs/corpus-history.md. Indexed in CLAUDE.md docs section.
Append-only convention; future migrations + cold-pack runs add
entries here so the operator log isn't only in the audit chain.
Derivation query (sqlite3 audit_events) embedded in the entry so
future re-derivation is one copy-paste.
Pinned the canonical shard count at M = 4 based on real-Wikipedia
ingest + query benchmark (bench/shard_count_sweep.py). Captured the
"when does SQLite stop being the right substrate" decision tree so
future operators know what bench would justify a fork or replacement.
Bench numbers (Wikipedia 2003 cur dump, 2000 docs, 4 cells of
M ∈ {1, 2, 4, 8}, 50 FTS queries per cell):
M chunks/s q_p50_ms q_p99_ms attach_ms
1 3,648 0.05 0.20 1.61
2 5,649 0.03 0.19 3.24
4 6,405 0.07 0.30 9.00
8 6,959 0.03 0.28 9.65
Key observations:
- M=1→M=2 is the biggest ingest win (+55%). Most gain happens there.
- M=2→M=4 is +13%. M=4→M=8 is only +9% — diminishing returns.
- Real wikitext canonicalization is per-worker Python CPU bound, not
SQLite-writer-lock bound. More shards don't unlock more CPU.
- Query p50/p99 is flat across M within noise (50 queries small).
- ATTACH cost grows linearly: 1.6 / 3.2 / 9.0 / 9.7 ms.
Why M=4 specifically:
- Captures 92% of peak ingest throughput (6,405 / 6,959).
- 6 ATTACH slots free under SQLite's 10 ceiling for aux DBs
(qa.db, snapshots.db, selfmodel-chain.db, crawl_*.db, future
mesh_*.db) — comfortable headroom. M=8 leaves only 2 slots.
- Mobile-tolerable: phone NAND attach is 5-10x slower than NVMe;
M=4 = 45-90 ms cold start (instant), M=8 = 50-100 ms (sluggish
with no headroom).
- Matches fox's current 4-shard layout = cheapest migration.
Decision tree for when SQLite stops being right (full text in
ticket §"When the SQLite-default substrate stops being right"):
A. ATTACH ceiling pressure (auxiliary DBs grow past 5) → bench
forked SQLite with SQLITE_MAX_ATTACHED=125, M ∈ {16, 32, 64};
if attach cost stays linear past M=10, fork viable but pays
permanent "no longer stock sqlite3" tax.
B. Ingest hits >10k chunks/s sustained ceiling → first tune
page_size / WAL checkpoint / mmap_size / synchronous. If
tuning gets 2-5x, stay on SQLite. If still ceiling-limited,
candidates: DuckDB (columnar, MVCC, FTS), libmdbx (B+tree no
FTS; we'd build it). In-house DB rejected without specific
failure of those.
C. Federation needs multi-writer-same-shard → SQLite writer-lock
serializes peers, becomes federation bottleneck. First try
leader-election (single-writer-per-shard with WAL replication
to followers). If true multi-writer required, SQLite is wrong;
candidates: FoundationDB, CRDT-on-KV-store. DuckDB does NOT
solve this — its MVCC is single-process.
Honest verdict: for current arborist workload (single-writer-per-
shard, read-mostly federation), stock python3 sqlite3 is the right
substrate. None of A/B/C are close to firing. The bench discipline
exists to know what to measure when something changes.
bench/results/shard-count-sweep-2026-05-26T16-20-48Z.csv (synthetic
baseline) + 2026-05-26T16-31-34Z.csv (real Wikipedia) committed as
the load-bearing measurement for the M=4 choice.
fox caught the imprecision: "the shards are teleported if we fix this
because we know the shard a doc will end up in."
Re-ingest = re-read source files, re-canonicalize, re-chunk, re-hash.
Hours-to-days from Wikipedia dumps.
What's actually needed = content-addressed rebalance. Every row in
the corpus is already addressed by content (document_root, leaf_hash,
merkle_nodes.hash, audit_event_hash). Migration reads existing rows,
computes new_shard_idx via the routing function, INSERTs into the new
M shards. No source-file work, no canonicalization, no LLM.
Updated the ticket's "Migration story" section with the per-table
teleport recipe:
documents: move by document_root → new shard
chunks (+content): follow doc to its new shard, re-assign chunk_id
chunks_fts: rebuild per new shard from chunks.content
merkle_nodes: move with document_root
edges: move by src_root
concept_relations, move by content-derived parent
derivations,
providence_cache,
citation_aliases,
term_aliases:
audit_events: ALL → shard 000 (canonical), re-numbered + re-hashed
snapshots, mesh_*: canonical shard 000
Audit chain handling: chose Option A (canonical shard 000) over
Option B (split-by-subject_root). Preserves global event ordering,
re-hashing happens once at migration time. The alternative loses
cross-shard event ordering for falsification + replay reasoning.
Estimated wall: ~20-40 min I/O-bound on current corpus
(35 GB across 4 shards → ~35 GB across 8 shards, moves once).
Hard discipline added: move into ~/.arborist/shards.v2/, keep
originals as rollback, atomic dir-swap when verified. Mistake in
routing function = corrupted federation, recoverable only by going
back to the originals.
Surfaced while sizing #000061's federation story. Today shard count
conflates two roles:
- producer ingest parallelism (wants = vCPU count)
- consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10)
On default Python sqlite3, the ATTACH ceiling is 10 and can't be
raised without a custom sqlite3 build (which violates CLAUDE.md's
"python3 + venv + sqlite3 is enough" property). Producer with 16
shards → consumer fails to attach the 11th, federation silently
breaks.
Design: introduce M = canonical shard count (corpus-wide constant,
default 8) decoupled from N = ingest workers. Document → shard
assignment becomes content-deterministic:
shard_idx = int(document_root[:8], 16) % M
Same input → same output across every peer. Today's "spray by ingest
order" is non-deterministic across peers — two peers re-ingesting
the same corpus put the same document_root in different shards. That
weakens federation more than it should.
Migration hard-constraint (fox: "this implies we will need to
reprocess all our data into shards"): re-ingest required. Current
layout is sprayed by ingest order; post-ticket is sprayed by content
hash. Two layouts are incompatible by construction. Captured in
ticket §Migration as the load-bearing operational note.
Phases laid out (0-4: design lock → read path → ingest path →
pack-restore → corpus migration tool). Open audit-chain re-numbering
question (per-shard event_hash chains break when rows rebalance
across shards). Kept as one ticket — do-not-proliferate.
Scaffold only. No code yet — design lock first.
Next ID bumped to 000066.
Dav1d's reviews of #000061 (Response A + Response B/FINAL in
~/Downloads, 2026-05-26) flagged a long list of items — most already
shipped in the SPV-split work. Three were genuine gaps worth folding
into #000061 before close:
Gap 1: manifest/latest pointer for new-peer discovery.
A fresh peer doing `cold list` got a list of metadata-pack hashes
but no obvious "which one is current for shard X." Added
get_latest_pointer + update_latest_pointer to the backend ABC.
push_pack writes manifest/latest.json on every successful metadata
pack push (read-modify-write keyed by snapshot_root). Mutable
pointer; content addressing of the packs themselves preserves the
trust root. Last-writer-wins on contention.
Gap 2: license_class field + producer-side refuse for public buckets.
Maps documents.source_type to a license bucket (wikipedia_cur /
textbook_tex → public_redistributable; html / grok / vcs → unknown;
anything else → unknown). Strictness order: public < unknown <
private. compute_shard_license_class() walks DISTINCT source_type
in documents. push_pack now refuses to upload if the shard's
strictest license is more restrictive than the operator's
allow_license_class (default: public_redistributable). The
metadata pack's manifest carries _license_class so consumers /
auditors can see the producer's classification without inspecting
source documents. ValueError on refusal — the bucket ACL is the
operator's call, but arborist refuses to participate in a
licensing/membership leak unless explicitly opted in.
Gap 3: cold_pending table for resumable uploads.
Killed mid-upload, push_pack left orphan multi-GB tempfiles in
/tmp with no DB trace. Added schema:
CREATE TABLE cold_pending (
tempfile_path TEXT PRIMARY KEY,
pack_hash TEXT NOT NULL,
kind TEXT NOT NULL,
backend_endpoint TEXT NOT NULL,
backend_bucket TEXT NOT NULL,
object_key TEXT NOT NULL,
started_at INTEGER NOT NULL,
state TEXT NOT NULL DEFAULT 'pending'
);
push_pack INSERTs a row before each upload + DELETEs on success.
A killed process leaves the row pointing at the orphan tempfile;
a recovery script (future) reads cold_pending, checks bucket for
the object, either deletes the row + tempfile (success was just
unreported) or re-uploads from the tempfile if it still exists.
Matches the same pattern as the audit chain — explicit state
rows beat inferring from chunks.content IS NULL.
Sibling tickets opened for the larger items the reviews flagged
(scaffold-only, no code; opening them captures the design in the
log without proliferating, per CLAUDE.md):
- #000063 Cold-object private-ciphertext mode (mesh-keyed object
keys for non-public corpora on public-read buckets). Needs mesh
group-key ABI + real non-public corpus before code.
- #000064 Cold-object operations toolkit (verify / diff / doctor /
repair-fts / gc-plan CLI + expanded audit-event taxonomy).
Bundled so the audit-event vocabulary gets one design pass.
5 new tests:
test_gap2_license_gate_refuses_unknown_class_to_public_bucket
test_gap2_license_class_in_metadata_manifest
test_gap1_latest_pointer_resolves_metadata_pack_per_snapshot
test_gap3_cold_pending_clears_on_successful_upload
test_gap3_cold_pending_records_inflight_upload
26 cold-object + 7 evict tests pass (33/33 green incl. boto3 wire).
Next ID bumped to 000065.
Live v3 SPV corpus run (bmq47x6t3) completed cleanly during this work.
Will report sizing + memory profile in the next message.
Three doc-only housekeeping items from Dav1d's de-novo reconciliation
of the architecture stack (no code changes; the active build is
#000061 cold-pack work, which is unaffected by this review):
1. Accepted-error formula. Dav1d corrected p_raw × (1-d)(1-r) to
p_raw × (1 - dr) where r is conditional on detection. Searched
docs/ and arborist/ — the wrong form does NOT appear in this
tree (it lives in the external recursive-truth-maintenance / RCO
manuscripts Dav1d cited). Nothing to fix here; recorded for the
manuscript authors.
2. #000060 H-ABCDEFG split. Folded the M/C/X axis split into the
harness scope: M (mechanism — does the substrate's plumbing work),
C (capability — does it improve task performance), X (external
adversarial — does it generalize outside author-designed fixtures).
ACCEPT requires clearing all three. Without X, the harness risks
self-validating benchmark theology — passing tests its own designers
picked. Tagged at fixture-definition time; aggregator emits per-axis
pass rates + combined ACCEPT verdict. Doesn't change the existing
BatteryResult row schema.
3. #000062 Mechanistic Witness — new scaffold-only ticket. Specifies
a content-addressed MechanisticWitnessRoot over (model_config,
capture_policy, contrastive_prompts, features/neurons, intervention
result, behavioral delta, safety policy) as a DIAGNOSTIC sidecar
feeding SelfModel + benchmark fixtures. Four hard guardrails:
diagnostic-only by default; sandbox intervention only; no production
steering without governance ACCEPT via #000060 M+C+X; feature labels
never become semantic proof. No code until a real falsifier use case
exists + guardrails are CLAUDE.md rules + #000060 harness gates
promotion. The dual-use risk (Pan et al. 2025: 0.1% MLP ablation
breaks refusal in 72B models) makes governance-first framing
load-bearing.
Next ID bumped to 000063.
No code change to arborist/. The in-flight v3 SPV corpus pack
(bmq47x6t3) continues unaffected.
v1 packs (chunks-only) were under-engineered: a new peer landing on
v1 packs would have chunk bodies indexed by leaf_hash but no documents
table, no audit chain, no merkle interior, no edges — couldn't actually
hydrate. fox: "isn't what I wanted you under engineered..."
v2 packs ship every load-bearing shard table alongside chunk bodies in
the same tar.zst:
manifest.jsonl # chunk catalog (unchanged)
tables/documents.jsonl # array-per-line columnar JSONL
tables/chunks.jsonl # without content column
tables/merkle_nodes.jsonl
tables/edges.jsonl # FAN-IN restructured
tables/audit_events.jsonl
tables/derivations.jsonl
tables/concept_relations.jsonl
tables/concept_token_idf.jsonl
tables/providence_cache.jsonl
tables/citation_aliases.jsonl
tables/term_aliases.jsonl
tables/snapshots.jsonl
tables/document_http_meta.jsonl
blobs/<hash[:2]>/<hash[2:]> # raw UTF-8 chunk bodies
Two compression strategies inside the pack:
1. Array-per-line JSONL ({"_columns": [...]} header line + ["v1","v2",...]
data lines) drops ~30% of uncompressed bytes vs object-per-row JSONL.
zstd recovers most of that on its own, but smaller uncompressed
footprint also speeds up stream-restore.
2. Edges fan-in restructure at pack-build time: 22M rows of
(src_root, edge_type, dst_root, dst_uri, anchor) → ~500k unique
(dst_uri, edge_type, anchor, dst_root) groups with src_roots as an
array. ~5-10x compressed savings on the dominant table. Reverses on
unpack into the per-edge live schema. Live queries unchanged.
NOT shipped (per-peer state): mesh_*, selfmodel_*, capital_ledger,
memory_*, controller_events, fork_score_branches, adapter_loss_reports,
falsifications, schema_meta, meta. NOT shipped (rebuildable): chunks_fts*,
documents_fts* — restored from chunks.content + documents.title on
unpack.
push_pack no longer appends `cold_pack_pushed` to the audit chain.
That event leaked into the next push's audit_events.jsonl dump and
broke the "two writers at the same corpus state produce identical
pack_hash" determinism property. The bucket/disc file IS the receipt;
the snapshot_root pinned inside the pack metadata binds it to a corpus
state. No load-bearing consumer of the audit row.
pull_pack restored to handle both v1 (chunks-only) and v2 (tables +
chunks) packs. For v2 it extracts tables/*.jsonl to a temp dir,
calls restore_shard_metadata (which INSERT OR IGNOREs into the live
schema and expands edges back to per-edge rows), then fills chunk
content for every leaf_hash in blobs/. Idempotent against populated
DBs (INSERT OR IGNORE all the way down). Self-cleaning temp dir.
Sizing measured 2026-05-26: ~2.1 GB per shard pack compressed (chunk
content 1.78 GB + metadata ~0.3 GB), ~8.5 GB total across 4 shards.
~20% more than v1 chunks-only for self-sufficient hydration.
24 cold-object + evict tests pass (+1 new test_push_pack_v2_hydrates_fresh_empty_db
that builds a pack from a populated DB and unpacks into a completely
empty DB to verify all tables restored). Full suite: 2558 passed,
28 skipped, 1 xfailed.
Ship arborist corpus state to new peers and DVD-R archival via
point-in-time tar.zst packs. One artifact serves both channels —
bucket+CDN delivery and physical-media archival.
Bucket holds packs only. Pack key = hash_leaf(manifest_bytes), so same
chunk set on two writers produces the same pack_hash and upload is
idempotent. Each pack pins the corpus snapshot_root it covers in audit
+ result body — packs are delayed snapshots, not live mirrors;
falsifications between repacks produce new pack_hashes.
stream_packs runs streaming zstd over tarfile, peeking compressed-buffer
size after each chunk via FLUSH_BLOCK (preserves dictionary). Default
cap 4_400_000_000 — 4.4 GB DVD-R safe-fit, ~6.5% buffer below the
4.7 GB marketing capacity to absorb ISO9660 overhead, growisofs
lead-in/lead-out, media variance, and drive-edge refusal. Each disc
fills to ~4.4 GB recorded data, not the ~1.5 GB an uncompressed cap
produced.
One backend class (S3CompatibleBackend via boto3 + endpoint_url) covers
AWS S3, DO Spaces, R2, B2, GCS S3-interop, MinIO. Optional dep
[object-store] = boto3>=1.34; dev extras pull moto for the wire test.
Voyeur: credentials via AWS_ACCESS_KEY_ID/_SECRET_ACCESS_KEY env or
~/.aws/credentials, never printed; only endpoint URL + bucket name
surface in logs.
CLI: arborist cold {pack,unpack,stats}. Makefile: cold-pack,
cold-pack-dvd (local-dir output for growisofs), cold-unpack, cold-stats.
Sizing for current shards (14.1M chunks, ~17 GB compressed): ~4 packs
at the default cap, ~\$0.34/mo DO Spaces storage, ~\$0.0001/fresh-peer
hydrate.
Always-on raw-UTF-8 leaf store (per ticket "Hard invariants") deferred
— packs-only for now, backfill later.
2557 passed, 28 skipped, 1 xfailed.
Address Grok's two minor-improvement flags on the docs.
New docs/diagrams/{three-layer-stack,cache-key-8dim,falsification-states}.{dot,svg,png}
embedded into docs/_source/concepts.rst — visual scaffolding for the
3-layer stack, 8-dim cache_key composition, and falsification state
machine (previously prose+tables only).
docs/_source/cookbook.rst gains a "Use arborist as a Python library"
section: open_store + ingest_documents, custom Source subclass,
audit-chain walk + verify, Merkle proof round-trip, programmatic
arborist.qa.query() with OpenAICompatibleClient + StubClient swap.
Every Python recipe smoke-tested against a scratch DB before publish.
make docs-api: 0 new warnings. make test: 2557 passed.
The §12.1 pipeline I added was second-hand from benchmark-matrix.md
and got several things wrong against the code:
- listed 4 verdicts; actual is 5 (missing FABRICATED — the
fabrication-vs-WRONG split that energy-cogs §5.5 leans on for the
qwen-fabricates / hermes-abstains finding)
- "θ=0.85" was right by accident — but it's the code-judge-pinned
_CODE_JUDGE_THETA_CONTRA constant, raised from the manifest 0.5
default after measuring 114 FPs in the 0.5-0.75 band
- omitted the short-answer entity-grounding fast path (which runs
BEFORE NLI per the 2026-05-19 Poland-Tusk smoke)
- omitted the HYBRID rescue ladder (NLI entail / entity rescue /
2026-05-21 verbatim-quote-on-topic rescue)
- conflated WRONG and FABRICATED (the subject-in-gold split is what
distinguishes "source has the topic but a different value" from
"source silent on the topic")
Rewrote §12.1 grounded in `bench/judge_code.py:judge()` (its own
docstring at line 501-528 is the truth on rule order), with file:line
citations and the verdict-mapping in full.
Also fixed a real artifact-vs-doc drift INSIDE the judge: the
module-top docstring still claimed θ_contra default 0.5 and omitted
the short-path and the WRONG/FABRICATED split. Updated to match the
authoritative judge() docstring + current code.
No behavior change — docstring + benchmarks doc only.
A stable façade so another Python app can use arborist as a
content-addressed / Merkle / audit-chained store without the CLI or a
wire protocol. Import from arborist.embed, not internal modules, so
refactors don't break embedders.
Surface: open_store(path), ingest_documents(conn, docs), search(conn, q),
plus re-exported Document/Edge/Source/Hit/IngestStats. Core only
(python+sqlite3) — no extras. _IterableSource adapts a plain doc iterable
into the Source contract.
This is the seam for using arborist as neopig's optional provenance
backend: neopig produces Documents from crawled pages, arborist gives
content-dedup (document_root) + FTS5 + an append-only audit chain
alongside neopig's existing md5/FileVault storage. Docs in
docs/embedding.md. 6 tests pin open/ingest/dedup/idempotence/edges/search.
New docs/crawler.md covering the crawl path we actually run: BFS
same-host discovery, robots/feed/sitemap handling, polite vs --fast,
the shared-session + no-HEAD + crawl-delay-fix speedups, and the
content-addressed payoff. Leads on the two store-derived diagnostics:
duplicate detection (group by document_root — body, not URI) and
partial-overlap (shared chunk leaves), plus orphan finding
(sitemap − BFS-reached) and the planned crawl-report webmaster tools.
Honest pros/cons: orphans invisible to crawl by design, single-host,
no JS execution, --fast is anti-social off your own turf.
- '.. class:: center' parsed as a Python class named 'center' (duplicate
across one-pager + two-pager) -> '.. rst-class:: center' (styling, no
object). Clears the duplicate-object-description warnings.
- two v8-consensus section underlines were shorter than their titles
('underline too short') -> extended to title length.
Pager + v8 pages now build with zero warnings.
The page was a markdown+LaTeX paper dumped into .rst — 499 errors/warnings,
big chunks rendering broken. Mechanical, content-preserving fixes:
- markdown code fences (```lang) -> RST .. code-block:: (48 blocks; the
dominant error source — RST read each ``` as an unclosed inline literal)
- display math [ ... ] -> literal blocks (34)
- code-block:: json -> text where bodies are schemas with <...>/[m,n]
placeholders, not valid JSON (10; same as the v8 fix)
- escape inline | in prose (math like |Z|, |pred(v)|) read as RST
substitution refs (37 lines; no pipe-tables in the file, so safe)
Residual 62 are scattered indented-math derivation lines RST treats as
block quotes — they still render, just warn; diminishing returns on an
advanced/optional theory page.
The simple-table (=== separators) had a wrapped multi-line cell that
docutils rejected as malformed -> the floor table rendered broken on the
live site. Converted to a list-table (alignment-proof). Also switched the
proposer-submission block from 'code-block:: json' to 'text' — it's a
schema with <...> placeholders, not valid JSON, so the json lexer warned.
Page now builds with no errors/warnings on these blocks.
The Substrate group is Dav1d's formal Merkle-AGI research that drove the
design — valuable, but it sat third in the nav and read like required
reading. IA-only fix (no content touched):
- re-caption 'Substrate' -> 'Substrate theory (advanced)' + a preamble
marking it optional (nothing in Getting started / API depends on it);
- reorder it BELOW the practical sections (now: Summary -> Getting started
-> API -> Substrate theory -> Project);
- pull 'bench' out of Substrate into Getting started (it's the practical
benchmark surface, not theory).
Practitioner path comes first; theory stays intact + credited, just last.
- Frame the solution as a Reverse RAG (Merkle Providence Reverse RAG) with
a link to the whitepaper (unfirehose.com/merkle-providence-reverse-rag-
whitepaper — note: published on unfirehose, not uncloseai).
- New differentiator: NO vector embeddings — retrieval is lexical-first
(FTS5 BM25 + Merkle), dense-vector optional + off by default; embedding
10M docs costs 10-100x more/doc + a vector index to store/maintain. A
big part of why COGS is low.
- COGS framed per 1,000 answers, labeled @ $0.33/kWh (intro + diagram cost
node). Cost node clarified: no embeddings, NO reasoning (reasoning is the
thing that would cost 4-6x, which we skip). Retrieval node + mapping
table updated to lexical-first / no vector index.
ranksep=1.2 over-stretched it vertically (1193x1483, lots of whitespace).
Drop to 0.4 + nodesep 0.25: 1202x1063, ~28% less area, no vertical sprawl,
stays roughly square. Width is floored by the widest node label; click-to-
zoom (zoom.js) covers reading detail.
Graphviz renders SVG as <object>, which swallows clicks and can't be
lightboxed. Vendored, dependency-free zoom.js converts each graphviz
<object> to a clickable <img> (crisp vector) and adds a fullscreen
overlay: click to open, scroll to zoom toward the cursor, drag to pan,
Esc / dbl-click / background-click to close. Works for any img.zoomable,
so future charts get it free. Degrades gracefully (no JS -> inline image
still renders). Wired via html_css_files / html_js_files.
TB alone left it landscape (1202x1101, h/w 0.92). Width is floored by the
widest node label (can't shrink without trimming labels), so bump
ranksep to stretch vertically: 1193x1483, h/w 1.24 — clearly top-down/
vertical for the docs column.
rankdir LR -> TB so the diagram reads top-down and fits the docs column
(roughly square ~1200x1100, fox: 'square is good') instead of a wide
horizontal strip.
Graphviz DOT diagram of the arborist pipeline as the answer to the Google
L5 system-design prompt 'design a RAG pipeline for 10M docs with zero
hallucination'. Maps our components onto the canonical 10-box RAG design
and shows the three extensions that buy zero-hallucination + near-zero
cost: deterministic verifier (not a model confidence score) -> honest
UNGROUNDED; Merkle-bound cache that skips the GPU; measured energy COGS
(~$0.07-0.16/1k answers, non-reasoning). Includes the mapping table +
the 3.47M->10M scaling math (+77GB, sourcing/storage not redesign).
Enables sphinx.ext.graphviz (SVG output) + graphviz apt package on the
RTD build; adds the page to the Summary-pages toctree. Builds clean
(page renders, DOT validates via dot -Tsvg).
Main tweet + follow-up (Merkle cache hit skips GPU, doesn't increment the
per-1k). Numbers are the measured claim_lattice figures from the energy
report; note attached to hold the arbitrage/forcing-function framing until
the value side is hardened (higher N + blinded judge).
$0.10/1k-q overstated the qwen-27B case. Honest range: ~$0.07-0.16 per
1,000 queries of GPU electricity — hermes-8B $0.07-0.09 (under a dime),
qwen-27B $0.12-0.16 (over a dime; claim_lattice dearer than quote from
more prefilled context). Fixes the §5.4 'either rig' claim.
The report is now cost AND value (quality-per-dollar), not cost-only.
§5.5 quality delta: substrate-vs-solo (code judge, n=30) on numeral +
stale fixtures, hermes-8B vs qwen-27B-nothink. Substrate lifts both
2-13x and they nearly converge (hermes 18/21, qwen 19/26 grounded-
correct) — grounding comes from retrieval+verification, not parametric
size, so the base model matters far less. Bare-model failure styles
diverge (hermes abstains, qwen fabricates). Quality-per-dollar: ~18-26/30
grounded for $0.085-0.158/1k-q.
Honest caveats recorded: CORRECT=grounded-in-2010-corpus (not current);
the stale fixture is no longer post-corpus; the qwen thinking bug (39c040c)
that voided the first run and the judge strengthening (2d31866). §1
reframed; §8 repro + §9 next updated (post-2010 fixture, SOTA judge for
residue).
fox 2026-05-21: two corpora exist (2003-05-16 + 2010); the 2010 one is
loaded in ~/.arborist/shards (verified: 866K docs/shard, has Barack
Obama / YouTube articles a 2003 dump can't). CLAUDE.md Live-endpoints and
the energy-COGS report now state 2010; fabrication-bait fixtures target
post-2010 events. Historical closed-ticket prose left as point-in-time
records.
- §2 now a two-rig table: 4090/Qwen-27B-Q4_K_XL/llama.cpp (isolated) vs
3090/Hermes-8B-FP8-Dynamic/vLLM (live/public). Spells out that the
cross-model rate gap confounds FOUR variables (params, quant/precision,
engine, GPU) — not '8B vs 27B' alone.
- §5.4 hermes results under live traffic. Method finding: the watt_bench
window integral is unusable on a contended card, but the slope
calibration survives clean (cancels the variable shared-load baseline)
— so on shared/public cards use watt_calibrate, not the window integral.
Rates: hermes 0.109/4.40 J/tok vs qwen 0.175/6.16; decode 35-40x prefill.
- Per-query DOLLARS both rigs @$0.33/kWh: a grounded substrate answer is
<$0.10 per 1000 queries (hermes $0.085, qwen $0.158); hermes ~half qwen.
- §5.2 budget confound corrected to flag Hermes-tuning honestly (was
rationalized). §9: fixed-budget apples-to-apples re-run + per-model
budget tuning added as next steps.
Full report for Dav1d: qwen-nothink on the dedicated 4090. Headline —
prefill 0.175 J/input-tok ($0.016/M) vs decode 6.157 J/output-tok
($0.564/M), decode 35x dearer per token. The substrate prefills ~6.6k
input tok/query (vs solo ~52, 127x): ~67% of its GPU energy is reading
the retrieved context, not generating. Per-query 1719 J (substrate-CL)
vs 95 J (solo) = $0.16 vs $0.009 per 1k queries; calibration predicts
measured within ~5%. Real cost is CPU latency, not GPU watts.
Documents the full methodology + the corrections that got here
(contamination -> isolation; blended mean-W -> measured states; gross
-> not-attributable; per-token -> separated input/output; len//4 ->
real usage), threats to validity, reproducibility (commits, config hash,
persisted samples), and next rigs (3090+Hermes, reasoning, prefill-cache
study).
fox 2026-05-21: 'we dont touch card during retrieval'. Retrieval + verify
are CPU/SQLite on the orchestrator; the GPU is idle (at the shared, always-
on model-resident floor) during them. So the substrate does NOT 'hold the
card' through its wall-clock — the gross window integral over-counts by
charging that always-on floor for the seconds we spend retrieving (energy
that exists regardless of the query; the card serves other traffic then).
Correct attribution: the GPU cost of a query is its GENERATION energy only
(the marginal — burst above the serving floor). Relabel: marginal is the
headline GPU COGS; gross is demoted to a reference 'window total, not
query-attributable'. Print + energy_cogs docstring + stock-v1-config doc
updated. No math change (marginal was already right) — this corrects the
narration. Measured qwen-nothink/4090: substrate GPU COGS is LOW
(claim_lattice $0.21/M-tok, quote $0.83) — its real overhead is latency
(CPU retrieval), not GPU watts.
fox 2026-05-21: compute cost-of-goods-sold by kWh vs tokens, with the
three power states (idle / warm-idle / generation) MEASURED per
card×model×server — never hardcoded (his 40/127/380 W were illustrative
of one 3090). The only operator input is --price-per-kwh (default 0.33
USD/kWh, a configurable site rate).
energy_cogs() (pure, unit-tested) decomposes measured generation energy
against the measured warm-idle baseline:
* gross — all measured joules over the window (all-in, includes the
warm-idle cost of keeping the model hot, amortized).
* marginal — joules ABOVE warm-idle: what one more request's burst
actually costs (clamped >=0).
kWh = J/3.6e6; $/1k-tok is the unit that compares to API pricing. Both
surface per cell + a COGS print line.
watt_bench's arborist arm now loads the frozen bench.stock_v1 policy
(--answer-mode, drift-guarded on non-reasoning) so cost is measured for
the SAME substrate the campaign grades. Cells record
window_start/end_unix so a post-hoc load_monitor queue-depth cross-ref
can flag organic-traffic contamination on the non-isolated single-slot
endpoints. 6 COGS tests; full suite 2534 passed.
fox 2026-05-21: characterize substrate-ON under BOTH answer shapes, so
answer_mode is a swept axis, not a single pinned value.
stock_v1.py now exposes STOCK_V1_POLICIES{quote,claim_lattice} +
STOCK_V1_GOVERNANCE_HASHES (quote 5b6ca4c5..., claim_lattice 036a4c79...),
policy_for(mode), and assert_not_drifted(mode). Shared pins (crosslang
OFF, repair OFF, quantifier dry-run, metacognition label-only,
soft-preflight OFF, claim cap 12, v2-acronym-aware) are frozen
identically across modes.
Wire the treatment arms to the pin (the consumer-side step that makes
the freeze real):
* control_ab --answer-mode {quote,claim_lattice}
* control_sweep --arborist-answer-mode {quote,claim_lattice}
Both default claim_lattice (prior behavior), call assert_not_drifted on
non-reasoning runs (halts the sweep if DEFAULT_QUERY_POLICY drifts), and
load the frozen policy_for(mode) instead of an inline
dict(DEFAULT_QUERY_POLICY, ...). Reasoning refs (phase 3) keep their
documented JSON overrides and skip the assert by design (different hash).
jaggedness is left standalone — it is a mode-agnostic retrieval
instrument, coupling it to the answer-policy freeze adds friction with no
correctness gain. Full suite 2528 passed.
