Each producer shard now optionally emits a THIRD pack alongside its
metadata and chunks packs: an "fts" pack containing the FTS5 shadow
tables (chunks_fts_data, chunks_fts_idx, chunks_fts_docsize,
chunks_fts_config + the documents_fts_* counterparts) packed as a
fresh SQLite file inside the tar so BLOB columns round-trip natively.
Consumer detects fts_pack_hashes in the metadata pack's manifest,
pulls each fts pack, ATTACHes the embedded sqlite, INSERTs every
shadow-table row into its target's empty shadow tables, and SKIPS
the local FTS rebuild entirely.
Producer side:
arborist/cold_object.py
+ PACK_KIND_FTS = "fts"
+ FTS_SHADOW_TABLES tuple (8 shadow tables)
+ build_fts_pack(src_db_path, ...)
creates a temp sqlite, applies SCHEMA_SQL (so destination
has FTS virtual tables → shadow tables auto-created), copies
every shadow-table row from src via cursor iteration, packs
the sqlite file into tar.zst
+ ParsedManifest.fts_pack_hashes
+ parse_manifest reads _fts_pack_hashes records
+ build_metadata_pack accepts fts_pack_hashes parameter and
writes the new manifest record
arborist/evict.py:push_pack
+ include_fts: bool = True parameter (CLI --no-fts opts out)
+ Phase B.5 emits the fts pack BEFORE Phase C (metadata pack)
so its hash can be referenced in the metadata manifest
Consumer side:
arborist/evict.py
+ _pull_fts_pack_into_targets() — pulls fts pack body, extracts
embedded sqlite, ATTACHes into each target, INSERT OR IGNORE
every shadow-table row. INSERT OR IGNORE protects against
rowid collisions on other targets that don't own these chunks.
+ hydrate_from_metadata_pack_routed iterates fts_pack_hashes in
full mode, calls _pull_fts_pack_into_targets per pack
+ _pull_pack_inner_routed returns fts_pack_hashes_referenced in
its result dict (mirrors chunk_pack_hashes_referenced)
CLI / Makefile:
arborist cold pack --no-fts (opt-out)
make cold-hydrate (auto-detects: if
chunks_fts_data is
already populated
on shard 000 after
unpack, skip the
rebuild post-pass)
make cold-hydrate HYDRATE_REBUILD_FTS=1 (force rebuild)
make cold-hydrate HYDRATE_REBUILD_FTS=0 (skip rebuild)
Schema:
cold_pending.kind CHECK extended to include 'fts'
pack_key() accepts kind="fts" → packs/<hash>.fts.tar.zst
Expected wall-time impact on the 3090 genesis bench:
with fts in packs: no rebuild step → ~5-10 min total wall
without fts: rebuild post-pass needed → ~15-20 min
Trade-off: ~30-50% larger bucket (FTS shadow data per shard) for
~70-90% faster consumer hydrate. Producer flips the trade via
--no-fts. The fts pack is optional in the manifest (empty list →
consumer falls back to rebuild) so old bucket data without fts
packs continues to work unchanged.
34 cold-unpack-routed + migrate + planner tests pass.
Open ticket. Today's hydrate_from_metadata_pack takes one conn and
writes every incoming row into one shard — fine when the corpus
was a single shard, broken now that #000065 put the producer in
M=4 hash-routed topology. A fresh peer pulling packs must land each
doc on `shard_for_document(document_root, M)` — same routing
function as the producer — or the consumer's M=4 ATTACH-and-route
assumption is just decoration over a single-shard reality.
Plan:
1. Add corpus_shard_count to pack manifest (read from source meta
during dump_shard_metadata) — pack carries the M it was built
against.
2. restore_shard_metadata_routed(targets, M, table_dir) in
cold_pack_metadata.py — mirrors _route_per_doc_table from
migrate.py (per-document tables route by document_root /
src_root / core_root; consolidated tables all go to target 0).
3. hydrate_from_metadata_pack gains a targets / shards_dir param.
4. arborist cold unpack --shards-dir DIR initialises M target
shards from the manifest's corpus_shard_count and routes.
5. Regression test: pack 2 shards → hydrate into fresh 4 shards
→ assert every doc on its hash-routed target.
Refactor question (raised, not decided): the routing rules
(ROUTED_BY_DOCUMENT_ROOT, CONSOLIDATED_TABLES) currently live in
migrate.py. Either duplicate them in cold_pack_metadata.py (fast)
or factor into arborist/multi_shard.py (cleaner, also serves
#000066 graft mode). Shared module is more honest.
Prerequisite for #46 (genesis on 3090 from cloud). Without this,
genesis is a 2-step α-kludge (hydrate-then-reshard) that wastes
~30 min and treats packed shards as if from an arbitrary topology.
Index entry bumped; next-id 67→68. Per-ticket spec doc to follow
when the implementation gates open.
2026-05-26 19:47 UTC. ~94 min wall. 3,468,226 globally-unique docs +
6,235,588 chunks + 90,592,990 edges + 3,468,403 audit events re-routed
from non-deterministic spray-by-ingest-order layout to canonical
content-hash M=4 layout (shard_idx = int(document_root[:8], 16) % 4).
Final state:
per-shard doc uniformity within ±0.04% (theoretical max ±0.05% for
first-32-bit SHA-256 prefix)
audit chain consolidated to shard 000 via Option A (re-sorted by ts,
re-chained; bodies preserved unchanged; tail event type=reshard
carries plan+result body, hash 8da3aa19…)
on-disk sizes: 000=11.0 / 001=8.8 / 002=8.8 / 003=8.8 GB
validation: chunks delta 176 (0.003%) + edges delta 547 (0.0006%)
are cross-shard dupes from re-ingest history, collapsed by
INSERT OR IGNORE; within the 1% tolerance gate
smoke queries: Barack Obama / YouTube / Albert Einstein all
returned proper evidence from correct (hash-routed) shards
chain-check-shards: 0 breaks on every shard
Two mid-flight defect fixes (also committed):
04edff7: derivations.src_root FK guard fired on legitimately
cross-shard refs → writer connection PRAGMA foreign_keys = OFF
c86d5ac: WAL accumulated ~37 GB across passes (SQLite auto-checkpoint
blocked by open reader cursors) → _checkpoint_truncate between
phases. Production migration was rescued mid-flight by manual
sibling-connection wal_checkpoint(TRUNCATE) freeing 27.7 GB.
Closes#000065 in both the index and the per-ticket file. The
per-ticket design doc stays open as a design-log artifact (its
content is still the right reference for the next reshard / for
graft mode #000066).
Follow-on tracked as tasks #44–#47:
#44 re-pack post-reshard shards into DO Spaces (current bucket
packs are stale, still in pre-reshard topology)
#45 verify bucket pack hydration is deterministic against the new
content-hash layout
#46 genesis a fresh peer on 3090-ai.foxhop.net from cloud — first
real SPV-wallet end-to-end test
#47 retire stale pre-reshard bucket packs after #46 confirms
Scaffold-only ticket. Captures the architecture for taking the
#000061 cold-pack format and adding a second mode beside hydrate:
overlay an existing pack onto a populated shard set ("graft").
Surfaced while running the #000065 reshard cutover and fox extended
the design: each pack carries a `corpus_name` field
(wikipedia-2010, wikipedia-current, arxiv-cs, ...), making
`arborist cold graft wikipedia-current` feel like `apt install`.
Three concerns analysed:
doc/chunk/edge overlay trivial (INSERT OR IGNORE on content-
addressed PKs collapses dupes)
FTS5 overlay trivial (new chunk_ids → new fts rows)
audit chain overlay the only hard part — three approaches:
A graft receipt (chosen): one event in
host chain carrying pack_hash +
event_count + first/last hashes; pack
file is the durable witness; zero
schema cost; aligned with v8/v9
witness pattern
B re-chain everything: rejected — graft
is frequent so invalidating external
refs is wrong tradeoff (different
story from the one-time reshard)
C chain forest with chain_id col: right
answer when graft dominates lifecycle
but premature now
Long-game payoff: mesh-peer-corpus-merge. Two peers diverge over a
partition, each carries packs the other lacks, reconciliation =
exchange + graft what's missing. Makes "mesh of arborists"
coherent rather than "fleet of arborists."
Scaffold gated on (a) #000065 lands+stabilises, (b) a second
corpus exists to graft, (c) at least two peers want to exchange.
No code until then; the design lock is what the ticket buys.
Index bumped Next ID 000066→000067.
Three pieces, all read-only or additive — no shard mutation, no
schema-version bump:
1. shard_for_document(document_root, M) in arborist/document.py.
Pure function: int(document_root[:8], 16) % M. 22 tests cover
determinism, range-bounds, near-uniform distribution (±5pp at
N=20k), and seven lock-in fixtures so peers will disagree
loudly if anyone changes the formula.
2. corpus_shard_count meta field + get/set helpers in store.py.
Lives in the existing key/value meta table; SCHEMA_VERSION
stays at v9.8.0 (the DDL doesn't change and source_root is
layout-independent, so cache records survive a reshard).
Legacy shards (without the field) return None; reshard tool
populates it on every target shard at migration time.
3. Pre-migration snapshot captured to
bench/results/pre-migration-snapshot.json:
docs 3,468,392 (3,468,226 globally unique)
chunks 6,235,764
edges 90,593,537
audit 3,468,403
This is the reference set post-reshard row counts must match.
4. Audit-event extraction script writes all 3.47M events from
all 4 shards to /tmp/audit-events.ndjson (2.0 GB) for the
Option-A canonical-chain consolidation step. Verifies chain
integrity on extract — all 4 source chains report 0 breaks.
5. Fixed a wrong chunk count in docs/corpus-history.md
(had ~3.54M/shard; actual is ~1.56M/shard) and added the
edge-count column (~22.6M/shard, 90.6M total). 6.24M chunks
total, not 14.12M.
Tests: 29 new pass (22 routing + 7 meta). No existing tests
touched.
The original ingest of Wikipedia 2010 into fox's 4 production shards
was 2026-04-27 21:33-23:13 UTC — ~100 minutes wall, 4-way parallel,
3.47M docs / 14.12M chunks. Captured here because the audit chain is
the only durable record but querying 3.47M rows to recover the
headline number is friction; one line in a doc removes it.
Also informs #000065 reshard planning: ingest rate ceiling on real
XML workload is ~2,350 chunks/sec aggregate (4-way), vs the
~6,400 chunks/sec the M-sweep bench measured on the 2003 cur dump
(which skips XML parsing). The teleport-style reshard should beat
both ceilings because it's just SQLite INSERT throughput, no XML
parse + canonicalize + edge extraction.
New file: docs/corpus-history.md. Indexed in CLAUDE.md docs section.
Append-only convention; future migrations + cold-pack runs add
entries here so the operator log isn't only in the audit chain.
Derivation query (sqlite3 audit_events) embedded in the entry so
future re-derivation is one copy-paste.
Pinned the canonical shard count at M = 4 based on real-Wikipedia
ingest + query benchmark (bench/shard_count_sweep.py). Captured the
"when does SQLite stop being the right substrate" decision tree so
future operators know what bench would justify a fork or replacement.
Bench numbers (Wikipedia 2003 cur dump, 2000 docs, 4 cells of
M ∈ {1, 2, 4, 8}, 50 FTS queries per cell):
M chunks/s q_p50_ms q_p99_ms attach_ms
1 3,648 0.05 0.20 1.61
2 5,649 0.03 0.19 3.24
4 6,405 0.07 0.30 9.00
8 6,959 0.03 0.28 9.65
Key observations:
- M=1→M=2 is the biggest ingest win (+55%). Most gain happens there.
- M=2→M=4 is +13%. M=4→M=8 is only +9% — diminishing returns.
- Real wikitext canonicalization is per-worker Python CPU bound, not
SQLite-writer-lock bound. More shards don't unlock more CPU.
- Query p50/p99 is flat across M within noise (50 queries small).
- ATTACH cost grows linearly: 1.6 / 3.2 / 9.0 / 9.7 ms.
Why M=4 specifically:
- Captures 92% of peak ingest throughput (6,405 / 6,959).
- 6 ATTACH slots free under SQLite's 10 ceiling for aux DBs
(qa.db, snapshots.db, selfmodel-chain.db, crawl_*.db, future
mesh_*.db) — comfortable headroom. M=8 leaves only 2 slots.
- Mobile-tolerable: phone NAND attach is 5-10x slower than NVMe;
M=4 = 45-90 ms cold start (instant), M=8 = 50-100 ms (sluggish
with no headroom).
- Matches fox's current 4-shard layout = cheapest migration.
Decision tree for when SQLite stops being right (full text in
ticket §"When the SQLite-default substrate stops being right"):
A. ATTACH ceiling pressure (auxiliary DBs grow past 5) → bench
forked SQLite with SQLITE_MAX_ATTACHED=125, M ∈ {16, 32, 64};
if attach cost stays linear past M=10, fork viable but pays
permanent "no longer stock sqlite3" tax.
B. Ingest hits >10k chunks/s sustained ceiling → first tune
page_size / WAL checkpoint / mmap_size / synchronous. If
tuning gets 2-5x, stay on SQLite. If still ceiling-limited,
candidates: DuckDB (columnar, MVCC, FTS), libmdbx (B+tree no
FTS; we'd build it). In-house DB rejected without specific
failure of those.
C. Federation needs multi-writer-same-shard → SQLite writer-lock
serializes peers, becomes federation bottleneck. First try
leader-election (single-writer-per-shard with WAL replication
to followers). If true multi-writer required, SQLite is wrong;
candidates: FoundationDB, CRDT-on-KV-store. DuckDB does NOT
solve this — its MVCC is single-process.
Honest verdict: for current arborist workload (single-writer-per-
shard, read-mostly federation), stock python3 sqlite3 is the right
substrate. None of A/B/C are close to firing. The bench discipline
exists to know what to measure when something changes.
bench/results/shard-count-sweep-2026-05-26T16-20-48Z.csv (synthetic
baseline) + 2026-05-26T16-31-34Z.csv (real Wikipedia) committed as
the load-bearing measurement for the M=4 choice.
fox caught the imprecision: "the shards are teleported if we fix this
because we know the shard a doc will end up in."
Re-ingest = re-read source files, re-canonicalize, re-chunk, re-hash.
Hours-to-days from Wikipedia dumps.
What's actually needed = content-addressed rebalance. Every row in
the corpus is already addressed by content (document_root, leaf_hash,
merkle_nodes.hash, audit_event_hash). Migration reads existing rows,
computes new_shard_idx via the routing function, INSERTs into the new
M shards. No source-file work, no canonicalization, no LLM.
Updated the ticket's "Migration story" section with the per-table
teleport recipe:
documents: move by document_root → new shard
chunks (+content): follow doc to its new shard, re-assign chunk_id
chunks_fts: rebuild per new shard from chunks.content
merkle_nodes: move with document_root
edges: move by src_root
concept_relations, move by content-derived parent
derivations,
providence_cache,
citation_aliases,
term_aliases:
audit_events: ALL → shard 000 (canonical), re-numbered + re-hashed
snapshots, mesh_*: canonical shard 000
Audit chain handling: chose Option A (canonical shard 000) over
Option B (split-by-subject_root). Preserves global event ordering,
re-hashing happens once at migration time. The alternative loses
cross-shard event ordering for falsification + replay reasoning.
Estimated wall: ~20-40 min I/O-bound on current corpus
(35 GB across 4 shards → ~35 GB across 8 shards, moves once).
Hard discipline added: move into ~/.arborist/shards.v2/, keep
originals as rollback, atomic dir-swap when verified. Mistake in
routing function = corrupted federation, recoverable only by going
back to the originals.
Surfaced while sizing #000061's federation story. Today shard count
conflates two roles:
- producer ingest parallelism (wants = vCPU count)
- consumer ATTACH fan-out (capped at SQLITE_MAX_ATTACHED=10)
On default Python sqlite3, the ATTACH ceiling is 10 and can't be
raised without a custom sqlite3 build (which violates CLAUDE.md's
"python3 + venv + sqlite3 is enough" property). Producer with 16
shards → consumer fails to attach the 11th, federation silently
breaks.
Design: introduce M = canonical shard count (corpus-wide constant,
default 8) decoupled from N = ingest workers. Document → shard
assignment becomes content-deterministic:
shard_idx = int(document_root[:8], 16) % M
Same input → same output across every peer. Today's "spray by ingest
order" is non-deterministic across peers — two peers re-ingesting
the same corpus put the same document_root in different shards. That
weakens federation more than it should.
Migration hard-constraint (fox: "this implies we will need to
reprocess all our data into shards"): re-ingest required. Current
layout is sprayed by ingest order; post-ticket is sprayed by content
hash. Two layouts are incompatible by construction. Captured in
ticket §Migration as the load-bearing operational note.
Phases laid out (0-4: design lock → read path → ingest path →
pack-restore → corpus migration tool). Open audit-chain re-numbering
question (per-shard event_hash chains break when rows rebalance
across shards). Kept as one ticket — do-not-proliferate.
Scaffold only. No code yet — design lock first.
Next ID bumped to 000066.
Dav1d's reviews of #000061 (Response A + Response B/FINAL in
~/Downloads, 2026-05-26) flagged a long list of items — most already
shipped in the SPV-split work. Three were genuine gaps worth folding
into #000061 before close:
Gap 1: manifest/latest pointer for new-peer discovery.
A fresh peer doing `cold list` got a list of metadata-pack hashes
but no obvious "which one is current for shard X." Added
get_latest_pointer + update_latest_pointer to the backend ABC.
push_pack writes manifest/latest.json on every successful metadata
pack push (read-modify-write keyed by snapshot_root). Mutable
pointer; content addressing of the packs themselves preserves the
trust root. Last-writer-wins on contention.
Gap 2: license_class field + producer-side refuse for public buckets.
Maps documents.source_type to a license bucket (wikipedia_cur /
textbook_tex → public_redistributable; html / grok / vcs → unknown;
anything else → unknown). Strictness order: public < unknown <
private. compute_shard_license_class() walks DISTINCT source_type
in documents. push_pack now refuses to upload if the shard's
strictest license is more restrictive than the operator's
allow_license_class (default: public_redistributable). The
metadata pack's manifest carries _license_class so consumers /
auditors can see the producer's classification without inspecting
source documents. ValueError on refusal — the bucket ACL is the
operator's call, but arborist refuses to participate in a
licensing/membership leak unless explicitly opted in.
Gap 3: cold_pending table for resumable uploads.
Killed mid-upload, push_pack left orphan multi-GB tempfiles in
/tmp with no DB trace. Added schema:
CREATE TABLE cold_pending (
tempfile_path TEXT PRIMARY KEY,
pack_hash TEXT NOT NULL,
kind TEXT NOT NULL,
backend_endpoint TEXT NOT NULL,
backend_bucket TEXT NOT NULL,
object_key TEXT NOT NULL,
started_at INTEGER NOT NULL,
state TEXT NOT NULL DEFAULT 'pending'
);
push_pack INSERTs a row before each upload + DELETEs on success.
A killed process leaves the row pointing at the orphan tempfile;
a recovery script (future) reads cold_pending, checks bucket for
the object, either deletes the row + tempfile (success was just
unreported) or re-uploads from the tempfile if it still exists.
Matches the same pattern as the audit chain — explicit state
rows beat inferring from chunks.content IS NULL.
Sibling tickets opened for the larger items the reviews flagged
(scaffold-only, no code; opening them captures the design in the
log without proliferating, per CLAUDE.md):
- #000063 Cold-object private-ciphertext mode (mesh-keyed object
keys for non-public corpora on public-read buckets). Needs mesh
group-key ABI + real non-public corpus before code.
- #000064 Cold-object operations toolkit (verify / diff / doctor /
repair-fts / gc-plan CLI + expanded audit-event taxonomy).
Bundled so the audit-event vocabulary gets one design pass.
5 new tests:
test_gap2_license_gate_refuses_unknown_class_to_public_bucket
test_gap2_license_class_in_metadata_manifest
test_gap1_latest_pointer_resolves_metadata_pack_per_snapshot
test_gap3_cold_pending_clears_on_successful_upload
test_gap3_cold_pending_records_inflight_upload
26 cold-object + 7 evict tests pass (33/33 green incl. boto3 wire).
Next ID bumped to 000065.
Live v3 SPV corpus run (bmq47x6t3) completed cleanly during this work.
Will report sizing + memory profile in the next message.
Three doc-only housekeeping items from Dav1d's de-novo reconciliation
of the architecture stack (no code changes; the active build is
#000061 cold-pack work, which is unaffected by this review):
1. Accepted-error formula. Dav1d corrected p_raw × (1-d)(1-r) to
p_raw × (1 - dr) where r is conditional on detection. Searched
docs/ and arborist/ — the wrong form does NOT appear in this
tree (it lives in the external recursive-truth-maintenance / RCO
manuscripts Dav1d cited). Nothing to fix here; recorded for the
manuscript authors.
2. #000060 H-ABCDEFG split. Folded the M/C/X axis split into the
harness scope: M (mechanism — does the substrate's plumbing work),
C (capability — does it improve task performance), X (external
adversarial — does it generalize outside author-designed fixtures).
ACCEPT requires clearing all three. Without X, the harness risks
self-validating benchmark theology — passing tests its own designers
picked. Tagged at fixture-definition time; aggregator emits per-axis
pass rates + combined ACCEPT verdict. Doesn't change the existing
BatteryResult row schema.
3. #000062 Mechanistic Witness — new scaffold-only ticket. Specifies
a content-addressed MechanisticWitnessRoot over (model_config,
capture_policy, contrastive_prompts, features/neurons, intervention
result, behavioral delta, safety policy) as a DIAGNOSTIC sidecar
feeding SelfModel + benchmark fixtures. Four hard guardrails:
diagnostic-only by default; sandbox intervention only; no production
steering without governance ACCEPT via #000060 M+C+X; feature labels
never become semantic proof. No code until a real falsifier use case
exists + guardrails are CLAUDE.md rules + #000060 harness gates
promotion. The dual-use risk (Pan et al. 2025: 0.1% MLP ablation
breaks refusal in 72B models) makes governance-first framing
load-bearing.
Next ID bumped to 000063.
No code change to arborist/. The in-flight v3 SPV corpus pack
(bmq47x6t3) continues unaffected.
v1 packs (chunks-only) were under-engineered: a new peer landing on
v1 packs would have chunk bodies indexed by leaf_hash but no documents
table, no audit chain, no merkle interior, no edges — couldn't actually
hydrate. fox: "isn't what I wanted you under engineered..."
v2 packs ship every load-bearing shard table alongside chunk bodies in
the same tar.zst:
manifest.jsonl # chunk catalog (unchanged)
tables/documents.jsonl # array-per-line columnar JSONL
tables/chunks.jsonl # without content column
tables/merkle_nodes.jsonl
tables/edges.jsonl # FAN-IN restructured
tables/audit_events.jsonl
tables/derivations.jsonl
tables/concept_relations.jsonl
tables/concept_token_idf.jsonl
tables/providence_cache.jsonl
tables/citation_aliases.jsonl
tables/term_aliases.jsonl
tables/snapshots.jsonl
tables/document_http_meta.jsonl
blobs/<hash[:2]>/<hash[2:]> # raw UTF-8 chunk bodies
Two compression strategies inside the pack:
1. Array-per-line JSONL ({"_columns": [...]} header line + ["v1","v2",...]
data lines) drops ~30% of uncompressed bytes vs object-per-row JSONL.
zstd recovers most of that on its own, but smaller uncompressed
footprint also speeds up stream-restore.
2. Edges fan-in restructure at pack-build time: 22M rows of
(src_root, edge_type, dst_root, dst_uri, anchor) → ~500k unique
(dst_uri, edge_type, anchor, dst_root) groups with src_roots as an
array. ~5-10x compressed savings on the dominant table. Reverses on
unpack into the per-edge live schema. Live queries unchanged.
NOT shipped (per-peer state): mesh_*, selfmodel_*, capital_ledger,
memory_*, controller_events, fork_score_branches, adapter_loss_reports,
falsifications, schema_meta, meta. NOT shipped (rebuildable): chunks_fts*,
documents_fts* — restored from chunks.content + documents.title on
unpack.
push_pack no longer appends `cold_pack_pushed` to the audit chain.
That event leaked into the next push's audit_events.jsonl dump and
broke the "two writers at the same corpus state produce identical
pack_hash" determinism property. The bucket/disc file IS the receipt;
the snapshot_root pinned inside the pack metadata binds it to a corpus
state. No load-bearing consumer of the audit row.
pull_pack restored to handle both v1 (chunks-only) and v2 (tables +
chunks) packs. For v2 it extracts tables/*.jsonl to a temp dir,
calls restore_shard_metadata (which INSERT OR IGNOREs into the live
schema and expands edges back to per-edge rows), then fills chunk
content for every leaf_hash in blobs/. Idempotent against populated
DBs (INSERT OR IGNORE all the way down). Self-cleaning temp dir.
Sizing measured 2026-05-26: ~2.1 GB per shard pack compressed (chunk
content 1.78 GB + metadata ~0.3 GB), ~8.5 GB total across 4 shards.
~20% more than v1 chunks-only for self-sufficient hydration.
24 cold-object + evict tests pass (+1 new test_push_pack_v2_hydrates_fresh_empty_db
that builds a pack from a populated DB and unpacks into a completely
empty DB to verify all tables restored). Full suite: 2558 passed,
28 skipped, 1 xfailed.
Ship arborist corpus state to new peers and DVD-R archival via
point-in-time tar.zst packs. One artifact serves both channels —
bucket+CDN delivery and physical-media archival.
Bucket holds packs only. Pack key = hash_leaf(manifest_bytes), so same
chunk set on two writers produces the same pack_hash and upload is
idempotent. Each pack pins the corpus snapshot_root it covers in audit
+ result body — packs are delayed snapshots, not live mirrors;
falsifications between repacks produce new pack_hashes.
stream_packs runs streaming zstd over tarfile, peeking compressed-buffer
size after each chunk via FLUSH_BLOCK (preserves dictionary). Default
cap 4_400_000_000 — 4.4 GB DVD-R safe-fit, ~6.5% buffer below the
4.7 GB marketing capacity to absorb ISO9660 overhead, growisofs
lead-in/lead-out, media variance, and drive-edge refusal. Each disc
fills to ~4.4 GB recorded data, not the ~1.5 GB an uncompressed cap
produced.
One backend class (S3CompatibleBackend via boto3 + endpoint_url) covers
AWS S3, DO Spaces, R2, B2, GCS S3-interop, MinIO. Optional dep
[object-store] = boto3>=1.34; dev extras pull moto for the wire test.
Voyeur: credentials via AWS_ACCESS_KEY_ID/_SECRET_ACCESS_KEY env or
~/.aws/credentials, never printed; only endpoint URL + bucket name
surface in logs.
CLI: arborist cold {pack,unpack,stats}. Makefile: cold-pack,
cold-pack-dvd (local-dir output for growisofs), cold-unpack, cold-stats.
Sizing for current shards (14.1M chunks, ~17 GB compressed): ~4 packs
at the default cap, ~\$0.34/mo DO Spaces storage, ~\$0.0001/fresh-peer
hydrate.
Always-on raw-UTF-8 leaf store (per ticket "Hard invariants") deferred
— packs-only for now, backfill later.
2557 passed, 28 skipped, 1 xfailed.
Address Grok's two minor-improvement flags on the docs.
New docs/diagrams/{three-layer-stack,cache-key-8dim,falsification-states}.{dot,svg,png}
embedded into docs/_source/concepts.rst — visual scaffolding for the
3-layer stack, 8-dim cache_key composition, and falsification state
machine (previously prose+tables only).
docs/_source/cookbook.rst gains a "Use arborist as a Python library"
section: open_store + ingest_documents, custom Source subclass,
audit-chain walk + verify, Merkle proof round-trip, programmatic
arborist.qa.query() with OpenAICompatibleClient + StubClient swap.
Every Python recipe smoke-tested against a scratch DB before publish.
make docs-api: 0 new warnings. make test: 2557 passed.
The §12.1 pipeline I added was second-hand from benchmark-matrix.md
and got several things wrong against the code:
- listed 4 verdicts; actual is 5 (missing FABRICATED — the
fabrication-vs-WRONG split that energy-cogs §5.5 leans on for the
qwen-fabricates / hermes-abstains finding)
- "θ=0.85" was right by accident — but it's the code-judge-pinned
_CODE_JUDGE_THETA_CONTRA constant, raised from the manifest 0.5
default after measuring 114 FPs in the 0.5-0.75 band
- omitted the short-answer entity-grounding fast path (which runs
BEFORE NLI per the 2026-05-19 Poland-Tusk smoke)
- omitted the HYBRID rescue ladder (NLI entail / entity rescue /
2026-05-21 verbatim-quote-on-topic rescue)
- conflated WRONG and FABRICATED (the subject-in-gold split is what
distinguishes "source has the topic but a different value" from
"source silent on the topic")
Rewrote §12.1 grounded in `bench/judge_code.py:judge()` (its own
docstring at line 501-528 is the truth on rule order), with file:line
citations and the verdict-mapping in full.
Also fixed a real artifact-vs-doc drift INSIDE the judge: the
module-top docstring still claimed θ_contra default 0.5 and omitted
the short-path and the WRONG/FABRICATED split. Updated to match the
authoritative judge() docstring + current code.
No behavior change — docstring + benchmarks doc only.
A stable façade so another Python app can use arborist as a
content-addressed / Merkle / audit-chained store without the CLI or a
wire protocol. Import from arborist.embed, not internal modules, so
refactors don't break embedders.
Surface: open_store(path), ingest_documents(conn, docs), search(conn, q),
plus re-exported Document/Edge/Source/Hit/IngestStats. Core only
(python+sqlite3) — no extras. _IterableSource adapts a plain doc iterable
into the Source contract.
This is the seam for using arborist as neopig's optional provenance
backend: neopig produces Documents from crawled pages, arborist gives
content-dedup (document_root) + FTS5 + an append-only audit chain
alongside neopig's existing md5/FileVault storage. Docs in
docs/embedding.md. 6 tests pin open/ingest/dedup/idempotence/edges/search.
New docs/crawler.md covering the crawl path we actually run: BFS
same-host discovery, robots/feed/sitemap handling, polite vs --fast,
the shared-session + no-HEAD + crawl-delay-fix speedups, and the
content-addressed payoff. Leads on the two store-derived diagnostics:
duplicate detection (group by document_root — body, not URI) and
partial-overlap (shared chunk leaves), plus orphan finding
(sitemap − BFS-reached) and the planned crawl-report webmaster tools.
Honest pros/cons: orphans invisible to crawl by design, single-host,
no JS execution, --fast is anti-social off your own turf.
- '.. class:: center' parsed as a Python class named 'center' (duplicate
across one-pager + two-pager) -> '.. rst-class:: center' (styling, no
object). Clears the duplicate-object-description warnings.
- two v8-consensus section underlines were shorter than their titles
('underline too short') -> extended to title length.
Pager + v8 pages now build with zero warnings.
The page was a markdown+LaTeX paper dumped into .rst — 499 errors/warnings,
big chunks rendering broken. Mechanical, content-preserving fixes:
- markdown code fences (```lang) -> RST .. code-block:: (48 blocks; the
dominant error source — RST read each ``` as an unclosed inline literal)
- display math [ ... ] -> literal blocks (34)
- code-block:: json -> text where bodies are schemas with <...>/[m,n]
placeholders, not valid JSON (10; same as the v8 fix)
- escape inline | in prose (math like |Z|, |pred(v)|) read as RST
substitution refs (37 lines; no pipe-tables in the file, so safe)
Residual 62 are scattered indented-math derivation lines RST treats as
block quotes — they still render, just warn; diminishing returns on an
advanced/optional theory page.
The simple-table (=== separators) had a wrapped multi-line cell that
docutils rejected as malformed -> the floor table rendered broken on the
live site. Converted to a list-table (alignment-proof). Also switched the
proposer-submission block from 'code-block:: json' to 'text' — it's a
schema with <...> placeholders, not valid JSON, so the json lexer warned.
Page now builds with no errors/warnings on these blocks.
The Substrate group is Dav1d's formal Merkle-AGI research that drove the
design — valuable, but it sat third in the nav and read like required
reading. IA-only fix (no content touched):
- re-caption 'Substrate' -> 'Substrate theory (advanced)' + a preamble
marking it optional (nothing in Getting started / API depends on it);
- reorder it BELOW the practical sections (now: Summary -> Getting started
-> API -> Substrate theory -> Project);
- pull 'bench' out of Substrate into Getting started (it's the practical
benchmark surface, not theory).
Practitioner path comes first; theory stays intact + credited, just last.
- Frame the solution as a Reverse RAG (Merkle Providence Reverse RAG) with
a link to the whitepaper (unfirehose.com/merkle-providence-reverse-rag-
whitepaper — note: published on unfirehose, not uncloseai).
- New differentiator: NO vector embeddings — retrieval is lexical-first
(FTS5 BM25 + Merkle), dense-vector optional + off by default; embedding
10M docs costs 10-100x more/doc + a vector index to store/maintain. A
big part of why COGS is low.
- COGS framed per 1,000 answers, labeled @ $0.33/kWh (intro + diagram cost
node). Cost node clarified: no embeddings, NO reasoning (reasoning is the
thing that would cost 4-6x, which we skip). Retrieval node + mapping
table updated to lexical-first / no vector index.
ranksep=1.2 over-stretched it vertically (1193x1483, lots of whitespace).
Drop to 0.4 + nodesep 0.25: 1202x1063, ~28% less area, no vertical sprawl,
stays roughly square. Width is floored by the widest node label; click-to-
zoom (zoom.js) covers reading detail.
Graphviz renders SVG as <object>, which swallows clicks and can't be
lightboxed. Vendored, dependency-free zoom.js converts each graphviz
<object> to a clickable <img> (crisp vector) and adds a fullscreen
overlay: click to open, scroll to zoom toward the cursor, drag to pan,
Esc / dbl-click / background-click to close. Works for any img.zoomable,
so future charts get it free. Degrades gracefully (no JS -> inline image
still renders). Wired via html_css_files / html_js_files.
TB alone left it landscape (1202x1101, h/w 0.92). Width is floored by the
widest node label (can't shrink without trimming labels), so bump
ranksep to stretch vertically: 1193x1483, h/w 1.24 — clearly top-down/
vertical for the docs column.
rankdir LR -> TB so the diagram reads top-down and fits the docs column
(roughly square ~1200x1100, fox: 'square is good') instead of a wide
horizontal strip.
Graphviz DOT diagram of the arborist pipeline as the answer to the Google
L5 system-design prompt 'design a RAG pipeline for 10M docs with zero
hallucination'. Maps our components onto the canonical 10-box RAG design
and shows the three extensions that buy zero-hallucination + near-zero
cost: deterministic verifier (not a model confidence score) -> honest
UNGROUNDED; Merkle-bound cache that skips the GPU; measured energy COGS
(~$0.07-0.16/1k answers, non-reasoning). Includes the mapping table +
the 3.47M->10M scaling math (+77GB, sourcing/storage not redesign).
Enables sphinx.ext.graphviz (SVG output) + graphviz apt package on the
RTD build; adds the page to the Summary-pages toctree. Builds clean
(page renders, DOT validates via dot -Tsvg).
Main tweet + follow-up (Merkle cache hit skips GPU, doesn't increment the
per-1k). Numbers are the measured claim_lattice figures from the energy
report; note attached to hold the arbitrage/forcing-function framing until
the value side is hardened (higher N + blinded judge).
$0.10/1k-q overstated the qwen-27B case. Honest range: ~$0.07-0.16 per
1,000 queries of GPU electricity — hermes-8B $0.07-0.09 (under a dime),
qwen-27B $0.12-0.16 (over a dime; claim_lattice dearer than quote from
more prefilled context). Fixes the §5.4 'either rig' claim.
The report is now cost AND value (quality-per-dollar), not cost-only.
§5.5 quality delta: substrate-vs-solo (code judge, n=30) on numeral +
stale fixtures, hermes-8B vs qwen-27B-nothink. Substrate lifts both
2-13x and they nearly converge (hermes 18/21, qwen 19/26 grounded-
correct) — grounding comes from retrieval+verification, not parametric
size, so the base model matters far less. Bare-model failure styles
diverge (hermes abstains, qwen fabricates). Quality-per-dollar: ~18-26/30
grounded for $0.085-0.158/1k-q.
Honest caveats recorded: CORRECT=grounded-in-2010-corpus (not current);
the stale fixture is no longer post-corpus; the qwen thinking bug (39c040c)
that voided the first run and the judge strengthening (2d31866). §1
reframed; §8 repro + §9 next updated (post-2010 fixture, SOTA judge for
residue).
fox 2026-05-21: two corpora exist (2003-05-16 + 2010); the 2010 one is
loaded in ~/.arborist/shards (verified: 866K docs/shard, has Barack
Obama / YouTube articles a 2003 dump can't). CLAUDE.md Live-endpoints and
the energy-COGS report now state 2010; fabrication-bait fixtures target
post-2010 events. Historical closed-ticket prose left as point-in-time
records.
- §2 now a two-rig table: 4090/Qwen-27B-Q4_K_XL/llama.cpp (isolated) vs
3090/Hermes-8B-FP8-Dynamic/vLLM (live/public). Spells out that the
cross-model rate gap confounds FOUR variables (params, quant/precision,
engine, GPU) — not '8B vs 27B' alone.
- §5.4 hermes results under live traffic. Method finding: the watt_bench
window integral is unusable on a contended card, but the slope
calibration survives clean (cancels the variable shared-load baseline)
— so on shared/public cards use watt_calibrate, not the window integral.
Rates: hermes 0.109/4.40 J/tok vs qwen 0.175/6.16; decode 35-40x prefill.
- Per-query DOLLARS both rigs @$0.33/kWh: a grounded substrate answer is
<$0.10 per 1000 queries (hermes $0.085, qwen $0.158); hermes ~half qwen.
- §5.2 budget confound corrected to flag Hermes-tuning honestly (was
rationalized). §9: fixed-budget apples-to-apples re-run + per-model
budget tuning added as next steps.
Full report for Dav1d: qwen-nothink on the dedicated 4090. Headline —
prefill 0.175 J/input-tok ($0.016/M) vs decode 6.157 J/output-tok
($0.564/M), decode 35x dearer per token. The substrate prefills ~6.6k
input tok/query (vs solo ~52, 127x): ~67% of its GPU energy is reading
the retrieved context, not generating. Per-query 1719 J (substrate-CL)
vs 95 J (solo) = $0.16 vs $0.009 per 1k queries; calibration predicts
measured within ~5%. Real cost is CPU latency, not GPU watts.
Documents the full methodology + the corrections that got here
(contamination -> isolation; blended mean-W -> measured states; gross
-> not-attributable; per-token -> separated input/output; len//4 ->
real usage), threats to validity, reproducibility (commits, config hash,
persisted samples), and next rigs (3090+Hermes, reasoning, prefill-cache
study).
fox 2026-05-21: 'we dont touch card during retrieval'. Retrieval + verify
are CPU/SQLite on the orchestrator; the GPU is idle (at the shared, always-
on model-resident floor) during them. So the substrate does NOT 'hold the
card' through its wall-clock — the gross window integral over-counts by
charging that always-on floor for the seconds we spend retrieving (energy
that exists regardless of the query; the card serves other traffic then).
Correct attribution: the GPU cost of a query is its GENERATION energy only
(the marginal — burst above the serving floor). Relabel: marginal is the
headline GPU COGS; gross is demoted to a reference 'window total, not
query-attributable'. Print + energy_cogs docstring + stock-v1-config doc
updated. No math change (marginal was already right) — this corrects the
narration. Measured qwen-nothink/4090: substrate GPU COGS is LOW
(claim_lattice $0.21/M-tok, quote $0.83) — its real overhead is latency
(CPU retrieval), not GPU watts.
fox 2026-05-21: compute cost-of-goods-sold by kWh vs tokens, with the
three power states (idle / warm-idle / generation) MEASURED per
card×model×server — never hardcoded (his 40/127/380 W were illustrative
of one 3090). The only operator input is --price-per-kwh (default 0.33
USD/kWh, a configurable site rate).
energy_cogs() (pure, unit-tested) decomposes measured generation energy
against the measured warm-idle baseline:
* gross — all measured joules over the window (all-in, includes the
warm-idle cost of keeping the model hot, amortized).
* marginal — joules ABOVE warm-idle: what one more request's burst
actually costs (clamped >=0).
kWh = J/3.6e6; $/1k-tok is the unit that compares to API pricing. Both
surface per cell + a COGS print line.
watt_bench's arborist arm now loads the frozen bench.stock_v1 policy
(--answer-mode, drift-guarded on non-reasoning) so cost is measured for
the SAME substrate the campaign grades. Cells record
window_start/end_unix so a post-hoc load_monitor queue-depth cross-ref
can flag organic-traffic contamination on the non-isolated single-slot
endpoints. 6 COGS tests; full suite 2534 passed.
fox 2026-05-21: characterize substrate-ON under BOTH answer shapes, so
answer_mode is a swept axis, not a single pinned value.
stock_v1.py now exposes STOCK_V1_POLICIES{quote,claim_lattice} +
STOCK_V1_GOVERNANCE_HASHES (quote 5b6ca4c5..., claim_lattice 036a4c79...),
policy_for(mode), and assert_not_drifted(mode). Shared pins (crosslang
OFF, repair OFF, quantifier dry-run, metacognition label-only,
soft-preflight OFF, claim cap 12, v2-acronym-aware) are frozen
identically across modes.
Wire the treatment arms to the pin (the consumer-side step that makes
the freeze real):
* control_ab --answer-mode {quote,claim_lattice}
* control_sweep --arborist-answer-mode {quote,claim_lattice}
Both default claim_lattice (prior behavior), call assert_not_drifted on
non-reasoning runs (halts the sweep if DEFAULT_QUERY_POLICY drifts), and
load the frozen policy_for(mode) instead of an inline
dict(DEFAULT_QUERY_POLICY, ...). Reasoning refs (phase 3) keep their
documented JSON overrides and skip the assert by design (different hash).
jaggedness is left standalone — it is a mode-agnostic retrieval
instrument, coupling it to the answer-policy freeze adds friction with no
correctness gain. Full suite 2528 passed.
Before the multi-day campaign (hermes 3090/4090 -> qwen 3090/4090 ->
reasoning variants) the substrate-ON treatment arm must NOT drift. It
previously inherited DEFAULT_QUERY_POLICY implicitly, so any mid-run
edit would silently change what 'substrate-ON' means.
bench/stock_v1.py snapshots DEFAULT_QUERY_POLICY + re-asserts the
load-bearing pins (answer_mode=quote, crosslang OFF, repair OFF,
quantifier caps dry-run, metacognition label-only, soft-preflight OFF,
claim ceiling 12, v2-acronym-aware), then hashes the whole effective
dict. assert_not_drifted() fails loudly if that hash ever changes —
re-pinning is a deliberate fox-gated V.2 bump, never silent. The whole
campaign is identified by one governance_policy_hash
(5b6ca4c5...aade4e). Non-reasoning + non-distributed are harness axes
(reasoning -> phase 3, mesh -> later fork), not policy fields.
docs/stock-v1-config.md documents V.1, substrate-OFF (control_ab arm A),
the campaign matrix, and the energy-COGS companion (#000057) — whose
power states (idle / warm-idle / generation) are MEASURED per
card+model+inference-server at runtime, never hardcoded; only $/kWh is
an operator flag.
v1 of the same-model substrate-delta harness's non-jagged metric.
For one corpus title, surface-perturb its question (numeral / accent /
hyphen / honorific / amp / brit) preserving the referent, then ask
whether retrieval surfaces the SAME target for canonical vs perturbed
phrasing. J_norm = XOR disagreement rate @k (lower = less jagged);
graded mean |Δrank| catches rank instability the binary metric misses.
Pure query --dry-run: no LLM, no verifier, no judge, no n=3 noise, no
5pp floor — the recall_at_k discipline. Reuses recall_at_k.probe +
mine_questions._surface_variant. Feeds #000012 ForkScore
ΔJaggednessReduction. A-vs-C answer-quality arm already exists under
#000057 (control_ab/control_sweep) — not rebuilt. Curvature + LLM-arm
jaggedness delta remain open (ticket §8).
make bench-jaggedness JAGGED_LIMIT=40 JAGGED_K=8
fox: a graveyard that keeps every wrong answer forever grows unbounded
like cancer and is counter to constraint-optimization. Bake in the
bound:
- earn-to-enter (recurrence-gated) — a failure earns a tombstone only
when its canonical claim-family re-fails; one-offs dropped.
- fingerprints not transcripts (UTXO-set analogy) — store
canonical_claim_family|falsifier_root|failure_class, prune bulk text.
- decay/compact to steady-state — evicts like a surface, never a core.
- off the hot path — dedicated graveyard shard, bloom-filter burden
check only.
Gossip-group falsifier admission inherits the same discipline:
difficulty-adjusted stable-rate ingestion (BTC block-rate target) +
per-window budget (#000036), enforced in #000012/mesh. BTC lesson is
bounded self-regulating ingestion, not store-everything. If it cannot
be bounded to a steady state, the graveyard is not built.
Three tickets from the 2026-05-20 Dav1dPrometheus "Protocol-Layer AGI"
working report (held outside the repo; referenced not committed):
- #000058 cache_key_9 verifier-policy mandatory-vs-legible decision +
doc reconcile. Records the five-step-#1 correction: verifier fields
already fold into governance_policy_hash, so the 9th dim is audit
legibility not a correctness gap.
- #000059 admission discipline: claim-graveyard burden-shift +
self-providence quarantine (guards the existing ingest-self-providence
self-confirmation loop).
- #000060 H-ABCDEFG same-model substrate-delta harness (jaggedness +
curvature); curvature-aware ForkScore folded into #000012 Phase 2,
not spawned as a sibling.
Reconciles CLAUDE.md cache_key invariant (8-dim -> 8 + optional 9th).
Next ID 000058 -> 000061.
The exact llama.cpp (qwen/4090) and vLLM (hermes/3090) launch commands,
recon'd 2026-05-20, so the foxhop-states salt states can be written
accurately rather than guessed. Notes the convention (salt manages the
systemd unit; engine binaries + model artifacts stay manual on
/mnt/data as documented prereqs) and the live-hermes cutover constraint
(keep >=1 hermes online; qwen is expendable).
Toward fox's next goal: score the full serving stack on quality AND
cost — {qwen, hermes} × {llama.cpp, vLLM} × {3090, 4090} × {solo,
arborist}, measuring CG% + GPU watts + CPU watts + joules/answer per
cell.
watt_bench.py — adds CpuSampler (Intel RAPL package energy via
/sys/class/powercap/intel-rapl:*/energy_uj). RAPL exposes a cumulative
microjoule counter, so energy-over-window is an end-minus-start diff
(handles wrap) — more accurate than integrating instantaneous power.
Sums multi-package. energy_uj is root-only by default (PLATYPUS /
CVE-2020-8694), so it degrades to available=False when locked;
--cpu-energy-cmd 'sudo cat {path}' supplies a privileged reader when a
sudo rule exists. Each cell now reports gpu/cpu/total joules-per-
question + gpu joules-per-token; the report records cpu_rapl_available.
Verified: graceful degradation when locked; RAPL diff math (1->4 MJ uJ
= 3.0 J, exact).
benchmark-matrix.md — expands the cost section to the full 16-cell
(model × engine × GPU × arm) design, the per-cell metric set (quality +
GPU + CPU energy), the serving-stack inventory from 2026-05-20 recon
(4090=qwen/llama.cpp, 3090=hermes/vLLM — each box has one engine + one
model today), and the buildout gap (vLLM+qwen, llama.cpp+hermes, cross-
GPU models). Notes idle-floor asymmetry (hermes/3090 ~127W vs
qwen/4090 ~20W) as a real optimizer input.
Harness is ready; the serving-config buildout + RAPL perm grant are the
remaining (ops, fox-directed) prerequisites to run the full matrix.
Two deliverables for the cost/energy axis of the constraint optimizer.
docs/benchmark-matrix.md — shareable spec of the control experiment:
the question, fixture (386 office-holder Qs with corpus-vintage gold),
the 3-model × 3-framing × 2-arm matrix (18 cells), the verdict
vocabulary + two reads (accuracy vs grounding-fidelity), the
deterministic code judge + its Opus calibration, the results-so-far
table, and the NEW cost dimension (tokens / latency / GPU watts /
joules-per-answer measured per GPU tier). Self-contained — readable
cold by David.
bench/watt_bench.py — GPU wattage harness. Samples nvidia-smi
power.draw on the inference GPU while driving a small representative
subset, reports mean/peak watts, trapezoid-integrated joules,
joules-per-question, and joules-per-token. Tags the GPU
(--gpu-label 3090|4090) so the optimizer can compare hardware tiers.
Idle-baseline sampling separates load draw from idle. Does NOT grade
(energy is independent of correctness); saves answers + per-question
timing to JSONL for a later quality-per-joule pass via
score_with_code_judge.
Designed to run ON the GPU box (the orchestrator has no GPU; the
3090/4090 live on the inference boxes). Degrades gracefully when
nvidia-smi is absent (energy fields null) so it is testable anywhere.
Verified: PowerSampler graceful degradation + trapezoid integration
(synthetic 100->200->200W over 2s = 350 J, exact).
The headline cost finding the optimizer must weight: qwen-think
reasoning = 1300-3300 tokens/answer vs qwen-nothink ~50-100 (20-50x),
for a workload where arborist+qwen-nothink already lands 82% CG. The
energy numbers will quantify whether reasoning's premium is ever
justified — grounding-fidelity per joule, not per answer.
Durable record of the #000057 sweep in the bench journal. Captures:
- The question: is Hermes-8B's confident present-day-officeholder
fabrication an 8B weakness, a framing artefact, or does retrieval
fix it? Crosses {hermes, qwen-nothink, qwen-think} × {plain,
source_relative, as_of_corpus} × {solo, arborist} on a 386-item
office-holder fixture with corpus-vintage gold.
- The judge methodology: Opus headless judge burned quota (79.5%
JUDGE_ERROR), replaced with the deterministic code judge
(bench/judge_code.py), calibrated against Opus's gradeable records
(CG agreement 13->47%, WRONG 56->89%, ABSTAINED 80->95%).
- Consolidated CG% scorecard, all arms on the identical final judge.
- Three findings:
1. Retrieval dominates — arb/qwen-nothink/plain 82% vs 7% solo;
no solo config approaches the retrieval arms.
2. Reasoning does NOT improve raw correctness — qwen-think/as_of
44% vs nothink 50%.
3. Reasoning's real cost is broken honest-abstention —
qwen-nothink/source_relative abstains 97% (clean); qwen-think
only 61%, reasoning itself into wrong parametric answers.
- Production recommendation: arborist + qwen-nothink, plain framing,
reasoning OFF (82% CG, ~0% abstain, 11% wrong-assert).
- Held cell noted: arborist+qwen-think running at write time, result
to be appended.
Bench %s are point-in-time measurements (not repo-derived counts),
so no AUTOCOUNT tags — consistent with addenda 1-7. test_doc_counts
3/3.
Opt-in third NLI inference backend (ARBORIST_NLI_BACKEND=tinygrad) in
qa/nli/shadow.py, parallel to torch/onnx-int8, behind the #000049
cage: shadow-only, never an audit_mode input, never auto-preempts the
proven path (guarded so it cannot regress torch/onnx). Loads the ONNX
export through tinygrad's frontend wrapped to the existing
model(**enc).logits contract so _nli_batch is byte-unchanged.
bench/scripts/nli_backend_ab.py + make bench-nli-backends: deterministic
A/B, gate is numerical agreement with the torch reference first,
latency second (a divergent engine = a different shadow signal = a
different nli_policy_hash). Instrument is honest — reports a requested
backend as unavailable rather than relabelling a fallback's numbers.
First CPU-smoke run already quantified that the deployed §7 #22 int8
export diverges Δmax≈0.42 from torch — the immunity property made
measurable, not a defect. Real tinygrad numbers pending a producer-box
run (tinygrad not an arborist dep; frontend op-coverage for the large
MNLI checkpoints unverified by design).
docs/onnx-vendor-capture-immunity.md: why the model-in-proof-path cage
makes the inference engine an interchangeable sidecar, never a trust
dependency — public-domain positioning capital. Indexed in CLAUDE.md.
Full suite 2498 passed (identical to baseline); 24/24 NLI tests green.
bench/control_ab.py + `make control-ab` (gated on judge-self-test
as a make dependency — instrument gate cannot be skipped). Same
model both arms; gold = target-article text by target_root; Arborist
[E…] scaffolding stripped (blinding — format can't betray the arm);
Arborist UNGROUNDED credited as honest abstention; hermetic Opus
judge; deterministic aggregate; self-auditing JSONL; threats-to-
validity printed in the report.
N=2 smoke: clean end-to-end, 0 JUDGE_ERROR — and already surfaced a
case AGAINST the treatment (solo correctly ABSTAINED; Arborist
HYBRID-WRONG). The instrument can falsify the Arborist value claim;
that is the point. n=2 proves nothing (report says so) — verdict
needs a real N.
fox ruled judge = Opus via `claude -p`. bench/judge.py:
hermetic (`env -u CLAUDECODE claude -p`, fresh process, context =
only (Q, answer, gold) — no arm label, no Arborist context, no
session), blinded-by-caller, reference-grounded against the fixed
gold (ignore parametric knowledge), structured via FINAL_VERDICT=
sentinel parsed LAST-match.
Instrument-before-experiment gate worked: first cut parsed
first-match over the model's chain-of-thought → 0/3 self-test. The
judge REASONED correctly; the parser was the defect (+ two bad test
fixtures, my error). Hardened (sentinel contract + fixed fixtures),
re-verified: `make judge-self-test` = 4/4 on known-verdict triples
via real claude -p. The make target is the precondition gate; no
control run trusts the judge until it passes.
Threat to validity recorded, not hidden: same model family judging;
mitigated (blind + no-stake + reference-grounded) not eliminated —
different-family SOTA cross-check is the only full removal.
Next: bench/control_ab.py + `make control-ab` (Hermes-solo vs
Arborist, gold=target-article text, blinded, judged) — NOT yet
built; no broken make target shipped for it.
fox: should the judge get the same docs, or the 8 retrieved/cited?
Neither. Per-arm retrieved docs (a) are undefined for Hermes-solo
(no retrieval → breaks A/B symmetry), (b) re-import circularity
(judging the treatment vs its own retrieval), (c) measure citation
self-consistency (Arborist's verifier already does that), not
truth. Judge's reference = the mined question's fixed ground-truth
target/gold, identical for both arms, independent of retrieval —
the ground-truth-carrying fixture supplies exactly this. Recorded a
distinct, non-headline faithfulness ablation (judge Arborist vs its
own context) so it is never conflated with the control delta.
Index row unchanged (skim surface; ticket body is the design log).
