Real consumer-side bottleneck for cold-pack genesis is the phase-2
chunks-content UPDATE loop: each `UPDATE chunks SET content=? WHERE
chunk_id=?` grows the row from NULL to ~500 bytes, triggering SQLite
page splits, which become ext4 metadata-journal events. With 6M
chunks × 4 parallel writers, those journal events serialize and
dominate consumer wall time (~130 of the 164-min v3-revert run).
Fix: producer dumps a synthetic `_content_size` column in chunks.jsonl
carrying the on-disk byte length of each chunk's content (constant-
time SQLite `length(content)`). Consumer's phase 1 INSERT pre-allocates
the row with `content = bytes(_content_size)` instead of letting it
default to NULL. Phase 2's UPDATE then replaces same-size bytes
in-place — no row growth, no page splits, no per-row journal events.
Implementation:
arborist/cold_pack_metadata.py
_dump_generic_table for `chunks`:
Emit synthetic `_content_size` column = length(content) at the
end of the columnar header. Underscore prefix avoids collision
with any future schema column.
_restore_routed_table:
Detect `_content_size` in the JSONL header; if present (and
table == chunks), build INSERT against [chunks_cols] + ['content']
and substitute a bytes(_content_size) placeholder for the
content position. Phase 2 UPDATE later replaces those bytes.
Forward compatibility:
- Old packs (no _content_size): consumer uses today's NULL-content
INSERT path. No behavior change.
- New packs: consumer auto-detects, uses pre-sized path.
SPV-wallet trade-off:
In just-enough mode the consumer pulls only the metadata pack so
chunks land with the placeholder bytes (NOT NULL anymore). That's
a SEMANTIC CHANGE for SPV — `chunks.content IS NULL` no longer
means "JIT-fetch later." Documented in code; if SPV-mode JIT-fetch
ever ships, it must distinguish placeholder bytes (where every
byte is 0) from real content.
New test (TestPreSizedChunks.test_chunks_content_pre_sized_after_metadata_restore):
Hydrate just-enough → chunks rows have non-NULL bytes content of
correct size. Catches the regression if a future change reverts
the placeholder logic.
Expected impact: ~50-70% reduction in phase-2 wall time. Real number
lands when the next 3090 bench-max iteration runs against re-packed
bucket. 6 cold-unpack-routed tests pass.
v4 bench (2026-05-27 03:09 UTC) measured a corrupt outcome:
chunks_fts=1,561,604 on EVERY target shard regardless of chunks
count. _pull_fts_pack_into_targets had been iterating all M targets
and INSERT'ing each fts pack's shadow tables into every one of them.
Each fts pack's chunks_fts_docsize entries reference chunk_ids that
were independently auto-assigned in its source producer shard (each
source DB has chunk_ids 1..1.56M independently). Inserting all 4
packs into all 4 targets → 4× the per-target FTS rows, pointing at
chunk_ids the target doesn't own. Body searches would return garbage.
Fix: sample one id from the fts pack's chunks_fts_docsize, look it
up in each target's chunks table, INSERT only into the target where
it's found. The other M-1 targets stay untouched and receive their
FTS data from their corresponding fts pack(s) in later iterations.
In post-reshard production topology, each producer source shard's
docs all hash to ONE consumer target, so this 1:1 mapping is exact.
The test fixture is artificial (single-shard producer with docs
hash-distributed across M=4 targets) but still validates the core
property: only one target receives FTS data; the others stay empty.
FTS5 shadow tables aren't subsettable per-row (segment data is
opaque, mixed entries for many docs in one segment) so we can't
filter FTS rows to "only chunks that exist on this target" — we
copy all-or-nothing per pack. That's why the producer's post-reshard
shape (each pack scoped to one target's docs) is the structural
prerequisite for fts packs to make sense.
New regression test
(TestFtsPackRoutingRegression.test_fts_pack_only_on_owning_target):
exactly 1 of M targets has chunks_fts_docsize > 0; the others
must have 0. Pre-fix this asserted on all-4 targets having FTS
data → failed. Post-fix passes.
Returns now include owning_target_idx for forensic visibility into
which target the fts pack landed on.
5 cold-unpack-routed tests pass.
Today's hydrate_from_metadata_pack writes every row into one shard.
With the corpus in M=4 hash-routed topology (#000065), a fresh peer
pulling packs must land each doc on shard_for_document(root, M) —
same routing function as the producer — or the consumer's M=4
ATTACH-and-route assumption is just decoration over a single-shard
reality.
Code (new entry points alongside the existing single-conn ones):
arborist/cold_pack_metadata.py
+ restore_shard_metadata_routed(targets, M, table_dir)
+ _restore_routed_table (per-document tables)
+ _restore_edges_fan_out_routed (edges by src_root)
+ _shard_idx_for_root helper (mirrors arborist.document)
+ routing rules: _ROUTED_BY_COL / _CONSOLIDATED_TO_SHARD_0
(mirror migrate.py's ROUTED_BY_DOCUMENT_ROOT / CONSOLIDATED_TABLES)
arborist/evict.py
+ hydrate_from_metadata_pack_routed(targets, backend, hash, M=, mode=)
+ _pull_pack_inner_routed (mirrors _pull_pack_inner; phase 2
chunk-body fill iterates every target shard — leaf_hash lookup
naturally hits at most one since each chunk's metadata row
landed on exactly one target during phase 1)
arborist/cli.py
arborist cold unpack
+ --hydrate-shards-dir DIR (M-aware genesis path)
+ --hydrate-M N (default 4 matches #000065)
legacy --db / --global-shards-dir path unchanged
Behaviour notes:
- FK enforcement off on target writes (cross-shard refs are valid
under hash routing, same fix as #000065 reshard executor)
- audit_events lands on target 0 (Option A consolidation)
- corpus-wide tables (snapshots / concepts / aliases / providence)
consolidate to target 0
- per-document tables route by document_root (documents,
document_http_meta, chunks, merkle_nodes)
- edges route by src_root (matches migrate.py)
- derivations route by core_root (matches migrate.py)
- existing single-conn API unchanged — callers that didn't pass a
shards-dir get the legacy single-shard behaviour
4 new tests:
test_pack_then_hydrate_routed — end-to-end pack → hydrate → assert
every doc on its hash-routed target, no doc on the wrong shard
test_corpus_shard_count_set_on_routed_hydrate — meta plumbing
test_M_mismatch_rejected
test_empty_targets_rejected
All 59 prior tests still pass.
Refactor opportunity (not taken): _ROUTED_BY_COL duplicates
migrate.py's ROUTED_BY_DOCUMENT_ROOT. A shared arborist/multi_shard.py
module would serve both reshard and graft (#000066). Left as
follow-up since the duplication is small and graft is still scaffold.
Unblocks #46 genesis on 3090: that's now a single arborist cold unpack
--hydrate-shards-dir ~/.arborist/shards --hydrate-M 4 invocation
instead of the α two-step kludge (hydrate-then-reshard).
Add PRAGMA wal_checkpoint(TRUNCATE) at two points in
_execute_all_at_once so committed WAL pages don't pin disk through
subsequent passes. Production migration on 2026-05-26 hit 7 GB free
disk (down from 89) because SQLite's auto-checkpoint can't run while
a reader cursor is open, and FTS rebuild keeps a SELECT cursor open
through 1.5M chunks per target. Across 4 targets the FTS rebuild
plus audit consolidate held ~37 GB of committed-but-unreclaimed WAL.
Manual sibling-connection wal_checkpoint(TRUNCATE) freed 27 GB
mid-migration.
Checkpoints land at:
* end of _rebuild_fts_on_target (after the SELECT cursor is
explicitly cur.close()'d so the TRUNCATE checkpoint can actually
fire — TRUNCATE/RESTART block on active readers)
* end of _consolidate_audit_chain (after the 3.47M-row giant
transaction commits, before the next phase touches the same
connection)
VACUUM is already implicitly a checkpoint, so the existing per-
target VACUUM pass continues to handle the final checkpoint
naturally.
Helper _checkpoint_truncate(conn) returns the (busy, log_frames,
checkpointed) tuple SQLite emits; for the serial executor, busy=1
is improbable since each phase finishes before moving on.
Regression test
(TestWalCheckpointing.test_no_large_wal_after_migration) asserts
no WAL file exceeds 4 MB after migration completes. Without the
checkpoint calls this would fail on real-sized corpora; with them
the test passes deterministically.
Doesn't affect the running migration (it loaded the module from
memory before this commit). Future reshards run with bounded WAL —
no near-ENOSPC scares.
The 2026-05-26 cutover crashed mid-build with sqlite3.IntegrityError
"FOREIGN KEY constraint failed" inside _route_per_doc_table on the
derivations table.
Root cause: derivations.src_root carries a FK to
documents.document_root, but under content-hash routing a derivation
row's src_root can legitimately reference a surface doc that hashes
to a DIFFERENT target shard than the derivation's core_root. The FK
is a single-shard-era guard; it must stay live for the runtime
write path (to catch typo'd inserts into the wrong shard) but must
be OFF for the migration writer which legitimately produces
cross-shard refs.
Fix: arborist/migrate.py _connect_target now applies
`PRAGMA foreign_keys = OFF` after SCHEMA_SQL executescript runs.
Schema's own `PRAGMA foreign_keys = ON` still applies to the schema
DDL pass (and runtime connect() / connect_query() still get FK=ON
since they don't touch this helper). Only the migration writer is
relaxed. Documented inline.
Regression test
(TestCrossShardForeignKeys.test_cross_shard_derivation_succeeds)
synthesizes a derivation row whose core_root and src_root hash to
different M=4 target shards, runs the migration, asserts the row
lands on core_root's target with src_root pointing cross-shard. Pre-
fix this raised IntegrityError; post-fix it passes.
Originals untouched on the production host — the executor crashed
BEFORE the atomic-promote step, so .db files are intact;
~/.arborist/shards/00X.db.new files from the failed run will be
cleared by the next attempt's "stale .new before opening" cleanup
hook (already in _execute_all_at_once).
Reshard no longer creates a parallel "shards.v2/" directory. Builds
write to "<target_dir>/00X.db.new" alongside originals; when the
executor's validation passes, each .db.new is atomically renamed to
its final 00X.db name via os.replace (POSIX-atomic per file).
For an in-place migration (target_dir == source_dir, which is the
canonical use case), the rename REPLACES the original shard files.
~/.arborist/shards/ never contains a parallel "v2" or "next" or
"backup" directory — it always holds exactly one corpus, just with
the new topology after promote completes.
Validation gate (refuses to promote on mismatch):
* --expected-row-counts <snapshot.json> threads the pre-migration
snapshot's documents/chunks/edges totals through to the executor.
* Tolerance: ±1% to absorb the dupe-collapse from INSERT OR IGNORE
on cross-shard duplicate document_roots (166 dupes measured
pre-migration; <0.005% drift).
* Mismatch → RuntimeError, .db.new files left in place for
inspection, no rename performed.
CLI:
arborist corpus reshard --to 4 \
--source-dir ~/.arborist/shards \
--target-dir ~/.arborist/shards \
--audit-events-ndjson /tmp/audit-events.ndjson \
--expected-row-counts bench/results/pre-migration-snapshot.json
3 new tests:
* test_target_dir_only_has_db_files_after_completion — no .db.new
sidecars survive a successful run
* test_inplace_reshard_overwrites_originals — target_dir ==
source_dir works end-to-end; final files are the new hash-routed
shards
* test_validation_failure_leaves_new_files — bad expected counts
trip the validation guard; .db.new files survive; no .db files
promoted
Stale .db.new files from a prior failed run are unlinked before
opening fresh targets, so a partial-fail-and-retry is idempotent.
WAL/SHM sidecars removed at promote time so the new live .db
produces fresh sidecars on next open.
Build the 'all_at_once' executor and wire it into the CLI as
`arborist corpus reshard`. Plan-only preview against fox's host
confirms the planner picks all_at_once at 95.5 GB free, 64.2 GB
peak draw, 31.3 GB free at peak (well above the 4 GB safety).
Executor (arborist/migrate.py):
* ROUTED_BY_DOCUMENT_ROOT — documents, document_http_meta, chunks,
merkle_nodes, edges (by src_root), derivations (by core_root).
Each row hashed to shard_for_document(root, M) and INSERT'd into
the chosen target.
* CONSOLIDATED_TABLES — snapshots, concept_relations,
concept_token_idf, citation_aliases, term_aliases,
providence_cache, falsifications all land on canonical
target shard 000.
* REBUILT_ON_TARGET — chunks_fts + documents_fts rebuilt from the
materialized data after content moves. chunks.content is
zstd-packed at rest so the rebuilder decompresses via
arborist.compress.unpack_chunk before inserting plaintext into
the FTS5 index.
* Audit chain consolidation (Option A) — all rows from
/tmp/audit-events.ndjson are sorted by (ts, src_shard, src_seq),
re-chained with fresh event_hash values, and INSERT'd into
target shard 000. Bodies preserved unchanged for forensic
fidelity.
* One 'reshard' audit event appended at the tail, carrying the
plan + result as the body. An operator months later can answer
"where did this corpus topology come from" from this one row.
* corpus_shard_count meta stamped on every target shard.
* VACUUM each target at end to reclaim INSERT-pattern fragmentation.
CLI: `arborist corpus reshard --to M --source-dir DIR --target-dir DIR
[--plan-only | --force-strategy X
| --allow-in-place | --dry-run]`
11 integration tests build a tiny 2-shard corpus, run the migration,
verify per-doc routing, chunk-follows-doc, audit chain integrity,
reshard event at tail, corpus_shard_count meta on every target,
chunks_fts searchability, doc count preservation, dry-run no-op,
error handling.
Also: get_meta() now indexes by position so callers without
row_factory=sqlite3.Row don't trip. No behavior change for callers
that DO use Row factory.
Strategy B (per_source_shard) and C (streaming_row) are stubbed in
the planner (peak-draw estimators wired) but execute_plan raises
NotImplementedError for them. Not needed at fox's current disk
(88+ GB free); skipping the build until that's ever the constrained
path.
Disk-aware strategy picker for the corpus reshard. Pure-compute API:
feed it (corpus facts, free bytes) and it returns a HydrationPlan
naming one of three strategies + a peak-draw estimate.
Strategies (preference order):
all_at_once - direct source→target with overlap; fastest;
peak draw = corpus + WAL × M_target + VACUUM
per_source_shard - pack→delete→hydrate per round; safest;
peak draw = pack + WAL + 1 target VACUUM
streaming_row - in-place mutation, --allow-in-place opt-in;
peak draw = WAL only
Safety budget: 4 GB margin above the strategy's peak draw — the
buffer that survives one bad sort + one badly-sized WAL grow.
Production-host preview at 95 GB free / 38 GB corpus:
strategy: all_at_once
peak draw est: 64.2 GB
free at peak: 31.2 GB (well above safety)
rationale: all_at_once: peak draw 64.2 GB + 4.3 GB safety ≤ free 95.4 GB
The plan's full readout (strategy, free bytes, peak draw, rationale)
is JSON-serializable so it goes into the migration audit event as a
single forensic record.
14 tests cover: strategy pick by free-disk level, --force_strategy
override, --allow-in-place gating, skewed shard sizes, json-
serializable audit body, target_M validation.
No execution yet — this is just the planner. Next: the strategy A
executor (direct source→target read+write).
Three pieces, all read-only or additive — no shard mutation, no
schema-version bump:
1. shard_for_document(document_root, M) in arborist/document.py.
Pure function: int(document_root[:8], 16) % M. 22 tests cover
determinism, range-bounds, near-uniform distribution (±5pp at
N=20k), and seven lock-in fixtures so peers will disagree
loudly if anyone changes the formula.
2. corpus_shard_count meta field + get/set helpers in store.py.
Lives in the existing key/value meta table; SCHEMA_VERSION
stays at v9.8.0 (the DDL doesn't change and source_root is
layout-independent, so cache records survive a reshard).
Legacy shards (without the field) return None; reshard tool
populates it on every target shard at migration time.
3. Pre-migration snapshot captured to
bench/results/pre-migration-snapshot.json:
docs 3,468,392 (3,468,226 globally unique)
chunks 6,235,764
edges 90,593,537
audit 3,468,403
This is the reference set post-reshard row counts must match.
4. Audit-event extraction script writes all 3.47M events from
all 4 shards to /tmp/audit-events.ndjson (2.0 GB) for the
Option-A canonical-chain consolidation step. Verifies chain
integrity on extract — all 4 source chains report 0 breaks.
5. Fixed a wrong chunk count in docs/corpus-history.md
(had ~3.54M/shard; actual is ~1.56M/shard) and added the
edge-count column (~22.6M/shard, 90.6M total). 6.24M chunks
total, not 14.12M.
Tests: 29 new pass (22 routing + 7 meta). No existing tests
touched.
Dav1d's reviews of #000061 (Response A + Response B/FINAL in
~/Downloads, 2026-05-26) flagged a long list of items — most already
shipped in the SPV-split work. Three were genuine gaps worth folding
into #000061 before close:
Gap 1: manifest/latest pointer for new-peer discovery.
A fresh peer doing `cold list` got a list of metadata-pack hashes
but no obvious "which one is current for shard X." Added
get_latest_pointer + update_latest_pointer to the backend ABC.
push_pack writes manifest/latest.json on every successful metadata
pack push (read-modify-write keyed by snapshot_root). Mutable
pointer; content addressing of the packs themselves preserves the
trust root. Last-writer-wins on contention.
Gap 2: license_class field + producer-side refuse for public buckets.
Maps documents.source_type to a license bucket (wikipedia_cur /
textbook_tex → public_redistributable; html / grok / vcs → unknown;
anything else → unknown). Strictness order: public < unknown <
private. compute_shard_license_class() walks DISTINCT source_type
in documents. push_pack now refuses to upload if the shard's
strictest license is more restrictive than the operator's
allow_license_class (default: public_redistributable). The
metadata pack's manifest carries _license_class so consumers /
auditors can see the producer's classification without inspecting
source documents. ValueError on refusal — the bucket ACL is the
operator's call, but arborist refuses to participate in a
licensing/membership leak unless explicitly opted in.
Gap 3: cold_pending table for resumable uploads.
Killed mid-upload, push_pack left orphan multi-GB tempfiles in
/tmp with no DB trace. Added schema:
CREATE TABLE cold_pending (
tempfile_path TEXT PRIMARY KEY,
pack_hash TEXT NOT NULL,
kind TEXT NOT NULL,
backend_endpoint TEXT NOT NULL,
backend_bucket TEXT NOT NULL,
object_key TEXT NOT NULL,
started_at INTEGER NOT NULL,
state TEXT NOT NULL DEFAULT 'pending'
);
push_pack INSERTs a row before each upload + DELETEs on success.
A killed process leaves the row pointing at the orphan tempfile;
a recovery script (future) reads cold_pending, checks bucket for
the object, either deletes the row + tempfile (success was just
unreported) or re-uploads from the tempfile if it still exists.
Matches the same pattern as the audit chain — explicit state
rows beat inferring from chunks.content IS NULL.
Sibling tickets opened for the larger items the reviews flagged
(scaffold-only, no code; opening them captures the design in the
log without proliferating, per CLAUDE.md):
- #000063 Cold-object private-ciphertext mode (mesh-keyed object
keys for non-public corpora on public-read buckets). Needs mesh
group-key ABI + real non-public corpus before code.
- #000064 Cold-object operations toolkit (verify / diff / doctor /
repair-fts / gc-plan CLI + expanded audit-event taxonomy).
Bundled so the audit-event vocabulary gets one design pass.
5 new tests:
test_gap2_license_gate_refuses_unknown_class_to_public_bucket
test_gap2_license_class_in_metadata_manifest
test_gap1_latest_pointer_resolves_metadata_pack_per_snapshot
test_gap3_cold_pending_clears_on_successful_upload
test_gap3_cold_pending_records_inflight_upload
26 cold-object + 7 evict tests pass (33/33 green incl. boto3 wire).
Next ID bumped to 000065.
Live v3 SPV corpus run (bmq47x6t3) completed cleanly during this work.
Will report sizing + memory profile in the next message.
Bidirectional sync: producer always emits both kinds; consumer chooses
how much to pull.
packs/<hash>.metadata.tar.zst — one per shard (~0.6 GB compressed)
packs/<hash>.metadata.manifest.ndjson
packs/<hash>.chunks.tar.zst — N per shard (each ≤ 4.4 GB cap)
packs/<hash>.chunks.manifest.ndjson
Each artifact is independently content-addressed by its own manifest
hash. The metadata pack's manifest carries _chunk_pack_hashes — every
chunk pack covering this shard's content — so a "full" consumer can
iterate them. Chunks packs are anonymous from the consumer side
(reachable only via the metadata pack's reference list).
Consumer sync modes:
arborist cold unpack <metadata_hash> # default: just-enough
arborist cold unpack <metadata_hash> --full # also pulls chunks
just-enough: pull only the metadata pack. Schema fully restored; every
chunks row has content=NULL. Node is immediately queryable for
metadata operations (documents, edges, audit chain, Merkle); chunk-body
queries return null until a future JIT-fetch path fills them on cache
miss.
full: after the metadata pack lands, iterate _chunk_pack_hashes and pull
every chunk pack. Final state: full corpus offline-queryable.
This is fox's original SPV-wallet framing — was right from day one.
Key API changes:
pack_key(hash, *, kind="chunks", manifest=False) — kind in bucket path
stream_packs(chunks, *, max_compressed_bytes, ...) -> Iterator[FilePack]
now emits chunk-only packs (no extra_members)
build_metadata_pack(table_files, *, snapshot_root, chunk_pack_hashes, ...)
single FilePack with v3 metadata manifest
pull_metadata_pack(conn, backend, hash) -> dict
pull_chunk_pack(conn, backend, hash) -> dict
hydrate_from_metadata_pack(conn, backend, metadata_hash, *, mode) -> dict
push_pack orchestrates: dump tables → stream chunk packs (collect hashes)
→ build metadata pack with chunk_pack_hashes → upload all. Returns
{metadata_pack_hash, chunk_pack_hashes, packs: [...]}.
Manifest format v3:
metadata pack:
{"_format_version": 3}
{"_kind": "metadata"}
{"_snapshot_root": "..."}
{"_chunk_pack_hashes": [...]}
{"table_file": "tables/X.jsonl", "hash": "...", "size": N} per table
chunks pack:
{"_format_version": 3}
{"_kind": "chunks"}
{"leaf_hash": "...", "size": N} per chunk
pack_hash for each = hash_leaf(manifest_bytes); content-addressed at
both layers. Two writers with the same shard state produce identical
metadata_pack_hash AND identical chunk_pack_hashes.
Cap-and-split applies only to chunk packs (chunks are bounded N).
Metadata pack is one file per shard; if a single table file is larger
than the cap, that's noted as future row-level split work.
cold list now surfaces kind per artifact (metadata vs chunks), plus
table_count + chunk_pack_hashes (for metadata packs) or chunk_count
(for chunks packs). Operators can quickly find the metadata pack hash
to feed `cold unpack`.
28 cold-object + evict + boto3 tests pass (3 new SPV-shape tests + 1
v3 manifest-shape test + tampered-metadata-pack test).
Caught a real defect on the live v2 corpus run: pack_hash was computed
from the chunk-only manifest, so two packs with the same chunk set but
different table content collided on pack_hash. Observed live: v1 packs
and v2 packs for the same shard produced the SAME pack_hash:
v1 (chunks-only) b3c427baaf9b40e3c33ace8438b4254e3abdc7b38bdcb7b1a8012588cb32848a
v2 (tables + chunks) b3c427baaf9b40e3c33ace8438b4254e3abdc7b38bdcb7b1a8012588cb32848a
↑ identical, different bytes
Consequences if shipped: bucket overwrites swap whose tables you get,
mesh peers A and B disagree on what "pack X" is while both claim to
have it, content-addressing story is broken for the metadata half.
Fix: manifest format v2 prepends a `{"_format_version": 2}` header,
then one row per shipped table:
{"table_file": "tables/audit_events.jsonl", "hash": "...", "size": N}
{"table_file": "tables/chunks.jsonl", "hash": "...", "size": N}
...
{"leaf_hash": "...", "size": N} # chunks come after
{"leaf_hash": "...", "size": N}
pack_hash = hash_leaf(manifest_bytes) now binds the full pack content.
Two writers with the same shard state produce the same pack_hash; same
chunks + different tables produce different pack_hashes.
Implementation:
- hash_file_leaf(path) — streaming sha256 with leaf 0x00 prefix, 1 MB
reads, used to fingerprint multi-GB tables/<name>.jsonl files without
loading them into RAM.
- stream_packs' extra_members now takes (member_name, source_path,
content_hash) tuples. push_pack computes the hash with hash_file_leaf
after the dump finishes.
- _finalize_pack writes the format header + table refs (sorted by name
for determinism) before chunk entries.
- parse_manifest returns a ParsedManifest (format_version, tables[],
chunks[]) instead of just list[PackEntry]. Backward-compatible with
v1 manifests (no header → format_version=1, tables=()).
- pull_pack pulls the manifest aside during tar walk, then after
extraction verifies each tables/<name>.jsonl file via hash_file_leaf
against the manifest's content_hash. Mismatch → ValueError, no rows
reach the live schema.
New tests:
- test_v2_pack_hash_binds_table_contents — asserts manifest format +
table refs + per-table hash & size.
- test_pull_pack_rejects_tampered_table — corrupts one table's bytes
in a real pack, confirms pull_pack raises on hash mismatch.
27 cold-object + evict tests pass.
Live v2 corpus run showed ~5 GB RSS per worker — RssAnon dominant, so
process heap, not mmap. Traced to two unfixed memory pits in the edges
fan-in dump path:
1. SQLite ORDER BY on edges (22M rows, no covering index for the v2
sort order dst_uri+edge_type+anchor) allocates a multi-GB in-memory
sort area before spilling. Adding:
CREATE INDEX IF NOT EXISTS idx_edges_dst_uri_type_anchor
ON edges(dst_uri, edge_type, anchor, dst_root, src_root)
means the ORDER BY walks the index in order — no in-memory sort.
First create takes ~30-60 s on a 22M-row shard; idempotent on
subsequent dumps. Disk cost ~1 GB per shard (4 shards × 1 GB ≈
2-3 % corpus footprint increase). Worth it.
2. Python groupby accumulator: src_roots = [row[4] for row in group]
materializes the entire src_root list per destination. For
en.wikipedia.org/wiki/* destinations with millions of inbound links,
this list is itself ~GB-sized. Switch to bounded batches:
_FAN_IN_BATCH = 10_000 # max src_roots per fan-in JSON row
A destination with N inbound links splits into ceil(N / batch) rows.
Restore path (INSERT OR IGNORE) handles multi-row destinations
correctly because PK includes src_root — accidental duplicates
collapse cleanly.
New regression test test_edges_fan_in_batches_huge_destinations builds
an edges table with FAN_IN_BATCH+137 rows pointing at one dst_uri,
verifies the dump produces the expected number of split rows and the
restore reconstructs all N edges with no loss or duplication.
25 cold-object + evict tests pass.
v1 packs (chunks-only) were under-engineered: a new peer landing on
v1 packs would have chunk bodies indexed by leaf_hash but no documents
table, no audit chain, no merkle interior, no edges — couldn't actually
hydrate. fox: "isn't what I wanted you under engineered..."
v2 packs ship every load-bearing shard table alongside chunk bodies in
the same tar.zst:
manifest.jsonl # chunk catalog (unchanged)
tables/documents.jsonl # array-per-line columnar JSONL
tables/chunks.jsonl # without content column
tables/merkle_nodes.jsonl
tables/edges.jsonl # FAN-IN restructured
tables/audit_events.jsonl
tables/derivations.jsonl
tables/concept_relations.jsonl
tables/concept_token_idf.jsonl
tables/providence_cache.jsonl
tables/citation_aliases.jsonl
tables/term_aliases.jsonl
tables/snapshots.jsonl
tables/document_http_meta.jsonl
blobs/<hash[:2]>/<hash[2:]> # raw UTF-8 chunk bodies
Two compression strategies inside the pack:
1. Array-per-line JSONL ({"_columns": [...]} header line + ["v1","v2",...]
data lines) drops ~30% of uncompressed bytes vs object-per-row JSONL.
zstd recovers most of that on its own, but smaller uncompressed
footprint also speeds up stream-restore.
2. Edges fan-in restructure at pack-build time: 22M rows of
(src_root, edge_type, dst_root, dst_uri, anchor) → ~500k unique
(dst_uri, edge_type, anchor, dst_root) groups with src_roots as an
array. ~5-10x compressed savings on the dominant table. Reverses on
unpack into the per-edge live schema. Live queries unchanged.
NOT shipped (per-peer state): mesh_*, selfmodel_*, capital_ledger,
memory_*, controller_events, fork_score_branches, adapter_loss_reports,
falsifications, schema_meta, meta. NOT shipped (rebuildable): chunks_fts*,
documents_fts* — restored from chunks.content + documents.title on
unpack.
push_pack no longer appends `cold_pack_pushed` to the audit chain.
That event leaked into the next push's audit_events.jsonl dump and
broke the "two writers at the same corpus state produce identical
pack_hash" determinism property. The bucket/disc file IS the receipt;
the snapshot_root pinned inside the pack metadata binds it to a corpus
state. No load-bearing consumer of the audit row.
pull_pack restored to handle both v1 (chunks-only) and v2 (tables +
chunks) packs. For v2 it extracts tables/*.jsonl to a temp dir,
calls restore_shard_metadata (which INSERT OR IGNOREs into the live
schema and expands edges back to per-edge rows), then fills chunk
content for every leaf_hash in blobs/. Idempotent against populated
DBs (INSERT OR IGNORE all the way down). Self-cleaning temp dir.
Sizing measured 2026-05-26: ~2.1 GB per shard pack compressed (chunk
content 1.78 GB + metadata ~0.3 GB), ~8.5 GB total across 4 shards.
~20% more than v1 chunks-only for self-sufficient hydration.
24 cold-object + evict tests pass (+1 new test_push_pack_v2_hydrates_fresh_empty_db
that builds a pack from a populated DB and unpacks into a completely
empty DB to verify all tables restored). Full suite: 2558 passed,
28 skipped, 1 xfailed.
New peers landing on a public bucket need to enumerate pack_hashes
before they can `arborist cold unpack <hash>`. `cold stats` only gives
a count; `cold list` returns the per-pack details:
- pack_hash (taken from the key)
- compressed_bytes (one HEAD round-trip via new object_size method)
- chunk_count (parsed from the small manifest sidecar; skippable
via --no-manifest for huge-bucket fast listing)
`cold stats` also now reports total bucket footprint (sums HEAD sizes).
Backend ABC gains `object_size(key) -> int | None` so both subcommands
get sizes without paying egress for the body. S3 impl uses HEAD;
MemoryBackend reads from the dict.
Verified live against DO Spaces NYC3: list-empty → push tiny pack →
list-with-manifest (pack_hash, size=5311 B, chunk_count=5) → list
--no-manifest → cold stats → cleanup.
23 passed in tests/test_cold_object.py + tests/test_evict.py.
Three improvements after the first DO Spaces smoke + bench:
1. ORDER BY c.leaf_hash on the chunk-selection SQL. Two writers running
cold pack against the same DB at the same snapshot now produce the
same pack_hashes — chunk-to-pack assignment is a function of (chunk
set, cap) and nothing else. Prerequisite for parallel per-shard pack
workers and for two replicas to converge on byte-identical bucket
state. Costs ~25% on build wall (real-bench 31s → 40s on 100k chunks)
due to sort over the leaf_hash index + documents JOIN; worth it.
New test pins the determinism property.
2. boto3 multipart upload via TransferConfig (8 MB threshold + 8 MB
parts + 10-way concurrency) on every put. Required anyway for packs
> 5 GB (DO Spaces single-PUT limit). Measured 5.5 MB/s → 9.0 MB/s
on 121 MB pack to DO Spaces NYC3 (1.6x; ceiling is closer to network
than to boto3 serialization).
3. Stream the SQL cursor in push_pack instead of fetchall(). At 14M
chunks × ~700 bytes/row the prior fetchall materialized ~10 GB of
Python heap before stream_packs ever ran. Cursor iteration bounds
memory by the in-progress pack (~few hundred MB at the 4.4 GB cap).
Full corpus extrapolation revises ~125 min (single-PUT) → ~104 min
(multipart, sequential per-shard). Real wins live in parallel per-shard
pack workers — deferred; the determinism work landed here is the
prerequisite.
22 passed in tests/test_cold_object.py + tests/test_evict.py.
Ship arborist corpus state to new peers and DVD-R archival via
point-in-time tar.zst packs. One artifact serves both channels —
bucket+CDN delivery and physical-media archival.
Bucket holds packs only. Pack key = hash_leaf(manifest_bytes), so same
chunk set on two writers produces the same pack_hash and upload is
idempotent. Each pack pins the corpus snapshot_root it covers in audit
+ result body — packs are delayed snapshots, not live mirrors;
falsifications between repacks produce new pack_hashes.
stream_packs runs streaming zstd over tarfile, peeking compressed-buffer
size after each chunk via FLUSH_BLOCK (preserves dictionary). Default
cap 4_400_000_000 — 4.4 GB DVD-R safe-fit, ~6.5% buffer below the
4.7 GB marketing capacity to absorb ISO9660 overhead, growisofs
lead-in/lead-out, media variance, and drive-edge refusal. Each disc
fills to ~4.4 GB recorded data, not the ~1.5 GB an uncompressed cap
produced.
One backend class (S3CompatibleBackend via boto3 + endpoint_url) covers
AWS S3, DO Spaces, R2, B2, GCS S3-interop, MinIO. Optional dep
[object-store] = boto3>=1.34; dev extras pull moto for the wire test.
Voyeur: credentials via AWS_ACCESS_KEY_ID/_SECRET_ACCESS_KEY env or
~/.aws/credentials, never printed; only endpoint URL + bucket name
surface in logs.
CLI: arborist cold {pack,unpack,stats}. Makefile: cold-pack,
cold-pack-dvd (local-dir output for growisofs), cold-unpack, cold-stats.
Sizing for current shards (14.1M chunks, ~17 GB compressed): ~4 packs
at the default cap, ~\$0.34/mo DO Spaces storage, ~\$0.0001/fresh-peer
hydrate.
Always-on raw-UTF-8 leaf store (per ticket "Hard invariants") deferred
— packs-only for now, backfill later.
2557 passed, 28 skipped, 1 xfailed.
A stable façade so another Python app can use arborist as a
content-addressed / Merkle / audit-chained store without the CLI or a
wire protocol. Import from arborist.embed, not internal modules, so
refactors don't break embedders.
Surface: open_store(path), ingest_documents(conn, docs), search(conn, q),
plus re-exported Document/Edge/Source/Hit/IngestStats. Core only
(python+sqlite3) — no extras. _IterableSource adapts a plain doc iterable
into the Source contract.
This is the seam for using arborist as neopig's optional provenance
backend: neopig produces Documents from crawled pages, arborist gives
content-dedup (document_root) + FTS5 + an append-only audit chain
alongside neopig's existing md5/FileVault storage. Docs in
docs/embedding.md. 6 tests pin open/ingest/dedup/idempotence/edges/search.
Full --fast crawl of russell.ballestrini.net (242 URIs): 26s -> ~5s.
Three changes, biggest first:
1. fast_mode now actually ignores crawl-delay (the ~5x). The delay was
only zeroed on the robots-200 path; a site with no robots.txt (404)
or a robots fetch error fell back to default_crawl_delay (2s). Under
--fast that made every concurrent fetch wave sleep ~2s — ~10 waves
x 2s dominated the wall time. _enforce_crawl_delay now short-circuits
when fast_mode, matching the documented "ignore crawl-delay"
contract regardless of robots status. Disallow is still honored
(separate path).
2. One shared ClientSession for the fetcher's lifetime (keepalive TCP
connector sized to page-worker width) instead of a fresh session per
fetch — ~3x on a 24-page wave. Lazily built in-loop via _get_session;
the bridge closes it in a finally (guarded on owning the fetcher).
3. Drop the per-page preflight HEAD. aiohttp exposes response headers
before the body is read, so the existing content-type binary guard
skips images/video/audio without downloading them — the HEAD was a
redundant round trip that doubled per-page latency.
Diverges arborist's AsyncWebFetcher from the agents.ai.unturf.com/core
verbatim lift (fox-approved); candidate to upstream. Regression tests
pin fast=no-delay / polite=delay, shared-session lifecycle, and bridge
session teardown (owned vs injected).
Two crawler-discovery changes surfaced while chasing fast-crawl wall
time on russell.ballestrini.net:
1. Feed-skip in BFS discovery: the bridge fetched feed/sitemap URLs
(a multi-MB atom.xml among them) only for ingest_crawled to discard
them. Gate enqueue on the existing _looks_like_feed_url so we never
fetch crawl-infrastructure URLs — less wasted work and one fewer
slow wave straggler.
2. lxml link extraction, DRY'd: the three duplicated BeautifulSoup
html.parser closures (fresh fetch + 2 cache paths) collapse into one
module-level extract_page_links() backed by lxml.html (C parser,
releases the GIL so to_thread actually parallelises) with a BS4
fallback for markup lxml rejects. Parse on a 24-page wave 3.5s->2.5s.
Honest scope: neither moves full-crawl wall time much — measurement
showed the dominant cost is the per-page HEAD+GET double round-trip on
a per-call ClientSession, not parsing. These are correct-and-cleaner
on their own; the wall-time lever (shared session + drop redundant
HEAD) is a separate change. lxml extraction is regression-pinned
against the BS4 fallback for parity.
The bridge BFS fetched pages one-at-a-time, so --fast only dropped the
crawl-delay (sequential, zero-wait). Fast_mode's CPU*3 page-worker
budget never reached the path operators actually run.
Replace the popleft loop with a wave loop: each iteration pulls up to
`fetcher.max_page_workers` URLs off the queue front and fetches them
with asyncio.gather. Width is CPU*3 under fast_mode, 1 otherwise, so
the polite path stays byte-for-byte sequential and the per-page
crawl-delay still serialises same-domain fetches. Wave size is capped
to the remaining max_pages budget; dedup moves from pop-time to
enqueue-time so a URL linked from two parents in one wave is fetched
exactly once.
Measured on russell.ballestrini.net (own host, robots 404): same
12-page work 23.1s polite -> 4.0s fast (5.7x); full 243-page crawl
~25s vs the ~486s polite floor (19x). Disallow still honored; only
the rate limit is lifted.
Tests: peak-in-flight pins (>1 fast, ==1 polite) plus all existing
BFS bound / dedup / depth / max-pages cases on the width=1 path.
When a re-crawl detects a real content delta (a just-ingested root that
supersedes a prior version — content hash changed, not redeploy/ETag
noise the idempotent ingest already no-op'd), the pipeline now surfaces
the page's document chain over time instead of just 'something changed'.
bridge.py: version_chain(conn, uri) walks a URI's documents by ingest_ts
(each content change = new content-addressed doc + supersedes edge);
delta_report() adds the word-level similarity of the latest change;
render_delta_report() prints it. ingest_crawled() detects superseding
roots, emits the lineage report to stderr per changed page, and returns
'deltas' in its summary. Validated on the live russell.ballestrini.net
re-crawl: 223 pages, full redeploy, exactly 1 content change (/about/),
rendered as a 2-version chain (90% similar to prior). 2 tests; suite
2551 passed.
The code judge bailed to JUDGE_ERROR on 40% of in-corpus answers: HYBRID
(partial grounding) with low NLI entail, where the entity-grounding
rescue needs ZERO unsourced specifics. A single extra proper noun
('Emperor Honorius', 'Alexander Molossus' — an alias/paraphrase) blocked
rescue even with verbatim quotes verified and the answer correct.
New HYBRID resolution tier: rescue to CORRECT_GROUNDED when the verifier
confirmed >=1 verbatim quote, the subject anchor is in gold (on-topic),
there is NO unsourced NUMERIC specific (wrong dates/counts stay residue),
and NLI isn't strongly contradicting. Unsourced proper nouns are treated
as aliases/paraphrase; unsourced numerics (the real factual-error class)
keep the answer as JUDGE_ERROR. Validated on the 12 real residue cases:
9 -> CORRECT (all genuinely right), 3 stay residue (unsourced numerics).
JUDGE_ERROR 40% -> ~10%. self-test 4/4; 2 new tier tests; suite 2549.
Root cause of 'arborist abstains on everything with qwen' (fox 2026-05-21):
Qwen3 thinking-on default burns the entire token budget on hidden <think>
reasoning over a 20K RAG context and returns EMPTY message.content
(measured: 768/768 completion tokens, content '') -> every arborist answer
UNGROUNDED. Bench harnesses passed enable_thinking=False via the MODELS
dict, but the CLI + control_ab did not, so the quality bench was measuring
a thinking-budget-exhaustion artifact, not abstention.
OpenAICompatibleClient now defaults Qwen3 to enable_thinking=False unless a
caller set it explicitly (reasoning-variant path passes True, preserved).
Verified: same France query goes empty/UNGROUNDED -> STRICT 'Nicolas
Sarkozy' with the flag. Fixes every caller (CLI, control_ab). 5 tests;
full suite 2547 passed. Today's qwen QUALITY numbers are void and need
re-running; energy numbers stand (real inference happened regardless).
fox 2026-05-21: account wattage for input and output separately. Prefill
(process all prompt tokens, parallel/compute-bound) and decode (generate
output, autoregressive/bandwidth-bound) are different GPU ops with
different J/token — a single per-token number can't represent both.
Slope calibration (no sub-request power alignment): sweep prompt length
at tiny max_tokens -> prefill J/input-tok (fixed overhead cancels in the
slope); fix a tiny prompt and sweep forced output length (ignore_eos) ->
decode J/output-tok. Prefill kept COLD (unique filler so cached_tokens=0).
Reuses watt_bench probes. Bad points (context overflow) skip, not abort.
Measured qwen-nothink/4090 @$0.33/kWh: prefill 0.175 J/tok
($0.016/M-input-tok), decode 6.16 J/tok ($0.564/M-output-tok) — decode
35x dearer per token. Predicts measured substrate J/q within ~5%. 14
tests (+ slope). Validated live.
fox 2026-05-21: (1) use REAL API token usage, not len//4; (2) the
substrate prefills a large retrieved CONTEXT as INPUT while solo feeds
~nothing, so per-completion-token over-charges the substrate — and per-
TOTAL-token UNDER-charges it (its mix is ~98% cheap prefill tokens).
Measured n=30 qwen-nothink/4090: substrate prefills ~6.6k input tok/query
(claim_lattice) vs solo ~52 — ~127x. Neither single per-token denominator
is honest; prefill (parallel, cheap/tok) and decode (autoregressive,
dear/tok) must be costed separately.
- OpenAICompatibleClient stashes data['usage'] as .last_usage (non-
invasive; return type unchanged).
- watt_bench captures real prompt_tokens + completion_tokens per call
(both arms), aggregates per cell, and energy_cogs reports gross +
marginal per BOTH 1k-total-tok and 1k-completion-tok plus the context
size. Prints the prompt/completion split.
- 12 tests incl. the prompt-context artifact (per-total cheap, per-
completion dear). Full suite 2540 passed.
The clean per-input-tok / per-output-tok split rides bench/watt_calibrate
(slope calibration; separate commit once validated live).
fox 2026-05-21: 'gen 200W' was a bug — joules/window blends the ~400W
generation bursts with the sub-100W gaps (retrieval/verify/network) into
a power state the card never sits at. A card occupies DISTINCT states
(idle / middle-idle = resident-between-requests / generation), differing
per card×model×server.
watt_probe.classify_power_bands(): largest-gap split of the window
samples into a low band (serving floor) and high band (generation draw)
+ duty cycle. Data-derived, never hardcoded — tested at two scales. The
worker emits the decomposition + raw samples; RemoteProbe/LocalProbe
expose band_stats() uniformly.
energy_cogs: marginal now taken against the measured SERVING FLOOR (the
standing cost of being ready), not deep idle; the blend is kept but
labelled window_mean_w. Reports idle/serving-floor/gen-draw/duty.
Cache-miss certainty (fox's question): the arborist arm runs
burn_existing=True (force-deletes any live providence row before
inference) and asserts cache_hits==0 with a loud warning + real_inference
flag — so we time real generation, never a SQLite lookup. Solo has no
cache path. 11 tests (energy math + band split). Validated live on the
isolated 4090: solo gen 308W/70%-duty vs substrate 396W/8.6%-duty —
substrate marginal/tok is LOWER, gross/tok higher (it holds the card
longer for retrieval).
fox 2026-05-21: compute cost-of-goods-sold by kWh vs tokens, with the
three power states (idle / warm-idle / generation) MEASURED per
card×model×server — never hardcoded (his 40/127/380 W were illustrative
of one 3090). The only operator input is --price-per-kwh (default 0.33
USD/kWh, a configurable site rate).
energy_cogs() (pure, unit-tested) decomposes measured generation energy
against the measured warm-idle baseline:
* gross — all measured joules over the window (all-in, includes the
warm-idle cost of keeping the model hot, amortized).
* marginal — joules ABOVE warm-idle: what one more request's burst
actually costs (clamped >=0).
kWh = J/3.6e6; $/1k-tok is the unit that compares to API pricing. Both
surface per cell + a COGS print line.
watt_bench's arborist arm now loads the frozen bench.stock_v1 policy
(--answer-mode, drift-guarded on non-reasoning) so cost is measured for
the SAME substrate the campaign grades. Cells record
window_start/end_unix so a post-hoc load_monitor queue-depth cross-ref
can flag organic-traffic contamination on the non-isolated single-slot
endpoints. 6 COGS tests; full suite 2534 passed.
Two surgical fixes unblock 'arborist with synthesis LLM = Qwen-on-
llama.cpp' as a viable arm in the control sweep. Pre-existing
docstring said 'Arborist×Qwen needs proof-path guided_json+extra_body
surgery — coupled follow-up'; this is that follow-up.
Fix 1 — multi-engine structured-output extras
The runner / query JSON-mode paths previously sent only vLLM's
'guided_json' key for the claim_lattice schema. llama.cpp silently
drops it, leaving Qwen un-enforced (the parse-tolerant fallback did
all the work). Helper
claim_lattice_structured_output_extras() in arborist/qa/verify.py
now returns a dict carrying the schema under all three engine
conventions:
- guided_json (vLLM grammar-constrained sampling)
- json_schema (llama.cpp native shorthand)
- response_format (OpenAI-spec, honoured by llama.cpp and newer vLLM)
Each engine recognises its own key and silently drops the others.
Used at both inference call sites (runner.py:740, query.py:3324).
Hermes/vLLM path is unchanged — it picks up 'guided_json' and
ignores the other two.
Fix 2 — query() accepts user-supplied extra_body, merges with defaults
query() grew a keyword-only extra_body parameter (default None).
Per-model knobs (Qwen's {'chat_template_kwargs': {'enable_thinking':
False}} toggle, future template knobs) can flow from the caller to
the synthesis chat-completion call. Schema-enforcement extras are
added inside query() and merge under user keys — common case is
disjoint namespaces, but if a caller wants to override 'guided_json'
they can.
bench/control_sweep.py now passes MODELS[arborist_ref]['extra']
through to query() in the arborist branch, so --arborist-ref
qwen-nothink runs with reasoning disabled and --arborist-ref
qwen-think runs with reasoning enabled. Phase 1's arborist arm with
--arborist-ref=hermes is unaffected (MODELS['hermes']['extra'] is
None, merges to no-op).
Tests
+ 3 new in tests/test_verify_json.py covering helper default shape,
alternate-schema reuse, and query()'s new extra_body parameter
220 affected tests still green (verify / claim_lattice / judge /
runner suite)
pytest test_verify_json: 27/27
Next: small smoke run --arborist-ref qwen-nothink against 4-8 items
to confirm end-to-end before any full sweep. Phase 2 (qwen-think solo)
still running in background, unaffected — it doesn't touch the
arborist arm.
The Arborist arm runs answer_mode='claim_lattice' (per control_sweep.py
:179, control_ab.py:155) so its answers arrive as the JSON envelope
{"claims":[{"text":"...","evidence_ids":["E1"]},...]}.
_descaffold strips the [E1] evidence-pointer markup but the JSON
braces + key syntax remain. The verifier's strategy-2 (span) and
strategy-3 (proper-noun) extractors see brace noise instead of the
inner claim prose — every Arborist record degraded to UNGROUNDED.
The 2026-05-19T17-01-17Z sweep, re-graded with the freshly calibrated
judge (5a17f61), surfaced this: Arborist arm reported 0 CG across all
three variants in the live phase 1 output (the live run was pre-
calibration), and 29/120 CG (24%) under the calibrated rescore — clear
improvement just from theta_contra=0.85, but the JSON envelope was
still hobbling the verifier paths.
Fix: _unwrap_claim_lattice_json runs BEFORE all downstream rules.
Detection is conservative (three independent signals: starts-with-
brace AND "claims" key AND "text" key) so plain-prose answers
pass through unchanged. Multi-claim envelopes concatenate as discrete
sentences (extract_claim_spans treats each as its own span).
Malformed JSON falls back to the original answer — no silent
rewriting on broken input.
Smoke result on the Iceland Arborist case
ans: {"claims":[{"text":"The current president of Iceland is
Ólafur Ragnar Grímsson.","evidence_ids":["E1"]}]}
gold: {{Infobox Political post |post = President |body = Iceland
|incumbent = [[Ólafur Ragnar Grímsson]] ...}}
before: UNGROUNDED → FABRICATED (then WRONG after calibration)
after: short_entity_grounded → CORRECT_GROUNDED
pytest: 27/27 (added 7 unwrap-coverage tests covering single-claim
envelopes, multi-claim concatenation, plain-prose passthrough,
malformed-JSON tolerance, unrelated-JSON passthrough, and the
end-to-end Arborist-envelope CG flow). Self-test 4/4 unchanged.
Re-rescores of 17:01 sweep + phase 1 sweep run after this commit
to measure final Arborist scorecard improvement.
bench/judge_code.py — drop-in alternative to bench/judge.py with the
same Verdict shape & closed verdict vocabulary (CG/W/F/A/JE) but zero
quota cost: composes verifier + NLI + abstention + specificity into a
fixed-order pipeline. fox 2026-05-19: 'data first, judging later' —
this is the data-collection arm; LLM-based judging (Opus batched
needle-haystack, or Grok credit-card) is a separate downstream
concern that operates on the residue this judge cannot classify
deterministically.
Pipeline (first hit decides):
1. empty / no-gold guards
2. explicit abstention phrases (lexical regex)
3. NLI contradiction (arborist.qa.nli.shadow_check) — strongest
signal: gold contradicts the claim → WRONG
4. lexical verifier (arborist.qa.verify.verify_quotes) →
STRICT → CORRECT_GROUNDED
HYBRID + NLI entail >= 0.55 → CORRECT_GROUNDED
UNGROUNDED + specifics-not-in-gold → FABRICATED
UNGROUNDED + no specifics → ABSTAINED
HYBRID without NLI corroboration → JUDGE_ERROR (residue
for an LLM judge)
Threshold note: _CODE_JUDGE_THETA_ENTAIL_CORROBORATE=0.55 is distinct
from the NLI manifest's entailment_block_veto=0.9. The manifest's
threshold is calibrated for OVERRIDING a STRICT lexical signal with
negative evidence — high bar. The corroboration use here is the
opposite direction: additive positive evidence on an already-positive
anchor — moderate bar appropriate. Self-test case 1 measures NLI
entail=0.769 (clearly entailed, clear margin above 0.55).
Specificity for FABRICATED layers three scanners:
- verifier's multi-word proper-noun extractor (Higgs Boson, ...)
- local single-word capitalised-token scanner (Napoleon, Mars, ...)
deliberately separate because the verifier's gate is conservative
by design (multi-word only)
- numerics (years, dates, large counts, money)
Self-test: same 4 fixtures as bench/judge.py:self_test() so the two
instruments can be cross-checked when fox re-fires the Opus judge on
the residue later. Result: 4/4 INSTRUMENT TRUSTWORTHY.
tests/test_judge_code.py — pulls the contract into make test
(18 cases): module identifiers pinned, dataclass shape parity,
empty / no-gold guards, parametrised abstention phrases, specificity
layer behaviour, the canonical 4-case self-test, batch helper, and
graceful NLI-unavailable degradation. 18/18 pass.
Pre-existing known limitation, documented in the docstring: terse
correct answers ('In 1945.' against gold containing '1945') route to
ABSTAINED because the verifier's span extractor needs prose shape;
NLI sees no clause-level overlap at very short claims. The conservative
ABSTAINED label is correct deferral; tuning this is a calibration
question for real bench data, not the instrument's contract.
No callers touched yet — control_sweep.py & control_ab.py still
import the disabled Opus judge. Wiring this in is a separate ticket
move per fox's data-first sequencing.
The review's central, correct finding: _title_query_tokens is
hot-path (every query AND title) and its fold set (hyphen #000007 +
numeral/accent/honorific/brit) changes which documents retrieve, but
that normalization's version was bound nowhere → a replay cannot
identify which token-normalization produced an old providence
record's sources. Same provenance class as the #000001 keyword gap.
Severity is honest: replay-provenance gap, NOT cache corruption —
different folds → different sources → different context_root →
different cache_key, so no false answer-cache aliasing or false
STRICT. Verifier/proof path unchanged.
Fix follows the repo's OWN #000001 §5/§6 decision (bind retrieval
transforms into the run-DAG RetrievalPlan/retrieval_plan_hash, NOT
governance_policy_hash). The review suggested governance "Option A";
repo precedent is run-DAG binding (same status as retrieval_keywords
and #000056 MT-engine identity) — the discrepancy is surfaced for
fox as an explicit call, not silently overridden.
- RetrievalPlan.title_token_policy (empty default → omitted from
canonical() → every prior retrieval_plan_hash byte-identical; the
§5 zero-churn discipline, same as the #000056 MT fields).
- _TITLE_TOKEN_POLICY single source of truth in query.py, bound at
the plan construction site; bump on any fold change.
- Plus the review's edge cases: Roman-substring-in-word not folded,
out-of-range not folded, Unicode Roman explicitly unsupported,
hyphen∘numeral composition. Full suite 2498, 0 regressions.
Declined (not engineering, per don't-proliferate): the review's
SelfModel/MemoryRoot/5S-5T-5F/capital-ledger ceremony — the ticket
design log is the single source of truth; scope recorded there.
Two more MEASURED fold-search wins on mined ground-truth fixtures
(deterministic recall, no LLM), both lifting at @1/@3/@8 (not
coarse-k artifacts):
honorific (Mt/St/Dr <-> Mount/Saint/Doctor): recall@1 45% -> 75%
(+30pp), @8 62% -> 85%, misses 15 -> 6
brit (British <-> American spelling): recall@1 50% -> 70%
(+20pp), @8 70% -> 85%, misses 12 -> 6
Both _*_fold_variants are additive+symmetric, strict closed sets
(no English-word collision), no-op outside their class (verified
independent: brit no-ops on honorific titles & vice versa), unioned
into _title_query_tokens beside hyphen(#000007)/numeral/accent.
Full suite 2488 passed, 0 regressions (hot-path); real-path tests
(FakeSource->ingest->query()->real _Hit).
Fold-search FINAL across the survey backlog, ranked by MEASURED @1
headroom (not prevalence — the instrument's job):
SHIPPED: numeral (a3ac653) accent (b573c59) honorific brit (here)
NO-BUILD: hyphen — existing #000007 already delivers 90%@1
(the measure-the-unmeasured-thing check pays off)
NO-BUILD: amp — 82%@1 with no fold (prevalence-overranked;
instrument killed it cheaply, like digit-ordinal pre-build)
Net: 4 deterministic retrieval wins + a reusable mined-recall
instrument + the discipline codified in CLAUDE.md, from a goal that
4 prior hypotheses died on because the bench couldn't measure them.
Second MEASURED fold-search win, and the instrument correcting my own
premature call. accent-fold ON vs OFF on the mined accent fixture:
recall@1 55% -> 85% (+30pp), rank-1 22/40 -> 34/40. recall@8 was
flat (95->98) — a too-lenient k nearly got a real lever wrongly
reverted; @1/@3 is the resolution that drives primary-source
selection. _accent_fold_variants: ASCII-fold then re-tokenise so a
diacritic title ("Béla Bartók", which _TITLE_TOKEN_RE otherwise
fragments to junk) matches the ASCII form a user types. Additive+
symmetric, no-op on pure-ASCII (zero effect on non-accent
queries/titles), mirrors _hyphen_fold_variants (#000007).
Also fixes a defect I shipped in a3ac653: an orphaned duplicate
body left as dead code after `return base` in _title_query_tokens
(unreachable — numeral-fold behaviour/measurement were valid — but
cruft; removed).
Fold-search factory, fanned out across the full survey backlog
(deterministic recall, no LLM, parallel — serial-by-caution was
halting in disguise):
- recall_at_k.py: returns rank -> recall@1/@3/@k from one retrieval
(verified offline). A coarse k hides rank-only lifts.
- mine_questions.py: numeral/accent/hyphen/honorific/amp/brit
ground-truth classes; fixtures committed.
- Measured @1 headroom verdicts: accent SHIP (this commit);
honorific 45% / brit 50% = real headroom (build next); hyphen
90% = existing #000007 already delivers, NOTHING to build (the
measure-the-unmeasured-thing check pays off); amp 82% = no fold
needed (prevalence-overranked, instrument kills it cheaply).
CLAUDE.md bench-maxing: two measured lessons codified — report
recall@1/@3/@k (a lenient k hides rank lifts; prevalence != miss-
rate), and fan out independent measurements (serial-by-caution is
halting). Full suite 2488 passed, 0 regressions (accent-fold is
hot-path in _title_query_tokens); real-path test (FakeSource->
ingest->query()->real _Hit).
The first MEASURED, above-noise retrieval win this thread. The 75-q
n=3 audit_mode bench couldn't resolve any single lever (every failure
class <=3-5 q, sub the 5pp floor — four hypotheses died there). Fix
the instrument, not just the lever:
- bench/mine_questions.py + bench/recall_at_k.py: mine questions from
corpus titles (ground-truth target known by construction), grade by
deterministic retrieval recall@k via `query --dry-run` — no LLM, no
verifier, no n=3 noise, scalable to the 22K-deep numeral pool. The
curated qa_questions.txt stays the separate verifier-honesty/trap
gate; mined fixtures measure the answerable long tail per class.
- _numeral_fold_variants in query.py: ordinal-word ("Alexander the
second") <-> multi-char Roman ("Alexander II"), additive+symmetric,
unioned into _title_query_tokens exactly like _hyphen_fold_variants
(#000007). Strict 2..40 Roman set → no English-word collision;
single-char Romans (I/V/X) intentionally out of scope (universal
len>1 token filter — stated before building, ~4 of 10 residual
misses).
Measured on the mined numeral fixture: recall@8 22/40 (55%) -> 30/40
(75%), +20pp; 20 hits now rank-1. Discipline applied end to end:
measured-first, mirrored precedent, full-suite regression run (2482
passed, 0 regressions — numeral-fold is hot-path in
_title_query_tokens), real-path test (FakeSource->ingest->query()->
real _Hit, not a hand-built object), measured-after on a noise-free
instrument. The ~6 multi-char residual misses are a different
downstream cause the instrument now exposes for future iteration.
Three workstreams, full suite 2482 passed, experimental paths default-OFF.
#000055 — Windows quickstart without make
tasks.py (pure-stdlib runner) + make.bat shim + .gitattributes;
README Windows section rewritten. Quickstart needs only Python
3.10+ (no make/bzip2/curl/bash). Mirrors the Makefile quickstart
subset; drift-pinned by tests/test_tasks_runner.py.
#000001 §7 Phase 0 — deterministic cross-language guard
arborist/qa/crosslang.py: non-English signal (¿/¡/non-ASCII) + an
es function-word stoppack. Fail-closed to UNGROUNDED before
retrieval/LLM (mirrors the quantifier reject-DAG) when no content
token survives, else strips es stopwords from the retrieval query
only. English path byte-identical by construction. Default OFF
(crosslang_guard_enabled). Measured: the anarcocapitalismo field
case 10.4s -> 1.6s.
#000056 — Operation Sandwich (cross-language grounding)
arborist/qa/mt/: opus-mt es/fr/ru<->en, lazy per-pair memoised
singleton (fixes the 88%-engine-error concurrency defect),
manifest-pinned, [mt] extra; entity_mask wrapper. Sandwich =
translate query in (retrieval + LLM prompt) -> English answer ->
UNTOUCHED verifier grounds English-vs-English -> translate the
verified answer out as display-only (banner-labelled, zero
grounding). question_hash + verifier_policy_hash invariant; MT
engine identity binds into RetrievalPlan, not governance. CLI
--crosslang-translate / make XLANG_MT=1. Default OFF; entity_mask
default OFF (measured net-negative at bench scale). Fan-out bench
(bench/*.py): Spanish ~0% -> 71% grounded vs the real no-support
baseline; the round-trip predictor was tried and refuted; the
entity-mask lever failed at scale (corpus-title anchoring untried).
CLAUDE.md: cross-language bright-line convention + module map.
Pre-existing modified diagram files are intentionally excluded.
Five rule tightenings, each targeting a specific bench-qa STRICT
false-positive shape (the xfail regressions from the previous commit):
1. claim-lattice bracket-artifact skip — sentences matching
`\[E\d+\s*\|` (pointer markup) or `..."\]` (truncation tail)
are no longer parsed as natural-language assertions; `Such a
thesis was..."]` no longer fires vacuous.
2. Circular-rule differentia cap — circular now requires the
predicate to be (a) entirely vacuous OR (b) leads with a subject
token AND has ≤ 2 non-subject non-filler differentia tokens.
"Michael Jordan's Restaurant was a restaurant in Chicago,
Illinois, named after the basketball player Michael Jordan"
has 6 differentia → no longer fires. Pre-existing positive test
"The entity is the entity referring to the State of Israel"
has 2 differentia (state, israel) → still fires (under threshold).
3. phrase_component_reuse translation-chain exception — when the
predicate ALSO contains a quoted phrase (translation /
definition / etymology context), token reuse with the subject's
quoted phrase is legitimate, not circular. "The name 'Rosebud
River' is a translation … 'the river of the roses'" no longer
fires.
4. Vacuous-rule short-acronym escape — `_coherence_predicate_has_short_acronym_content`
recognizes title-cased element-symbols (Au, Fe, Pb…) and all-caps
2-5-char acronyms (DNA, FBI, USB, NASA…) as content even though
they're below the ≥3-char content-token filter. "The chemical
symbol for gold is Au." no longer fires; "Iron has the chemical
symbol Fe." also clean; tautology "DNA stands for DNA." still
correctly flagged circular.
5. (the 'term <X>' idiom xfail stays xfail — borderline, no clean
lexical fix.)
5 xfail → passing (the regression coverage is now executable proof
of fix); 1 xfail remains. Pooled bench-qa STRICT FP rate test ceiling
tightened from 7% to 2%. All 7 pre-existing positive coherence tests
still fire correctly. 110 total tests pass; 1 xfailed; no regressions
on doc-counts / nli / relevance.
§3.1 diagnose_coherence (now 19 tests, +10 from the parallel session's 9):
- 3 more positive shapes (multi-sentence vacuous, named-entity circular,
grammar-term phrase_component_reuse).
- 5 xfail regression tests for SHAPES THAT FALSE-FIRE on real bench-qa
STRICT data (44/808 = 5.4% FP rate measured on the pooled n=1+3+5
STRICT answers). Each xfail names the exact shape + why it should
ideally be 'ok' + which rule needs tightening:
* 'The chemical symbol for gold is Au.' → vacuous (short predicate)
* 'Michael Jordan's Restaurant was a restaurant ... named after
Michael Jordan.' → circular (named-after re-use)
* 'The Western X was the western half of the X' → circular
* 'The name <Phrase> is a translation ... of the <derivative>' →
phrase_component_reuse (translation/etymology)
* claim-lattice [E1 | … …"] tails → vacuous (truncated bracket
fragment)
* 'The term <X>' → phrase_component_reuse (idiomatic English)
- 1 load-bearing real-traffic test: FP rate on 808-cell pooled STRICT
must stay ≤ 7% (current 5.4%) — fires loud if a future change
regresses it. Skips on fresh-checkout (bench/qa_results/ gitignored).
§3.2 ShadowRelevance (now 20 tests, +7 from the round-1 scaffold):
- Manifest tests for round-2 primary (bge-reranker-large), the size
spectrum coverage (50-560MB), the candidate-bench findings block
(biggest-within-family / not-across-families / deeper-not-better /
capacity-floor).
- Pair-kind distinction (question_answer vs claim_source recorded
separately for downstream telemetry / governance hashing).
- Batch-order preservation (_score_batch must return scores in input
order — load-bearing for downstream zip-back).
- Empty-input handling (Q empty, D empty, whitespace-only).
- Zionist-entity discriminator sanity (on-topic > off-topic logit).
- demote_below_score-stays-null invariant (the §7 #18→#27 discipline:
no hardcoded threshold; must come from a real-traffic shadow sweep).
Total: 101 passed + 6 xfailed (5 §3.1 regressions documented + 1 from
parallel session). The 5 xfails are the bench-maxing receipts — they
document EXACTLY which shapes §3.1 false-fires on, with the rule that
needs tightening named in each reason.
arborist/qa/relevance/ — manifest pins cross-encoder/ms-marco-MiniLM-L-6-v2
(~80MB, Apache) as primary; alternates: L-12, BAAI/bge-reranker-base,
ms-marco-electra-base. demote_below_score=null on purpose — the
#000049 §7 #18→#27 discipline (proved 6× that clean-eval thresholds
don't transfer to bench-qa data) requires the threshold to be set by a
shadow sweep against pooled real STRICT, not by a literature number.
ShadowRelevance class mirrors ShadowNLI (lazy [nli]-extra import, cuda
auto-detect via ARBORIST_RELEVANCE_DEVICE or ARBORIST_NLI_DEVICE,
batched _score_batch, graceful degrade-to-available=False). Two surface
methods: check_question_answer (deflection / Q-A drift) and
check_claim_source (topic-collision mis-cite). 13 tests.
Sanity on the motivating field case (Zionist entity): ON-topic +9.96
vs OFF-topic -9.04 → 18-pt margin. Mona Lisa Q→A deflection: on +10.45
vs deflect +3.56 → ~7-pt margin. The model CLEANLY discriminates the
failure modes #000052 §1 named. It does NOT catch the
recombination-where-the-different-entity-clause-also-mentions-the-target
case (Kilimanjaro/Mount Kenya) — and that's the right architectural
split: aboutness (#000052 §3.2) and entailment (#000049 NLI) are
orthogonal axes; the Kilimanjaro recombination case needs the semantic
candidate selector (#000050/#000051 vec hybrid).
Remaining: build candidate-bench eval (~20-30 deflection + mis-cite
fixtures), shadow-sweep θ over pooled bench-qa STRICT (expect another
walk-back per the #000049 lesson), recall-side realism check, then
fox+dav1d sign-off. Still SHADOW; production verifier unchanged.
`arborist/concepts/extract.py:acronym_parens_synonym` — new
corpus-agnostic extractor. Scans each doc's lead chunk (first 4000
chars) for `<Multi-Word Phrase> (ACRO)` where the all-caps acronym's
letters strictly match the content-word initials of the phrase, in
order, after function-word filtering. Emits bidirectional synonym
edges between the lowercased acronym and each ≥3-char content token
of the phrase, evidence_kind="acronym_parens", anchored to that doc's
document_root. Idempotent like link_reciprocity_synonym.
Why this complements link_reciprocity: Wikipedia represents
abbreviation→expansion as a one-way *redirect* (CPU →
Central processing unit), which the ingest does not record as an
edge — so the existing reciprocal-link extractor never learned the
relation. The relation IS in body text by near-universal convention
("Central processing unit (CPU) is..."), which this extractor reads.
Corpus-agnostic: HTML, blogs, textbooks benefit equally.
Conservative: strict 1:1 acronym-to-atom match (rejects HTTP-shape,
where letters land mid-word), function words filtered, repeated
definitions deduped per doc, ≥3-char target floor. 8 new tests
covering CPU bidirectional emit, RAM idempotency, FBI function-word
filter, HTTP length-mismatch reject, XYZ initial-mismatch reject,
ROM hyphenated-word handling, per-doc dedupe, registry presence.
Retrieval-side only — synonym edges reshape FTS5 candidate selection
via synonym_expand at query time, never enter audit_mode / cache_key
/ audit_event_hash. No governance hash bump, no cache invalidation.
Closes#000050 §2a's CPU/GPU abbreviation rows *upstream* of vec;
the Orwell-shape conceptual-allusion row remains the genuine #000050
justification. Operational follow-up (not code): run on each shard
via `arborist concepts derive --extractor acronym_parens` (CLI
surface itself is aspirational in docstrings; extractors are called
programmatically today). Next ID 000054 -> 000055.
`arborist.qa.evidence._content_tokens` dropped every token under 4
chars, so a short all-caps acronym (CPU, GPU, DNA, FBI, USB…) never
registered as a content token — which defeated Rule 8
(_claim_title_overlap / TITLE_MISMATCH), the subject-tokens-absent
check (Rule 9), the bare-name-claim guard, and spotlight-excerpt token
selection whenever a question/claim's topic IS an acronym. The field
case: `what is a CPU?` cited to the "CPU design" article tripped
TITLE_MISMATCH even though claim and title both contain "CPU".
Fix: keep a token if it's an all-caps 2-3-char alpha run in the source
text; everything else unchanged. The change only ever ADDS tokens, so
TITLE_MISMATCH / SUBJECT_TOKENS_ABSENT / BARE_NAME_CLAIM can only stop
firing, never start — monotone toward fewer spurious demotes; no
STRICT→non-STRICT transition is possible from it.
Versioned: `content_token_rules: "v2-acronym-aware"` added to
runner.DEFAULT_POLICY + query.DEFAULT_QUERY_POLICY +
keys._VERIFIER_POLICY_FIELDS → folds into verifier_policy_hash, prior
cache records orphan on lookup (by design; same discipline as
base_version / hyphen_fold_v1). Does NOT touch the retrieval
abbreviation→expansion gap (CPU→Central processing unit — #000050 vec
hybrid / concepts/ synonym edges; the root cause of the satellite
retrieval). 8 new tests; full suite green (2502); bench-qa-smoke clean.
Next ID 000053 -> 000054.
Speedup (§3 plan): ShadowNLI._nli_batch batches forwards
(ARBORIST_NLI_BATCH=64); device auto-detect (ARBORIST_NLI_DEVICE, else
cuda-if-available); auto-prefer an ONNX export — bench/scripts/export_nli_onnx.py
/ make export-nli-onnx exports + int8-dynamic-quantizes the pinned
checkpoint into ~/.arborist/models/nli/<ver>/onnx/ (operator state, NOT
committed), _ensure_loaded loads model_quantized.onnx via
optimum.onnxruntime (backend onnx-int8), falls back to torch silently.
torch-cpu-batch1 ~120ms/pair → onnx-int8-cpu-batched ~32ms/pair (~4x);
seconds on a 4090. optimum[onnxruntime] added to the [nli] extra; 24
tests.
Gate-item-4 verdict at proper n: ARBORIST_NLI_SHADOW=1 make bench-qa
BENCH_QA_N=1 → 223 cells (89 STRICT / 90 HYBRID / 44 UNGROUNDED; also
surfaced + fixed a lone-surrogate bug). Shadow sweep over those: NLI-as-
runtime-veto on STRICT has ~26% FP at θc 0.5, ~8% at θc 0.90, ~0% only
at θc 0.99 — and θc 0.99 gives up most recombination recall (hard
synthetic recombinations bottom out ~0.76). FAILS the §7 #12 gate on
this design. Only untried path that might pass: a Phase-3 runtime hook
running NLI on the verifier's actual matched clauses (1-3), not
top-6-by-overlap. Until then: runtime NLI demotion stays off; the 2
fixtures stay permanent boundary markers; θc stays 0.5. Production
verifier unchanged; falsification-hard stays 10/12.
Per-sentence shape check (no model) emitting kind ∈
{phrase_component_reuse, circular, vacuous, ok, empty}:
- circular: subject content-tokens ⊆ predicate's and the predicate
leads with a subject token ("Water is water").
- phrase_component_reuse: subject quotes a phrase, predicate reuses
one of that phrase's own tokens as a bare "the/a/an <token>"
referent — the 2026-05-12 field case ("the phrase 'Zionist entity'
is used as the entity"), a token collision the verifier +
deflection + title-relevance all pass and NLI returns neutral on.
Copulas inside a quoted span are skipped so 'war is peace' doesn't
break the subject/predicate split.
- vacuous: predicate is only placeholder hypernyms + filler ("X is
a thing").
Conservative — no full token-salad parsing; legit definitions pass ok.
Surfaced in inspect_cache_key + the `arborist inspect` human view
(· incoherent: <kind>). Advisory only — never writes providence_cache
/ audit_events / run_dag_root; demote-only verifier hook deliberately
not wired. 9 tests; full suite green (2500 passed).
candidate_clauses() — NLI now runs only on the top-N source clauses by
content-token overlap with the answer claim (max_candidate_clauses=6),
not the whole context; records n_candidate_clauses / best_clause_overlap
/ recombination_risk. Synthetic sweep unchanged (28/28 recombination,
0/26 legit FP, mean 1.45 candidate clauses/record). Real-traffic smoke
re-run: STRICT would-demote 30% → 20%, overall 47% → 33% — better, not
fixed; recombination-risk split doesn't separate either. Residual STRICT
false-contras at ~0.83-0.92 → θc would need ≈ 0.90 (vs the clean-set
0.5); at θc=0.90 the data in hand gives 27/28 synthetic recall, 0/26
legit FP, 0/10 smoke STRICT FP — but n=10 is too small to set on.
Next: a fuller ARBORIST_NLI_SHADOW=1 bench-qa run → sweep θc on hundreds
of STRICT cells → confirm → set it. θc stays 0.5; runtime NLI demotion
stays off. Production verifier unchanged; falsification-hard stays 10/12.
arborist/qa/nli/ — SHADOW ONLY (never an audit_mode input; manifest not
yet in governance_policy_hash per §7 #2). manifest.json pins
cross-encoder/nli-MiniLM2-L6-H768 @ a fixed HF revision + the
bench-validated θc 0.5/θe 0.9 + 2 alternates + the Phase-3 TODO;
shadow.py = ShadowNLI/shadow_check (lazy transformers+torch behind a new
[nli] extra, clauses() segmenter, the §7 #5 clause-level Demote()
decision, degrades to available=False when [nli] absent);
bench/scripts/nli_shadow_sweep.py + make bootstrap-nli / bench-nli-shadow
(the gate-item-4 instrument); 16 tests.
First sweep (116 records — 5f-falsification packs + the arborist-nli-bench
eval sets): 28/28 synth recombination demoted, 0/26 FP on legit summaries,
0/9 fires on already-STRICT_SPAN records, 25/50 on UNGROUNDED (the
contradiction half; quiet on non-sequiturs). Gate items 1/2/3/5/6 clear
on available data; item 4 — shadow FP rate on a real live-bench-qa
sample — remains the open measurement. Production verifier unchanged;
falsification-hard stays 10/12.
The cache lookup in ask()/query()/canonical-persist runs outside the
write transaction, so two concurrent callers on the same cache_key can
both miss and both reach the providence_cache INSERT — the loser raised
`UNIQUE constraint failed: providence_cache.cache_key` and ask() crashed
(MOAD-0005 / TOCTOU; sibling of the af870bb append_audit fix). All three
write sites (qa/runner.py, qa/query.py, qa/canonical_cache.py) now end in
`ON CONFLICT(cache_key) DO NOTHING`, so the loser no-ops (its answer is
equivalent — same question/model/policy ⇒ same cache_key; canonical
answers are deterministic). With busy_timeout on every connection
(af870bb) the loser waits on the writer's lock then no-ops.
test_qa.py::test_concurrent_ask_same_cache_key_no_unique_crash — 6
threads run ask() on the same question concurrently; must not raise;
exactly one cache row lands. Verified it fails without the fix (5 of 6
threads raise IntegrityError).
Closes the 8 mis-segments #000046 left in formulate-hard-v1.jsonl.
The parser was line/bullet-only — one line ⇒ one claim — so a line
that crammed several pointered claims onto one row ("Water is wet
[E1]; fire is hot [E2]", "X happened [E1]. Y followed [E2]") became
one monolithic claim with all the pointers, and a wrapped bullet
became two.
arborist/qa/parse_claims.py: _SEGMENT_SEP_RE splits a line on ';',
sentence boundaries ('. '/'! '/'? ' then a Capital), spaced dashes
(' - '/' — '/' – '), ' and '/' or '/' because '/' although '/' since
'/' while ', inline '(N)' enumeration markers, and commas — with
'(?![^\[]*\])' so a comma inside a [E1, E2] bracket never splits it.
_segment_line keeps the split ONLY IF every resulting non-empty
segment is a well-pointered claim — a legit single claim ("The cat
is black and white [E1].", "The cast: A, B, C [E1].") is never
broken because splitting it would manufacture pointer-less prose
fragments → guard rejects; a leading colon-terminated header with no
pointer ("Two facts:", "Key points:") is allowed and dropped. Plus a
wrapped-bullet join: a continuation line (leading whitespace then a
lowercase letter, no bullet glyph) folds its text + pointers into the
previous claim.
Effect: formulate-hard rate 4/12 → 12/12 (the pack is now at ceiling
— a harder Formulate tier would re-open below-ceiling headroom; a
#000046 follow-up). Remaining #000048 headroom: 2 STRICT_PARAPHRASE
recombinations in falsification-hard (Mercury, Einstein — step 2.2).
Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells; parse_pointer_claims
feeds the 450 claim_lattice_pointer + claim_lattice cells) after
(bench/qa_results/2026-05-11T20-26-37Z) vs the pre-step-2.4 baseline
(...T17-12-41Z = HEAD's parse_claims.py). STRICT-rate quote 0.54→0.55,
pointer 0.22→0.22, lattice 0.43→0.45 — all within the 5-pp noise
floor. Per-row diff: the segmenter changed the parsed-claim count on
the SAME answer text for 7 of the 450 lattice cells (0 in
claim_lattice, 7 in claim_lattice_pointer); of those, 2 caused an
audit_mode change — both correct: a wrap-join recovered an answer's
intended structure (4 claims, 2 pointer-less wrap-fragments → HYBRID)
into 2 well-pointered claims → STRICT; and a crammed-one-line blob (1
monolithic claim, all pointers → STRICT) split into 8 claims, some
not individually verifying → HYBRID (the honest verdict — false-
positive STRICT was the corruption). Every other lattice/quote delta
is LLM re-answer variance. No regression — the segmenter's only
visible effects on real traffic are honest improvements. Summarized
in qa-modes-bench.md Addendum 7 + ticket-000048 §5 step 2.4.
Tests: 8 new in test_claim_lattice.py (semicolon/sentence/conjunction
splits; pointerless-fragment + cast-list guards; leading-colon-header
drop; wrapped-bullet join; pointer-order/multi-pointer); existing
parse_pointer_claims tests pass untouched; test_5f_formulate_hard_pack
re-pinned 4/12 → 12/12. make test 2358 passed, 28 skipped.
#000048 → steps 2.1 + 2.4 landed; #000046 / #000012 §8 / TICKETS.md /
Makefile / fixture _meta + notes updated.
Closes the 4 HYBRID_ENTITY over-grounds #000046 left in
falsification-hard-v1.jsonl. The entity strategy grants HYBRID when a
multi-word proper noun matches the source — but "Insulin was
discovered by Alexander Fleming" against "Penicillin was discovered by
Alexander Fleming" matches on the shared "Alexander Fleming" while the
swapped subject "Insulin" (the falsehood) is ignored.
arborist/qa/verify.py: _entity_salient_disagrees(answer_text, norm_ctx)
flags a >4-char Capitalized content token (stopword-filtered) or a
digit-number in the answer absent from the source.
_is_single_sentence(text) — no internal '. '/'! '/'? ' break. Gated in
verify_quotes' entity branch (proximity policy) in the weakest-grounding
slot only: not cluster AND len(verified) <= 1 AND _is_single_sentence
AND _entity_salient_disagrees → UNGROUNDED. The narrow caller-gate is
what keeps a structured multi-claim summary untouched — the Matrix cast
list (many entities, a tight cluster) and the TMNT answer (a numbered
list with parenthetical nicknames the source omits): model-added
accurate detail in a real summary isn't a contradiction, only the
single-sentence-one-weak-match shape is. The Matrix/TMNT/hybrid
entity-path regression tests still pass, pinned untouched.
Effect: falsification-hard rate 6/12 → 10/12 = 0.833 (Insulin / Berlin
/ 1889 / Pacific now correctly UNGROUNDED). The 2 live-pack fixtures it
newly demotes — 5f-fal-live-003 (the exact gap #000046 built its hard
pack around) and 5f-fal-live-028 — had expected_reason updated
HYBRID_ENTITY → UNGROUNDED (the live pack records what verify_quotes
actually does). Remaining hard-pack headroom: 2 STRICT_PARAPHRASE
recombinations (Mercury, Einstein — step 2.2) + 8 Formulate
mis-segments (step 2.4).
Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells) after
(bench/qa_results/2026-05-11T17-12-41Z) vs the pre-step-2.1 baseline
(...T14-19-51Z = HEAD's verify.py). STRICT-rate quote 0.50→0.54,
pointer 0.25→0.22, lattice 0.45→0.43 — all within the 5-pp noise
floor. Per-row diff (675 common cells, 30 quote-mode rows changed
audit_mode): 0 quote-mode rows demoted to UNGROUNDED from the entity
path — the gate fired on 0 legitimate QA answers in the whole bench.
Every transition was LLM re-answer variance (verifier quote→quote with
the verdict flipping); pointer/lattice deltas are noise too (the gate
is in verify_quotes / quote mode, not the claim-lattice verifier). No
regression — the gate is provably narrow on real traffic. Summarized in
qa-modes-bench.md Addendum 6 + ticket-000048 §5 step 2.1.
Tests: 4 new in test_verify.py (_is_single_sentence helper,
_entity_salient_disagrees helper, swapped-subject → UNGROUNDED,
gate-narrow-on-multi-claim); test_5f_falsification_hard_pack_below_ceiling
re-pinned 6/12 → 10/12; test_fork_score_positive_gamma_5f_... updated
(positive γ·Δ5f on the real lift — possibly MARGINAL given the ÷5
dilution; ACCEPT via a degraded-parent sub-scenario).
make test 2343 passed, 28 skipped.
#000048 → step 2.1 landed; #000046 / #000012 §8 / TICKETS.md /
Makefile / fixture _meta + notes / baseline JSON updated.
Wire vector quantization (the §3.1 production knob): arborist embed
--quant int8 [--rebuild]. The chunk_vecs vec0 column becomes int8[384]
vs float[384] per quant; the quant folds into VEC_BACKEND_VERSION
(...-384int8-... / ...-384float32-...) and vec_meta records it per
shard. Switching quant on an existing chunk_vecs requires --rebuild
(the vec0 element type can't be altered in place — embed_documents
raises ValueError telling you to --rebuild).
int8 serialization: scale each bge component by 127 (theoretical
[-1,1] range), clamp to [-127,127], round, serialize_int8. The same
scaling on the query vector → distances comparable; cosine is
scale-invariant so the uniform x127 cancels in the ranking.
sqlite-vec v0.1.9 quirk worked around: a bare blob inserted into a
vec0 column is interpreted as float32 regardless of the column's
declared type — int8 vectors MUST be wrapped in vec_int8(...). So
the INSERT and the MATCH now wrap the blob in vec_f32(?) (float32)
or vec_int8(?) (int8) — constructor name from a fixed dict, no
injection surface. (Discovered the hard way: a bare int8 blob into
an int8[384] column → "expected int8, but a float32 vector was
provided".)
VecBackend reads the quant from the existing chunk_vecs schema (or
defaults to float32) so search uses the matching wrapper. New module
exports: QUANTS, EMBED_QUANT, vec_backend_version(quant), existing_quant.
int8 head-to-head on crawl_appliedcombinatorics_org.db (168 chunks):
- storage: float32 1,597,440 B -> int8 417,792 B = 3.8x smaller
(~4x at corpus scale where the 1024-vector blocks fill; the
~28 KB of vec0 metadata doesn't quarter, hence 3.8 not 4.0).
- recall vs the float32 baseline:
Q "how many ways to choose k things from n":
identical top-5 (Combinations, Permutations, Exercises,
Derangements, Graph Coloring).
Q "pigeonhole principle counting":
identical top-2 (Graph Coloring, Exercises); ranks 3-4 swap
Derangements <-> Permutations at Δdistance 0.002 — sub-noise.
- embed speed unchanged (~3.9 chunks/s — model-load-dominated).
Conclusion: int8 is the obvious production config (§3.1's +6%-tax
recommendation confirmed empirically). v1 default stays float32 for
max fidelity; flipping the default to int8 is a fox call.
CLI (arborist/cli.py): arborist embed --quant {float32,int8}; output
JSON gains "quant"; embed_documents ValueError → exit 2 with the
"--rebuild" hint.
tests/test_search_vec.py (9 -> 16): test_int8_quant_roundtrips
(int8[384] schema, vec_meta version, search round-trip, quant
inferred by VecBackend), test_quant_mismatch_requires_rebuild,
test_invalid_quant_rejected.
#000039 status updated. Full suite: 2343 passed, 28 skipped.
(Unrelated parallel-clone work in the tree — Makefile, arborist/qa/
verify.py, bench/fixtures/5f/*, tests/test_bench_batteries.py,
tests/test_verify.py — is #000046's hard-fixture tier, not touched.)
