Bench-v5 (5 themes × 3 questions × 2 paths, Hermes-3-8B) AFTER the
fold-stack lift + apply_title_boost wire-up to use _title_query_tokens
shows the SAME 12 regressions as the pre-wire-up baseline:
- 5 wrong-primary picks (Dr Who, Albert/Ahmed/Alaric the third/first,
Casa Batlló error)
- 7 STRICT→HYBRID demotes on correct primaries (Spider-Man, Ampère,
Dr Syn, Dr V64, Finnish Defence Forces, Hopewell Centre, labor
economics)
Diagnosis: apply_title_boost only reranks docs ALREADY in the
candidate set. body-only retrieval (default with multi_route=False)
never surfaces "Doctor (Doctor Who)" so no fold-aware rerank can
promote it. Legacy surfaces it via the title route. FTS5 porter
stemmer handles plurals but NOT Dr→Doctor or third→III, so the
SqliteShardCorpus.fts_title method also doesn't help here without
the folds applied at retrieval-token-gen time, not rank time.
Updated ticket with the honest assessment: 5 Path A stages across
v1/v2/v3 have now proven legacy query() doesn't decompose into a
library of helpers. Three forward paths offered for fox to decide:
A. leave default at providence, accept fold regressions (env
escape hatch ARBORIST_LEGACY_QUERY=1 already shipped)
B. flip default back to legacy, treat providence as
infrastructure for cloud-query/corpus-query only
C. keep both alive long-term — separate query2 command
Re-bench legacy vs providence_query on fold themes (accent, hyphen,
honorific, brit, numeral) after the proxy memory fix.
Result (15 question-pairs through Hermes-3-8B): 12 regressions,
2 improvements, 1 tie. Net-negative on these themes, BUT all 12
regressions trace to a single root cause — the 5 fold-variants
helpers (_hyphen, _numeral, _accent, _honorific, _brit) live inside
_title_query_tokens at query.py:288-325 and providence_query
lazy-imports the WRAPPER without lifting the fold helpers.
Same gap manifests two ways:
- Wrong primary (5): Dr Who → pathology; Albert/Ahmed/Alaric the
third/first → wrong articles; Casa Batlló → error
- STRICT → HYBRID on correct primary (7): the verifier's Rule 8
title-overlap check calls the SAME _title_query_tokens —
without folds, "Andre-Marie" (claim) and "André-Marie" (title)
are distinct tokens, overlap fails, audit_mode demotes
Path A v3 surfaces: lift the fold-variants stack to _text_norm.py,
re-export from query.py, drop the lazy-imports in source_roles.py +
retrieval_routes.py. ~250 LOC moved + ~50 LOC import-rewrites,
half-day. Lower risk than v1 (pure code motion, helpers are
identical between paths).
Themes deliberately skipped this round (need their own gates ported
separately): quantifier_subset, metacog_subset, warrant_chain_probe,
es, fr. Re-bench AFTER v3 lands.
Also commits bench/legacy_vs_providence_bench.py + the result JSONL
so the regression set is reproducible.
bench/chunk_fetch_speed.py — measure per-chunk fetch latency for the
two cloud paths a future JUST_ENOUGH=1 blob-publish move would
compare against: (1) apsw HttpRangeVFS on the big shard .db (current
FtsSidecarShardClient fallback path), (2) direct HTTP GET on a
same-bucket object (proxy for per-chunk blob fetch).
Measured 2026-05-31 against clones/full-bench/000.db (12.5 GB) +
clones/sidecars-fts/000.idx.db:
apsw median: 704 ms / chunk (mean 773, first 1594, warmup ~500)
blob median: 91 ms / chunk (mean 92, flat — no warmup effect)
speedup: 7.7×
Real-world: ~2.6 s saved per fresh 4-chunk query. For cache-miss
flows that already pay 5-15 s on the LLM call this is real but not
transformative. The big win for blobs is cache HITS that don't go
to LLM (returns drop from ~100 ms via cached chunks to sub-50 ms
via blobs) and bulk bench runs (400 fetches = 4 min vs 30 s).
Interactive single-question flow with LLM in the loop is fine on
the apsw path; blobs stay as future optimization, not blocker.
Also archives bench/three_way_results/*.jsonl (3 runs across the
#000072 Phase 1 progression) + bench/slim_fts_parity_results/ so
the journey from "cloud diverges from local" through "cloud matches
local 5/5" is on disk for the design-log record.
New bench/teacher_judge.py reads judge_input.jsonl, cross-judges
each (question, model, answer) using a DIFFERENT model than the
answerer (default: hermes-answers → qwen judges, qwen-answers →
hermes judges). Writes verdicts.jsonl + markdown summary including
a false-STRICT highlight section.
Key implementation details:
- Reuses _judge_prompt from cross_model_selfplay so iterating on
the prompt template doesn't require re-running benches.
- Sends chat_template_kwargs:{enable_thinking:false} on every call
— required for qwen.ai.unturf.com (llama.cpp deepseek-reasoning
format) where reasoning eats max_tokens before producing content.
vLLM (hermes) silently ignores the unknown kwarg.
- Prompt template tightened: previously said "Reply in format
VERDICT: <rationale>" which qwen took literally, replying with
the word VERDICT instead of the verdict token. Now explicit:
"write the chosen verdict word itself".
Live cross-judge result on 151-row 76-question bench:
hermes (judged by qwen): 44 CORRECT, 23 WRONG, 9 PARTIAL → 58% accuracy
qwen (judged by hermes): 45 CORRECT, 6 WRONG, 11 PARTIAL → 73% accuracy
Qwen is materially more accurate despite costing 1.8× more — matches
fox's "qwen slightly outperforms" intuition with hard numbers.
8 false-STRICTs surfaced including Q12 (hermes conflated Roman
Empire with HRE), Q48 (hermes answered with Niger River info on a
Nile question), and Q72 (both models missed the Game of Thrones
reference in "winter is coming").
STRICT audit ≠ factually correct. The verifier passes any answer whose
quotes match source text; a model can quote correctly and still draw a
wrong conclusion. Without a judge, the bench can't detect false-STRICTs.
This flag emits <ts>.judge_input.jsonl alongside the regular results —
one row per (question, model) with a model-agnostic teacher prompt.
Feed to any teacher (claude -p per row, remote API, Hermes self-judge)
to get CORRECT|WRONG|PARTIAL|UNCERTAIN verdicts. Skips rows that
errored or returned empty answers.
Grounded in observed reality from the 76-question 2010-wiki bench:
Hermes produced 2 confidently-wrong STRICTs (Q12 conflated Roman Empire
with Holy Roman Empire; Q48 answered the Niger River when asked about
the Nile). Qwen had 0 false STRICTs over the same fixture. Naming the
flag --teacher-model-judge (not --opus-judge) keeps the harness
provider-agnostic.
Replaces the hand-rolled BM25 sidecar (arborist/wallet/sidecar.py, ~690
LOC) with a slim SQLite file that just COPIES the source shard's FTS5
shadow tables verbatim + minimal doc/chunk metadata. Cloud retrieval
then runs SQLite FTS5 bm25() on the same bytes the local shard uses —
bit-for-bit parity by construction. 5/5 source + audit_mode agreement
on the smoke fixture between local corpus-query and cloud-query against
the new manifest-fts.json.
Why
---
Custom binary sidecar was per-document BM25; main encyclopedia articles
got length-normalized so hard that on "why did the dinosaurs go extinct?"
"Edwina, the Dinosaur Who Didn't Know She Was Extinct" beat "Dinosaur"
(measured cloud-vs-local divergence). Local FTS5 indexes per-chunk so
each chunk is a moderate-length doc and the main article wins multiply.
Different granularity, not a tuning knob — fix is to use the same
indexer cloud-side.
What ships
----------
- arborist/wallet/fts_sidecar_build.py — builder. ATTACH source shard,
copy documents (root/uri/title only), copy chunks (id/root/idx/leaf
only, NO content), CREATE VIRTUAL TABLE chunks_fts/documents_fts with
same DDL as source, bulk-copy the four shadow tables verbatim,
VACUUM. 8.78 GB shard → 2.15 GB sidecar (24.5%) in ~45 s; full
4-shard wiki corpus 37.4 GB → 8.1 GB (21.7%) in ~3 min.
- arborist/wallet/bucket.py: FtsSidecarShardClient — downloads slim
sidecar once into ~/.arborist/sidecar-fts-cache/<hash>.idx.db, opens
read-only sqlite3 (check_same_thread=False for parallel shard fan-
out), runs FTS5 MATCH locally. Chunk content fetches via blobs/<hash>
with HTTP-range big-shard fallback when blobs aren't published.
MultiShardSidecarCorpus simplified to fts_sidecar_url ∨ bucket-direct
(both are FTS5 backends; merge by raw bm25 MIN ascending).
- arborist/qa/corpus.py: SidecarBucketCorpus.higher_is_better=False
(FTS5 bm25 is negative, lower=better). chunks_for_doc dispatches on
fetch_chunk_body attr for the slim-FTS5 client. apply_title_boost
imports tokenizer helpers from new arborist/qa/_text_norm.py.
- arborist/qa/_text_norm.py — fold_accents, numeral_expand,
tokenize_text, STOPWORDS — extracted from the deleted sidecar.py so
apply_title_boost keeps its lexical shape.
- arborist/cli.py: `arborist sidecar build-fts` subcommand; old
`sidecar build`/`sidecar search` removed. cloud_query recognizes
fts_sidecar_url + sidecar_url alike.
- Makefile: `sidecar-build-fts` + `sidecar-build-fts-all` targets;
`sidecar-build` + `sidecar-search` removed.
- scripts/upload_fts_sidecars.py — boto3 producer: uploads slim
sidecars to clones/sidecars-fts/<n>.idx.db, publishes
clones/manifest-fts.json (4 wikipedia shards inherit existing
shard_url for content fallback; ACL public-read; idempotent on
size match). Existing manifest-sidecar.json untouched.
- tests/test_qa_corpus_functional.py + test_qa_corpus_integration.py
converted from build_sidecar → build_fts_sidecar; 6 fixtures pass.
- bench/slim_fts_parity_bench.py — local 3-way bench
(legacy/corpus/slim_fts) over the smoke fixture.
Validation
----------
- Per-shard FTS5 parity: slim sidecar returns IDENTICAL rowids + bm25
scores to the source shard for top-10 of "dinosaurs extinct".
- 3-way bench (legacy local / corpus local / slim-FTS5 over real
bucket fallback): 5/5 source agreement AND 5/5 audit_mode agreement
between corpus and slim_fts. Q5 legacy disagreement (Edwina vs
Dinosaur) is the pre-existing 2000-line query() retrieval quirk,
unrelated.
- End-to-end cloud query against published manifest-fts.json (cold-
start, ~149 s sidecar download once): STRICT · Dinosaur, every
quote verified (2/2).
- Full pytest suite: 2737 passed, 28 skipped, 1 xfailed. One pre-
existing failure (tests/test_doc_counts.py — claim_pack docs row-
count drift) and one pre-existing cold_object failure, both reproduce
on main HEAD.
Bucket state
------------
- s3://arborist/clones/sidecars-fts/00[0-3].idx.db (8.1 GB) — new
- s3://arborist/clones/manifest-fts.json — new
- s3://arborist/clones/manifest-sidecar.json — kept live (deprecated
but still readable; downstream callers should switch to
manifest-fts.json)
New ticket for Joseph (@TrudoJo)'s 6-dim procedural spatial-anchor
framework as the first verifier kernel under the #000013 v7-W reserved
namespace. A single committed 32-byte SHA-256 hash deterministically
expands into six 32-byte regions H1..H6 via the HMAC-SHA-512 KDF already
shipped in arborist/substrate/anchor_prg.py (#000035); each region drives
one quantized object dimension under a fixed canonical mapper. Domain
separation from #000035 by dedicated spatial_anchor_seed published in
the v7-W manifest -- preserves #000035's KAT freeze + dav1d 2026-05-11
final review intact.
Bundle:
- docs/tickets/ticket-000070-spatial-anchor-pi-w-object.md (718 lines):
full spec with 8 design-choice subsections, working Python sketch,
12-NOT scope boundaries, 8-row cross-references, five-step deletions,
10 open questions for dav1d.
- bench/spatial_anchor_validation.py: pure-stdlib pre-review evidence
(~2s, RNG-seed-pinned, reproducible). Five benches: avalanche,
cell-distribution uniformity, collision vs birthday-bound, cross-region
independence, domain separation.
- bench/spatial_anchor_validation_results.md: report from first run.
- Makefile: 'make bench-spatial-anchor [SPATIAL_N=N]' target + PHONY.
- docs/TICKETS.md: index row + Next ID 000070 -> 000071.
Bench headlines (N=10000):
- Avalanche mean 767.85 bits (PRF null 768, z=-0.49) -> PASS
- Cell-distribution chi^2 |z|<1 at L=2,3,4 -> PASS
- Birthday-bound ratio obs/exp 0.989/1.038 at populated L -> PASS
- Cross-region Pearson all 15 pairs < 2sigma -> PASS
- Domain separation Arm A 767.91 / Arm B exact 0 collision -> PASS
Five of dav1d's ten open questions (Q1 seed source, Q2 segmentation,
Q3 position mapper, Q8 endianness, Q9 KAT adversarial vectors) now
resolve with measurements rather than appeals to PRF authority.
Q4/Q5/Q6/Q7/Q10 remain non-empirical design decisions.
Status: open, awaiting dav1d review + fox go/no-go. No registry slot
booked, no substrate-paper amendment landed, no kernel module created.
The Hank-Scorpio-vs-Mr.-Burns problem: body BM25 alone outranks the
canonical primary-source article ("Homer Simpson") below sibling
articles that incidentally mention the same tokens. The sidecar
pipeline already had title-boost-with-extras-penalty; lift it into
a shared utility that the orchestrator applies to ANY adapter's
fts_body output.
arborist/qa/corpus.py:apply_title_boost(hits, query, *,
higher_is_better)
Stems both sides (possessive + plural collapse), numeral-expands
(7↔VII), accent-folds (é→e). Effective bonus per hit:
max(0, overlap - extras/2) * boost
where extras = title tokens NOT in query. Score direction honors
each adapter's convention (BM25 negative → subtract; sidecar BM25
positive → add).
arborist/qa/corpus_query.py:run_query
Now oversamples fts_body by 4× and reranks via apply_title_boost
before slicing to top_k. Without oversampling the right primary
can sit at rank 7-15 in the body BM25 output and get cut before
the rerank sees it.
bench/three_way_bench.py: NEW — drives same fixture through
- `arborist query` (legacy 2000-line pipeline)
- `arborist corpus-query` (new, local shards via Corpus + run_query)
- `arborist cloud query` (new, sidecar via Corpus + run_query)
prints 3-column table + flags primary-source disagreements.
Smoke fixture (5 questions, all paths):
4/5 all-paths agree on primary source (was 2/5 pre-fix)
1/5 disagrees: dinosaur-extinction
legacy: UNGROUNDED · Edwina (children's book) ← wrong
corpus: STRICT · Dinosaur (main article) ← right
cloud: HYBRID · Edwina (children's book) ← wrong
The new title-boost lifted corpus-query above the legacy here;
cloud still picks Edwina because SidecarReader.search applies
title-boost INTERNALLY (it predates the shared util), so the
extras-penalty stacks weirdly when run_query applies it again.
Next fix: disable internal boost in SidecarReader once shared
util is the single source of truth.
26 corpus/wallet/sidecar tests still green.
`arborist/qa/corpus_query.py:run_query(corpus, question, chat_client, ...)`
is now the single retrieval + evidence + LLM + verify + annotate
pipeline. Takes any Corpus adapter (SqliteShardCorpus | SidecarBucketCorpus
| future edge-proxy), returns the same result-dict shape today's
cloud-query emits (audit_mode, sources w/ used + pointer-ids, capacity,
timings, raw_answer, rendered answer).
`_cmd_cloud_query` reduced from ~310 lines of inline pipeline to ~80
lines of corpus construction + run_query call + progress emission +
render. Behavior identical: same audit_mode, same sources, same
capacity/timings tail.
Before: cli._cmd_cloud_query owned chunk-pull, evidence-build, prompt
construction, LLM call, verifier call, source annotation,
spotlight render — 310 lines of duplication with the local
query() pipeline.
After : cli._cmd_cloud_query owns ONLY corpus construction +
progress emission + render layer. The pipeline lives in
corpus_query.run_query and will be the single source of
truth once query.py refactor moves local onto the protocol.
Tests (corpus_query unit, 4 passing):
* verbatim quote → STRICT (n_verified ≥ 1, used annotation correct)
* capacity + timings dict populated with expected keys
* empty retrieval → UNGROUNDED + zero LLM calls (StubClient.calls == [])
* corpus.name leaks into result for bench attribution
Bench parity preserved (cloud_vs_local.py smoke, 5 questions):
before refactor: 1 regression (Mercury Seven STRICT → HYBRID)
after refactor : 1 regression (same — LLM-stochastic, primary source
identical both sides)
Behavior-preserving. Foundation laid for the local query() refactor:
when query.py learns to take a Corpus parameter, it'll delegate to
the same run_query() and every quality fix lands once for both.
32 prior wallet + corpus tests still green.
Renamed DEFAULT_PRICE_CENTS (per-call) → DEFAULT_DOLLARS_PER_1K_GROUNDED.
Per fox 2026-05-31: at owned-hardware scale, COGS is naturally
expressed per-1000-grounded-answers (hermes $0.09, qwen $0.16),
not per-call cents. The baseline already amortizes ungrounded calls.
Report now shows: per-model implied spend, effective COGS in
$/1k grounded, and cascade delta as % vs always-most-expensive.
20-question live bench (real run): cascade beats always-qwen by
-39% COGS AND grounds more answers (19/20 vs 17/20). Hermes and
qwen are equally reliable (17/20 each) but catch *different*
questions — additive coverage.
Adds bench/cross_model_selfplay.py + `make bench-cross-model` target.
For each question in a fixture, runs `arborist query` once per
configured model (default: Hermes + Qwen) and tabulates:
* audit_mode per model (EVIDENCE-WARRANTED → POINTER-LINKED → UNGROUNDED)
* agreement on primary source URI
* grounding rate per model
* estimated $/grounded-answer (per-call prices configurable)
* cheap-first cascade analysis (try cheapest, escalate on UNGROUNDED)
This is the "ask twice for two options" pattern from the agent
perspective — bakes it in as a benchmark so we can measure whether
the cascade beats always-using-the-stronger-model on $/grounded.
First live run (2 questions × 2 models, $0.41):
- Hermes: 1/1 grounded (1 timeout — operational issue)
- Qwen: 2/2 grounded STRICT
- Cascade: 2/2 grounded for $0.25 — beats always-Qwen ($0.32)
when Hermes succeeds on its first call.
Output: bench/cross_model_results/<utc-iso>.{jsonl,md} (gitignored).
Cloud-query was picking sibling articles (Mona Lisa's Revenge instead
of Mona Lisa, Republics of the Soviet Union instead of Soviet Union,
Mercury 13 instead of Mercury Seven) because:
1. Title-boost counted overlap but not extra title tokens. Both
"Mona Lisa" and "Mona Lisa's Revenge" overlapped query by 2 →
same boost → BM25 favored the shorter movie article.
2. RRF merge squashed per-shard rank-1 hits into a 1/(60+1) tie
across 4 sidecar shards. Tie-breaking was undefined; the right
article was as likely to lose as win.
Two fixes:
sidecar.search title-boost:
+ Filter title tokens through STOPWORDS + len-1 cutoff so 's', 'of',
'the' don't count as extras.
+ Penalty: extras = |title_tokens - query_tokens|; effective bonus
is `max(0, overlap - extras/2) * title_boost`. "Mona Lisa" gets
full bonus; "Mona Lisa's Revenge" gets half.
MultiShardSidecarCorpus.fts_search merge:
+ When all CONTRIBUTING shards have sidecars (their BM25 + boost
scores are directly comparable), merge by max raw score across
shards. RRF was masking score discrimination at the top of the
list.
+ Mixed (sidecar + bucket-direct FTS5) falls back to RRF since
those scales aren't comparable.
Bench (5-question smoke, cloud_vs_local.py):
before fix: 4 regressions / 5
after fix : 1 regression / 5 (and that one is the right source,
only the audit_mode dropped STRICT
→ HYBRID due to LLM-stochastic answer
phrasing)
Phase 2 — bench instrumentation + measurement run
bench/qa_sweep.py picks up the answerability sidecar projection per row
(answerability_fired, answerability_confidence, answerability_denial_
pattern, answerability_answer_type, answerability_candidate_count) and
aggregates per-mode (answerability_fires + S/M/W confidence breakdown)
into a new column in the markdown summary table.
Measurement run on bench/qa_results/phase2-sidecar-on/2026-05-27T14-
16-22Z (76 questions × n=3 × claim_lattice × Hermes-3-8B × tail layout,
228 runs). Headline:
sidecar fires 2/228 (0.88%)
confidence dist 2 strong / 0 medium / 0 weak
precision 100% (2/2 fires were the Ballestrini fixture)
recall on Ballestrini 2/3 across n=3 (third run model extracted
correctly -> sidecar silent,
correct behavior)
false positives 0/226 non-Ballestrini runs
verifier verdict both fires labeled STRICT by the binary
verifier (the verifier-blind class, exactly
as predicted)
Detection rule's three-clause conjunction (denial + extraction-shape +
candidate proximity near cleaned subject tokens) is operating at the
precision floor. The strong-confidence-only firing pattern is what
calibrates Phase 3's demote threshold.
Phase 3 — opt-in demote flag (default OFF per Dav1d Phase 4 NO-GO)
arborist/qa/keys.py: answerability_demote_enabled added to
_VERIFIER_POLICY_FIELDS so flipping the flag partitions cache via
verifier_policy_hash. Justification: when on, the rendered audit_mode
changes (EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL), which IS a
verifier-output property; verifier hash must move accordingly. The
other answerability_* fields stay governance-only (sidecar
diagnostic, no audit_mode mutation).
arborist/cli.py:_render_audit_label extended with answerability +
demote_enabled kwargs. Logic:
demote_triggers = (
demote_enabled
and answerability["answerability_warning"] is True
and answerability["confidence_class"] in ("strong", "medium")
)
lattice modes:
EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL (rung transition)
POINTER-LINKED / ANCHOR-WARRANTED -> "rung · missed-answer"
(tail tag; rung itself already
signals degradation)
non-lattice modes (quote/span/entity/paraphrase):
audit_mode token unchanged + "· missed-answer" tail tag
weak confidence: NEVER demotes (Phase 2 saw zero weak fires on real
failures; reserved for future expanded detection ladder)
CLI flag --demote-on-missed-answer on both `arborist query` and
`arborist ask`, default OFF. Flows into call_policy[
"answerability_demote_enabled"] and through to result[
"answerability_demote_enabled"] so the renderer reads it without
needing the policy dict.
End-to-end verified live: 4 fresh Hermes-3-8B runs with --demote-on-
missed-answer on `songs by veronica ballestrini`, all 4 rendered
EVIDENCE-MISSED-PARTIAL · via claim_lattice (Hermes hit the failure
mode in all 4, sidecar fired strong, demote logic transformed the
label).
Phase 4 (default flip to demote-on) — NO-GO per Dav1d 2026-05-27 §3.4:
"a false sidecar warning is tolerable; a false audit-label demotion
can damage trust in correct abstentions." Phase 2 precision is 100%
but n=2 fires is too few samples to claim precision floor empirically.
Default flip blocks on wider bench + human spot-check of the warnings.
Tests: 47 total (36 Phase 1 + 11 new Phase 3 covering hash partitioning
discipline + render-label projection across all four rung/confidence
matrices). Full suite 2794 passed (delta +22 from prior 2772).
Bench output (bench/qa_results/phase2-sidecar-on/) intentionally not
committed — bench/qa_results/ is gitignored per existing convention;
the ticket carries the headline numbers + path for re-inspection.
format_user_payload helper in arborist/qa/prompts.py becomes the single
source of truth for the user-turn payload. Three layouts:
tail (default) evidence first, question at end (prior behavior)
bookend question repeated before AND after evidence — counters
lost-in-the-middle on small models (≤8B)
per_chunk bookend + a one-line [for: <q>] reminder before each
evidence block; for list/extraction queries
USER_PAYLOAD_LAYOUTS constant exported; unknown layout raises ValueError.
The six _user_payload closures in query.py (3) and runner.py (3) all
delegate to format_user_payload. Quote-mode passes per_chunk_marker=None
to fall back to bookend on flat document/sources context.
Wired through both DEFAULT_QUERY_POLICY and DEFAULT_POLICY. Folds into
governance_policy_hash (the layout changes the user-turn content the
model sees, so the policy hash partitions cleanly per layout); does NOT
fold into verifier_policy_hash (verifier rules unchanged).
Makefile gets LAYOUT_DEFAULT ?= tail and LAYOUT ?= $(LAYOUT_DEFAULT) so
operators can flip per-call (LAYOUT=bookend make query Q="...") or
session-wide (LAYOUT_DEFAULT=bookend make query Q="..."). Recommendation
matrix in the Makefile comment block above the query target encodes the
2026-05-27 bench finding.
Motivating case: the Veronica-Ballestrini "songs by" failure. Hermes-3-
8B under tail layout returned "specific songs by her are not mentioned
in the provided evidence" when evidence E2 literally contained the song
names. Same query under bookend recovered the answer (with conflation
between Veronica Ballestrini and The Veronicas); under per_chunk
recovered AND disambiguated three entities. Qwen-27B unaffected by
layout. The Ballestrini case is added to bench/qa_questions.txt as a
regression fixture under "entity list", with a 4-line comment pointing
to docs/user-payload-layout.md.
2026-05-27 bench (n=3 × 75q, claim_lattice mode, Hermes-3-8B):
tail STRICT 94/225 (0.418) — control
bookend STRICT 95/225 (0.422) — +0.44pp (noise, 5pp floor)
per_chunk STRICT 72/225 (0.320) — -9.78pp (significant regression)
Verdict: tail stays default (cache-preserving and bench-confirmed
neutral). Bookend/per_chunk available as opt-in operator knobs. Per_chunk
regresses in aggregate because the per-chunk reminder over-anchors the
model on every chunk (TOO_MANY_EVIDENCE_IDS violations rose from 20 →
54; mean answer chars in 32-64KB bucket doubled from 720 → 1660). The
Ballestrini-class failure is real but rare across the curated set; a
layout fix that helps the rare case at the cost of 10pp aggregate is a
bad default trade. Documented in full in docs/user-payload-layout.md
along with the Dav1d 2026-05-27 review framing (GO for opt-in, NO-GO
for default promotion, ADD companion missed-answer guard).
CLI changes (--user-payload-layout flag on `query` and `ask`) landed
separately in commit e5ee283 alongside the #54 busy_timeout fix.
Each producer shard now optionally emits a THIRD pack alongside its
metadata and chunks packs: an "fts" pack containing the FTS5 shadow
tables (chunks_fts_data, chunks_fts_idx, chunks_fts_docsize,
chunks_fts_config + the documents_fts_* counterparts) packed as a
fresh SQLite file inside the tar so BLOB columns round-trip natively.
Consumer detects fts_pack_hashes in the metadata pack's manifest,
pulls each fts pack, ATTACHes the embedded sqlite, INSERTs every
shadow-table row into its target's empty shadow tables, and SKIPS
the local FTS rebuild entirely.
Producer side:
arborist/cold_object.py
+ PACK_KIND_FTS = "fts"
+ FTS_SHADOW_TABLES tuple (8 shadow tables)
+ build_fts_pack(src_db_path, ...)
creates a temp sqlite, applies SCHEMA_SQL (so destination
has FTS virtual tables → shadow tables auto-created), copies
every shadow-table row from src via cursor iteration, packs
the sqlite file into tar.zst
+ ParsedManifest.fts_pack_hashes
+ parse_manifest reads _fts_pack_hashes records
+ build_metadata_pack accepts fts_pack_hashes parameter and
writes the new manifest record
arborist/evict.py:push_pack
+ include_fts: bool = True parameter (CLI --no-fts opts out)
+ Phase B.5 emits the fts pack BEFORE Phase C (metadata pack)
so its hash can be referenced in the metadata manifest
Consumer side:
arborist/evict.py
+ _pull_fts_pack_into_targets() — pulls fts pack body, extracts
embedded sqlite, ATTACHes into each target, INSERT OR IGNORE
every shadow-table row. INSERT OR IGNORE protects against
rowid collisions on other targets that don't own these chunks.
+ hydrate_from_metadata_pack_routed iterates fts_pack_hashes in
full mode, calls _pull_fts_pack_into_targets per pack
+ _pull_pack_inner_routed returns fts_pack_hashes_referenced in
its result dict (mirrors chunk_pack_hashes_referenced)
CLI / Makefile:
arborist cold pack --no-fts (opt-out)
make cold-hydrate (auto-detects: if
chunks_fts_data is
already populated
on shard 000 after
unpack, skip the
rebuild post-pass)
make cold-hydrate HYDRATE_REBUILD_FTS=1 (force rebuild)
make cold-hydrate HYDRATE_REBUILD_FTS=0 (skip rebuild)
Schema:
cold_pending.kind CHECK extended to include 'fts'
pack_key() accepts kind="fts" → packs/<hash>.fts.tar.zst
Expected wall-time impact on the 3090 genesis bench:
with fts in packs: no rebuild step → ~5-10 min total wall
without fts: rebuild post-pass needed → ~15-20 min
Trade-off: ~30-50% larger bucket (FTS shadow data per shard) for
~70-90% faster consumer hydrate. Producer flips the trade via
--no-fts. The fts pack is optional in the manifest (empty list →
consumer falls back to rebuild) so old bucket data without fts
packs continues to work unchanged.
34 cold-unpack-routed + migrate + planner tests pass.
Self-contained benchmark for the SPV-wallet validation. Records:
PRODUCER bucket state + pack count + compressed bytes
CONSUMER wall time, exit status, post-hydrate shard sizes,
per-shard documents/chunks/edges counts
Driver runs against the real DO Spaces bucket + the real fresh peer
on 3090-ai.foxhop.net. Writes one JSON artifact per run to
bench/results/cold-pack-roundtrip-<ISO>.json so a future operator
can diff hydrate times across pack-format changes (#000061 v3 →
graft mode #000066 → mesh-pull future).
Consumer command uses /usr/bin/time -v wrapped around
`arborist cold unpack --hydrate-shards-dir … --hydrate-M 4 --full`.
Hydrates into ~/.arborist/shards-genesis-test/ so it doesn't
clobber anything on the 3090.
No new ticket — this is task #45/#46 instrumentation. Existing
tests untouched.
Three pieces, all read-only or additive — no shard mutation, no
schema-version bump:
1. shard_for_document(document_root, M) in arborist/document.py.
Pure function: int(document_root[:8], 16) % M. 22 tests cover
determinism, range-bounds, near-uniform distribution (±5pp at
N=20k), and seven lock-in fixtures so peers will disagree
loudly if anyone changes the formula.
2. corpus_shard_count meta field + get/set helpers in store.py.
Lives in the existing key/value meta table; SCHEMA_VERSION
stays at v9.8.0 (the DDL doesn't change and source_root is
layout-independent, so cache records survive a reshard).
Legacy shards (without the field) return None; reshard tool
populates it on every target shard at migration time.
3. Pre-migration snapshot captured to
bench/results/pre-migration-snapshot.json:
docs 3,468,392 (3,468,226 globally unique)
chunks 6,235,764
edges 90,593,537
audit 3,468,403
This is the reference set post-reshard row counts must match.
4. Audit-event extraction script writes all 3.47M events from
all 4 shards to /tmp/audit-events.ndjson (2.0 GB) for the
Option-A canonical-chain consolidation step. Verifies chain
integrity on extract — all 4 source chains report 0 breaks.
5. Fixed a wrong chunk count in docs/corpus-history.md
(had ~3.54M/shard; actual is ~1.56M/shard) and added the
edge-count column (~22.6M/shard, 90.6M total). 6.24M chunks
total, not 14.12M.
Tests: 29 new pass (22 routing + 7 meta). No existing tests
touched.
Pinned the canonical shard count at M = 4 based on real-Wikipedia
ingest + query benchmark (bench/shard_count_sweep.py). Captured the
"when does SQLite stop being the right substrate" decision tree so
future operators know what bench would justify a fork or replacement.
Bench numbers (Wikipedia 2003 cur dump, 2000 docs, 4 cells of
M ∈ {1, 2, 4, 8}, 50 FTS queries per cell):
M chunks/s q_p50_ms q_p99_ms attach_ms
1 3,648 0.05 0.20 1.61
2 5,649 0.03 0.19 3.24
4 6,405 0.07 0.30 9.00
8 6,959 0.03 0.28 9.65
Key observations:
- M=1→M=2 is the biggest ingest win (+55%). Most gain happens there.
- M=2→M=4 is +13%. M=4→M=8 is only +9% — diminishing returns.
- Real wikitext canonicalization is per-worker Python CPU bound, not
SQLite-writer-lock bound. More shards don't unlock more CPU.
- Query p50/p99 is flat across M within noise (50 queries small).
- ATTACH cost grows linearly: 1.6 / 3.2 / 9.0 / 9.7 ms.
Why M=4 specifically:
- Captures 92% of peak ingest throughput (6,405 / 6,959).
- 6 ATTACH slots free under SQLite's 10 ceiling for aux DBs
(qa.db, snapshots.db, selfmodel-chain.db, crawl_*.db, future
mesh_*.db) — comfortable headroom. M=8 leaves only 2 slots.
- Mobile-tolerable: phone NAND attach is 5-10x slower than NVMe;
M=4 = 45-90 ms cold start (instant), M=8 = 50-100 ms (sluggish
with no headroom).
- Matches fox's current 4-shard layout = cheapest migration.
Decision tree for when SQLite stops being right (full text in
ticket §"When the SQLite-default substrate stops being right"):
A. ATTACH ceiling pressure (auxiliary DBs grow past 5) → bench
forked SQLite with SQLITE_MAX_ATTACHED=125, M ∈ {16, 32, 64};
if attach cost stays linear past M=10, fork viable but pays
permanent "no longer stock sqlite3" tax.
B. Ingest hits >10k chunks/s sustained ceiling → first tune
page_size / WAL checkpoint / mmap_size / synchronous. If
tuning gets 2-5x, stay on SQLite. If still ceiling-limited,
candidates: DuckDB (columnar, MVCC, FTS), libmdbx (B+tree no
FTS; we'd build it). In-house DB rejected without specific
failure of those.
C. Federation needs multi-writer-same-shard → SQLite writer-lock
serializes peers, becomes federation bottleneck. First try
leader-election (single-writer-per-shard with WAL replication
to followers). If true multi-writer required, SQLite is wrong;
candidates: FoundationDB, CRDT-on-KV-store. DuckDB does NOT
solve this — its MVCC is single-process.
Honest verdict: for current arborist workload (single-writer-per-
shard, read-mostly federation), stock python3 sqlite3 is the right
substrate. None of A/B/C are close to firing. The bench discipline
exists to know what to measure when something changes.
bench/results/shard-count-sweep-2026-05-26T16-20-48Z.csv (synthetic
baseline) + 2026-05-26T16-31-34Z.csv (real Wikipedia) committed as
the load-bearing measurement for the M=4 choice.
The §12.1 pipeline I added was second-hand from benchmark-matrix.md
and got several things wrong against the code:
- listed 4 verdicts; actual is 5 (missing FABRICATED — the
fabrication-vs-WRONG split that energy-cogs §5.5 leans on for the
qwen-fabricates / hermes-abstains finding)
- "θ=0.85" was right by accident — but it's the code-judge-pinned
_CODE_JUDGE_THETA_CONTRA constant, raised from the manifest 0.5
default after measuring 114 FPs in the 0.5-0.75 band
- omitted the short-answer entity-grounding fast path (which runs
BEFORE NLI per the 2026-05-19 Poland-Tusk smoke)
- omitted the HYBRID rescue ladder (NLI entail / entity rescue /
2026-05-21 verbatim-quote-on-topic rescue)
- conflated WRONG and FABRICATED (the subject-in-gold split is what
distinguishes "source has the topic but a different value" from
"source silent on the topic")
Rewrote §12.1 grounded in `bench/judge_code.py:judge()` (its own
docstring at line 501-528 is the truth on rule order), with file:line
citations and the verdict-mapping in full.
Also fixed a real artifact-vs-doc drift INSIDE the judge: the
module-top docstring still claimed θ_contra default 0.5 and omitted
the short-path and the WRONG/FABRICATED split. Updated to match the
authoritative judge() docstring + current code.
No behavior change — docstring + benchmarks doc only.
The code judge bailed to JUDGE_ERROR on 40% of in-corpus answers: HYBRID
(partial grounding) with low NLI entail, where the entity-grounding
rescue needs ZERO unsourced specifics. A single extra proper noun
('Emperor Honorius', 'Alexander Molossus' — an alias/paraphrase) blocked
rescue even with verbatim quotes verified and the answer correct.
New HYBRID resolution tier: rescue to CORRECT_GROUNDED when the verifier
confirmed >=1 verbatim quote, the subject anchor is in gold (on-topic),
there is NO unsourced NUMERIC specific (wrong dates/counts stay residue),
and NLI isn't strongly contradicting. Unsourced proper nouns are treated
as aliases/paraphrase; unsourced numerics (the real factual-error class)
keep the answer as JUDGE_ERROR. Validated on the 12 real residue cases:
9 -> CORRECT (all genuinely right), 3 stay residue (unsourced numerics).
JUDGE_ERROR 40% -> ~10%. self-test 4/4; 2 new tier tests; suite 2549.
Was a stale hardcoded 'same Hermes; judge=Opus hermetic' label that
misreported any run with --model/--judge overrides (e.g. qwen + code
judge). Now reflects the real config — honest header for the artifact.
fox 2026-05-21: account wattage for input and output separately. Prefill
(process all prompt tokens, parallel/compute-bound) and decode (generate
output, autoregressive/bandwidth-bound) are different GPU ops with
different J/token — a single per-token number can't represent both.
Slope calibration (no sub-request power alignment): sweep prompt length
at tiny max_tokens -> prefill J/input-tok (fixed overhead cancels in the
slope); fix a tiny prompt and sweep forced output length (ignore_eos) ->
decode J/output-tok. Prefill kept COLD (unique filler so cached_tokens=0).
Reuses watt_bench probes. Bad points (context overflow) skip, not abort.
Measured qwen-nothink/4090 @$0.33/kWh: prefill 0.175 J/tok
($0.016/M-input-tok), decode 6.16 J/tok ($0.564/M-output-tok) — decode
35x dearer per token. Predicts measured substrate J/q within ~5%. 14
tests (+ slope). Validated live.
fox 2026-05-21: (1) use REAL API token usage, not len//4; (2) the
substrate prefills a large retrieved CONTEXT as INPUT while solo feeds
~nothing, so per-completion-token over-charges the substrate — and per-
TOTAL-token UNDER-charges it (its mix is ~98% cheap prefill tokens).
Measured n=30 qwen-nothink/4090: substrate prefills ~6.6k input tok/query
(claim_lattice) vs solo ~52 — ~127x. Neither single per-token denominator
is honest; prefill (parallel, cheap/tok) and decode (autoregressive,
dear/tok) must be costed separately.
- OpenAICompatibleClient stashes data['usage'] as .last_usage (non-
invasive; return type unchanged).
- watt_bench captures real prompt_tokens + completion_tokens per call
(both arms), aggregates per cell, and energy_cogs reports gross +
marginal per BOTH 1k-total-tok and 1k-completion-tok plus the context
size. Prints the prompt/completion split.
- 12 tests incl. the prompt-context artifact (per-total cheap, per-
completion dear). Full suite 2540 passed.
The clean per-input-tok / per-output-tok split rides bench/watt_calibrate
(slope calibration; separate commit once validated live).
fox 2026-05-21: 'we dont touch card during retrieval'. Retrieval + verify
are CPU/SQLite on the orchestrator; the GPU is idle (at the shared, always-
on model-resident floor) during them. So the substrate does NOT 'hold the
card' through its wall-clock — the gross window integral over-counts by
charging that always-on floor for the seconds we spend retrieving (energy
that exists regardless of the query; the card serves other traffic then).
Correct attribution: the GPU cost of a query is its GENERATION energy only
(the marginal — burst above the serving floor). Relabel: marginal is the
headline GPU COGS; gross is demoted to a reference 'window total, not
query-attributable'. Print + energy_cogs docstring + stock-v1-config doc
updated. No math change (marginal was already right) — this corrects the
narration. Measured qwen-nothink/4090: substrate GPU COGS is LOW
(claim_lattice $0.21/M-tok, quote $0.83) — its real overhead is latency
(CPU retrieval), not GPU watts.
fox 2026-05-21: 'gen 200W' was a bug — joules/window blends the ~400W
generation bursts with the sub-100W gaps (retrieval/verify/network) into
a power state the card never sits at. A card occupies DISTINCT states
(idle / middle-idle = resident-between-requests / generation), differing
per card×model×server.
watt_probe.classify_power_bands(): largest-gap split of the window
samples into a low band (serving floor) and high band (generation draw)
+ duty cycle. Data-derived, never hardcoded — tested at two scales. The
worker emits the decomposition + raw samples; RemoteProbe/LocalProbe
expose band_stats() uniformly.
energy_cogs: marginal now taken against the measured SERVING FLOOR (the
standing cost of being ready), not deep idle; the blend is kept but
labelled window_mean_w. Reports idle/serving-floor/gen-draw/duty.
Cache-miss certainty (fox's question): the arborist arm runs
burn_existing=True (force-deletes any live providence row before
inference) and asserts cache_hits==0 with a loud warning + real_inference
flag — so we time real generation, never a SQLite lookup. Solo has no
cache path. 11 tests (energy math + band split). Validated live on the
isolated 4090: solo gen 308W/70%-duty vs substrate 396W/8.6%-duty —
substrate marginal/tok is LOWER, gross/tok higher (it holds the card
longer for retrieval).
fox 2026-05-21: compute cost-of-goods-sold by kWh vs tokens, with the
three power states (idle / warm-idle / generation) MEASURED per
card×model×server — never hardcoded (his 40/127/380 W were illustrative
of one 3090). The only operator input is --price-per-kwh (default 0.33
USD/kWh, a configurable site rate).
energy_cogs() (pure, unit-tested) decomposes measured generation energy
against the measured warm-idle baseline:
* gross — all measured joules over the window (all-in, includes the
warm-idle cost of keeping the model hot, amortized).
* marginal — joules ABOVE warm-idle: what one more request's burst
actually costs (clamped >=0).
kWh = J/3.6e6; $/1k-tok is the unit that compares to API pricing. Both
surface per cell + a COGS print line.
watt_bench's arborist arm now loads the frozen bench.stock_v1 policy
(--answer-mode, drift-guarded on non-reasoning) so cost is measured for
the SAME substrate the campaign grades. Cells record
window_start/end_unix so a post-hoc load_monitor queue-depth cross-ref
can flag organic-traffic contamination on the non-isolated single-slot
endpoints. 6 COGS tests; full suite 2534 passed.
fox 2026-05-21: characterize substrate-ON under BOTH answer shapes, so
answer_mode is a swept axis, not a single pinned value.
stock_v1.py now exposes STOCK_V1_POLICIES{quote,claim_lattice} +
STOCK_V1_GOVERNANCE_HASHES (quote 5b6ca4c5..., claim_lattice 036a4c79...),
policy_for(mode), and assert_not_drifted(mode). Shared pins (crosslang
OFF, repair OFF, quantifier dry-run, metacognition label-only,
soft-preflight OFF, claim cap 12, v2-acronym-aware) are frozen
identically across modes.
Wire the treatment arms to the pin (the consumer-side step that makes
the freeze real):
* control_ab --answer-mode {quote,claim_lattice}
* control_sweep --arborist-answer-mode {quote,claim_lattice}
Both default claim_lattice (prior behavior), call assert_not_drifted on
non-reasoning runs (halts the sweep if DEFAULT_QUERY_POLICY drifts), and
load the frozen policy_for(mode) instead of an inline
dict(DEFAULT_QUERY_POLICY, ...). Reasoning refs (phase 3) keep their
documented JSON overrides and skip the assert by design (different hash).
jaggedness is left standalone — it is a mode-agnostic retrieval
instrument, coupling it to the answer-policy freeze adds friction with no
correctness gain. Full suite 2528 passed.
Before the multi-day campaign (hermes 3090/4090 -> qwen 3090/4090 ->
reasoning variants) the substrate-ON treatment arm must NOT drift. It
previously inherited DEFAULT_QUERY_POLICY implicitly, so any mid-run
edit would silently change what 'substrate-ON' means.
bench/stock_v1.py snapshots DEFAULT_QUERY_POLICY + re-asserts the
load-bearing pins (answer_mode=quote, crosslang OFF, repair OFF,
quantifier caps dry-run, metacognition label-only, soft-preflight OFF,
claim ceiling 12, v2-acronym-aware), then hashes the whole effective
dict. assert_not_drifted() fails loudly if that hash ever changes —
re-pinning is a deliberate fox-gated V.2 bump, never silent. The whole
campaign is identified by one governance_policy_hash
(5b6ca4c5...aade4e). Non-reasoning + non-distributed are harness axes
(reasoning -> phase 3, mesh -> later fork), not policy fields.
docs/stock-v1-config.md documents V.1, substrate-OFF (control_ab arm A),
the campaign matrix, and the energy-COGS companion (#000057) — whose
power states (idle / warm-idle / generation) are MEASURED per
card+model+inference-server at runtime, never hardcoded; only $/kWh is
an operator flag.
Answers fox's 2026-05-21 question — 'are we being swamped because we're
open to internet?' — before the multi-day GPU bench, where uncontrolled
internet traffic on the single-slot hermes (3090/vLLM) and qwen
(4090/llama.cpp) endpoints would contaminate wattage + throughput.
Three stdlib subcommands (urllib + sqlite3 + hand-rolled SVG, nothing to
install):
poll — scrape each endpoint's Prometheus /metrics on an interval into
SQLite; queue depth (num_requests_waiting / requests_deferred)
is the swamp signal a single GPU slot exposes. Prunes past
--retention-days each cycle (bounded store, no cancer growth).
graph — multi-panel SVG: queue depth, running, req/s, tok/s, e2e latency.
access — parse Caddy/nginx access log for the real client IPs the backend
can't see behind the proxy hop; top talkers + per-IP rate SVG.
Backend /metrics = HOW MUCH; proxy log = WHO. make monitor-poll /
monitor-graph / monitor-access.
v1 of the same-model substrate-delta harness's non-jagged metric.
For one corpus title, surface-perturb its question (numeral / accent /
hyphen / honorific / amp / brit) preserving the referent, then ask
whether retrieval surfaces the SAME target for canonical vs perturbed
phrasing. J_norm = XOR disagreement rate @k (lower = less jagged);
graded mean |Δrank| catches rank instability the binary metric misses.
Pure query --dry-run: no LLM, no verifier, no judge, no n=3 noise, no
5pp floor — the recall_at_k discipline. Reuses recall_at_k.probe +
mine_questions._surface_variant. Feeds #000012 ForkScore
ΔJaggednessReduction. A-vs-C answer-quality arm already exists under
#000057 (control_ab/control_sweep) — not rebuilt. Curvature + LLM-arm
jaggedness delta remain open (ticket §8).
make bench-jaggedness JAGGED_LIMIT=40 JAGGED_K=8
Implements fox's 2026-05-20 architecture: don't ship shards to the
tight-on-disk GPU boxes; run the benchmark FROM the laptop (local
shards + retrieval + judge + workload loop, driving the worker's LLM
endpoint over the network) and have the worker boxes REPORT their own
power.
Adds a probe abstraction with two implementations behind one
start/stop/gpu_stats/cpu_stats interface:
- LocalProbe: wraps PowerSampler + CpuSampler (watt_bench runs ON the
GPU box — the original mode)
- RemoteProbe: orchestrates bench/watt_probe.py on a remote worker
over SSH (scp the stdlib probe once, launch --until-file detached,
touch the stop-file after the workload, fetch the JSON). The worker
needs no shards / arborist / venv.
--remote-gpu-host HOST selects RemoteProbe; --endpoint points at the
worker's endpoint. Idle baseline now records gpu_util_mean_pct and
flags ">5% ⇒ contaminated by live traffic" so the 3090-style
contamination (live hermes traffic polluting the baseline) is visible
in the output, not silent.
Verified end-to-end: laptop-driven n=2 solo qwen, power sampled on the
4090 over SSH — idle gpu 20.2W @ 0% util (clean) + cpu 86W; load
populated; report tagged mode=remote. The on-box LocalProbe path is
unchanged (omit --remote-gpu-host).
Report schema gains mode / remote_gpu_host / gpu_available /
cpu_energy_available. Per-cell records gpu+cpu+total joules/question.
Splits the power-sampling half out for the laptop-driver / worker-
reporter architecture (fox 2026-05-20): don't ship shards to the tight-
on-disk GPU boxes; instead drive the benchmark FROM the laptop (which
holds the shards + judge + workload loop) and have the worker boxes
just serve the model and REPORT their own power.
Power sampling must run on the box (nvidia-smi + RAPL are host-local),
but nothing else does. watt_probe.py is stdlib-only — python3 +
nvidia-smi + readable RAPL, no venv, no arborist, no shards. Copy it to
the worker and run; it samples GPU power (nvidia-smi, trapezoid-
integrated to joules) + CPU package energy (RAPL energy-diff) over a
window and emits the same energy schema watt_bench's local samplers
produce. Two window modes: --duration (fixed) or --until-file (laptop
touches a stop-file — the workload-correlated mode). Records
gpu_util_mean_pct so live-traffic contamination can be detected.
Takes only the top-level RAPL package (intel-rapl:N, single colon),
not the :N:M core/uncore subdomains, so no double-count.
Verified on the 4090: scp'd to /tmp, ran with no install, real readings
(GPU 20.2W idle @ 0% util, CPU 82W, joules computed, 6 samples / 3.2s).
Next: laptop-side orchestration in watt_bench (--remote-gpu-host) that
SSH-starts this probe for each cell's window while the driver runs
retrieval + judge locally and drives the worker's LLM endpoint.
Toward fox's next goal: score the full serving stack on quality AND
cost — {qwen, hermes} × {llama.cpp, vLLM} × {3090, 4090} × {solo,
arborist}, measuring CG% + GPU watts + CPU watts + joules/answer per
cell.
watt_bench.py — adds CpuSampler (Intel RAPL package energy via
/sys/class/powercap/intel-rapl:*/energy_uj). RAPL exposes a cumulative
microjoule counter, so energy-over-window is an end-minus-start diff
(handles wrap) — more accurate than integrating instantaneous power.
Sums multi-package. energy_uj is root-only by default (PLATYPUS /
CVE-2020-8694), so it degrades to available=False when locked;
--cpu-energy-cmd 'sudo cat {path}' supplies a privileged reader when a
sudo rule exists. Each cell now reports gpu/cpu/total joules-per-
question + gpu joules-per-token; the report records cpu_rapl_available.
Verified: graceful degradation when locked; RAPL diff math (1->4 MJ uJ
= 3.0 J, exact).
benchmark-matrix.md — expands the cost section to the full 16-cell
(model × engine × GPU × arm) design, the per-cell metric set (quality +
GPU + CPU energy), the serving-stack inventory from 2026-05-20 recon
(4090=qwen/llama.cpp, 3090=hermes/vLLM — each box has one engine + one
model today), and the buildout gap (vLLM+qwen, llama.cpp+hermes, cross-
GPU models). Notes idle-floor asymmetry (hermes/3090 ~127W vs
qwen/4090 ~20W) as a real optimizer input.
Harness is ready; the serving-config buildout + RAPL perm grant are the
remaining (ops, fox-directed) prerequisites to run the full matrix.
Two deliverables for the cost/energy axis of the constraint optimizer.
docs/benchmark-matrix.md — shareable spec of the control experiment:
the question, fixture (386 office-holder Qs with corpus-vintage gold),
the 3-model × 3-framing × 2-arm matrix (18 cells), the verdict
vocabulary + two reads (accuracy vs grounding-fidelity), the
deterministic code judge + its Opus calibration, the results-so-far
table, and the NEW cost dimension (tokens / latency / GPU watts /
joules-per-answer measured per GPU tier). Self-contained — readable
cold by David.
bench/watt_bench.py — GPU wattage harness. Samples nvidia-smi
power.draw on the inference GPU while driving a small representative
subset, reports mean/peak watts, trapezoid-integrated joules,
joules-per-question, and joules-per-token. Tags the GPU
(--gpu-label 3090|4090) so the optimizer can compare hardware tiers.
Idle-baseline sampling separates load draw from idle. Does NOT grade
(energy is independent of correctness); saves answers + per-question
timing to JSONL for a later quality-per-joule pass via
score_with_code_judge.
Designed to run ON the GPU box (the orchestrator has no GPU; the
3090/4090 live on the inference boxes). Degrades gracefully when
nvidia-smi is absent (energy fields null) so it is testable anywhere.
Verified: PowerSampler graceful degradation + trapezoid integration
(synthetic 100->200->200W over 2s = 350 J, exact).
The headline cost finding the optimizer must weight: qwen-think
reasoning = 1300-3300 tokens/answer vs qwen-nothink ~50-100 (20-50x),
for a workload where arborist+qwen-nothink already lands 82% CG. The
energy numbers will quantify whether reasoning's premium is ever
justified — grounding-fidelity per joule, not per answer.
fox was right ("or you have defects still"): the arborist+qwen-think
empties were NOT an inherent reasoning limitation, they were
max_tokens set too low. Diagnosed with finish_reason + token usage on
a realistic large arborist-style context:
max_tokens=1024 → 4/4 empty, ALL finish_reason='length', all hit
exactly 1024 tokens
max_tokens=4096 → 0/4 empty, ALL finish_reason='stop', used
1339-3295 completion tokens
qwen-think spends 1300-3300 tokens on its (internal) reasoning trace
BEFORE emitting the tiny claim-lattice JSON. The arborist arm's budget
comes from DEFAULT_QUERY_POLICY["max_tokens"]=512 (correct for non-
reasoning single-line JSON), so reasoning refs hit finish='length'
mid-trace and return EMPTY. Fix: arb_policy["max_tokens"]=8192 for
reasoning refs (generous headroom over the 3295 observed ceiling).
This also CORRECTS a wrong claim in commit aa9d9c8's message: the
json-schema grammar does NOT suppress the reasoning trace. The
reasoning happens (it burns those 1300-3300 completion tokens); the
grammar only shapes the final emitted answer into schema-valid JSON.
So arborist+qwen-think is a genuinely distinct config, not a clone of
arborist+qwen-nothink. The earlier "grammar suppresses reasoning"
read was an artefact of toy-prompt diagnostics where reasoning fit
under 1024 tokens.
The empty-retry from aa9d9c8 stays as belt-and-suspenders for any
residual model-side empties, but with an adequate budget it should
rarely fire.
Cost note (fox's standing point, now quantified): reasoning = 1300-
3300 tokens/answer vs qwen-nothink's ~50-100 = 20-50x token cost. For
a workload where arborist+qwen-nothink already lands 82% CG, that cost
multiplier is the open question — not whether the cell is measurable
(it now is), but whether the marginal lift justifies 20-50x spend.
Two defects blocked the arborist+qwen-think cell, both diagnosed
2026-05-20:
Defect 1 — stop sequence truncates to empty. The claim_lattice path
sets stop=['\n\n'] (runaway guard tuned for single-line Hermes JSON).
A reasoning model's output trips it immediately → 100% empty answers
→ 100% ABSTAINED (measured on the first 23 items of the killed run).
Fix: MODELS gains an explicit flag; the arborist arm
clears claim_lattice_json_stop_sequences for reasoning refs. Direct
A/B confirmed: stop=['\n\n'] → ''; stop=None → valid JSON.
Defect 2 — intermittent empty completions. Even with the stop cleared,
qwen-think under json-schema grammar emits an empty completion ~1/3 of
calls (a llama.cpp reasoning+grammar artefact; qwen-nothink phase 3
had ~0 spurious empties). Fix: empty-output self-heal — reasoning refs
retry up to 3 attempts, burning the cached empty each retry. Never
fabricates: a still-empty answer after retries is recorded as empty.
6-item smoke: 0/6 (broken) → 5/6 valid JSON; residual ~1/6 are
questions that reliably break (4 consecutive empties), a documented
artefact.
Structural finding (to fold into Addendum 8): json-schema grammar
enforcement SUPPRESSES the reasoning trace — output is pure single-
line JSON, no <think> block. So arborist+qwen-think is structurally
≈ arborist+qwen-nothink; the thinking lever that moved the solo arm
is neutralized by the claim_lattice grammar. The re-run will confirm
empirically.
Non-reasoning refs (hermes, qwen-nothink) unchanged: single pass,
stop sequence intact.
fox 2026-05-19: 'we don't need to redo anything'. The arborist+qwen-
nothink sweep would re-run qwen-nothink solo on 1158 records that
phase 1 already produced (control_sweep_2026-05-19T21-52-56Z.jsonl,
calibrated rescore on disk). Wasteful — qwen-nothink solo behaviour
doesn't depend on retrieval being on, so phase 1's numbers stand.
--skip-solo runs ONLY the arborist arm. Implementation: skip the
per-variant model loop in _process_item when the flag is set; the
arborist arm below it still runs if arborist_on. The spend banner
zeroes the solo-call count so the operator sees the actual LLM
budget for the arborist-only run.
Example: a full arborist+qwen-nothink sweep against the 386-item
fixture goes from 2316 LLM calls (1158 solo + 1158 arborist) to
1158 calls (arborist only). Halves wall-clock on single-worker
llama.cpp.
pytest 27/27, imports clean, --help shows the flag.
Reorder rule 3 (short-answer entity grounding) above rule 4 (NLI
contradiction) so positive lexical evidence cannot be overridden by
NLI clause-level noise. Surfaced by the 2026-05-19 arborist+qwen-
nothink smoke:
i=3 · who is the prime minister of Poland?
ans: 'Donald Tusk is listed as the Prime Minister of Poland.'
gold: ...lists Tusk + Marcinkiewicz + Belka + Kaczynski + Kopacz...
NLI contradiction p=0.892 (above 0.85 threshold)
NLI entailment p=0.744 (also high on the correct clause)
Tusk WAS PM in 2010 (served 2007-2014); answer is correct against
the corpus-vintage gold. The NLI contradiction signal came from
clause-level candidate selection picking a NON-Tusk PM the source
also mentions; entailment was high on the Tusk clause. Mixed signal
that the WRONG rule then over-confidently resolved.
The fix is a rule reorder, not a threshold change — the fast path's
positive-evidence combination (specifics-in-gold AND subject-in-gold)
is a strictly stronger signal than NLI's clause-level max
contradiction, so when it fires it should win. The combination
discriminates Poland-Tusk (Tusk ∈ gold, Poland ∈ gold → CG) from
Anthony-Albanese (Albanese ∉ gold → fast path declines → falls
through to UNGROUNDED-subject-in-gold → WRONG, unchanged).
Self-test 4/4 INSTRUMENT TRUSTWORTHY unchanged. pytest 27/27.
Poland-Tusk regression smoke: now CG via short_entity_grounded ✓.
No regression risk on the existing reconciliation cells:
- Iceland CG: short_entity_grounded was already winning (was rule
4, now rule 3 — same outcome, earlier exit)
- WWII-1812 WRONG: '1812' ∉ gold → fast path declines, NLI fires ✓
- Higgs-cafe FABRICATED: 'Higgs' ∉ gold → fast path declines ✓
- Anthony Albanese WRONG: 'Albanese' ∉ gold → fast path declines ✓
- Abstention phrases: rule 2 still fires first ✓
Two surgical fixes unblock 'arborist with synthesis LLM = Qwen-on-
llama.cpp' as a viable arm in the control sweep. Pre-existing
docstring said 'Arborist×Qwen needs proof-path guided_json+extra_body
surgery — coupled follow-up'; this is that follow-up.
Fix 1 — multi-engine structured-output extras
The runner / query JSON-mode paths previously sent only vLLM's
'guided_json' key for the claim_lattice schema. llama.cpp silently
drops it, leaving Qwen un-enforced (the parse-tolerant fallback did
all the work). Helper
claim_lattice_structured_output_extras() in arborist/qa/verify.py
now returns a dict carrying the schema under all three engine
conventions:
- guided_json (vLLM grammar-constrained sampling)
- json_schema (llama.cpp native shorthand)
- response_format (OpenAI-spec, honoured by llama.cpp and newer vLLM)
Each engine recognises its own key and silently drops the others.
Used at both inference call sites (runner.py:740, query.py:3324).
Hermes/vLLM path is unchanged — it picks up 'guided_json' and
ignores the other two.
Fix 2 — query() accepts user-supplied extra_body, merges with defaults
query() grew a keyword-only extra_body parameter (default None).
Per-model knobs (Qwen's {'chat_template_kwargs': {'enable_thinking':
False}} toggle, future template knobs) can flow from the caller to
the synthesis chat-completion call. Schema-enforcement extras are
added inside query() and merge under user keys — common case is
disjoint namespaces, but if a caller wants to override 'guided_json'
they can.
bench/control_sweep.py now passes MODELS[arborist_ref]['extra']
through to query() in the arborist branch, so --arborist-ref
qwen-nothink runs with reasoning disabled and --arborist-ref
qwen-think runs with reasoning enabled. Phase 1's arborist arm with
--arborist-ref=hermes is unaffected (MODELS['hermes']['extra'] is
None, merges to no-op).
Tests
+ 3 new in tests/test_verify_json.py covering helper default shape,
alternate-schema reuse, and query()'s new extra_body parameter
220 affected tests still green (verify / claim_lattice / judge /
runner suite)
pytest test_verify_json: 27/27
Next: small smoke run --arborist-ref qwen-nothink against 4-8 items
to confirm end-to-end before any full sweep. Phase 2 (qwen-think solo)
still running in background, unaffected — it doesn't touch the
arborist arm.
The Arborist arm runs answer_mode='claim_lattice' (per control_sweep.py
:179, control_ab.py:155) so its answers arrive as the JSON envelope
{"claims":[{"text":"...","evidence_ids":["E1"]},...]}.
_descaffold strips the [E1] evidence-pointer markup but the JSON
braces + key syntax remain. The verifier's strategy-2 (span) and
strategy-3 (proper-noun) extractors see brace noise instead of the
inner claim prose — every Arborist record degraded to UNGROUNDED.
The 2026-05-19T17-01-17Z sweep, re-graded with the freshly calibrated
judge (5a17f61), surfaced this: Arborist arm reported 0 CG across all
three variants in the live phase 1 output (the live run was pre-
calibration), and 29/120 CG (24%) under the calibrated rescore — clear
improvement just from theta_contra=0.85, but the JSON envelope was
still hobbling the verifier paths.
Fix: _unwrap_claim_lattice_json runs BEFORE all downstream rules.
Detection is conservative (three independent signals: starts-with-
brace AND "claims" key AND "text" key) so plain-prose answers
pass through unchanged. Multi-claim envelopes concatenate as discrete
sentences (extract_claim_spans treats each as its own span).
Malformed JSON falls back to the original answer — no silent
rewriting on broken input.
Smoke result on the Iceland Arborist case
ans: {"claims":[{"text":"The current president of Iceland is
Ólafur Ragnar Grímsson.","evidence_ids":["E1"]}]}
gold: {{Infobox Political post |post = President |body = Iceland
|incumbent = [[Ólafur Ragnar Grímsson]] ...}}
before: UNGROUNDED → FABRICATED (then WRONG after calibration)
after: short_entity_grounded → CORRECT_GROUNDED
pytest: 27/27 (added 7 unwrap-coverage tests covering single-claim
envelopes, multi-claim concatenation, plain-prose passthrough,
malformed-JSON tolerance, unrelated-JSON passthrough, and the
end-to-end Arborist-envelope CG flow). Self-test 4/4 unchanged.
Re-rescores of 17:01 sweep + phase 1 sweep run after this commit
to measure final Arborist scorecard improvement.
Opus is trusted; this commit closes the systematic gaps surfaced by
B's rescore on the 17:01 sweep (2289 records, of which 468 had real
Opus verdicts). Off-diagonal cells in descending size and the
root-cause fix for each:
cell n=114 code:WRONG · opus:CORRECT_GROUNDED
Root cause: NLI fires contradiction p in [0.5, 0.75] on factual
answers like 'Ólafur Ragnar Grímsson is president of Iceland'
against wikitext-shaped infobox gold — clause-level candidate
selection picks up co-mentioned earlier office-holders, reads
temporal-frame mismatch as contradiction. The TRUE contradiction
signal (WW2 1812 self-test fixture) measures p=0.985 — clean
margin above noise.
Fix: raise theta_contra 0.5 → 0.85 (code-judge override of the
NLI manifest's 0.5 contradiction_veto).
cell n=60 code:FABRICATED · opus:WRONG
Root cause: 'Anthony Albanese' answer vs Julia Gillard gold gets
FABRICATED (specifics-not-in-gold) but Opus correctly distinguishes
WRONG (source contradicts by naming someone else) from
FABRICATED (source silent on topic).
Fix: when verifier UNGROUNDED + specifics not in gold AND the
question's subject anchor IS in gold, demote FABRICATED → WRONG.
Subject anchor uses proper-noun-shaped terms from the question
(Iceland / Australia / Higgs) — not the last-content-token
heuristic, which mis-fires on coincidental matches like 'cafe'
appearing in a 'gold does not mention any cafe' denial.
cell n=18 code:WRONG · opus:ABSTAINED
Root cause: abstention patterns missed Hermes's most common
refusal phrasings — 'I do not have accurate information', 'I do
not have access to a reference knowledge base', 'I lack access
to'. Original patterns required determine/know/tell verbs right
after 'do not'.
Fix: three new patterns for the 'do not have / lack ...
information / access / knowledge' family.
cell n=13 code:ABSTAINED · opus:CORRECT_GROUNDED
Root cause: verifier's strategy-2 needs prose shape; terse-name
answers ('Pratibha Patil', 'Jalal Talabani') fall to
UNGROUNDED-no-specifics → ABSTAINED, missing valid CG.
Fix: short-answer entity-grounding fast path. When answer is
short (≤15 tokens) AND every specific asserted is present in
gold (no unsourced) AND at least one specific WAS asserted AND
the question's subject anchor is in gold → CG. Guards against
'wrong topic, right name' false-positives via the subject check.
Structural reorder: NLI contradiction now runs AFTER the
abstention check and short-answer fast path (instead of preempting
the verifier), so the verifier's STRICT/HYBRID positive signal
isn't overridden by NLI noise. NLI still leads the path on truly
unbounded answers — verifier UNGROUNDED + NLI ≥ 0.85 contradiction
keeps the WRONG label.
Self-test 4/4 INSTRUMENT TRUSTWORTHY. pytest contract 18/18.
v2 rescore on the same 17:01 sweep runs in the background to
measure agreement-matrix improvement empirically.
Added bench/analyze_judge_disagreement.py — the harness that drove
this calibration (reads B's rescore JSONL, bucketises off-diagonal
cells, dumps configurable samples per cell with question / answer /
gold / both rationales). Reusable for the next calibration round.
Reads an existing sweep JSONL (Opus verdicts already recorded), re-fetches
gold per record via _gold(), runs the code judge on (question_asked,
answer, gold), and emits:
- markdown scorecard: agreement matrix (code × original judge), per-arm
/ per-model / per-variant code-judge tallies, and a residue table of
the JUDGE_ERROR records (the natural input to a later LLM-batch
needle-haystack pass — Opus or Grok);
- JSONL with one row per sweep record (code_verdict + code_rationale
+ code_decision), joinable on (i, arm, model, variant) to the
source sweep.
Zero LLM calls. Reads sweep JSONL + shards read-only. Pairs with the
new --judge switch (a2e9b49): the switch decides what NEW data uses;
this script decides what the ALREADY-COLLECTED data looks like under
the deterministic judge.
Usage (parameter default matches control_sweep.py default fixture):
python -m bench.score_with_code_judge --in <sweep>.jsonl
Currently running against control_sweep_2026-05-19T17-01-17Z.jsonl
(the 2289-record sweep that ran on the prior huge-N pass before the
Opus quota burned out). Output will land at
bench/qa_results/control_sweep_2026-05-19T17-01-17Z_code_judge.{md,jsonl}.
Wire bench/judge_code.py into the sweep harnesses as the default judge.
Both control_sweep.py and control_ab.py grow a --judge {code,opus} CLI
arg; both share the same Verdict shape so the dispatch is a pointer
assignment + threading the judge_fn through _process_item.
Behaviour:
- DEFAULT = code: zero LLM, zero quota, deterministic. Self-test gate
is the code judge's 4-fixture contract.
- --judge opus: original gated Opus path; needs ARBORIST_JUDGE_ENABLE=1
set per 1cabfe6's fail-closed gate, otherwise every record returns
JUDGE_ERROR with rationale 'disabled — set ARBORIST_JUDGE_ENABLE=1'
and the sweep records that label honestly.
Reporting:
- Header line now records which judge ran ('Judge = code (...)' or
'Judge = opus (...)') so partial-reports & resumes don't lie about
provenance.
- Spend banner shows '0 LLM calls' for the code path so the no-burn
property is visible in the operator output.
Test surface: pytest sweep across tests/ still 136/136 (no regressions);
new --judge flag visible in --help on both harnesses.
Next: bench/score_with_code_judge.py to re-grade existing sweep JSONLs
(written under the gated-Opus run) with the code judge; agreement
matrix surfaces residue size for the eventual LLM-batch needle-haystack.
bench/judge_code.py — drop-in alternative to bench/judge.py with the
same Verdict shape & closed verdict vocabulary (CG/W/F/A/JE) but zero
quota cost: composes verifier + NLI + abstention + specificity into a
fixed-order pipeline. fox 2026-05-19: 'data first, judging later' —
this is the data-collection arm; LLM-based judging (Opus batched
needle-haystack, or Grok credit-card) is a separate downstream
concern that operates on the residue this judge cannot classify
deterministically.
Pipeline (first hit decides):
1. empty / no-gold guards
2. explicit abstention phrases (lexical regex)
3. NLI contradiction (arborist.qa.nli.shadow_check) — strongest
signal: gold contradicts the claim → WRONG
4. lexical verifier (arborist.qa.verify.verify_quotes) →
STRICT → CORRECT_GROUNDED
HYBRID + NLI entail >= 0.55 → CORRECT_GROUNDED
UNGROUNDED + specifics-not-in-gold → FABRICATED
UNGROUNDED + no specifics → ABSTAINED
HYBRID without NLI corroboration → JUDGE_ERROR (residue
for an LLM judge)
Threshold note: _CODE_JUDGE_THETA_ENTAIL_CORROBORATE=0.55 is distinct
from the NLI manifest's entailment_block_veto=0.9. The manifest's
threshold is calibrated for OVERRIDING a STRICT lexical signal with
negative evidence — high bar. The corroboration use here is the
opposite direction: additive positive evidence on an already-positive
anchor — moderate bar appropriate. Self-test case 1 measures NLI
entail=0.769 (clearly entailed, clear margin above 0.55).
Specificity for FABRICATED layers three scanners:
- verifier's multi-word proper-noun extractor (Higgs Boson, ...)
- local single-word capitalised-token scanner (Napoleon, Mars, ...)
deliberately separate because the verifier's gate is conservative
by design (multi-word only)
- numerics (years, dates, large counts, money)
Self-test: same 4 fixtures as bench/judge.py:self_test() so the two
instruments can be cross-checked when fox re-fires the Opus judge on
the residue later. Result: 4/4 INSTRUMENT TRUSTWORTHY.
tests/test_judge_code.py — pulls the contract into make test
(18 cases): module identifiers pinned, dataclass shape parity,
empty / no-gold guards, parametrised abstention phrases, specificity
layer behaviour, the canonical 4-case self-test, batch helper, and
graceful NLI-unavailable degradation. 18/18 pass.
Pre-existing known limitation, documented in the docstring: terse
correct answers ('In 1945.' against gold containing '1945') route to
ABSTAINED because the verifier's span extractor needs prose shape;
NLI sees no clause-level overlap at very short claims. The conservative
ABSTAINED label is correct deferral; tuning this is a calibration
question for real bench data, not the instrument's contract.
No callers touched yet — control_sweep.py & control_ab.py still
import the disabled Opus judge. Wiring this in is a separate ticket
move per fox's data-first sequencing.
2026-05-19: huge-N #000057 control sweep (f63b00d → 9dc02e4) burned
our Opus quota. Disable judge.py by default so a stray re-run can't
re-burn — every call short-circuits to JUDGE_ERROR with rationale
'disabled — set ARBORIST_JUDGE_ENABLE=1 ...' and zero subprocess
spawn (0ms in the disabled path, smoke-tested).
Why a gate, not a model swap:
- judge.py uses Opus deliberately as EXTERNAL SOTA outside both arms;
swapping the judge to Hermes/Qwen would corrupt the experiment
(Hermes is itself an arm under test). The hygiene comment at
judge.py:26-29 already names same-family-judging as the live threat
to validity at Opus level; downgrading further changes the science.
- Gating instead preserves the science when fox re-enables, and gives
us the data-first workflow he asked for: deterministic tool
pre-filters (verifier / NLI / recall@k) up front, judge only on
residue worth Opus tokens, with explicit go.
Behaviour:
- control_sweep.py + control_ab.py already treat JUDGE_ERROR
non-fatally (counted as JE in _bucket); disabled runs degrade to
100% JE in the tally and surface the disable reason in rationale —
the loudest possible 'judge did not run here' signal.
- Re-enable per-run: ARBORIST_JUDGE_ENABLE=1 python -m bench.control_sweep ...
- self_test() will report 4× JUDGE_ERROR when gated — intentional;
if the instrument is off, the self-test must NOT silently pass.
Smoke-test (without flag): label='JUDGE_ERROR' rationale='disabled — ...'
dt=0.0ms · no claude subprocess spawned.
Cross-referenced from CLAUDE.md '## Live endpoints' /
'Budget discipline' subsection added in 2365bd1.
fox: 'not 11 hours it shouldn't take that long'. Probed endpoints —
Qwen-27B absorbs 8 concurrent with 0 errors (0.5->2.1s); the
bottleneck is the serialized claude -p Opus judge, so more workers
≈ near-linear speedup. control_sweep.py gains --resume PATH: appends
to an existing JSONL, skips items already COMPLETE (full
models×variants for solo + variants for arborist if i<=arborist-n),
re-runs partial items; _aggregate now dedupes (i,arm,model,variant)
last-wins so a killed-mid-unit restart never double-counts, and
_load_recs tolerates a truncated trailing line from the kill.
Makefile control-sweep gains CONTROL_SWEEP_WORKERS / _RESUME / _ARB_N
so make stays the interface. 6-worker run killed cleanly (specific
pids, no pkill), relaunched resume @ 12 workers — 12 done items
preserved, 374 to run, ~5h -> ~2-2.5h.
fox: 'make the n huge huge, check in every ~7 turns'.
- stale fixture re-mined to the FULL pool: 386 questions (180 pres /
119 PM / 34 premier / 20 gov-gen / 17 chancellor / 15 CM / 1 FM),
deterministic.
- control_sweep.py rebuilt: ThreadPoolExecutor work-unit-per-item
(per-unit qa_db -> zero SQLite write contention; shards read-only
safe under concurrent readers), incremental flushed JSONL, and a
--report-only PATH mode that aggregates a PARTIAL file with ZERO
LLM/judge spend (the interim check-in path).
- huge N goes on the CONTROL (solo x3 models x3 framings) where the
open statistical question lives; Arborist A/B = fixed --arborist-n
(default 40), not re-measured 386x (power belongs on the control,
and query() over ~40GB shards is the heaviest call).
- bench-maxing doctrine applied: independent hermetic judge calls
fanned out, 'serial-by-caution is halting in disguise'.
N=3 (prior run) already shows the coherent, review-relevant story:
plain -> all 3 models confidently assert (0% abstain); source_relative
-> abstain jumps (hermes 33 / qwen-think 67 / qwen-nothink 100%);
as_of_corpus -> ~100% correct all models. The gap is largely
framing+snapshot, not a universal capability deficit.
fox ruling: 'we both do not know which framing is right, measure all
benchmarks and bring results forward for review'; 'we have qwen with
and without reasoning to use'.
bench/control_sweep.py: sweeps the CONTROL (solo) arm across
{Hermes-3-8B, Qwen3.6-27B reasoning, Qwen3.6-27B no-reasoning} ×
{plain, source-relative, as-of-corpus-era} questions, judged vs the
fixed corpus-vintage gold; Arborist-Hermes treatment reference run
alongside at the same N. Presents the SAME judge verdicts under both
the accuracy framing (the naive read fox flagged as unfair-as-truth,
shown for contrast) and the grounding-fidelity framing (the
defensible read), plus an explicit note on the faithfulness-ablation
framing + Arborist×Qwen — both deliberately NOT run (proof-path
surgery / different instrument, not a relabelling).
Answers fox's open question empirically: does a 27B *reasoning*
control honestly ABSTAIN where the 8B fabricates (gap = weak-small-
model artefact) or still confidently assert post-corpus (gap = real,
scale-independent)? as_of_corpus separates 'can't recall the era'
from 'won't constrain to a source'.
Qwen toggles probed live: reasoning answer in message.content (CoT in
separate reasoning_content, not surfaced — logged limitation);
no-reasoning via chat_template_kwargs{enable_thinking:false}. Gate =
in-script judge self_test (aborts on fail) — stronger than a make
edge, no double self-test spend.
Opt-in third NLI inference backend (ARBORIST_NLI_BACKEND=tinygrad) in
qa/nli/shadow.py, parallel to torch/onnx-int8, behind the #000049
cage: shadow-only, never an audit_mode input, never auto-preempts the
proven path (guarded so it cannot regress torch/onnx). Loads the ONNX
export through tinygrad's frontend wrapped to the existing
model(**enc).logits contract so _nli_batch is byte-unchanged.
bench/scripts/nli_backend_ab.py + make bench-nli-backends: deterministic
A/B, gate is numerical agreement with the torch reference first,
latency second (a divergent engine = a different shadow signal = a
different nli_policy_hash). Instrument is honest — reports a requested
backend as unavailable rather than relabelling a fallback's numbers.
First CPU-smoke run already quantified that the deployed §7 #22 int8
export diverges Δmax≈0.42 from torch — the immunity property made
measurable, not a defect. Real tinygrad numbers pending a producer-box
run (tinygrad not an arborist dep; frontend op-coverage for the large
MNLI checkpoints unverified by design).
docs/onnx-vendor-capture-immunity.md: why the model-in-proof-path cage
makes the inference engine an interchangeable sidecar, never a trust
dependency — public-domain positioning capital. Indexed in CLAUDE.md.
Full suite 2498 passed (identical to baseline); 24/24 NLI tests green.
