# Benchmarks — orientation, harnesses, fixtures, discipline Arborist ships two bench harnesses, four question fixtures, and a signal-floor discipline pinned across the substrate. This doc is the canonical entry point. Three companion docs cover specifics: | File | Purpose | |----------------------------------------|--------------------------------------------| | `docs/bench-maxing.md` | Discipline rules (5pp signal floor, etc.) | | `docs/qa-modes-bench.md` | Historical journal — running addendum log | | `docs/bench-emergent-design.md` | Random-word stress-test design rationale | | `docs/qa-modes-bench-2026-04-30.md` | Frozen point-in-time bench (Apr-30 baseline) | | `docs/energy-cogs-benchmark.md` | **qwen vs hermes** — GPU energy COGS + quality-per-dollar (Dav1d-audience report) | | `docs/benchmark-matrix.md` | Control-arm (solo vs substrate) A/B matrix + the deterministic code judge | Read those when you need the why or the long-form data. This file covers the what, where, and how. ## 1. Three harnesses, three purposes ### 1.1 `bench/qa_sweep.py` — curated QA-quality bench Runs a fixture file × answer modes × n samples. Persists a JSONL of per-row results plus a markdown summary. Used for: - **Regression checks** before merging substrate changes. - **A/B cycles** on policy knobs (cap on/off, reminder on/off, preflight enabled/disabled). - **Per-mode strict-rate tracking** over time. Run via `make bench-qa` or `bench/qa_sweep.py` directly. Bench rows carry ~25 fields per cell, including the four-rung label (`audit_mode`), the verifier-method, the violation kinds, the preflight QuestionState projection (`preflight_logical_statuses`, `preflight_question_shape`, etc.), the quantifier classifier output (`quantifier_intensity`, `scope_bound_hint`, `claim_cap_applied`), the preflight stage hash 12-char prefix (#000009 §7.2), and a per-stage timing breakdown (`preflight_ms`, `soft_preflight_ms`, `search_ms`, `context_ms`, `llm_ms`, `persist_ms`, `total_ms`). ### 1.2 `scripts/bench_emergent.py` — random-word stress test Picks three random words from `/usr/share/dict/words`, asks Hermes (at temp 0.8) to weave them into a creative question paragraph, then sends that paragraph to arborist. Logs every cycle to `bench/emergent_log.jsonl` as one JSONL line. Designed for **blue-moon cadence**, not every-commit benching — the combinatoric word space surfaces failure shapes the curated bench doesn't reach. The teacher review step is intentionally **not automated.** Fox brings interesting log entries to a teacher model (Claude Opus 4.7 in the current setup) and asks for guidance: did the answer match the question? Novelty class (known_truth_grounding / emergent_synthesis / novel_claim / no_signal)? Which hyperparam to tune next? Run via `make bench-emergent` (default `EMERGENT_N=10`) or `make bench-emergent EMERGENT_N=100` for a longer cycle. See `docs/bench-emergent-design.md` for the design rationale. ### 1.3 `bench/control_ab.py` + `watt_*` — cross-model cost & quality-per-dollar This is how we benchmark **qwen vs hermes with arborist**, on two axes that multiply into **quality-per-dollar**. Full living report (and the Dav1d-facing writeup) in `docs/energy-cogs-benchmark.md`; this is the orientation. **Two rigs, run on the same harness** (the GPU work happens on the producer box, never in arborist's `python+sqlite3` core): | | rig A | rig B | |---|---|---| | model | `Qwen3.6-27B` UD-Q4_K_XL (llama.cpp) | `Hermes-3-8B` FP8 (vLLM) | | GPU | RTX 4090, `ai.foxhop.net` (isolated for the bench) | RTX 3090, `3090-ai.foxhop.net` (live/public) | **Value axis — `bench/control_ab.py`.** Runs the *same* model two ways: **solo** (question only, no retrieval/verifier) vs **substrate-ON** (full `query()` — 4-route FTS5 → context → LLM → verifier), n=30 per fixture, grades each answer with the judge (§12). Headline finding: the substrate lifts both models 2–13× and they **nearly converge** (an 8B and a 27B reach similar grounded quality — grounding comes from retrieval+verify, not parametric size). **Cost axis — `bench/watt_bench.py` / `watt_calibrate.py` / `watt_probe.py`.** Measures **GPU joules per token** by slope calibration (prefill J/input-tok and decode J/output-tok measured separately — they differ ~35–40×), times the *real* token mix per query, at a site `--price-per-kwh`. Power states are measured at runtime, never hardcoded; cache-miss is forced so we time generation not a SQLite lookup. Result: a grounded answer costs **~$0.07–0.16 per 1,000 queries** of GPU electricity (hermes-8B cheaper than qwen-27B — but that's a 4-way confound: params × quant × engine × card, not "8B vs 27B" alone). The dominant cost is **prefilling the retrieved context** (~⅔ of substrate GPU energy), not generating the answer. Run the value axis via `make control-ab` (`N=… FIXTURE=…`); the watt instruments run directly — `bench/watt_calibrate.py` / `watt_bench.py`, with `make rapl-access` first for CPU-package power. Invocations and raw artifacts are in `docs/energy-cogs-benchmark.md` §8. **All value-axis grading is done by the deterministic code judge (§12)** — an Opus LLM-judge was attempted for #000057 and abandoned (too expensive per call, ~79.5 % `JUDGE_ERROR` rate-limited, burned coding quota); it never produced usable verdicts at scale and is gated off so it can't re-burn. ## 2. Four question fixtures ``` bench/qa_questions.txt — 75 questions bench/qa_questions_smoke.txt — smoke test bench/qa_questions_quantifier_subset.txt — 9 questions bench/qa_questions_quantifier_baseline.txt — 1 question bench/qa_questions_metacog_subset.txt — 28 questions ``` | Fixture | Use case | Cell size at n=3 × 3 modes | |--------------------------------------|---------------------------------------------------------------------------------------------|----------------------------| | `qa_questions.txt` | Full regression bench. Run before merging substrate changes. Wide coverage + corner cases. | 675 runs, ~3-3.5h | | `qa_questions_smoke.txt` | Smoke test on 1-2 questions. Use to confirm a CLI flag wires through correctly. | 3-6 runs, ~30s | | `qa_questions_quantifier_subset.txt` | 9 broad-quantifier questions (7 unbounded + 2 bounded). Use for A/B cycles on #000008 knobs. | 81 runs, ~7-15min | | `qa_questions_quantifier_baseline.txt` | Single broad question for narrow n=3 baseline measurement. | 9 runs, ~2-4min | | `qa_questions_metacog_subset.txt` | 28 questions targeting each #000010 detector (temporal, contradiction, false-premise, out-of-corpus) plus mixed multi-trigger + well-formed controls. | 252 runs, ~25-35min | When choosing a fixture: ask whether you're measuring the **substrate as a whole** (full bench), a **specific feature** (broad-quantifier subset, metacog subset), or just **wiring correctness** (smoke). ## 3. The signal floor — n=3 × 9 = 27 sample minimum, 5pp delta floor Per `docs/bench-maxing.md`: - **5pp floor** for STRICT-rate deltas at n=3 × any-fixture sample size. Differences below this are Hermes nondeterminism, not signal. Quote-mode often jitters ±10pp on 27-sample cells. - **Mean-ratio + UNGROUNDED-rate** are tighter signals at the same sample size — they aggregate per-row values rather than counting discrete verdicts. - **n=5 verification** when an A/B cell sits within the noise band (e.g. 0.30 vs 0.33 STRICT-rate is 3pp — within noise; n=5 on the closer cell tightens the variance). - **vLLM concurrency saturates at c=3-4.** Going higher slows per-call latency without parallelizing harder. Default `c=4`. Don't tune to a single bench cycle. Always have a baseline + an A/B cell + (sometimes) a verification cell. ## 4. Make targets cheat sheet ``` # Curated QA bench (default: bench/qa_questions.txt × 3 modes × n=3) make bench-qa # Same but with limit / different file .venv/bin/python bench/qa_sweep.py \ --questions bench/qa_questions_quantifier_subset.txt \ --shards-dir ~/.arborist/shards --n 3 --concurrency 4 \ --out-dir bench/qa_results # A/B cycle: flip a policy knob for one bench cell .venv/bin/python bench/qa_sweep.py \ --questions bench/qa_questions_quantifier_subset.txt \ --shards-dir ~/.arborist/shards --n 3 --concurrency 4 \ --policy quantifier_guard_apply_caps=true \ --policy quantifier_reminder_enabled=true # Resume an interrupted bench (same --seed, append to existing JSONL) .venv/bin/python bench/qa_sweep.py \ --resume bench/qa_results/2026-05-03T19-30-13Z.jsonl \ --seed 0 # Smoke test — quick sanity of the wiring .venv/bin/python bench/qa_sweep.py \ --questions bench/qa_questions_smoke.txt --n 1 --concurrency 1 # Random-word stress test (blue-moon cadence) make bench-emergent # n=10 default make bench-emergent EMERGENT_N=100 ``` ## 5. The bench-row schema Every row in `bench/qa_results/.jsonl` carries (current shape — fields are additive across versions): **Identity & verdict:** - `question`, `answer_mode`, `audit_mode`, `verifier_method` - `n_quotes`, `n_verified`, `ratio` - `cache_key` (12-char prefix) - `preflight_hash` (12-char prefix; see #000009 §7.2) - `model_profile_id` **Diagnostics:** - `format_collapsed` (pointer-mode only signal; #000008) - `violation_kinds` (sorted list of unique kinds) - `lazy_anchor_ratio`, `pointer_id_distribution` - `answer_brackets`, `answer_pointer_count`, `answer_chars_with_brackets`, `raw_meaningful_line_count` - `deflection_kind`, `subject_anchor`, `subject_in_answer` - `metaphor_deflection_kind`, `metaphor_cue_count`, `metaphor_overlap_count` **Preflight projection (#000010):** - `preflight_logical_statuses` (list) - `preflight_question_shape`, `preflight_result` - `preflight_temporal_sensitivity` - `preflight_has_false_premise`, `preflight_has_contradiction` - `preflight_corpus_requirement` **Quantifier classifier (#000008):** - `quantifier_intensity`, `quantifier_matched_token` - `scope_bound_hint`, `quantifier_explicit_count` - `claim_cap_applied` **Capacity:** - `prompt_chars_total`, `prompt_chars_evidence`, `prompt_chars_system`, `prompt_chars_question` - `answer_chars` **Directive compliance** (per-row pass/fail for the seven-point program directives observable from a single bench row): - `directive_compliance` dict keyed on D2/D3/D4/D6/D7 **Time:** - `elapsed_s`, `iso_ts`, `error` The markdown summary alongside the JSONL renders aggregate views (per-mode strict-rate, format-collapse + violation kinds table, strict-rate-by-prompt-size buckets, directive coverage). ## 6. The journal — where headlines live `docs/qa-modes-bench.md` is the **rolling research log** for the QA bench. Each substantial bench cycle adds a new addendum (or refines an existing one). The 2026-05-02 baseline is the frozen authoritative state of the substrate before #000008/9/10/11 landed; subsequent addenda compare against it. Current addenda: - **2026-05-02 baseline** — pre-preflight authoritative state. - **Addendum 1** (broad-quantifier A/B from #000008 §12.6). - **Addendum 2** (preflight on vs off, 9-question broad subset). - **Addendum 3** (full 75-question regression check post-flip). - **Addendum 4** (metacog-trigger detector validation, 6 → 28 question fixture). When a new bench cycle produces a substantive finding, add an addendum or refine an existing one. Bench artifacts (JSONL + markdown) live in `bench/qa_results/` and are gitignored — only headlines + cross-references go in the journal. Per-ticket bench data lives in the relevant ticket file's `§12` or `§13` (e.g. `docs/tickets/ticket-000008-...md` §12 carries the four-cell A/B; #000010 §13 has three sub-validations; #000006 has the rolling-amend pattern across multiple cycle counts). ## 7. The bench-emergent log `bench/emergent_log.jsonl` is a long-running append-only log of random-word stress cycles. Each entry has: ```json { "ts": , "iso_ts": "2026-05-04T...", "words": ["dismally", "heats", "Jamaicans"], "question": "", "answer": "", "audit_mode": "STRICT|HYBRID|UNGROUNDED", "verifier_method": "claim_lattice", "n_quotes": 3, "n_verified": 0, "violation_kinds": ["CITATION_MISMATCH", "TOO_MANY_EVIDENCE_IDS"], "metaphor_deflection_kind": "no_signal", "metaphor_cue_count": 2, "metaphor_overlap_count": 2, "sources": [{"title": "...", "uri": "...", "used": false}, ...], "answer_seconds": , "total_seconds": , "teacher": } ``` Findings live in `docs/tickets/ticket-000006-bench-emergent-findings.md` as rolling amends — each new cycle batch produces a new amend with the verdict shift, violation profile, and tuning candidates. ## 8. How to run a focused A/B Pattern from #000008 §12 (broad-quantifier four-cell A/B): 1. **Pick a fixture** matching the feature under test (broad-subset for quantifier work; metacog-subset for metacognition work; smoke for wiring sanity). 2. **Run the baseline** — same fixture, default policy. 3. **Run the cell** — same fixture, one policy knob flipped via `--policy KEY=VALUE`. Repeat for each independent knob. 4. **Compare** — STRICT-rate, mean-ratio, UNGROUNDED-rate, FORMAT_COLLAPSED rate, dominant violation kinds. Apply the 5pp floor to STRICT-rate; mean-ratio and UNGROUNDED tend to be tighter. 5. **Verify with n=5** if a cell sits within the noise band. 6. **Document** — add an addendum to `docs/qa-modes-bench.md` or a new `§12.x` / `§13.x` to the relevant ticket file. The four-cell A/B in #000008 (preflight off / reminder only / cap only / cap+reminder) is the canonical pattern for measuring multi-knob features. ## 9. How to interpret results **STRICT-rate** = the fraction of cells where the verifier could prove every claim grounded against cited evidence. Most direct quality metric, but noisy at small sample sizes. **HYBRID-rate** = mixed verdicts (some claims grounded, some not). Often more informative than STRICT-rate when a feature's job is to rescue UNGROUNDED → HYBRID rather than HYBRID → STRICT. **UNGROUNDED-rate** = honest "the verifier could not ground." Lower is generally better, but UNGROUNDED on questions the corpus genuinely can't answer is the *correct* outcome — the substrate preferring UNGROUNDED to fabricated STRICT is the architectural win named in `docs/seven-point-program.md` D7. **Mean ratio** = mean of `n_verified / n_quotes` across rows. Aggregates per-row values rather than counting verdicts; tighter at small sample sizes than STRICT-rate. **FORMAT_COLLAPSED rate** = pointer-mode-specific gauge of whether the model is following the `[E\d+]` citation protocol. Zero is the goal; non-zero means the model emitted free prose without pointer tags. **Violation kind distribution** = per-mode tallies of which hard checks fired most often. Gives the operator a per-failure-mode view of the substrate's weaknesses. **Directive compliance** = per-row pass/fail for the seven-point-program directives (D2/D3/D4/D6/D7). Substrate-level hygiene check; should be ~100% on lattice modes. **Audit-line tails** (rendered by `arborist/cli.py:_render_warrant_tail`): - Hard tails: `· warrant missing`, `· title mismatch`, `· format collapsed`, `· broad cap N`, `· broad rejected`, `· broad unbounded`, `· broad runaway`, `· false premise`, `· contradictory`, `· stale risk`, `· out of corpus`, `· frame ambiguous`. - Soft tails (advisory): `· soft: