docs: surface qwen-vs-hermes cost bench + the judge in benchmarks orientation

docs/benchmarks.md

@@ -10,11 +10,13 @@ the canonical entry point. Three companion docs cover specifics:
 | `docs/qa-modes-bench.md`               | Historical journal — running addendum log  |
 | `docs/bench-emergent-design.md`        | Random-word stress-test design rationale   |
 | `docs/qa-modes-bench-2026-04-30.md`    | Frozen point-in-time bench (Apr-30 baseline) |
+| `docs/energy-cogs-benchmark.md`        | **qwen vs hermes** — GPU energy COGS + quality-per-dollar (Dav1d-audience report) |
+| `docs/benchmark-matrix.md`             | Control-arm (solo vs substrate) A/B matrix + the deterministic code judge |
 
 Read those when you need the why or the long-form data. This file
 covers the what, where, and how.
 
-## 1. Two harnesses, two purposes
+## 1. Three harnesses, three purposes
 
 ### 1.1 `bench/qa_sweep.py` — curated QA-quality bench
 
@@ -58,6 +60,50 @@ Run via `make bench-emergent` (default `EMERGENT_N=10`) or
 `make bench-emergent EMERGENT_N=100` for a longer cycle. See
 `docs/bench-emergent-design.md` for the design rationale.
 
+### 1.3 `bench/control_ab.py` + `watt_*` — cross-model cost & quality-per-dollar
+
+This is how we benchmark **qwen vs hermes with arborist**, on two axes
+that multiply into **quality-per-dollar**. Full living report (and the
+Dav1d-facing writeup) in `docs/energy-cogs-benchmark.md`; this is the
+orientation.
+
+**Two rigs, run on the same harness** (the GPU work happens on the
+producer box, never in arborist's `python+sqlite3` core):
+
+| | rig A | rig B |
+|---|---|---|
+| model | `Qwen3.6-27B` UD-Q4_K_XL (llama.cpp) | `Hermes-3-8B` FP8 (vLLM) |
+| GPU | RTX 4090, `ai.foxhop.net` (isolated for the bench) | RTX 3090, `3090-ai.foxhop.net` (live/public) |
+
+**Value axis — `bench/control_ab.py`.** Runs the *same* model two ways:
+**solo** (question only, no retrieval/verifier) vs **substrate-ON** (full
+`query()` — 4-route FTS5 → context → LLM → verifier), n=30 per fixture,
+grades each answer with the judge (§12). Headline finding: the substrate
+lifts both models 2–13× and they **nearly converge** (an 8B and a 27B
+reach similar grounded quality — grounding comes from retrieval+verify,
+not parametric size).
+
+**Cost axis — `bench/watt_bench.py` / `watt_calibrate.py` / `watt_probe.py`.**
+Measures **GPU joules per token** by slope calibration (prefill J/input-tok
+and decode J/output-tok measured separately — they differ ~35–40×), times
+the *real* token mix per query, at a site `--price-per-kwh`. Power states
+are measured at runtime, never hardcoded; cache-miss is forced so we time
+generation not a SQLite lookup. Result: a grounded answer costs
+**~$0.07–0.16 per 1,000 queries** of GPU electricity (hermes-8B cheaper
+than qwen-27B — but that's a 4-way confound: params × quant × engine ×
+card, not "8B vs 27B" alone). The dominant cost is **prefilling the
+retrieved context** (~⅔ of substrate GPU energy), not generating the
+answer.
+
+Run the value axis via `make control-ab` (`N=…  FIXTURE=…`); the watt
+instruments run directly — `bench/watt_calibrate.py` / `watt_bench.py`,
+with `make rapl-access` first for CPU-package power. Invocations and raw
+artifacts are in `docs/energy-cogs-benchmark.md` §8. **All value-axis
+grading is done by the deterministic code judge (§12)** — an Opus
+LLM-judge was attempted for #000057 and abandoned (too expensive per
+call, ~79.5 % `JUDGE_ERROR` rate-limited, burned coding quota); it never
+produced usable verdicts at scale and is gated off so it can't re-burn.
+
 ## 2. Four question fixtures
 
 ```
@@ -373,3 +419,66 @@ replace**, the curated adversarial set (the verifier-honesty gate).
     make answers more consistent across variants than the bare model?)
     is the LLM+judge version — rides `bench/control_sweep.py` over
     variant groups, gated v2.
+
+## 12. The judge — how bench answers are graded
+
+A grading judge appears **only** in the cross-model value bench (§1.3) —
+deciding whether a *model's answer* is correct-and-grounded. It is
+external bench science and must not be confused with arborist's in-system
+**verifier** (`arborist/qa/verify.py`), which decides `audit_mode` on the
+proof path and is bound by the "no LLM-as-judge in the verifier" rule.
+The bench judge sits *outside* both arms (solo and substrate) and grades
+post-hoc; it never touches either system's internals.
+
+Verdict vocabulary (closed, so aggregation is deterministic):
+`CORRECT_GROUNDED` / `WRONG` / `ABSTAINED` / `JUDGE_ERROR`.
+
+**The judge is the deterministic code judge.** Every value-axis number
+in `docs/energy-cogs-benchmark.md` was graded by it. There is no
+LLM-judge running on this path.
+
+### 12.1 Code judge — `bench/judge_code.py`
+
+Deterministic, **no LLM, no quota**. First hit decides:
+empty/no-gold guard → explicit-abstention regex → short-answer
+entity-grounding fast path → **NLI contradiction** (θ=0.85, runs on the
+GPU box) → lexical verifier (quote/span/entity/paraphrase) →
+WRONG-vs-FABRICATED tie-break on subject-in-gold. Claim-lattice JSON
+envelopes are unwrapped to prose first so the substrate arm grades on
+equal terms with solo. It is a *proxy* for grounding (lexical + NLI), not
+SOTA reading. The `JUDGE_ERROR` residue (~10 % after the `2d31866`
+HYBRID-resolution strengthening; see energy-cogs §5.5) is the residue
+worth handing to a stronger judge if one is ever wired in.
+**Full pipeline:** `docs/benchmark-matrix.md` "## The judge"; used in
+context in `docs/energy-cogs-benchmark.md` §5.5.
+
+### 12.2 Why there is no LLM judge in active use — `bench/judge.py` (abandoned)
+
+An Opus headless judge (`claude -p`) was attempted for #000057 as
+"external bench science." It did not work in practice:
+
+- **~79.5 % of its verdicts came back `JUDGE_ERROR`** — rate-limited by
+  the Anthropic API on a huge-N sweep, per `docs/qa-modes-bench.md`.
+- **Each call was expensive enough** that completing a real sweep would
+  have consumed the coding-agent's quota for unrelated work; the
+  experiment burned Opus quota and wasted dev time wiring it up
+  (`f63b00d` → `9dc02e4`, the wakeup call for the CLAUDE.md "Budget
+  discipline" rule).
+- It is now **fail-closed gated off** by an env flag (`ARBORIST_JUDGE_ENABLE=1`)
+  so a stray re-run can't re-burn the budget — a guardrail, not a
+  feature flag a reviewer should treat as available.
+
+The intended methodological hygiene (hermetic / blinded / grounded /
+structured / audited; stated same-family threat-to-validity) is preserved
+in the `bench/judge.py` module docstring as a record of what *would* be
+required of any future external SOTA judge — it is **not** a description
+of an active grading path. Re-introducing it needs a different-family
+SOTA model (per the same-family threat note) AND fox's explicit go on
+spend.
+
+**To forward to a reviewer (Dav1d):** `docs/energy-cogs-benchmark.md`
+(qwen-vs-hermes, already Dav1d-audience — shows the code judge grading
+in §5.5/§8) + this §12. The separate
+`docs/relevance-and-veto-synthesis-for-dav1d.md` covers the in-system
+*verifier / veto* layer (proof-path; a different thing from bench
+grading) and should not be confused with this section.