diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 53852df..06ce8b4 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -10,11 +10,13 @@ the canonical entry point. Three companion docs cover specifics: | `docs/qa-modes-bench.md` | Historical journal — running addendum log | | `docs/bench-emergent-design.md` | Random-word stress-test design rationale | | `docs/qa-modes-bench-2026-04-30.md` | Frozen point-in-time bench (Apr-30 baseline) | +| `docs/energy-cogs-benchmark.md` | **qwen vs hermes** — GPU energy COGS + quality-per-dollar (Dav1d-audience report) | +| `docs/benchmark-matrix.md` | Control-arm (solo vs substrate) A/B matrix + the deterministic code judge | Read those when you need the why or the long-form data. This file covers the what, where, and how. -## 1. Two harnesses, two purposes +## 1. Three harnesses, three purposes ### 1.1 `bench/qa_sweep.py` — curated QA-quality bench @@ -58,6 +60,50 @@ Run via `make bench-emergent` (default `EMERGENT_N=10`) or `make bench-emergent EMERGENT_N=100` for a longer cycle. See `docs/bench-emergent-design.md` for the design rationale. +### 1.3 `bench/control_ab.py` + `watt_*` — cross-model cost & quality-per-dollar + +This is how we benchmark **qwen vs hermes with arborist**, on two axes +that multiply into **quality-per-dollar**. Full living report (and the +Dav1d-facing writeup) in `docs/energy-cogs-benchmark.md`; this is the +orientation. + +**Two rigs, run on the same harness** (the GPU work happens on the +producer box, never in arborist's `python+sqlite3` core): + +| | rig A | rig B | +|---|---|---| +| model | `Qwen3.6-27B` UD-Q4_K_XL (llama.cpp) | `Hermes-3-8B` FP8 (vLLM) | +| GPU | RTX 4090, `ai.foxhop.net` (isolated for the bench) | RTX 3090, `3090-ai.foxhop.net` (live/public) | + +**Value axis — `bench/control_ab.py`.** Runs the *same* model two ways: +**solo** (question only, no retrieval/verifier) vs **substrate-ON** (full +`query()` — 4-route FTS5 → context → LLM → verifier), n=30 per fixture, +grades each answer with the judge (§12). Headline finding: the substrate +lifts both models 2–13× and they **nearly converge** (an 8B and a 27B +reach similar grounded quality — grounding comes from retrieval+verify, +not parametric size). + +**Cost axis — `bench/watt_bench.py` / `watt_calibrate.py` / `watt_probe.py`.** +Measures **GPU joules per token** by slope calibration (prefill J/input-tok +and decode J/output-tok measured separately — they differ ~35–40×), times +the *real* token mix per query, at a site `--price-per-kwh`. Power states +are measured at runtime, never hardcoded; cache-miss is forced so we time +generation not a SQLite lookup. Result: a grounded answer costs +**~$0.07–0.16 per 1,000 queries** of GPU electricity (hermes-8B cheaper +than qwen-27B — but that's a 4-way confound: params × quant × engine × +card, not "8B vs 27B" alone). The dominant cost is **prefilling the +retrieved context** (~⅔ of substrate GPU energy), not generating the +answer. + +Run the value axis via `make control-ab` (`N=… FIXTURE=…`); the watt +instruments run directly — `bench/watt_calibrate.py` / `watt_bench.py`, +with `make rapl-access` first for CPU-package power. Invocations and raw +artifacts are in `docs/energy-cogs-benchmark.md` §8. **All value-axis +grading is done by the deterministic code judge (§12)** — an Opus +LLM-judge was attempted for #000057 and abandoned (too expensive per +call, ~79.5 % `JUDGE_ERROR` rate-limited, burned coding quota); it never +produced usable verdicts at scale and is gated off so it can't re-burn. + ## 2. Four question fixtures ``` @@ -373,3 +419,66 @@ replace**, the curated adversarial set (the verifier-honesty gate). make answers more consistent across variants than the bare model?) is the LLM+judge version — rides `bench/control_sweep.py` over variant groups, gated v2. + +## 12. The judge — how bench answers are graded + +A grading judge appears **only** in the cross-model value bench (§1.3) — +deciding whether a *model's answer* is correct-and-grounded. It is +external bench science and must not be confused with arborist's in-system +**verifier** (`arborist/qa/verify.py`), which decides `audit_mode` on the +proof path and is bound by the "no LLM-as-judge in the verifier" rule. +The bench judge sits *outside* both arms (solo and substrate) and grades +post-hoc; it never touches either system's internals. + +Verdict vocabulary (closed, so aggregation is deterministic): +`CORRECT_GROUNDED` / `WRONG` / `ABSTAINED` / `JUDGE_ERROR`. + +**The judge is the deterministic code judge.** Every value-axis number +in `docs/energy-cogs-benchmark.md` was graded by it. There is no +LLM-judge running on this path. + +### 12.1 Code judge — `bench/judge_code.py` + +Deterministic, **no LLM, no quota**. First hit decides: +empty/no-gold guard → explicit-abstention regex → short-answer +entity-grounding fast path → **NLI contradiction** (θ=0.85, runs on the +GPU box) → lexical verifier (quote/span/entity/paraphrase) → +WRONG-vs-FABRICATED tie-break on subject-in-gold. Claim-lattice JSON +envelopes are unwrapped to prose first so the substrate arm grades on +equal terms with solo. It is a *proxy* for grounding (lexical + NLI), not +SOTA reading. The `JUDGE_ERROR` residue (~10 % after the `2d31866` +HYBRID-resolution strengthening; see energy-cogs §5.5) is the residue +worth handing to a stronger judge if one is ever wired in. +**Full pipeline:** `docs/benchmark-matrix.md` "## The judge"; used in +context in `docs/energy-cogs-benchmark.md` §5.5. + +### 12.2 Why there is no LLM judge in active use — `bench/judge.py` (abandoned) + +An Opus headless judge (`claude -p`) was attempted for #000057 as +"external bench science." It did not work in practice: + +- **~79.5 % of its verdicts came back `JUDGE_ERROR`** — rate-limited by + the Anthropic API on a huge-N sweep, per `docs/qa-modes-bench.md`. +- **Each call was expensive enough** that completing a real sweep would + have consumed the coding-agent's quota for unrelated work; the + experiment burned Opus quota and wasted dev time wiring it up + (`f63b00d` → `9dc02e4`, the wakeup call for the CLAUDE.md "Budget + discipline" rule). +- It is now **fail-closed gated off** by an env flag (`ARBORIST_JUDGE_ENABLE=1`) + so a stray re-run can't re-burn the budget — a guardrail, not a + feature flag a reviewer should treat as available. + +The intended methodological hygiene (hermetic / blinded / grounded / +structured / audited; stated same-family threat-to-validity) is preserved +in the `bench/judge.py` module docstring as a record of what *would* be +required of any future external SOTA judge — it is **not** a description +of an active grading path. Re-introducing it needs a different-family +SOTA model (per the same-family threat note) AND fox's explicit go on +spend. + +**To forward to a reviewer (Dav1d):** `docs/energy-cogs-benchmark.md` +(qwen-vs-hermes, already Dav1d-audience — shows the code judge grading +in §5.5/§8) + this §12. The separate +`docs/relevance-and-veto-synthesis-for-dav1d.md` covers the in-system +*verifier / veto* layer (proof-path; a different thing from bench +grading) and should not be confused with this section.