# Energy COGS benchmark — arborist substrate vs bare model

**Status:** living report. Two rigs measured: qwen-27B-Q4 on a dedicated
RTX 4090 (isolated) and Hermes-3-8B-FP8 on a 3090 (live/public,
contended). Reasoning variants follow on the same harness.
**Ticket:** #000057 (watt bench) / #000060 (same-model substrate delta,
cost axis). **Audience:** Dav1d de-novo review + maintainers.

## 1. Question

What does the arborist substrate **cost** (GPU energy per token) and what
is it **worth** (grounded-correct answers), versus the same base model
answering alone (no retrieval, no verifier)? §1–§5.4 are the cost axis
(energy is correctness-independent, measured separately); §5.5 is the
value axis (substrate-vs-solo answer quality + a cross-model comparison).
Together they give **quality-per-dollar**.

## 2. System under test

Two rigs:

| | rig A | rig B |
|---|---|---|
| GPU | **RTX 4090** (24 GB), `ai.foxhop.net` | **RTX 3090** (24 GB), `3090-ai.foxhop.net` |
| model | `Qwen3.6-27B-UD-Q4_K_XL.gguf` | `Hermes-3-Llama-3.1-8B-FP8-Dynamic` |
| params | **27 B** | **8 B** |
| precision / quant | **~4-bit** (unsloth-dynamic Q4_K_XL GGUF) | **FP8-Dynamic** (8-bit) |
| inference engine | **llama.cpp** (`llama-server`) | **vLLM** |
| warm-idle (resident) | ~19 W | ~196–350 W |
| isolation | **dedicated** (public blocked) | **live/public** (contended) |

Common to both: arborist `query()` pipeline, **STOCK V.1** frozen config
(`claim_lattice` hash `036a4c79…`, `quote` `5b6ca4c5…`,
`bench/stock_v1.py`); **Wikipedia 2010 corpus** (~866K docs/shard,
`~/.arborist/shards`); orchestrator drives the endpoint, power sampled on
the box; **$0.33/kWh** (the only non-measured input, `--price-per-kwh`).
(Fabrication-bait questions target post-2010 events, so the substrate
correctly abstains where bare models answer from parametric memory.)

**Two arms, same model (the substrate-delta design):**
- **solo** — question only, neutral system prompt, no retrieval, no verifier.
- **substrate-ON** — full `query()`: 4-route FTS5 retrieval → context →
  LLM synthesis → deterministic verifier. Two answer modes
  (`claim_lattice` JSON, `quote` prose) — STOCK V.1 is a two-cell family.

**Cross-rig comparison confounds FOUR variables at once** — params
(8 B vs 27 B), quantization/precision (FP8-Dynamic vs ~4-bit Q4_K_XL),
inference engine (vLLM vs llama.cpp), and GPU (3090 vs 4090). The
per-token rate gap (hermes cheaper) is the **bundle**, not any single
factor; do not read it as "8 B vs 27 B" alone. Within a rig the
solo-vs-substrate delta is clean (same model/quant/engine/GPU).

## 3. Why the card had to be isolated

The 4090 served live internet traffic through a Caddy proxy. A first
pilot measured a **contaminated** result: the no-request idle baseline
read **258 W / 52 % util** (organic generation mid-window), *higher* than
the bench's own generation window — so the marginal calculation clamped
to a meaningless **$0.00**. A single-slot GPU under shared load cannot be
energy-profiled from a short snapshot.

Resolution: the endpoint is now a **dedicated benchmark rig** — public
access blocked at the proxy (`@public → 403`; our trusted IP still
reaches it). External traffic drained the card from 100 %/443 W to
**0 %/~20 W within ~1 min**, confirmed isolated from a third-party IP
(403). All numbers below are from the isolated card.

## 4. Instrument

Three properties a Dav1d review should check, each a defense against a
specific way the number could be a lie:

### 4.1 Measured power states, never hardcoded
A card occupies distinct states (idle, middle-idle = model resident
between requests, generation burst) and they differ per
card×model×inference-server. `watt_probe.classify_power_bands` splits the
0.5 s samples by the **largest gap** in the sorted distribution — a
data-derived threshold, not a fixed watt cut (unit-tested at two scales).
Reporting `joules / window` as one number ("mean W") is rejected: it
blends ~400 W bursts with sub-100 W gaps into a state the card never
occupies.

### 4.2 Cache-miss guaranteed (no timing a SQLite lookup)
If the substrate arm hit the providence cache it would return a stored
answer with **no LLM call** — we'd be timing a lookup, not generation.
Guard: the arm runs `burn_existing=True` (force-deletes any matching live
providence row before inference) and asserts `cache_hits == 0`, emitting
a loud warning + `real_inference=false` in the record otherwise. Verified
0 hits on every run. (Solo has no cache path.)

### 4.3 GPU work is prefill + decode only — retrieval is CPU
Retrieval (4-route FTS5 over SQLite) and verification run on the
orchestrator CPU; **the GPU is untouched during them.** The GPU does two
things per query: **prefill** (process the whole input prompt in
parallel) and **decode** (generate output tokens autoregressively). The
wall-clock *gross* window integral is **not** query-attributable — it
charges the always-on model-resident floor for the seconds spent in
CPU-side retrieval, energy that exists regardless of the query while the
card is free for other traffic. Gross is reported for reference only.

### 4.4 Input and output costed separately (calibrated)
Prefill (input) and decode (output) are physically different and have
different J/token, so they **cannot share a per-token denominator** — a
blended number is cheap-prefill-diluted in one direction and
decode-inflated in the other. `bench/watt_calibrate.py` measures each by
**slope calibration**: sweep prompt length at tiny `max_tokens` →
prefill J/input-tok (fixed overhead cancels in the slope); fix a tiny
prompt and sweep forced output length (`ignore_eos`) → decode
J/output-tok. Prefill is kept cold (unique filler → `cached_tokens=0`),
because at our scale there is no prefill/KV-cache reuse policy (we don't
yet know which contexts are worth keeping hot) — so context is prefilled
fresh every query.

## 5. Results (n=30, isolated 4090, qwen-nothink, $0.33/kWh)

### 5.1 Calibrated per-token energy
```
prefill (input)  : 0.175 J/tok  =  $0.016 / M-input-tok    (parallel, compute-bound)
decode  (output) : 6.157 J/tok  =  $0.564 / M-output-tok   (autoregressive, bandwidth-bound)
→ decode is 35× prefill per token
```
Clean linear fits over 3 points each (`cached_tokens=0` throughout).

### 5.2 Measured token mix (per query, n=30)
The substrate prefills the retrieved context; solo prefills almost
nothing. This — not a higher draw — is why the substrate costs more.

| arm / mode | input tok/q | output tok/q | input ×vs solo |
|---|---|---|---|
| solo | 52 | 14 | 1× |
| substrate · claim_lattice | 6,617 | 91 | **127×** |
| substrate · quote | 4,424 | 88 | 85× |

The two substrate modes differ in **input** tokens by POLICY, not noise:
`max_context_chars_by_mode` caps `quote` at 24 KB and `claim_lattice` at
48 KB (2×), so claim_lattice prefills ~1.5× more context.

**CONFOUND — not apples-to-apples.** Those caps were bench-tuned on
**Hermes-3-8B** (`docs/qa-modes-bench.md` Sprint 1b), then baked into
`DEFAULT_QUERY_POLICY` and applied to every model. So (a) the cross-mode
energy gap is mostly the *budget* difference, not anything intrinsic to
the mode; and (b) **qwen-27B here runs Hermes-tuned budgets** — its own
quality-optimal context size is unmeasured. For a clean cross-mode
energy comparison, hold the context budget constant across modes; for
cross-model, re-tune the budget per model first. The prefill lever is
real (≈⅔ of substrate GPU energy is prefill, so the budget directly sets
prefill COGS) — but the value driving it is currently a Hermes artifact.

### 5.3 Per-query GPU energy, decomposed (calibration applied)
`energy ≈ 0.175·input_tok + 6.157·output_tok`

| arm / mode | prefill J | decode J | **total J/q** | input share | **$/1k queries** |
|---|---|---|---|---|---|
| solo | 9 | 86 | **95** | 10 % | $0.009 |
| substrate · claim_lattice | 1,159 | 560 | **1,719** | **67 %** | $0.158 |
| substrate · quote | 775 | 542 | **1,317** | 59 % | $0.121 |

**Validation:** the calibration model predicts substrate-CL **1,719 J/q**
vs the independently **measured 1,804 J/q** (~5 %) — the input/output
decomposition reconstructs the directly-measured energy.

### 5.4 Second rig — 3090 / Hermes-8B, under live traffic
Run on the **public, contended** 3090 (0–4 concurrent web requests
throughout; idle baseline flagged 69–97 % util). Key method result:

- **The watt_bench per-query window integral is unusable here** — the
  contaminated idle baseline made marginal clamp toward $0 (`unimodal —
  no gen state`). Expected: a shared card can't be profiled by a window
  integral.
- **The slope calibration survived the traffic clean.** The per-token
  slope cancels the variable shared-load baseline across reps, so the
  rates came out clean linear fits (`cached_tokens=0` throughout):

```
                 prefill (input)         decode (output)
hermes / 3090    0.109 J/tok $0.010/M    4.40 J/tok $0.403/M   (decode 40× prefill)
qwen   / 4090    0.175 J/tok $0.016/M    6.16 J/tok $0.564/M   (decode 35× prefill)
```

  **On a shared/public card, use `watt_calibrate` (slope), not the
  per-query window integral.**

Per-query energy = calibration rate × **real** token mix (the token
counts are usage-reported, contamination-independent), sidestepping the
contaminated window integral:

| arm / mode | hermes/3090 J/q | **hermes $/1k-q** | qwen/4090 J/q | **qwen $/1k-q** |
|---|---|---|---|---|
| solo | ~98 | **$0.0090** | ~95 | $0.0087 |
| substrate · claim_lattice | **927** | **$0.085** | 1,719 | $0.158 |
| substrate · quote | 765 | $0.070 | 1,317 | $0.121 |

Hermes substrate ≈ **half** qwen's per-query GPU energy — but that is the
four-way confound (§2), dominated by 8 B vs 27 B. A grounded answer costs
**~$0.07–0.16 per 1,000 queries** of GPU electricity — hermes-8B
$0.07–0.09, qwen-27B $0.12–0.16 (claim_lattice dearer than quote, more
context prefilled).

### 5.5 Quality delta (value side) — does the substrate earn its cost?

Cost is meaningless without value. `bench/control_ab.py` runs the SAME
model **solo** (question only, no retrieval/verifier) vs **substrate-ON**
(full `query()`), grades each with the deterministic **code judge**
(`bench/judge_code.py`, NLI-assisted, no quota), n=30 per fixture. Two
fixtures: **numeral** (stable historical facts, in-corpus) and **stale**
(world-leader questions — also in-corpus now that the corpus is 2010, see
below). Verdicts: CORRECT_GROUNDED / WRONG / ABSTAINED / JUDGE_ERROR.

**Substrate-ON — grounded-correct (CG / WRONG / JUDGE_ERROR):**

| fixture | hermes-8B | qwen-27B-nothink |
|---|---|---|
| numeral | 18 / 5 / 6 | 19 / 6 / 3 |
| stale   | 21 / 6 / 3 | **26 / 2 / 1** |

**Solo (bare model) — and the failure *style* differs:**

| fixture | hermes-8B | qwen-27B-nothink |
|---|---|---|
| numeral | 9 CG / 4 WR / **14 ABSTAIN** | 5 CG / **12 WR** / 3 AB |
| stale   | 5 CG / 20 WR | 2 CG / 22 WR |

Findings:

1. **The substrate lifts both models 2–13×** (hermes 9→18 / 5→21; qwen
   5→19 / 2→26) and they **nearly converge** under it (18 vs 19, 21 vs
   26). An 8 B and a 27 B reach *similar* grounded quality — the
   substrate's thesis: grounding comes from retrieval+verification, not
   parametric size, so the base model matters far less.
2. **qwen-27B is the marginally stronger base** (26 vs 21 on stale, fewer
   WRONG/JUDGE_ERROR), but the gap is small relative to the lift.
3. **Bare-model failure styles diverge:** bare hermes **abstains** on
   obscure entities (14/30 — honest "don't know"); bare qwen **fabricates**
   (12 WRONG, 3 abstain — confidently wrong). The substrate fixes both.

**Quality-per-dollar (the synthesis):** the substrate delivers ~18–26/30
grounded-correct answers (vs 2–9 bare) for **$0.085–0.158 per 1,000
queries** of GPU electricity (§5.3–5.4). The grounding is nearly free; the
base model is largely interchangeable.

Two honest caveats (a reviewer will press these):
- **CORRECT_GROUNDED = grounded in the 2010 corpus, not factually current.**
  On leader questions the substrate grounds the *2010* answer (Sarkozy for
  France); bare models give the *current* answer (Macron) from parametric
  memory → ungrounded → WRONG-vs-source. That is the substrate's contract
  working: answer *from the corpus*, not from memory.
- **The "stale" fixture stopped being post-corpus** once the 2010 corpus
  was loaded — those leaders are now *in* corpus, so it measures grounding
  (like numeral), not abstention. A truly post-2010 fabrication-bait set is
  needed to re-test the honest-abstention path (§9).

**Validity note (why this is trustworthy now):** an earlier qwen quality
run showed arborist abstaining 30/30 — a **bug**, not epistemics: Qwen3
defaults to thinking-on and burned its whole token budget on hidden
`<think>` over the 20 K context, returning empty `content` → false
UNGROUNDED. Fixed in commit `39c040c` (client defaults Qwen3
`enable_thinking=False`). The energy numbers were unaffected (real
inference happened; bench harnesses already passed the no-think flag via
MODELS). The code judge was then strengthened (`2d31866`) to resolve
HYBRID answers with a verified verbatim quote + on-topic subject + no
unsourced numeric, cutting JUDGE_ERROR from 40 % to ~10 %.

## 6. Interpretation

- **The substrate's cost is reading the evidence, not generating it.**
  ~60–67 % of a substrate query's GPU energy is **prefill of the
  retrieved context**; solo's energy is ~90 % decode. Same model, same
  J/token rates — the substrate just feeds ~85–127× more input.
- **Per output token both arms are identical** (same decode rate); the
  delta is entirely the input context the substrate must prefill.
- **`claim_lattice` vs `quote`**: claim_lattice retrieves *more* context
  (6.6k vs 4.4k input tok) so its prefill is larger, but both decode a
  similar ~90 output tok. Net: claim_lattice 1,719 J/q, quote 1,317 J/q.
- **Absolute GPU COGS is tiny** at $0.33/kWh — **$0.16 per 1,000
  substrate queries** (~18× solo's $0.009). The dominant arborist cost
  is CPU-side latency (retrieval/verify), not GPU electricity.
- **Lever:** prefill/KV-cache reuse would cut the substrate's dominant
  (input) cost — but needs a policy for which contexts are worth keeping
  hot. Not viable at current scale; a real optimization at higher QPS.

## 7. Threats to validity

1. **Single fixture / model / card** — `qa_questions_stale_map` on one
   4090 + one Qwen-27B Q4 quant, llama.cpp. 3090 + Hermes + reasoning
   variants pending (§9). Token mixes (esp. retrieved-context size) are
   corpus- and policy-dependent.
2. **Calibration is linear + endpoint-specific** — assumes
   `E ≈ prefill·in + decode·out` with constant per-token rates; batching
   / KV-cache / different quants shift the coefficients. Re-calibrate per
   rig. Fit validated to ~5 % against measured here.
3. **$/kWh** is a single site assumption (0.33); linear to rescale.
4. **idle / serving-floor bands** are a measured proxy for the
   model-resident state; the per-query decomposition above relies on the
   *calibrated* rates, not the bands, and is the trustworthy number.
5. **Tokens are now real API `usage`** (prompt + completion), not the
   former `len//4` estimate — this corrected an earlier draft that
   ignored input entirely.

## 8. Reproducibility

- Energy instruments: `bench/watt_bench.py`, `bench/watt_probe.py`,
  `bench/watt_calibrate.py` (commits `5b1cbee` → `892d9ed`); math + band
  split + slope unit-tested in `tests/test_watt_cogs.py` (14 tests).
- Quality (§5.5): `bench/control_ab.py` + `bench/judge_code.py` (code
  judge, NLI-assisted, no quota; HYBRID-resolution strengthened in
  `2d31866`, self-test 4/4). Qwen3 no-think fix `39c040c`.
- Real token usage: `OpenAICompatibleClient.last_usage`.
- Frozen substrate: `bench/stock_v1.py` (`assert_not_drifted`).
- Raw per-cell power samples, window timestamps, and calibration sweeps
  persisted in `bench/qa_results/watt_{4090,3090}_*.json` /
  `watt_calibrate_{4090,3090}_*.json` (gitignored — local artifacts;
  numbers reproduced in §5).
- Isolation: qwen closed to public at the proxy
  (`proxy.unturf.com` `ingress/Caddyfile`, commit `4b1fed1`) — dedicated
  4090 rig. The 3090/hermes run was on the **live public** endpoint;
  contention quantified via vLLM `/metrics` queue depth.

## 9. Next

- **Apples-to-apples re-run with a FIXED context budget** — the per-mode
  budgets (`max_context_chars_by_mode`) are Hermes-tuned (§5.2 confound),
  so the cross-mode prefill gap is a budget artifact. A constant-budget
  pass would isolate intrinsic mode cost.
- **Reasoning variants** (qwen-think, hermes-reasoning) — phase 3; the
  decode coefficient dominates (long reasoning traces = many output
  tokens at 4–6 J each).
- **Prefill-cache study** — measure `cached_tokens` lift under repeated
  contexts to size the KV-reuse lever (§6).
- **Per-model context-budget tuning** — qwen's quality-optimal budget is
  unmeasured (only Hermes was tuned); needed before any cross-model
  *quality*-per-joule claim.
- **Truly-post-2010 fabrication-bait fixture** — the "stale" set is now
  in-corpus (§5.5 caveat); a genuinely post-2010 set is needed to re-test
  the honest-abstention path that the corpus upgrade erased.
- **Stronger non-code judge for the residue** — ~10 % JUDGE_ERROR remains
  (mostly bare-solo ungrounded answers); a SOTA blinded judge would grade
  those (the open #000057 "which judge" decision).