arborist/docs/energy-cogs-benchmark.md

Energy COGS benchmark — arborist substrate vs bare model

Status: living report. First rig: qwen-27B on a dedicated RTX 4090. 3090 + Hermes-3-8B and the reasoning variants follow on the same harness. Ticket: #000057 (watt bench) / #000060 (same-model substrate delta, cost axis). Audience: Dav1d de-novo review + maintainers.

1. Question

What does the arborist substrate cost in GPU energy per token, versus the same base model answering alone (no retrieval, no verifier)? Energy is the cost axis that pairs with the quality axis (control_sweep CG%); this report is cost only — it does not grade answers (energy is correctness-independent).

2. System under test

component	value
GPU	NVIDIA RTX 4090 (24 GB), ai.foxhop.net
model	Qwen3.6-27B-UD-Q4_K_XL.gguf, llama.cpp, enable_thinking=false
substrate	arborist query() pipeline, STOCK V.1 frozen config
substrate hash	claim_lattice 036a4c79…, quote 5b6ca4c5… (bench/stock_v1.py)
driver	orchestrator drives the LLM endpoint; power sampled on the box
price	$0.33/kWh (configurable --price-per-kwh; the only non-measured input)

Two arms, same model (the substrate-delta design):

• solo — question only, neutral system prompt, no retrieval, no verifier.
• substrate-ON — full query(): 4-route FTS5 retrieval → context → LLM synthesis → deterministic verifier. Two answer modes (claim_lattice JSON, quote prose) — STOCK V.1 is a two-cell family.

3. Why the card had to be isolated

The 4090 served live internet traffic through a Caddy proxy. A first pilot measured a contaminated result: the no-request idle baseline read 258 W / 52 % util (organic generation mid-window), higher than the bench's own generation window — so the marginal calculation clamped to a meaningless $0.00. A single-slot GPU under shared load cannot be energy-profiled from a short snapshot.

Resolution: the endpoint is now a dedicated benchmark rig — public access blocked at the proxy (@public → 403; our trusted IP still reaches it). External traffic drained the card from 100 %/443 W to 0 %/~20 W within ~1 min, confirmed isolated from a third-party IP (403). All numbers below are from the isolated card.

4. Instrument

Three properties a Dav1d review should check, each a defense against a specific way the number could be a lie:

4.1 Measured power states, never hardcoded

A card occupies distinct states (idle, middle-idle = model resident between requests, generation burst) and they differ per card×model×inference-server. watt_probe.classify_power_bands splits the 0.5 s samples by the largest gap in the sorted distribution — a data-derived threshold, not a fixed watt cut (unit-tested at two scales). Reporting joules / window as one number ("mean W") is rejected: it blends ~400 W bursts with sub-100 W gaps into a state the card never occupies.

4.2 Cache-miss guaranteed (no timing a SQLite lookup)

If the substrate arm hit the providence cache it would return a stored answer with no LLM call — we'd be timing a lookup, not generation. Guard: the arm runs burn_existing=True (force-deletes any matching live providence row before inference) and asserts cache_hits == 0, emitting a loud warning + real_inference=false in the record otherwise. Verified 0 hits on every run. (Solo has no cache path.)

4.3 GPU work is prefill + decode only — retrieval is CPU

Retrieval (4-route FTS5 over SQLite) and verification run on the orchestrator CPU; the GPU is untouched during them. The GPU does two things per query: prefill (process the whole input prompt in parallel) and decode (generate output tokens autoregressively). The wall-clock gross window integral is not query-attributable — it charges the always-on model-resident floor for the seconds spent in CPU-side retrieval, energy that exists regardless of the query while the card is free for other traffic. Gross is reported for reference only.

4.4 Input and output costed separately (calibrated)

Prefill (input) and decode (output) are physically different and have different J/token, so they cannot share a per-token denominator — a blended number is cheap-prefill-diluted in one direction and decode-inflated in the other. bench/watt_calibrate.py measures each by slope calibration: sweep prompt length at tiny max_tokens → prefill J/input-tok (fixed overhead cancels in the slope); fix a tiny prompt and sweep forced output length (ignore_eos) → decode J/output-tok. Prefill is kept cold (unique filler → cached_tokens=0), because at our scale there is no prefill/KV-cache reuse policy (we don't yet know which contexts are worth keeping hot) — so context is prefilled fresh every query.

5. Results (n=30, isolated 4090, qwen-nothink, $0.33/kWh)

5.1 Calibrated per-token energy

prefill (input)  : 0.175 J/tok  =  $0.016 / M-input-tok    (parallel, compute-bound)
decode  (output) : 6.157 J/tok  =  $0.564 / M-output-tok   (autoregressive, bandwidth-bound)
→ decode is 35× prefill per token

Clean linear fits over 3 points each (cached_tokens=0 throughout).

5.2 Measured token mix (per query, n=30)

The substrate prefills the retrieved context; solo prefills almost nothing. This — not a higher draw — is why the substrate costs more.

arm / mode	input tok/q	output tok/q	input ×vs solo
solo	52	14	1×
substrate · claim_lattice	6,617	91	127×
substrate · quote	4,424	88	85×

5.3 Per-query GPU energy, decomposed (calibration applied)

energy ≈ 0.175·input_tok + 6.157·output_tok

arm / mode	prefill J	decode J	total J/q	input share	$/1k queries
solo	9	86	95	10 %	$0.009
substrate · claim_lattice	1,159	560	1,719	67 %	$0.158
substrate · quote	775	542	1,317	59 %	$0.121

Validation: the calibration model predicts substrate-CL 1,719 J/q vs the independently measured 1,804 J/q (~5 %) — the input/output decomposition reconstructs the directly-measured energy.

6. Interpretation

• The substrate's cost is reading the evidence, not generating it. ~60–67 % of a substrate query's GPU energy is prefill of the retrieved context; solo's energy is ~90 % decode. Same model, same J/token rates — the substrate just feeds ~85–127× more input.
• Per output token both arms are identical (same decode rate); the delta is entirely the input context the substrate must prefill.
• claim_lattice vs quote: claim_lattice retrieves more context (6.6k vs 4.4k input tok) so its prefill is larger, but both decode a similar ~90 output tok. Net: claim_lattice 1,719 J/q, quote 1,317 J/q.
• Absolute GPU COGS is tiny at $0.33/kWh — $0.16 per 1,000 substrate queries (~18× solo's $0.009). The dominant arborist cost is CPU-side latency (retrieval/verify), not GPU electricity.
• Lever: prefill/KV-cache reuse would cut the substrate's dominant (input) cost — but needs a policy for which contexts are worth keeping hot. Not viable at current scale; a real optimization at higher QPS.

7. Threats to validity

1. Single fixture / model / card — qa_questions_stale_map on one 4090 + one Qwen-27B Q4 quant, llama.cpp. 3090 + Hermes + reasoning variants pending (§9). Token mixes (esp. retrieved-context size) are corpus- and policy-dependent.
2. Calibration is linear + endpoint-specific — assumes E ≈ prefill·in + decode·out with constant per-token rates; batching / KV-cache / different quants shift the coefficients. Re-calibrate per rig. Fit validated to ~5 % against measured here.
3. $/kWh is a single site assumption (0.33); linear to rescale.
4. idle / serving-floor bands are a measured proxy for the model-resident state; the per-query decomposition above relies on the calibrated rates, not the bands, and is the trustworthy number.
5. Tokens are now real API usage (prompt + completion), not the former len//4 estimate — this corrected an earlier draft that ignored input entirely.

8. Reproducibility

• Instruments: bench/watt_bench.py, bench/watt_probe.py, bench/watt_calibrate.py (commits 5b1cbee → 892d9ed); math + band split + slope unit-tested in tests/test_watt_cogs.py (14 tests).
• Real token usage: OpenAICompatibleClient.last_usage.
• Frozen substrate: bench/stock_v1.py (assert_not_drifted).
• Raw per-cell power samples, window timestamps, and the calibration sweep persisted in bench/qa_results/watt_4090_*.json / watt_calibrate_4090_*.json for offline re-analysis.
• Isolation: qwen closed to public at the proxy (proxy.unturf.com ingress/Caddyfile, commit 4b1fed1) — dedicated benchmark rig.

9. Next

• 3090 + Hermes-3-8B on the same harness + a fresh calibration (--gpu-label 3090 --models hermes --arborist-ref hermes; re-run watt_calibrate --model-key hermes).
• Reasoning variants (qwen-think, hermes-reasoning) — phase 3; the decode coefficient will dominate (long reasoning traces = many output tokens at 6 J each).
• Prefill-cache study — measure cached_tokens lift under repeated contexts to size the KV-reuse lever (§6).