arborist/docs/energy-cogs-benchmark.md

Energy COGS benchmark — arborist substrate vs bare model

Status: living report. Two rigs measured: qwen-27B-Q4 on a dedicated RTX 4090 (isolated) and Hermes-3-8B-FP8 on a 3090 (live/public, contended). Reasoning variants follow on the same harness. Ticket: #000057 (watt bench) / #000060 (same-model substrate delta, cost axis). Audience: Dav1d de-novo review + maintainers.

1. Question

What does the arborist substrate cost in GPU energy per token, versus the same base model answering alone (no retrieval, no verifier)? Energy is the cost axis that pairs with the quality axis (control_sweep CG%); this report is cost only — it does not grade answers (energy is correctness-independent).

2. System under test

Two rigs:

	rig A	rig B
GPU	RTX 4090 (24 GB), ai.foxhop.net	RTX 3090 (24 GB), 3090-ai.foxhop.net
model	Qwen3.6-27B-UD-Q4_K_XL.gguf	Hermes-3-Llama-3.1-8B-FP8-Dynamic
params	27 B	8 B
precision / quant	~4-bit (unsloth-dynamic Q4_K_XL GGUF)	FP8-Dynamic (8-bit)
inference engine	llama.cpp (llama-server)	vLLM
warm-idle (resident)	~19 W	~196–350 W
isolation	dedicated (public blocked)	live/public (contended)

Common to both: arborist query() pipeline, STOCK V.1 frozen config (claim_lattice hash 036a4c79…, quote 5b6ca4c5…, bench/stock_v1.py); Wikipedia 2010 corpus (~866K docs/shard, ~/.arborist/shards); orchestrator drives the endpoint, power sampled on the box; $0.33/kWh (the only non-measured input, --price-per-kwh). (Fabrication-bait questions target post-2010 events, so the substrate correctly abstains where bare models answer from parametric memory.)

Two arms, same model (the substrate-delta design):

• solo — question only, neutral system prompt, no retrieval, no verifier.
• substrate-ON — full query(): 4-route FTS5 retrieval → context → LLM synthesis → deterministic verifier. Two answer modes (claim_lattice JSON, quote prose) — STOCK V.1 is a two-cell family.

Cross-rig comparison confounds FOUR variables at once — params (8 B vs 27 B), quantization/precision (FP8-Dynamic vs ~4-bit Q4_K_XL), inference engine (vLLM vs llama.cpp), and GPU (3090 vs 4090). The per-token rate gap (hermes cheaper) is the bundle, not any single factor; do not read it as "8 B vs 27 B" alone. Within a rig the solo-vs-substrate delta is clean (same model/quant/engine/GPU).

3. Why the card had to be isolated

The 4090 served live internet traffic through a Caddy proxy. A first pilot measured a contaminated result: the no-request idle baseline read 258 W / 52 % util (organic generation mid-window), higher than the bench's own generation window — so the marginal calculation clamped to a meaningless $0.00. A single-slot GPU under shared load cannot be energy-profiled from a short snapshot.

Resolution: the endpoint is now a dedicated benchmark rig — public access blocked at the proxy (@public → 403; our trusted IP still reaches it). External traffic drained the card from 100 %/443 W to 0 %/~20 W within ~1 min, confirmed isolated from a third-party IP (403). All numbers below are from the isolated card.

4. Instrument

Three properties a Dav1d review should check, each a defense against a specific way the number could be a lie:

4.1 Measured power states, never hardcoded

A card occupies distinct states (idle, middle-idle = model resident between requests, generation burst) and they differ per card×model×inference-server. watt_probe.classify_power_bands splits the 0.5 s samples by the largest gap in the sorted distribution — a data-derived threshold, not a fixed watt cut (unit-tested at two scales). Reporting joules / window as one number ("mean W") is rejected: it blends ~400 W bursts with sub-100 W gaps into a state the card never occupies.

4.2 Cache-miss guaranteed (no timing a SQLite lookup)

If the substrate arm hit the providence cache it would return a stored answer with no LLM call — we'd be timing a lookup, not generation. Guard: the arm runs burn_existing=True (force-deletes any matching live providence row before inference) and asserts cache_hits == 0, emitting a loud warning + real_inference=false in the record otherwise. Verified 0 hits on every run. (Solo has no cache path.)

4.3 GPU work is prefill + decode only — retrieval is CPU

Retrieval (4-route FTS5 over SQLite) and verification run on the orchestrator CPU; the GPU is untouched during them. The GPU does two things per query: prefill (process the whole input prompt in parallel) and decode (generate output tokens autoregressively). The wall-clock gross window integral is not query-attributable — it charges the always-on model-resident floor for the seconds spent in CPU-side retrieval, energy that exists regardless of the query while the card is free for other traffic. Gross is reported for reference only.

4.4 Input and output costed separately (calibrated)

Prefill (input) and decode (output) are physically different and have different J/token, so they cannot share a per-token denominator — a blended number is cheap-prefill-diluted in one direction and decode-inflated in the other. bench/watt_calibrate.py measures each by slope calibration: sweep prompt length at tiny max_tokens → prefill J/input-tok (fixed overhead cancels in the slope); fix a tiny prompt and sweep forced output length (ignore_eos) → decode J/output-tok. Prefill is kept cold (unique filler → cached_tokens=0), because at our scale there is no prefill/KV-cache reuse policy (we don't yet know which contexts are worth keeping hot) — so context is prefilled fresh every query.

5. Results (n=30, isolated 4090, qwen-nothink, $0.33/kWh)

5.1 Calibrated per-token energy

prefill (input)  : 0.175 J/tok  =  $0.016 / M-input-tok    (parallel, compute-bound)
decode  (output) : 6.157 J/tok  =  $0.564 / M-output-tok   (autoregressive, bandwidth-bound)
→ decode is 35× prefill per token

Clean linear fits over 3 points each (cached_tokens=0 throughout).

5.2 Measured token mix (per query, n=30)

The substrate prefills the retrieved context; solo prefills almost nothing. This — not a higher draw — is why the substrate costs more.

arm / mode	input tok/q	output tok/q	input ×vs solo
solo	52	14	1×
substrate · claim_lattice	6,617	91	127×
substrate · quote	4,424	88	85×

The two substrate modes differ in input tokens by POLICY, not noise: max_context_chars_by_mode caps quote at 24 KB and claim_lattice at 48 KB (2×), so claim_lattice prefills ~1.5× more context.

CONFOUND — not apples-to-apples. Those caps were bench-tuned on Hermes-3-8B (docs/qa-modes-bench.md Sprint 1b), then baked into DEFAULT_QUERY_POLICY and applied to every model. So (a) the cross-mode energy gap is mostly the budget difference, not anything intrinsic to the mode; and (b) qwen-27B here runs Hermes-tuned budgets — its own quality-optimal context size is unmeasured. For a clean cross-mode energy comparison, hold the context budget constant across modes; for cross-model, re-tune the budget per model first. The prefill lever is real (≈⅔ of substrate GPU energy is prefill, so the budget directly sets prefill COGS) — but the value driving it is currently a Hermes artifact.

5.3 Per-query GPU energy, decomposed (calibration applied)

energy ≈ 0.175·input_tok + 6.157·output_tok

arm / mode	prefill J	decode J	total J/q	input share	$/1k queries
solo	9	86	95	10 %	$0.009
substrate · claim_lattice	1,159	560	1,719	67 %	$0.158
substrate · quote	775	542	1,317	59 %	$0.121

Validation: the calibration model predicts substrate-CL 1,719 J/q vs the independently measured 1,804 J/q (~5 %) — the input/output decomposition reconstructs the directly-measured energy.

5.4 Second rig — 3090 / Hermes-8B, under live traffic

Run on the public, contended 3090 (0–4 concurrent web requests throughout; idle baseline flagged 69–97 % util). Key method result:

• The watt_bench per-query window integral is unusable here — the contaminated idle baseline made marginal clamp toward $0 (unimodal — no gen state). Expected: a shared card can't be profiled by a window integral.
• The slope calibration survived the traffic clean. The per-token slope cancels the variable shared-load baseline across reps, so the rates came out clean linear fits (cached_tokens=0 throughout):

                 prefill (input)         decode (output)
hermes / 3090    0.109 J/tok $0.010/M    4.40 J/tok $0.403/M   (decode 40× prefill)
qwen   / 4090    0.175 J/tok $0.016/M    6.16 J/tok $0.564/M   (decode 35× prefill)

On a shared/public card, use watt_calibrate (slope), not the per-query window integral.

Per-query energy = calibration rate × real token mix (the token counts are usage-reported, contamination-independent), sidestepping the contaminated window integral:

arm / mode	hermes/3090 J/q	hermes $/1k-q	qwen/4090 J/q	qwen $/1k-q
solo	~98	$0.0090	~95	$0.0087
substrate · claim_lattice	927	$0.085	1,719	$0.158
substrate · quote	765	$0.070	1,317	$0.121

Hermes substrate ≈ half qwen's per-query GPU energy — but that is the four-way confound (§2), dominated by 8 B vs 27 B. A grounded answer costs < $0.10 / 1,000 queries of GPU electricity on either rig.

6. Interpretation

• The substrate's cost is reading the evidence, not generating it. ~60–67 % of a substrate query's GPU energy is prefill of the retrieved context; solo's energy is ~90 % decode. Same model, same J/token rates — the substrate just feeds ~85–127× more input.
• Per output token both arms are identical (same decode rate); the delta is entirely the input context the substrate must prefill.
• claim_lattice vs quote: claim_lattice retrieves more context (6.6k vs 4.4k input tok) so its prefill is larger, but both decode a similar ~90 output tok. Net: claim_lattice 1,719 J/q, quote 1,317 J/q.
• Absolute GPU COGS is tiny at $0.33/kWh — $0.16 per 1,000 substrate queries (~18× solo's $0.009). The dominant arborist cost is CPU-side latency (retrieval/verify), not GPU electricity.
• Lever: prefill/KV-cache reuse would cut the substrate's dominant (input) cost — but needs a policy for which contexts are worth keeping hot. Not viable at current scale; a real optimization at higher QPS.

7. Threats to validity

1. Single fixture / model / card — qa_questions_stale_map on one 4090 + one Qwen-27B Q4 quant, llama.cpp. 3090 + Hermes + reasoning variants pending (§9). Token mixes (esp. retrieved-context size) are corpus- and policy-dependent.
2. Calibration is linear + endpoint-specific — assumes E ≈ prefill·in + decode·out with constant per-token rates; batching / KV-cache / different quants shift the coefficients. Re-calibrate per rig. Fit validated to ~5 % against measured here.
3. $/kWh is a single site assumption (0.33); linear to rescale.
4. idle / serving-floor bands are a measured proxy for the model-resident state; the per-query decomposition above relies on the calibrated rates, not the bands, and is the trustworthy number.
5. Tokens are now real API usage (prompt + completion), not the former len//4 estimate — this corrected an earlier draft that ignored input entirely.

8. Reproducibility

• Instruments: bench/watt_bench.py, bench/watt_probe.py, bench/watt_calibrate.py (commits 5b1cbee → 892d9ed); math + band split + slope unit-tested in tests/test_watt_cogs.py (14 tests).
• Real token usage: OpenAICompatibleClient.last_usage.
• Frozen substrate: bench/stock_v1.py (assert_not_drifted).
• Raw per-cell power samples, window timestamps, and calibration sweeps persisted in bench/qa_results/watt_{4090,3090}_*.json / watt_calibrate_{4090,3090}_*.json (gitignored — local artifacts; numbers reproduced in §5).
• Isolation: qwen closed to public at the proxy (proxy.unturf.com ingress/Caddyfile, commit 4b1fed1) — dedicated 4090 rig. The 3090/hermes run was on the live public endpoint; contention quantified via vLLM /metrics queue depth.

9. Next

• Apples-to-apples re-run with a FIXED context budget — the per-mode budgets (max_context_chars_by_mode) are Hermes-tuned (§5.2 confound), so the cross-mode prefill gap is a budget artifact. A constant-budget pass would isolate intrinsic mode cost.
• Reasoning variants (qwen-think, hermes-reasoning) — phase 3; the decode coefficient dominates (long reasoning traces = many output tokens at 4–6 J each).
• Prefill-cache study — measure cached_tokens lift under repeated contexts to size the KV-reuse lever (§6).
• Per-model context-budget tuning — qwen's quality-optimal budget is unmeasured (only Hermes was tuned); needed before any cross-model quality-per-joule claim.