arborist/docs/energy-cogs-benchmark.md

Energy COGS benchmark — arborist substrate vs bare model

Status: living report. Two rigs measured: qwen-27B-Q4 on a dedicated RTX 4090 (isolated) and Hermes-3-8B-FP8 on a 3090 (live/public, contended). Reasoning variants follow on the same harness. Ticket: #000057 (watt bench) / #000060 (same-model substrate delta, cost axis). Audience: Dav1d de-novo review + maintainers.

1. Question

What does the arborist substrate cost (GPU energy per token) and what is it worth (grounded-correct answers), versus the same base model answering alone (no retrieval, no verifier)? §1–§5.4 are the cost axis (energy is correctness-independent, measured separately); §5.5 is the value axis (substrate-vs-solo answer quality + a cross-model comparison). Together they give quality-per-dollar.

2. System under test

Two rigs:

	rig A	rig B
GPU	RTX 4090 (24 GB), ai.foxhop.net	RTX 3090 (24 GB), 3090-ai.foxhop.net
model	Qwen3.6-27B-UD-Q4_K_XL.gguf	Hermes-3-Llama-3.1-8B-FP8-Dynamic
params	27 B	8 B
precision / quant	~4-bit (unsloth-dynamic Q4_K_XL GGUF)	FP8-Dynamic (8-bit)
inference engine	llama.cpp (llama-server)	vLLM
warm-idle (resident)	~19 W	~196–350 W
isolation	dedicated (public blocked)	live/public (contended)

Common to both: arborist query() pipeline, STOCK V.1 frozen config (claim_lattice hash 036a4c79…, quote 5b6ca4c5…, bench/stock_v1.py); Wikipedia 2010 corpus (~866K docs/shard, ~/.arborist/shards); orchestrator drives the endpoint, power sampled on the box; $0.33/kWh (the only non-measured input, --price-per-kwh). (Fabrication-bait questions target post-2010 events, so the substrate correctly abstains where bare models answer from parametric memory.)

Two arms, same model (the substrate-delta design):

• solo — question only, neutral system prompt, no retrieval, no verifier.
• substrate-ON — full query(): 4-route FTS5 retrieval → context → LLM synthesis → deterministic verifier. Two answer modes (claim_lattice JSON, quote prose) — STOCK V.1 is a two-cell family.

Cross-rig comparison confounds FOUR variables at once — params (8 B vs 27 B), quantization/precision (FP8-Dynamic vs ~4-bit Q4_K_XL), inference engine (vLLM vs llama.cpp), and GPU (3090 vs 4090). The per-token rate gap (hermes cheaper) is the bundle, not any single factor; do not read it as "8 B vs 27 B" alone. Within a rig the solo-vs-substrate delta is clean (same model/quant/engine/GPU).

3. Why the card had to be isolated

The 4090 served live internet traffic through a Caddy proxy. A first pilot measured a contaminated result: the no-request idle baseline read 258 W / 52 % util (organic generation mid-window), higher than the bench's own generation window — so the marginal calculation clamped to a meaningless $0.00. A single-slot GPU under shared load cannot be energy-profiled from a short snapshot.

Resolution: the endpoint is now a dedicated benchmark rig — public access blocked at the proxy (@public → 403; our trusted IP still reaches it). External traffic drained the card from 100 %/443 W to 0 %/~20 W within ~1 min, confirmed isolated from a third-party IP (403). All numbers below are from the isolated card.

4. Instrument

Three properties a Dav1d review should check, each a defense against a specific way the number could be a lie:

4.1 Measured power states, never hardcoded

A card occupies distinct states (idle, middle-idle = model resident between requests, generation burst) and they differ per card×model×inference-server. watt_probe.classify_power_bands splits the 0.5 s samples by the largest gap in the sorted distribution — a data-derived threshold, not a fixed watt cut (unit-tested at two scales). Reporting joules / window as one number ("mean W") is rejected: it blends ~400 W bursts with sub-100 W gaps into a state the card never occupies.

4.2 Cache-miss guaranteed (no timing a SQLite lookup)

If the substrate arm hit the providence cache it would return a stored answer with no LLM call — we'd be timing a lookup, not generation. Guard: the arm runs burn_existing=True (force-deletes any matching live providence row before inference) and asserts cache_hits == 0, emitting a loud warning + real_inference=false in the record otherwise. Verified 0 hits on every run. (Solo has no cache path.)

4.3 GPU work is prefill + decode only — retrieval is CPU

Retrieval (4-route FTS5 over SQLite) and verification run on the orchestrator CPU; the GPU is untouched during them. The GPU does two things per query: prefill (process the whole input prompt in parallel) and decode (generate output tokens autoregressively). The wall-clock gross window integral is not query-attributable — it charges the always-on model-resident floor for the seconds spent in CPU-side retrieval, energy that exists regardless of the query while the card is free for other traffic. Gross is reported for reference only.

4.4 Input and output costed separately (calibrated)

Prefill (input) and decode (output) are physically different and have different J/token, so they cannot share a per-token denominator — a blended number is cheap-prefill-diluted in one direction and decode-inflated in the other. bench/watt_calibrate.py measures each by slope calibration: sweep prompt length at tiny max_tokens → prefill J/input-tok (fixed overhead cancels in the slope); fix a tiny prompt and sweep forced output length (ignore_eos) → decode J/output-tok. Prefill is kept cold (unique filler → cached_tokens=0), because at our scale there is no prefill/KV-cache reuse policy (we don't yet know which contexts are worth keeping hot) — so context is prefilled fresh every query.

5. Results (n=30, isolated 4090, qwen-nothink, $0.33/kWh)

5.1 Calibrated per-token energy

prefill (input)  : 0.175 J/tok  =  $0.016 / M-input-tok    (parallel, compute-bound)
decode  (output) : 6.157 J/tok  =  $0.564 / M-output-tok   (autoregressive, bandwidth-bound)
→ decode is 35× prefill per token

Clean linear fits over 3 points each (cached_tokens=0 throughout).

5.2 Measured token mix (per query, n=30)

The substrate prefills the retrieved context; solo prefills almost nothing. This — not a higher draw — is why the substrate costs more.

arm / mode	input tok/q	output tok/q	input ×vs solo
solo	52	14	1×
substrate · claim_lattice	6,617	91	127×
substrate · quote	4,424	88	85×

The two substrate modes differ in input tokens by POLICY, not noise: max_context_chars_by_mode caps quote at 24 KB and claim_lattice at 48 KB (2×), so claim_lattice prefills ~1.5× more context.

CONFOUND — not apples-to-apples. Those caps were bench-tuned on Hermes-3-8B (docs/qa-modes-bench.md Sprint 1b), then baked into DEFAULT_QUERY_POLICY and applied to every model. So (a) the cross-mode energy gap is mostly the budget difference, not anything intrinsic to the mode; and (b) qwen-27B here runs Hermes-tuned budgets — its own quality-optimal context size is unmeasured. For a clean cross-mode energy comparison, hold the context budget constant across modes; for cross-model, re-tune the budget per model first. The prefill lever is real (≈⅔ of substrate GPU energy is prefill, so the budget directly sets prefill COGS) — but the value driving it is currently a Hermes artifact.

5.3 Per-query GPU energy, decomposed (calibration applied)

energy ≈ 0.175·input_tok + 6.157·output_tok

arm / mode	prefill J	decode J	total J/q	input share	$/1k queries
solo	9	86	95	10 %	$0.009
substrate · claim_lattice	1,159	560	1,719	67 %	$0.158
substrate · quote	775	542	1,317	59 %	$0.121

Validation: the calibration model predicts substrate-CL 1,719 J/q vs the independently measured 1,804 J/q (~5 %) — the input/output decomposition reconstructs the directly-measured energy.

5.4 Second rig — 3090 / Hermes-8B, under live traffic

Run on the public, contended 3090 (0–4 concurrent web requests throughout; idle baseline flagged 69–97 % util). Key method result:

• The watt_bench per-query window integral is unusable here — the contaminated idle baseline made marginal clamp toward $0 (unimodal — no gen state). Expected: a shared card can't be profiled by a window integral.
• The slope calibration survived the traffic clean. The per-token slope cancels the variable shared-load baseline across reps, so the rates came out clean linear fits (cached_tokens=0 throughout):

                 prefill (input)         decode (output)
hermes / 3090    0.109 J/tok $0.010/M    4.40 J/tok $0.403/M   (decode 40× prefill)
qwen   / 4090    0.175 J/tok $0.016/M    6.16 J/tok $0.564/M   (decode 35× prefill)

On a shared/public card, use watt_calibrate (slope), not the per-query window integral.

Per-query energy = calibration rate × real token mix (the token counts are usage-reported, contamination-independent), sidestepping the contaminated window integral:

arm / mode	hermes/3090 J/q	hermes $/1k-q	qwen/4090 J/q	qwen $/1k-q
solo	~98	$0.0090	~95	$0.0087
substrate · claim_lattice	927	$0.085	1,719	$0.158
substrate · quote	765	$0.070	1,317	$0.121

Hermes substrate ≈ half qwen's per-query GPU energy — but that is the four-way confound (§2), dominated by 8 B vs 27 B. A grounded answer costs ~$0.07–0.16 per 1,000 queries of GPU electricity — hermes-8B $0.07–0.09, qwen-27B $0.12–0.16 (claim_lattice dearer than quote, more context prefilled).

5.5 Quality delta (value side) — does the substrate earn its cost?

Cost is meaningless without value. bench/control_ab.py runs the SAME model solo (question only, no retrieval/verifier) vs substrate-ON (full query()), grades each with the deterministic code judge (bench/judge_code.py, NLI-assisted, no quota), n=30 per fixture. Two fixtures: numeral (stable historical facts, in-corpus) and stale (world-leader questions — also in-corpus now that the corpus is 2010, see below). Verdicts: CORRECT_GROUNDED / WRONG / ABSTAINED / JUDGE_ERROR.

Substrate-ON — grounded-correct (CG / WRONG / JUDGE_ERROR):

fixture	hermes-8B	qwen-27B-nothink
numeral	18 / 5 / 6	19 / 6 / 3
stale	21 / 6 / 3	26 / 2 / 1

Solo (bare model) — and the failure style differs:

fixture	hermes-8B	qwen-27B-nothink
numeral	9 CG / 4 WR / 14 ABSTAIN	5 CG / 12 WR / 3 AB
stale	5 CG / 20 WR	2 CG / 22 WR

Findings:

1. The substrate lifts both models 2–13× (hermes 9→18 / 5→21; qwen 5→19 / 2→26) and they nearly converge under it (18 vs 19, 21 vs 26). An 8 B and a 27 B reach similar grounded quality — the substrate's thesis: grounding comes from retrieval+verification, not parametric size, so the base model matters far less.
2. qwen-27B is the marginally stronger base (26 vs 21 on stale, fewer WRONG/JUDGE_ERROR), but the gap is small relative to the lift.
3. Bare-model failure styles diverge: bare hermes abstains on obscure entities (14/30 — honest "don't know"); bare qwen fabricates (12 WRONG, 3 abstain — confidently wrong). The substrate fixes both.

Quality-per-dollar (the synthesis): the substrate delivers ~18–26/30 grounded-correct answers (vs 2–9 bare) for $0.085–0.158 per 1,000 queries of GPU electricity (§5.3–5.4). The grounding is nearly free; the base model is largely interchangeable.

Two honest caveats (a reviewer will press these):

• CORRECT_GROUNDED = grounded in the 2010 corpus, not factually current. On leader questions the substrate grounds the 2010 answer (Sarkozy for France); bare models give the current answer (Macron) from parametric memory → ungrounded → WRONG-vs-source. That is the substrate's contract working: answer from the corpus, not from memory.
• The "stale" fixture stopped being post-corpus once the 2010 corpus was loaded — those leaders are now in corpus, so it measures grounding (like numeral), not abstention. A truly post-2010 fabrication-bait set is needed to re-test the honest-abstention path (§9).

Validity note (why this is trustworthy now): an earlier qwen quality run showed arborist abstaining 30/30 — a bug, not epistemics: Qwen3 defaults to thinking-on and burned its whole token budget on hidden <think> over the 20 K context, returning empty content → false UNGROUNDED. Fixed in commit 39c040c (client defaults Qwen3 enable_thinking=False). The energy numbers were unaffected (real inference happened; bench harnesses already passed the no-think flag via MODELS). The code judge was then strengthened (2d31866) to resolve HYBRID answers with a verified verbatim quote + on-topic subject + no unsourced numeric, cutting JUDGE_ERROR from 40 % to ~10 %.

6. Interpretation

• The substrate's cost is reading the evidence, not generating it. ~60–67 % of a substrate query's GPU energy is prefill of the retrieved context; solo's energy is ~90 % decode. Same model, same J/token rates — the substrate just feeds ~85–127× more input.
• Per output token both arms are identical (same decode rate); the delta is entirely the input context the substrate must prefill.
• claim_lattice vs quote: claim_lattice retrieves more context (6.6k vs 4.4k input tok) so its prefill is larger, but both decode a similar ~90 output tok. Net: claim_lattice 1,719 J/q, quote 1,317 J/q.
• Absolute GPU COGS is tiny at $0.33/kWh — $0.16 per 1,000 substrate queries (~18× solo's $0.009). The dominant arborist cost is CPU-side latency (retrieval/verify), not GPU electricity.
• Lever: prefill/KV-cache reuse would cut the substrate's dominant (input) cost — but needs a policy for which contexts are worth keeping hot. Not viable at current scale; a real optimization at higher QPS.

7. Threats to validity

1. Single fixture / model / card — qa_questions_stale_map on one 4090 + one Qwen-27B Q4 quant, llama.cpp. 3090 + Hermes + reasoning variants pending (§9). Token mixes (esp. retrieved-context size) are corpus- and policy-dependent.
2. Calibration is linear + endpoint-specific — assumes E ≈ prefill·in + decode·out with constant per-token rates; batching / KV-cache / different quants shift the coefficients. Re-calibrate per rig. Fit validated to ~5 % against measured here.
3. $/kWh is a single site assumption (0.33); linear to rescale.
4. idle / serving-floor bands are a measured proxy for the model-resident state; the per-query decomposition above relies on the calibrated rates, not the bands, and is the trustworthy number.
5. Tokens are now real API usage (prompt + completion), not the former len//4 estimate — this corrected an earlier draft that ignored input entirely.

8. Reproducibility

• Energy instruments: bench/watt_bench.py, bench/watt_probe.py, bench/watt_calibrate.py (commits 5b1cbee → 892d9ed); math + band split + slope unit-tested in tests/test_watt_cogs.py (14 tests).
• Quality (§5.5): bench/control_ab.py + bench/judge_code.py (code judge, NLI-assisted, no quota; HYBRID-resolution strengthened in 2d31866, self-test 4/4). Qwen3 no-think fix 39c040c.
• Real token usage: OpenAICompatibleClient.last_usage.
• Frozen substrate: bench/stock_v1.py (assert_not_drifted).
• Raw per-cell power samples, window timestamps, and calibration sweeps persisted in bench/qa_results/watt_{4090,3090}_*.json / watt_calibrate_{4090,3090}_*.json (gitignored — local artifacts; numbers reproduced in §5).
• Isolation: qwen closed to public at the proxy (proxy.unturf.com ingress/Caddyfile, commit 4b1fed1) — dedicated 4090 rig. The 3090/hermes run was on the live public endpoint; contention quantified via vLLM /metrics queue depth.

9. Next

• Apples-to-apples re-run with a FIXED context budget — the per-mode budgets (max_context_chars_by_mode) are Hermes-tuned (§5.2 confound), so the cross-mode prefill gap is a budget artifact. A constant-budget pass would isolate intrinsic mode cost.
• Reasoning variants (qwen-think, hermes-reasoning) — phase 3; the decode coefficient dominates (long reasoning traces = many output tokens at 4–6 J each).
• Prefill-cache study — measure cached_tokens lift under repeated contexts to size the KV-reuse lever (§6).
• Per-model context-budget tuning — qwen's quality-optimal budget is unmeasured (only Hermes was tuned); needed before any cross-model quality-per-joule claim.
• Truly-post-2010 fabrication-bait fixture — the "stale" set is now in-corpus (§5.5 caveat); a genuinely post-2010 set is needed to re-test the honest-abstention path that the corpus upgrade erased.
• Stronger non-code judge for the residue — ~10 % JUDGE_ERROR remains (mostly bare-solo ungrounded answers); a SOTA blinded judge would grade those (the open #000057 "which judge" decision).