docs(#000057): energy-COGS report — separated prefill/decode, n=30, Dav1d-ready
Full report for Dav1d: qwen-nothink on the dedicated 4090. Headline — prefill 0.175 J/input-tok ($0.016/M) vs decode 6.157 J/output-tok ($0.564/M), decode 35x dearer per token. The substrate prefills ~6.6k input tok/query (vs solo ~52, 127x): ~67% of its GPU energy is reading the retrieved context, not generating. Per-query 1719 J (substrate-CL) vs 95 J (solo) = $0.16 vs $0.009 per 1k queries; calibration predicts measured within ~5%. Real cost is CPU latency, not GPU watts. Documents the full methodology + the corrections that got here (contamination -> isolation; blended mean-W -> measured states; gross -> not-attributable; per-token -> separated input/output; len//4 -> real usage), threats to validity, reproducibility (commits, config hash, persisted samples), and next rigs (3090+Hermes, reasoning, prefill-cache study).
This commit is contained in:
parent
892d9ed037
commit
9d9e530466
1 changed files with 186 additions and 0 deletions
186
docs/energy-cogs-benchmark.md
Normal file
186
docs/energy-cogs-benchmark.md
Normal file
|
|
@ -0,0 +1,186 @@
|
|||
# Energy COGS benchmark — arborist substrate vs bare model
|
||||
|
||||
**Status:** living report. First rig: qwen-27B on a dedicated RTX 4090.
|
||||
3090 + Hermes-3-8B and the reasoning variants follow on the same harness.
|
||||
**Ticket:** #000057 (watt bench) / #000060 (same-model substrate delta,
|
||||
cost axis). **Audience:** Dav1d de-novo review + maintainers.
|
||||
|
||||
## 1. Question
|
||||
|
||||
What does the arborist substrate cost in **GPU energy per token**,
|
||||
versus the same base model answering alone (no retrieval, no verifier)?
|
||||
Energy is the *cost* axis that pairs with the quality axis
|
||||
(`control_sweep` CG%); this report is cost only — it does **not** grade
|
||||
answers (energy is correctness-independent).
|
||||
|
||||
## 2. System under test
|
||||
|
||||
| component | value |
|
||||
|---|---|
|
||||
| GPU | NVIDIA RTX 4090 (24 GB), `ai.foxhop.net` |
|
||||
| model | `Qwen3.6-27B-UD-Q4_K_XL.gguf`, llama.cpp, `enable_thinking=false` |
|
||||
| substrate | arborist `query()` pipeline, **STOCK V.1** frozen config |
|
||||
| substrate hash | `claim_lattice` `036a4c79…`, `quote` `5b6ca4c5…` (`bench/stock_v1.py`) |
|
||||
| driver | orchestrator drives the LLM endpoint; power sampled on the box |
|
||||
| price | $0.33/kWh (configurable `--price-per-kwh`; the only non-measured input) |
|
||||
|
||||
**Two arms, same model (the substrate-delta design):**
|
||||
- **solo** — question only, neutral system prompt, no retrieval, no verifier.
|
||||
- **substrate-ON** — full `query()`: 4-route FTS5 retrieval → context →
|
||||
LLM synthesis → deterministic verifier. Two answer modes
|
||||
(`claim_lattice` JSON, `quote` prose) — STOCK V.1 is a two-cell family.
|
||||
|
||||
## 3. Why the card had to be isolated
|
||||
|
||||
The 4090 served live internet traffic through a Caddy proxy. A first
|
||||
pilot measured a **contaminated** result: the no-request idle baseline
|
||||
read **258 W / 52 % util** (organic generation mid-window), *higher* than
|
||||
the bench's own generation window — so the marginal calculation clamped
|
||||
to a meaningless **$0.00**. A single-slot GPU under shared load cannot be
|
||||
energy-profiled from a short snapshot.
|
||||
|
||||
Resolution: the endpoint is now a **dedicated benchmark rig** — public
|
||||
access blocked at the proxy (`@public → 403`; our trusted IP still
|
||||
reaches it). External traffic drained the card from 100 %/443 W to
|
||||
**0 %/~20 W within ~1 min**, confirmed isolated from a third-party IP
|
||||
(403). All numbers below are from the isolated card.
|
||||
|
||||
## 4. Instrument
|
||||
|
||||
Three properties a Dav1d review should check, each a defense against a
|
||||
specific way the number could be a lie:
|
||||
|
||||
### 4.1 Measured power states, never hardcoded
|
||||
A card occupies distinct states (idle, middle-idle = model resident
|
||||
between requests, generation burst) and they differ per
|
||||
card×model×inference-server. `watt_probe.classify_power_bands` splits the
|
||||
0.5 s samples by the **largest gap** in the sorted distribution — a
|
||||
data-derived threshold, not a fixed watt cut (unit-tested at two scales).
|
||||
Reporting `joules / window` as one number ("mean W") is rejected: it
|
||||
blends ~400 W bursts with sub-100 W gaps into a state the card never
|
||||
occupies.
|
||||
|
||||
### 4.2 Cache-miss guaranteed (no timing a SQLite lookup)
|
||||
If the substrate arm hit the providence cache it would return a stored
|
||||
answer with **no LLM call** — we'd be timing a lookup, not generation.
|
||||
Guard: the arm runs `burn_existing=True` (force-deletes any matching live
|
||||
providence row before inference) and asserts `cache_hits == 0`, emitting
|
||||
a loud warning + `real_inference=false` in the record otherwise. Verified
|
||||
0 hits on every run. (Solo has no cache path.)
|
||||
|
||||
### 4.3 GPU work is prefill + decode only — retrieval is CPU
|
||||
Retrieval (4-route FTS5 over SQLite) and verification run on the
|
||||
orchestrator CPU; **the GPU is untouched during them.** The GPU does two
|
||||
things per query: **prefill** (process the whole input prompt in
|
||||
parallel) and **decode** (generate output tokens autoregressively). The
|
||||
wall-clock *gross* window integral is **not** query-attributable — it
|
||||
charges the always-on model-resident floor for the seconds spent in
|
||||
CPU-side retrieval, energy that exists regardless of the query while the
|
||||
card is free for other traffic. Gross is reported for reference only.
|
||||
|
||||
### 4.4 Input and output costed separately (calibrated)
|
||||
Prefill (input) and decode (output) are physically different and have
|
||||
different J/token, so they **cannot share a per-token denominator** — a
|
||||
blended number is cheap-prefill-diluted in one direction and
|
||||
decode-inflated in the other. `bench/watt_calibrate.py` measures each by
|
||||
**slope calibration**: sweep prompt length at tiny `max_tokens` →
|
||||
prefill J/input-tok (fixed overhead cancels in the slope); fix a tiny
|
||||
prompt and sweep forced output length (`ignore_eos`) → decode
|
||||
J/output-tok. Prefill is kept cold (unique filler → `cached_tokens=0`),
|
||||
because at our scale there is no prefill/KV-cache reuse policy (we don't
|
||||
yet know which contexts are worth keeping hot) — so context is prefilled
|
||||
fresh every query.
|
||||
|
||||
## 5. Results (n=30, isolated 4090, qwen-nothink, $0.33/kWh)
|
||||
|
||||
### 5.1 Calibrated per-token energy
|
||||
```
|
||||
prefill (input) : 0.175 J/tok = $0.016 / M-input-tok (parallel, compute-bound)
|
||||
decode (output) : 6.157 J/tok = $0.564 / M-output-tok (autoregressive, bandwidth-bound)
|
||||
→ decode is 35× prefill per token
|
||||
```
|
||||
Clean linear fits over 3 points each (`cached_tokens=0` throughout).
|
||||
|
||||
### 5.2 Measured token mix (per query, n=30)
|
||||
The substrate prefills the retrieved context; solo prefills almost
|
||||
nothing. This — not a higher draw — is why the substrate costs more.
|
||||
|
||||
| arm / mode | input tok/q | output tok/q | input ×vs solo |
|
||||
|---|---|---|---|
|
||||
| solo | 52 | 14 | 1× |
|
||||
| substrate · claim_lattice | 6,617 | 91 | **127×** |
|
||||
| substrate · quote | 4,424 | 88 | 85× |
|
||||
|
||||
### 5.3 Per-query GPU energy, decomposed (calibration applied)
|
||||
`energy ≈ 0.175·input_tok + 6.157·output_tok`
|
||||
|
||||
| arm / mode | prefill J | decode J | **total J/q** | input share | **$/1k queries** |
|
||||
|---|---|---|---|---|---|
|
||||
| solo | 9 | 86 | **95** | 10 % | $0.009 |
|
||||
| substrate · claim_lattice | 1,159 | 560 | **1,719** | **67 %** | $0.158 |
|
||||
| substrate · quote | 775 | 542 | **1,317** | 59 % | $0.121 |
|
||||
|
||||
**Validation:** the calibration model predicts substrate-CL **1,719 J/q**
|
||||
vs the independently **measured 1,804 J/q** (~5 %) — the input/output
|
||||
decomposition reconstructs the directly-measured energy.
|
||||
|
||||
## 6. Interpretation
|
||||
|
||||
- **The substrate's cost is reading the evidence, not generating it.**
|
||||
~60–67 % of a substrate query's GPU energy is **prefill of the
|
||||
retrieved context**; solo's energy is ~90 % decode. Same model, same
|
||||
J/token rates — the substrate just feeds ~85–127× more input.
|
||||
- **Per output token both arms are identical** (same decode rate); the
|
||||
delta is entirely the input context the substrate must prefill.
|
||||
- **`claim_lattice` vs `quote`**: claim_lattice retrieves *more* context
|
||||
(6.6k vs 4.4k input tok) so its prefill is larger, but both decode a
|
||||
similar ~90 output tok. Net: claim_lattice 1,719 J/q, quote 1,317 J/q.
|
||||
- **Absolute GPU COGS is tiny** at $0.33/kWh — **$0.16 per 1,000
|
||||
substrate queries** (~18× solo's $0.009). The dominant arborist cost
|
||||
is CPU-side latency (retrieval/verify), not GPU electricity.
|
||||
- **Lever:** prefill/KV-cache reuse would cut the substrate's dominant
|
||||
(input) cost — but needs a policy for which contexts are worth keeping
|
||||
hot. Not viable at current scale; a real optimization at higher QPS.
|
||||
|
||||
## 7. Threats to validity
|
||||
|
||||
1. **Single fixture / model / card** — `qa_questions_stale_map` on one
|
||||
4090 + one Qwen-27B Q4 quant, llama.cpp. 3090 + Hermes + reasoning
|
||||
variants pending (§9). Token mixes (esp. retrieved-context size) are
|
||||
corpus- and policy-dependent.
|
||||
2. **Calibration is linear + endpoint-specific** — assumes
|
||||
`E ≈ prefill·in + decode·out` with constant per-token rates; batching
|
||||
/ KV-cache / different quants shift the coefficients. Re-calibrate per
|
||||
rig. Fit validated to ~5 % against measured here.
|
||||
3. **$/kWh** is a single site assumption (0.33); linear to rescale.
|
||||
4. **idle / serving-floor bands** are a measured proxy for the
|
||||
model-resident state; the per-query decomposition above relies on the
|
||||
*calibrated* rates, not the bands, and is the trustworthy number.
|
||||
5. **Tokens are now real API `usage`** (prompt + completion), not the
|
||||
former `len//4` estimate — this corrected an earlier draft that
|
||||
ignored input entirely.
|
||||
|
||||
## 8. Reproducibility
|
||||
|
||||
- Instruments: `bench/watt_bench.py`, `bench/watt_probe.py`,
|
||||
`bench/watt_calibrate.py` (commits `5b1cbee` → `892d9ed`); math + band
|
||||
split + slope unit-tested in `tests/test_watt_cogs.py` (14 tests).
|
||||
- Real token usage: `OpenAICompatibleClient.last_usage`.
|
||||
- Frozen substrate: `bench/stock_v1.py` (`assert_not_drifted`).
|
||||
- Raw per-cell power samples, window timestamps, and the calibration
|
||||
sweep persisted in `bench/qa_results/watt_4090_*.json` /
|
||||
`watt_calibrate_4090_*.json` for offline re-analysis.
|
||||
- Isolation: qwen closed to public at the proxy
|
||||
(`proxy.unturf.com` `ingress/Caddyfile`, commit `4b1fed1`) — dedicated
|
||||
benchmark rig.
|
||||
|
||||
## 9. Next
|
||||
|
||||
- **3090 + Hermes-3-8B** on the same harness + a fresh calibration
|
||||
(`--gpu-label 3090 --models hermes --arborist-ref hermes`; re-run
|
||||
`watt_calibrate --model-key hermes`).
|
||||
- **Reasoning variants** (qwen-think, hermes-reasoning) — phase 3; the
|
||||
decode coefficient will dominate (long reasoning traces = many output
|
||||
tokens at 6 J each).
|
||||
- **Prefill-cache study** — measure `cached_tokens` lift under repeated
|
||||
contexts to size the KV-reuse lever (§6).
|
||||
Loading…
Add table
Add a link
Reference in a new issue