diff --git a/docs/energy-cogs-benchmark.md b/docs/energy-cogs-benchmark.md new file mode 100644 index 0000000..473ebd1 --- /dev/null +++ b/docs/energy-cogs-benchmark.md @@ -0,0 +1,186 @@ +# Energy COGS benchmark — arborist substrate vs bare model + +**Status:** living report. First rig: qwen-27B on a dedicated RTX 4090. +3090 + Hermes-3-8B and the reasoning variants follow on the same harness. +**Ticket:** #000057 (watt bench) / #000060 (same-model substrate delta, +cost axis). **Audience:** Dav1d de-novo review + maintainers. + +## 1. Question + +What does the arborist substrate cost in **GPU energy per token**, +versus the same base model answering alone (no retrieval, no verifier)? +Energy is the *cost* axis that pairs with the quality axis +(`control_sweep` CG%); this report is cost only — it does **not** grade +answers (energy is correctness-independent). + +## 2. System under test + +| component | value | +|---|---| +| GPU | NVIDIA RTX 4090 (24 GB), `ai.foxhop.net` | +| model | `Qwen3.6-27B-UD-Q4_K_XL.gguf`, llama.cpp, `enable_thinking=false` | +| substrate | arborist `query()` pipeline, **STOCK V.1** frozen config | +| substrate hash | `claim_lattice` `036a4c79…`, `quote` `5b6ca4c5…` (`bench/stock_v1.py`) | +| driver | orchestrator drives the LLM endpoint; power sampled on the box | +| price | $0.33/kWh (configurable `--price-per-kwh`; the only non-measured input) | + +**Two arms, same model (the substrate-delta design):** +- **solo** — question only, neutral system prompt, no retrieval, no verifier. +- **substrate-ON** — full `query()`: 4-route FTS5 retrieval → context → + LLM synthesis → deterministic verifier. Two answer modes + (`claim_lattice` JSON, `quote` prose) — STOCK V.1 is a two-cell family. + +## 3. Why the card had to be isolated + +The 4090 served live internet traffic through a Caddy proxy. A first +pilot measured a **contaminated** result: the no-request idle baseline +read **258 W / 52 % util** (organic generation mid-window), *higher* than +the bench's own generation window — so the marginal calculation clamped +to a meaningless **$0.00**. A single-slot GPU under shared load cannot be +energy-profiled from a short snapshot. + +Resolution: the endpoint is now a **dedicated benchmark rig** — public +access blocked at the proxy (`@public → 403`; our trusted IP still +reaches it). External traffic drained the card from 100 %/443 W to +**0 %/~20 W within ~1 min**, confirmed isolated from a third-party IP +(403). All numbers below are from the isolated card. + +## 4. Instrument + +Three properties a Dav1d review should check, each a defense against a +specific way the number could be a lie: + +### 4.1 Measured power states, never hardcoded +A card occupies distinct states (idle, middle-idle = model resident +between requests, generation burst) and they differ per +card×model×inference-server. `watt_probe.classify_power_bands` splits the +0.5 s samples by the **largest gap** in the sorted distribution — a +data-derived threshold, not a fixed watt cut (unit-tested at two scales). +Reporting `joules / window` as one number ("mean W") is rejected: it +blends ~400 W bursts with sub-100 W gaps into a state the card never +occupies. + +### 4.2 Cache-miss guaranteed (no timing a SQLite lookup) +If the substrate arm hit the providence cache it would return a stored +answer with **no LLM call** — we'd be timing a lookup, not generation. +Guard: the arm runs `burn_existing=True` (force-deletes any matching live +providence row before inference) and asserts `cache_hits == 0`, emitting +a loud warning + `real_inference=false` in the record otherwise. Verified +0 hits on every run. (Solo has no cache path.) + +### 4.3 GPU work is prefill + decode only — retrieval is CPU +Retrieval (4-route FTS5 over SQLite) and verification run on the +orchestrator CPU; **the GPU is untouched during them.** The GPU does two +things per query: **prefill** (process the whole input prompt in +parallel) and **decode** (generate output tokens autoregressively). The +wall-clock *gross* window integral is **not** query-attributable — it +charges the always-on model-resident floor for the seconds spent in +CPU-side retrieval, energy that exists regardless of the query while the +card is free for other traffic. Gross is reported for reference only. + +### 4.4 Input and output costed separately (calibrated) +Prefill (input) and decode (output) are physically different and have +different J/token, so they **cannot share a per-token denominator** — a +blended number is cheap-prefill-diluted in one direction and +decode-inflated in the other. `bench/watt_calibrate.py` measures each by +**slope calibration**: sweep prompt length at tiny `max_tokens` → +prefill J/input-tok (fixed overhead cancels in the slope); fix a tiny +prompt and sweep forced output length (`ignore_eos`) → decode +J/output-tok. Prefill is kept cold (unique filler → `cached_tokens=0`), +because at our scale there is no prefill/KV-cache reuse policy (we don't +yet know which contexts are worth keeping hot) — so context is prefilled +fresh every query. + +## 5. Results (n=30, isolated 4090, qwen-nothink, $0.33/kWh) + +### 5.1 Calibrated per-token energy +``` +prefill (input) : 0.175 J/tok = $0.016 / M-input-tok (parallel, compute-bound) +decode (output) : 6.157 J/tok = $0.564 / M-output-tok (autoregressive, bandwidth-bound) +→ decode is 35× prefill per token +``` +Clean linear fits over 3 points each (`cached_tokens=0` throughout). + +### 5.2 Measured token mix (per query, n=30) +The substrate prefills the retrieved context; solo prefills almost +nothing. This — not a higher draw — is why the substrate costs more. + +| arm / mode | input tok/q | output tok/q | input ×vs solo | +|---|---|---|---| +| solo | 52 | 14 | 1× | +| substrate · claim_lattice | 6,617 | 91 | **127×** | +| substrate · quote | 4,424 | 88 | 85× | + +### 5.3 Per-query GPU energy, decomposed (calibration applied) +`energy ≈ 0.175·input_tok + 6.157·output_tok` + +| arm / mode | prefill J | decode J | **total J/q** | input share | **$/1k queries** | +|---|---|---|---|---|---| +| solo | 9 | 86 | **95** | 10 % | $0.009 | +| substrate · claim_lattice | 1,159 | 560 | **1,719** | **67 %** | $0.158 | +| substrate · quote | 775 | 542 | **1,317** | 59 % | $0.121 | + +**Validation:** the calibration model predicts substrate-CL **1,719 J/q** +vs the independently **measured 1,804 J/q** (~5 %) — the input/output +decomposition reconstructs the directly-measured energy. + +## 6. Interpretation + +- **The substrate's cost is reading the evidence, not generating it.** + ~60–67 % of a substrate query's GPU energy is **prefill of the + retrieved context**; solo's energy is ~90 % decode. Same model, same + J/token rates — the substrate just feeds ~85–127× more input. +- **Per output token both arms are identical** (same decode rate); the + delta is entirely the input context the substrate must prefill. +- **`claim_lattice` vs `quote`**: claim_lattice retrieves *more* context + (6.6k vs 4.4k input tok) so its prefill is larger, but both decode a + similar ~90 output tok. Net: claim_lattice 1,719 J/q, quote 1,317 J/q. +- **Absolute GPU COGS is tiny** at $0.33/kWh — **$0.16 per 1,000 + substrate queries** (~18× solo's $0.009). The dominant arborist cost + is CPU-side latency (retrieval/verify), not GPU electricity. +- **Lever:** prefill/KV-cache reuse would cut the substrate's dominant + (input) cost — but needs a policy for which contexts are worth keeping + hot. Not viable at current scale; a real optimization at higher QPS. + +## 7. Threats to validity + +1. **Single fixture / model / card** — `qa_questions_stale_map` on one + 4090 + one Qwen-27B Q4 quant, llama.cpp. 3090 + Hermes + reasoning + variants pending (§9). Token mixes (esp. retrieved-context size) are + corpus- and policy-dependent. +2. **Calibration is linear + endpoint-specific** — assumes + `E ≈ prefill·in + decode·out` with constant per-token rates; batching + / KV-cache / different quants shift the coefficients. Re-calibrate per + rig. Fit validated to ~5 % against measured here. +3. **$/kWh** is a single site assumption (0.33); linear to rescale. +4. **idle / serving-floor bands** are a measured proxy for the + model-resident state; the per-query decomposition above relies on the + *calibrated* rates, not the bands, and is the trustworthy number. +5. **Tokens are now real API `usage`** (prompt + completion), not the + former `len//4` estimate — this corrected an earlier draft that + ignored input entirely. + +## 8. Reproducibility + +- Instruments: `bench/watt_bench.py`, `bench/watt_probe.py`, + `bench/watt_calibrate.py` (commits `5b1cbee` → `892d9ed`); math + band + split + slope unit-tested in `tests/test_watt_cogs.py` (14 tests). +- Real token usage: `OpenAICompatibleClient.last_usage`. +- Frozen substrate: `bench/stock_v1.py` (`assert_not_drifted`). +- Raw per-cell power samples, window timestamps, and the calibration + sweep persisted in `bench/qa_results/watt_4090_*.json` / + `watt_calibrate_4090_*.json` for offline re-analysis. +- Isolation: qwen closed to public at the proxy + (`proxy.unturf.com` `ingress/Caddyfile`, commit `4b1fed1`) — dedicated + benchmark rig. + +## 9. Next + +- **3090 + Hermes-3-8B** on the same harness + a fresh calibration + (`--gpu-label 3090 --models hermes --arborist-ref hermes`; re-run + `watt_calibrate --model-key hermes`). +- **Reasoning variants** (qwen-think, hermes-reasoning) — phase 3; the + decode coefficient will dominate (long reasoning traces = many output + tokens at 6 J each). +- **Prefill-cache study** — measure `cached_tokens` lift under repeated + contexts to size the KV-reuse lever (§6).