engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(#000057): GPU COGS = generation only — retrieval/verify don't touch the card

Browse source 
fox 2026-05-21: 'we dont touch card during retrieval'. Retrieval + verify
are CPU/SQLite on the orchestrator; the GPU is idle (at the shared, always-
on model-resident floor) during them. So the substrate does NOT 'hold the
card' through its wall-clock — the gross window integral over-counts by
charging that always-on floor for the seconds we spend retrieving (energy
that exists regardless of the query; the card serves other traffic then).

Correct attribution: the GPU cost of a query is its GENERATION energy only
(the marginal — burst above the serving floor). Relabel: marginal is the
headline GPU COGS; gross is demoted to a reference 'window total, not
query-attributable'. Print + energy_cogs docstring + stock-v1-config doc
updated. No math change (marginal was already right) — this corrects the
narration. Measured qwen-nothink/4090: substrate GPU COGS is LOW
(claim_lattice $0.21/M-tok, quote $0.83) — its real overhead is latency
(CPU retrieval), not GPU watts.
This commit is contained in:
russell@unturf.com 2026-05-21 11:07:37 -04:00
parent 5b1cbeed80
commit 32aeb37086
No known key found for this signature in database
2 changed files with 41 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

34
bench/watt_bench.py
View file

@ -387,12 +387,17 @@ def energy_cogs(gpu_joules: float | None, window_s: float | None,
ready to answer.
* gen_draw_w — the actual generation draw (high band), with
``gen_duty`` = fraction of the window generating.
* gross COGS — ALL measured joules / tokens: the all-in cost,
amortized across throughput (state-agnostic, so
this number was always correct).
* marginal COGS — joules ABOVE the serving floor / tokens: what
one more request's generation actually adds.
Falls back to idle floor if no band split.
* marginal COGS — THE GPU COST OF A QUERY: joules ABOVE the
serving floor / tokens — the generation burst.
Retrieval/verify are CPU/SQLite (GPU untouched),
so this generation-only number is what the query
actually causes on the GPU. Falls back to idle
floor if no band split.
* gross — wall-clock window joules / tokens. NOT this
query's GPU cost: it charges the always-on
model-resident floor for the seconds we spend in
CPU-side retrieval (the card is free for other
traffic then). Kept for reference, not as COGS.
kWh = J / 3.6e6; ``$/1k-tok`` is the unit that compares to API pricing.
Only ``price_per_kwh`` is an operator input. See docs/stock-v1-config.md.
@ -668,12 +673,17 @@ def main() -> int:
+ ("" if b.get("bimodal") else " [unimodal — no gen state]"))
cg = cell["energy_cogs"]
if cg.get("available"):
print(f" COGS @${cg['price_per_kwh']}/kWh: "
f"gross ${cg['gross_usd_per_1k_tok']}/1k-tok"
+ (f" · marginal ${cg['marginal_usd_per_1k_tok']}/1k-tok "
f"(vs {cg['marginal_floor']}-floor {cg['marginal_floor_w']}W)"
if "marginal_usd_per_1k_tok" in cg else
" · marginal n/a (no floor)"))
# GPU COGS = generation energy ABOVE the always-on model-
# resident floor (the marginal). Retrieval/verify are CPU/
# SQLite — the GPU is untouched then, so the gross window
# integral OVER-counts (it charges the shared floor during our
# retrieval gap) and is NOT this query's GPU cost (fox).
print(f" GPU COGS @${cg['price_per_kwh']}/kWh: "
+ (f"${cg['marginal_usd_per_1k_tok']}/1k-tok generation "
f"(above {cg['marginal_floor']}-floor {cg['marginal_floor_w']}W)"
if "marginal_usd_per_1k_tok" in cg else "n/a")
+ f" · [window-total ${cg['gross_usd_per_1k_tok']}/1k-tok "
f"incl. shared idle during retrieval — NOT query-attributable]")
for mkey in models:
run_cell("solo", mkey)

30
docs/stock-v1-config.md
View file

@ -84,18 +84,26 @@ by `bench/jaggedness.py` (#000060).
## Cost axis — energy COGS (companion, #000057)
`bench/watt_bench.py` adds the cost side: GPU energy per question / per
completion-token, on each card. Energy COGS decomposes into the power
states the card actually occupies — idle, warm-idle (model resident,
waiting), and generation (request burst):
`bench/watt_bench.py` adds the cost side: GPU energy per completion-token,
on each card. The card occupies DISTINCT measured states — idle,
middle-idle (model resident, between requests), generation burst — split
data-derived by `watt_probe.classify_power_bands` (largest-gap, no
hardcoded watts). **Crucially: retrieval + verification are CPU/SQLite
work; the GPU is untouched during them** (fox 2026-05-21). So:
- **marginal** `(P_gen − P_warm_idle) · t_gen / tokens` — what one more
token actually costs;
- **gross / amortized** — all-in, including the warm-idle cost of
keeping the model hot, divided across throughput (the COGS-vs-
utilization curve);
- **dollar COGS** — `joules / 3.6e6 → kWh × $/kWh`, then `÷ tokens`
for `$/1k-tok` against API pricing.
- **marginal = THE GPU COGS of a query** — generation energy above the
serving floor, `÷ tokens`. This is what the query actually causes on
the GPU. Measured qwen-nothink/4090 (n=10, isolated): claim_lattice
**$0.21/M-tok**, quote **$0.83/M-tok**, bare solo ~$0.5–0.65/M-tok.
- **gross = wall-clock window total, NOT query-attributable.** It charges
the always-on model-resident floor for the seconds spent in CPU-side
retrieval — energy that exists regardless of the query (the card is
free for other traffic then). Reference only, never the COGS.
- **dollars** — `joules / 3.6e6 → kWh × $/kWh ÷ tokens` for `$/1k-tok`
against API pricing.
- **duty cycle** exposes the shape: a substrate query generates only
~9–25 % of its wall-clock; the rest is CPU retrieval. The substrate's
real overhead is **latency, not GPU watts.**
**Every power state is MEASURED per (card, model, inference-server) at
runtime — never hardcoded.** `P_warm_idle` comes from the no-request idle