engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(#000057): CPU wattage (RAPL) in watt_bench + expanded cost/quality matrix doc

Browse source 
Toward fox's next goal: score the full serving stack on quality AND
cost — {qwen, hermes} × {llama.cpp, vLLM} × {3090, 4090} × {solo,
arborist}, measuring CG% + GPU watts + CPU watts + joules/answer per
cell.

watt_bench.py — adds CpuSampler (Intel RAPL package energy via
/sys/class/powercap/intel-rapl:*/energy_uj). RAPL exposes a cumulative
microjoule counter, so energy-over-window is an end-minus-start diff
(handles wrap) — more accurate than integrating instantaneous power.
Sums multi-package. energy_uj is root-only by default (PLATYPUS /
CVE-2020-8694), so it degrades to available=False when locked;
--cpu-energy-cmd 'sudo cat {path}' supplies a privileged reader when a
sudo rule exists. Each cell now reports gpu/cpu/total joules-per-
question + gpu joules-per-token; the report records cpu_rapl_available.
Verified: graceful degradation when locked; RAPL diff math (1->4 MJ uJ
= 3.0 J, exact).

benchmark-matrix.md — expands the cost section to the full 16-cell
(model × engine × GPU × arm) design, the per-cell metric set (quality +
GPU + CPU energy), the serving-stack inventory from 2026-05-20 recon
(4090=qwen/llama.cpp, 3090=hermes/vLLM — each box has one engine + one
model today), and the buildout gap (vLLM+qwen, llama.cpp+hermes, cross-
GPU models). Notes idle-floor asymmetry (hermes/3090 ~127W vs
qwen/4090 ~20W) as a real optimizer input.

Harness is ready; the serving-config buildout + RAPL perm grant are the
remaining (ops, fox-directed) prerequisites to run the full matrix.
This commit is contained in:
russell@unturf.com 2026-05-20 14:45:49 -04:00
parent 5260161e6f
commit ab8df76792
No known key found for this signature in database
2 changed files with 173 additions and 28 deletions
Download patch file Download diff file Expand all files Collapse all files

68
docs/benchmark-matrix.md
View file

@ -78,29 +78,59 @@ NLI), not Opus-grade reading; JUDGE_ERROR residue is the natural input
to a later LLM-batch pass. NLI runs on a GPU (currently the 4090 `ai`
box); the rest is pure Python.
## Cost dimension (new — for the constraint optimizer)
## Cost dimension — for the constraint optimizer (expanded goal)
The matrix has been scored on **quality** (CG%) but not yet on **cost**.
Adding the energy axis so the constraint optimizer can trade grounding
fidelity against power:
The next-goal matrix scores **quality AND cost** across the serving
stack, not just the model. Every cell is a deployable configuration; the
optimizer trades grounding-fidelity against energy to pick the config to
ship.
| cost metric | unit | source |
|-------------|------|--------|
| tokens / answer | completion tokens | API usage field |
| latency / answer | seconds | wall-clock per cell |
| **GPU power** | **watts (mean / peak)** | **`nvidia-smi power.draw`, sampled on the inference GPU during the cell** |
| **energy / answer** | **joules** | **∫ power dt / questions** |
**Expanded axes (the full cost/quality matrix):**
Measured per GPU (**3090** and **4090**) so the optimizer knows the
energy cost of each config on each hardware tier. The wattage harness
(`bench/watt_bench.py`) samples `nvidia-smi` on the GPU box while driving
a small representative subset of the matrix.
| axis | values |
|------|--------|
| model | qwen-27B · hermes-8B |
| **engine** | **llama.cpp · vLLM** (where the model/engine combo is supported) |
| **GPU** | **RTX 3090 · RTX 4090** |
| arm | solo · arborist (retrieval) |
**Why it matters:** reasoning chains are the headline cost finding —
`qwen-think` spends 1300-3300 completion tokens/answer vs `qwen-nothink`'s
~50-100 (**20-50× the token cost**), for a workload where
`arborist + qwen-nothink` already lands **82 % CG**. The optimizer should
weight that: grounding-fidelity *per joule*, not per answer.
Cross product = 2 model × 2 engine × 2 GPU × 2 arm = up to **16
quality+cost cells**. Each cell records:
| metric | unit | source |
|--------|------|--------|
| quality (CG% / abstain%) | rate | code judge |
| tokens / answer | completion tokens | API usage |
| latency / answer | seconds | wall-clock |
| **GPU energy** | **watts mean/peak · joules · J/answer · J/token** | `nvidia-smi power.draw` on the inference GPU |
| **CPU energy** | **watts mean · joules · J/answer** | Intel RAPL `energy_uj` (package sum) on the box |
`bench/watt_bench.py` samples both GPU (`nvidia-smi`) and CPU (RAPL) on
the inference box while driving the subset. RAPL `energy_uj` is root-only
by default (PLATYPUS / CVE-2020-8694 mitigation) — pass
`--cpu-energy-cmd 'sudo cat {path}'` with a sudo rule, or relax the
sysfs perm, to capture CPU watts; GPU watts need no special perm.
**Serving-stack inventory (recon 2026-05-20):**
| box | host | GPU | engine present | model present |
|-----|------|-----|----------------|---------------|
| 4090 | `ai.foxhop.net:18888` | RTX 4090, i9-14900K (32t) | llama.cpp | qwen-27B gguf |
| 3090 | `3090-ai.foxhop.net:18888` | RTX 3090, i9-12900K (24t) | vLLM (venv) | hermes-8B |
To fill the matrix, the missing serving configs must be stood up:
vLLM+qwen, llama.cpp+hermes (gguf), and each model on the other GPU.
That's an infra buildout (install engines, fetch models, manage the
single-slot live endpoints), tracked separately from the harness.
**Why cost matters most here:** reasoning chains are the headline cost
finding — `qwen-think` spends 1300-3300 completion tokens/answer vs
`qwen-nothink`'s ~50-100 (**20-50× the token cost**), for a workload
where `arborist + qwen-nothink` already lands **82 % CG**. And hermes-8B
on a 3090 idles at ~127 W vs qwen-27B on a 4090 at ~20 W — idle floor,
model size, engine, and GPU tier all move the joules-per-grounded-answer
the optimizer cares about. The metric is **grounding-fidelity per joule**,
not per answer.
## Results so far (CG%, all arms on the identical calibrated judge)