feat(#000057): benchmark matrix doc (for David) + GPU wattage harness
Two deliverables for the cost/energy axis of the constraint optimizer. docs/benchmark-matrix.md — shareable spec of the control experiment: the question, fixture (386 office-holder Qs with corpus-vintage gold), the 3-model × 3-framing × 2-arm matrix (18 cells), the verdict vocabulary + two reads (accuracy vs grounding-fidelity), the deterministic code judge + its Opus calibration, the results-so-far table, and the NEW cost dimension (tokens / latency / GPU watts / joules-per-answer measured per GPU tier). Self-contained — readable cold by David. bench/watt_bench.py — GPU wattage harness. Samples nvidia-smi power.draw on the inference GPU while driving a small representative subset, reports mean/peak watts, trapezoid-integrated joules, joules-per-question, and joules-per-token. Tags the GPU (--gpu-label 3090|4090) so the optimizer can compare hardware tiers. Idle-baseline sampling separates load draw from idle. Does NOT grade (energy is independent of correctness); saves answers + per-question timing to JSONL for a later quality-per-joule pass via score_with_code_judge. Designed to run ON the GPU box (the orchestrator has no GPU; the 3090/4090 live on the inference boxes). Degrades gracefully when nvidia-smi is absent (energy fields null) so it is testable anywhere. Verified: PowerSampler graceful degradation + trapezoid integration (synthetic 100->200->200W over 2s = 350 J, exact). The headline cost finding the optimizer must weight: qwen-think reasoning = 1300-3300 tokens/answer vs qwen-nothink ~50-100 (20-50x), for a workload where arborist+qwen-nothink already lands 82% CG. The energy numbers will quantify whether reasoning's premium is ever justified — grounding-fidelity per joule, not per answer.
This commit is contained in:
parent
14cc9c891a
commit
5260161e6f
2 changed files with 443 additions and 0 deletions
139
docs/benchmark-matrix.md
Normal file
139
docs/benchmark-matrix.md
Normal file
|
|
@ -0,0 +1,139 @@
|
|||
# #000057 benchmark test matrix
|
||||
|
||||
Shareable spec of the control-experiment benchmark — what we measure,
|
||||
how, and the cells in the matrix. Companion to the rolling results in
|
||||
`docs/qa-modes-bench.md` (Addendum 8).
|
||||
|
||||
## The question
|
||||
|
||||
A small language model, asked "who is the president of France?", answers
|
||||
with the *present-day* office-holder ("Emmanuel Macron") even when the
|
||||
grounding corpus is a fixed ~2010 vintage (where the answer is Sarkozy).
|
||||
The model isn't lying — it's reciting its training-data present — but it
|
||||
fails to recognise it should defer to the supplied source, or abstain
|
||||
when it has none. The matrix isolates **what fixes that**: model scale,
|
||||
question framing, retrieval, or reasoning.
|
||||
|
||||
## Fixture
|
||||
|
||||
- **386 office-holder questions** (`bench/qa_questions_stale_map.json`),
|
||||
each paired with a **fixed corpus-vintage gold article** (~2010-2011,
|
||||
verified from the artefact). The gold is the single source of truth a
|
||||
grounded answer must match.
|
||||
- Adversarial by construction: the corpus answer (Sarkozy) differs from
|
||||
the model's training-data present (Macron), so a model reciting its
|
||||
prior is measurably *wrong against the source*.
|
||||
|
||||
## Axes
|
||||
|
||||
| axis | values | what it isolates |
|
||||
|------|--------|------------------|
|
||||
| **model** | `hermes` (Hermes-3-8B, vLLM) · `qwen-nothink` (Qwen3.6-27B, llama.cpp, reasoning off) · `qwen-think` (same, reasoning on) | scale (8B vs 27B) and reasoning |
|
||||
| **framing** | `plain` ("who is the president of France?") · `source_relative` ("According to the reference knowledge base, who is …") · `as_of_corpus` ("As of 2010, who was …") | whether prompt framing alone fixes the drift |
|
||||
| **arm** | `solo` (model alone — measures the parametric prior) · `arborist` (model + retrieval over the 2010 corpus, `answer_mode=claim_lattice`) | whether retrieval fixes it |
|
||||
|
||||
Full cross product = **3 models × 3 framings × 2 arms = 18 cells**, each
|
||||
over up to 386 items. (The `arborist` arm currently runs `hermes` and
|
||||
`qwen-nothink`; `qwen-think` is measurable but cost-prohibitive — see
|
||||
"Cost dimension".)
|
||||
|
||||
## Metrics
|
||||
|
||||
Every (question, answer, gold) triple gets one verdict from a closed
|
||||
vocabulary:
|
||||
|
||||
| verdict | meaning |
|
||||
|---------|---------|
|
||||
| **CORRECT_GROUNDED** (CG) | answer matches the gold source |
|
||||
| **WRONG** (W) | gold states something the answer contradicts |
|
||||
| **FABRICATED** (F) | answer asserts specifics absent from gold |
|
||||
| **ABSTAINED** (A) | answer honestly declines ("I don't have that") |
|
||||
| **JUDGE_ERROR** (JE) | judge could not classify deterministically |
|
||||
|
||||
Two reads of the same verdicts:
|
||||
|
||||
- **accuracy** — "did it say the corpus answer?" Penalises a model for
|
||||
knowing the *current* office-holder. Reported but flagged as
|
||||
misleading.
|
||||
- **grounding-fidelity** (the defensible read) — "did the model do the
|
||||
right conversational thing: answer when grounded, abstain when not?"
|
||||
W + F are failures (confident ungrounded assertion); A is success on
|
||||
the questions the model can't ground. **CG% and abstain% are the
|
||||
headline numbers.**
|
||||
|
||||
## The judge
|
||||
|
||||
Deterministic, no LLM, no quota: `bench/judge_code.py`. Pipeline (first
|
||||
hit decides): empty/no-gold guard → explicit-abstention regex →
|
||||
short-answer entity-grounding fast path → NLI contradiction (θ=0.85) →
|
||||
lexical verifier (quote/span/entity/paraphrase) → WRONG-vs-FABRICATED
|
||||
tie-break on subject-in-gold. Claim-lattice JSON envelopes are unwrapped
|
||||
to prose before grading so the `arborist` arm grades on equal terms with
|
||||
`solo`.
|
||||
|
||||
The judge was calibrated against an Opus (SOTA) reference judge on the
|
||||
records Opus could grade: agreement CG 13→47 %, WRONG 56→89 %, ABSTAINED
|
||||
80→95 %. The deterministic verdict is a *proxy* for grounding (lexical +
|
||||
NLI), not Opus-grade reading; JUDGE_ERROR residue is the natural input
|
||||
to a later LLM-batch pass. NLI runs on a GPU (currently the 4090 `ai`
|
||||
box); the rest is pure Python.
|
||||
|
||||
## Cost dimension (new — for the constraint optimizer)
|
||||
|
||||
The matrix has been scored on **quality** (CG%) but not yet on **cost**.
|
||||
Adding the energy axis so the constraint optimizer can trade grounding
|
||||
fidelity against power:
|
||||
|
||||
| cost metric | unit | source |
|
||||
|-------------|------|--------|
|
||||
| tokens / answer | completion tokens | API usage field |
|
||||
| latency / answer | seconds | wall-clock per cell |
|
||||
| **GPU power** | **watts (mean / peak)** | **`nvidia-smi power.draw`, sampled on the inference GPU during the cell** |
|
||||
| **energy / answer** | **joules** | **∫ power dt / questions** |
|
||||
|
||||
Measured per GPU (**3090** and **4090**) so the optimizer knows the
|
||||
energy cost of each config on each hardware tier. The wattage harness
|
||||
(`bench/watt_bench.py`) samples `nvidia-smi` on the GPU box while driving
|
||||
a small representative subset of the matrix.
|
||||
|
||||
**Why it matters:** reasoning chains are the headline cost finding —
|
||||
`qwen-think` spends 1300-3300 completion tokens/answer vs `qwen-nothink`'s
|
||||
~50-100 (**20-50× the token cost**), for a workload where
|
||||
`arborist + qwen-nothink` already lands **82 % CG**. The optimizer should
|
||||
weight that: grounding-fidelity *per joule*, not per answer.
|
||||
|
||||
## Results so far (CG%, all arms on the identical calibrated judge)
|
||||
|
||||
| arm / model | plain | source_relative | as_of_corpus |
|
||||
|-------------|-------|-----------------|--------------|
|
||||
| solo / hermes | 9 % | 5 % | 18 % |
|
||||
| solo / qwen-nothink | 7 % | 0 % | 50 % |
|
||||
| solo / qwen-think | 6 % | 5 % | 44 % |
|
||||
| arborist / hermes (n=40) | 60 % | 62 % | 25 % |
|
||||
| **arborist / qwen-nothink** | **82 %** | **65 %** | 50 % |
|
||||
|
||||
Headline findings: (1) **retrieval dominates** — no solo config
|
||||
approaches the arborist arms; (2) **reasoning doesn't improve raw
|
||||
correctness** (qwen-think/as_of 44 % vs nothink 50 %) and **breaks
|
||||
honest-abstention** under source_relative framing (nothink abstains
|
||||
98 %, think only 62 %); (3) **production answer: arborist + qwen-nothink,
|
||||
plain framing, reasoning off — 82 % CG.** The cost axis will confirm
|
||||
whether reasoning's energy premium is ever justified.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```
|
||||
# full matrix cell (one model, all framings, both arms)
|
||||
python -m bench.control_sweep --judge code --models qwen-nothink \
|
||||
--arborist-ref qwen-nothink --n 386 --arborist-n 386 --max-workers 1
|
||||
|
||||
# arborist-only (solo data already collected)
|
||||
python -m bench.control_sweep ... --skip-solo
|
||||
|
||||
# wattage subset (run ON the GPU box)
|
||||
python -m bench.watt_bench --models qwen-nothink --n 20 --gpu-label 4090
|
||||
```
|
||||
|
||||
Artifacts land in `bench/qa_results/`; scorecards via
|
||||
`bench/score_with_code_judge.py`; judge-vs-judge reconciliation via
|
||||
`bench/analyze_judge_disagreement.py`.
|
||||
Loading…
Add table
Add a link
Reference in a new issue