# #000057 benchmark test matrix

Shareable spec of the control-experiment benchmark — what we measure,
how, and the cells in the matrix. Companion to the rolling results in
`docs/qa-modes-bench.md` (Addendum 8).

## The question

A small language model, asked "who is the president of France?", answers
with the *present-day* office-holder ("Emmanuel Macron") even when the
grounding corpus is a fixed ~2010 vintage (where the answer is Sarkozy).
The model isn't lying — it's reciting its training-data present — but it
fails to recognise it should defer to the supplied source, or abstain
when it has none. The matrix isolates **what fixes that**: model scale,
question framing, retrieval, or reasoning.

## Fixture

- **386 office-holder questions** (`bench/qa_questions_stale_map.json`),
  each paired with a **fixed corpus-vintage gold article** (~2010-2011,
  verified from the artefact). The gold is the single source of truth a
  grounded answer must match.
- Adversarial by construction: the corpus answer (Sarkozy) differs from
  the model's training-data present (Macron), so a model reciting its
  prior is measurably *wrong against the source*.

## Axes

| axis | values | what it isolates |
|------|--------|------------------|
| **model** | `hermes` (Hermes-3-8B, vLLM) · `qwen-nothink` (Qwen3.6-27B, llama.cpp, reasoning off) · `qwen-think` (same, reasoning on) | scale (8B vs 27B) and reasoning |
| **framing** | `plain` ("who is the president of France?") · `source_relative` ("According to the reference knowledge base, who is …") · `as_of_corpus` ("As of 2010, who was …") | whether prompt framing alone fixes the drift |
| **arm** | `solo` (model alone — measures the parametric prior) · `arborist` (model + retrieval over the 2010 corpus, `answer_mode=claim_lattice`) | whether retrieval fixes it |

Full cross product = **3 models × 3 framings × 2 arms = 18 cells**, each
over up to 386 items. (The `arborist` arm currently runs `hermes` and
`qwen-nothink`; `qwen-think` is measurable but cost-prohibitive — see
"Cost dimension".)

## Metrics

Every (question, answer, gold) triple gets one verdict from a closed
vocabulary:

| verdict | meaning |
|---------|---------|
| **CORRECT_GROUNDED** (CG) | answer matches the gold source |
| **WRONG** (W) | gold states something the answer contradicts |
| **FABRICATED** (F) | answer asserts specifics absent from gold |
| **ABSTAINED** (A) | answer honestly declines ("I don't have that") |
| **JUDGE_ERROR** (JE) | judge could not classify deterministically |

Two reads of the same verdicts:

- **accuracy** — "did it say the corpus answer?" Penalises a model for
  knowing the *current* office-holder. Reported but flagged as
  misleading.
- **grounding-fidelity** (the defensible read) — "did the model do the
  right conversational thing: answer when grounded, abstain when not?"
  W + F are failures (confident ungrounded assertion); A is success on
  the questions the model can't ground. **CG% and abstain% are the
  headline numbers.**

## The judge

Deterministic, no LLM, no quota: `bench/judge_code.py`. Pipeline (first
hit decides): empty/no-gold guard → explicit-abstention regex →
short-answer entity-grounding fast path → NLI contradiction (θ=0.85) →
lexical verifier (quote/span/entity/paraphrase) → WRONG-vs-FABRICATED
tie-break on subject-in-gold. Claim-lattice JSON envelopes are unwrapped
to prose before grading so the `arborist` arm grades on equal terms with
`solo`.

The judge was calibrated against an Opus (SOTA) reference judge on the
records Opus could grade: agreement CG 13→47 %, WRONG 56→89 %, ABSTAINED
80→95 %. The deterministic verdict is a *proxy* for grounding (lexical +
NLI), not Opus-grade reading; JUDGE_ERROR residue is the natural input
to a later LLM-batch pass. NLI runs on a GPU (currently the 4090 `ai`
box); the rest is pure Python.

## Cost dimension — for the constraint optimizer (expanded goal)

The next-goal matrix scores **quality AND cost** across the serving
stack, not just the model. Every cell is a deployable configuration; the
optimizer trades grounding-fidelity against energy to pick the config to
ship.

**Expanded axes (the full cost/quality matrix):**

| axis | values |
|------|--------|
| model | qwen-27B · hermes-8B |
| **engine** | **llama.cpp · vLLM** (where the model/engine combo is supported) |
| **GPU** | **RTX 3090 · RTX 4090** |
| arm | solo · arborist (retrieval) |

Cross product = 2 model × 2 engine × 2 GPU × 2 arm = up to **16
quality+cost cells**. Each cell records:

| metric | unit | source |
|--------|------|--------|
| quality (CG% / abstain%) | rate | code judge |
| tokens / answer | completion tokens | API usage |
| latency / answer | seconds | wall-clock |
| **GPU energy** | **watts mean/peak · joules · J/answer · J/token** | `nvidia-smi power.draw` on the inference GPU |
| **CPU energy** | **watts mean · joules · J/answer** | Intel RAPL `energy_uj` (package sum) on the box |

`bench/watt_bench.py` samples both GPU (`nvidia-smi`) and CPU (RAPL) on
the inference box while driving the subset. RAPL `energy_uj` is root-only
by default (PLATYPUS / CVE-2020-8694 mitigation) — pass
`--cpu-energy-cmd 'sudo cat {path}'` with a sudo rule, or relax the
sysfs perm, to capture CPU watts; GPU watts need no special perm.

**Serving-stack inventory (recon 2026-05-20):**

| box | host | GPU | engine present | model present |
|-----|------|-----|----------------|---------------|
| 4090 | `ai.foxhop.net:18888` | RTX 4090, i9-14900K (32t) | llama.cpp | qwen-27B gguf |
| 3090 | `3090-ai.foxhop.net:18888` | RTX 3090, i9-12900K (24t) | vLLM (venv) | hermes-8B |

To fill the matrix, the missing serving configs must be stood up:
vLLM+qwen, llama.cpp+hermes (gguf), and each model on the other GPU.
That's an infra buildout (install engines, fetch models, manage the
single-slot live endpoints), to be captured as **salt states in
`~/git/foxhop-states`** (where `ai.foxhop.net`'s `llama-qwen` state
already lives; the 3090 box is not yet salt-managed).

**Known-good serving invocations (captured 2026-05-20 — the basis for
the salt states):**

- **qwen / llama.cpp / 4090** (live, salt `llama-qwen/init.sls`):
  ```
  /mnt/data/llm/llama.cpp/build/bin/llama-server \
    -m /mnt/data/llm/cache/Qwen3.6-27B-UD-Q4_K_XL.gguf \
    --host 0.0.0.0 --port 18888 -ngl 99 -c 65536 -fa on \
    --cache-type-k q8_0 --cache-type-v q8_0 -np 1 \
    --spec-type draft-mtp --spec-draft-n-max 6 \
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.5
  ```
- **hermes / vLLM / 3090** (live, MANUAL tmux — not yet salt; vLLM
  0.10.1.1 on python3.12):
  ```
  python -m vllm.entrypoints.openai.api_server \
    --model adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic \
    --host 0.0.0.0 --port 18888 --max-model-len 82000 \
    --enable-auto-tool-choice --tool-call-parser hermes
  ```

Per the existing convention, salt manages the **systemd unit**; the big
artifacts (engine binaries, model files / gguf) live on `/mnt/data` and
are provisioned manually, documented as state prereqs. Migrating the
live 3090 hermes from its manual tmux session to a salt systemd unit
must keep ≥1 hermes online (qwen is the expendable/exploratory
workload) — a careful cutover, not a blind highstate.

**Why cost matters most here:** reasoning chains are the headline cost
finding — `qwen-think` spends 1300-3300 completion tokens/answer vs
`qwen-nothink`'s ~50-100 (**20-50× the token cost**), for a workload
where `arborist + qwen-nothink` already lands **82 % CG**. And hermes-8B
on a 3090 idles at ~127 W vs qwen-27B on a 4090 at ~20 W — idle floor,
model size, engine, and GPU tier all move the joules-per-grounded-answer
the optimizer cares about. The metric is **grounding-fidelity per joule**,
not per answer.

## Results so far (CG%, all arms on the identical calibrated judge)

| arm / model | plain | source_relative | as_of_corpus |
|-------------|-------|-----------------|--------------|
| solo / hermes | 9 % | 5 % | 18 % |
| solo / qwen-nothink | 7 % | 0 % | 50 % |
| solo / qwen-think | 6 % | 5 % | 44 % |
| arborist / hermes (n=40) | 60 % | 62 % | 25 % |
| **arborist / qwen-nothink** | **82 %** | **65 %** | 50 % |

Headline findings: (1) **retrieval dominates** — no solo config
approaches the arborist arms; (2) **reasoning doesn't improve raw
correctness** (qwen-think/as_of 44 % vs nothink 50 %) and **breaks
honest-abstention** under source_relative framing (nothink abstains
98 %, think only 62 %); (3) **production answer: arborist + qwen-nothink,
plain framing, reasoning off — 82 % CG.** The cost axis will confirm
whether reasoning's energy premium is ever justified.

## Reproduce

```
# full matrix cell (one model, all framings, both arms)
python -m bench.control_sweep --judge code --models qwen-nothink \
    --arborist-ref qwen-nothink --n 386 --arborist-n 386 --max-workers 1

# arborist-only (solo data already collected)
python -m bench.control_sweep ... --skip-solo

# wattage subset (run ON the GPU box)
python -m bench.watt_bench --models qwen-nothink --n 20 --gpu-label 4090
```

Artifacts land in `bench/qa_results/`; scorecards via
`bench/score_with_code_judge.py`; judge-vs-judge reconciliation via
`bench/analyze_judge_disagreement.py`.