russell@unturf.com
69a9642296
The exact llama.cpp (qwen/4090) and vLLM (hermes/3090) launch commands, recon'd 2026-05-20, so the foxhop-states salt states can be written accurately rather than guessed. Notes the convention (salt manages the systemd unit; engine binaries + model artifacts stay manual on /mnt/data as documented prereqs) and the live-hermes cutover constraint (keep >=1 hermes online; qwen is expendable).
199 lines
9.1 KiB
Markdown
199 lines
9.1 KiB
Markdown
# #000057 benchmark test matrix
|
||
|
||
Shareable spec of the control-experiment benchmark — what we measure,
|
||
how, and the cells in the matrix. Companion to the rolling results in
|
||
`docs/qa-modes-bench.md` (Addendum 8).
|
||
|
||
## The question
|
||
|
||
A small language model, asked "who is the president of France?", answers
|
||
with the *present-day* office-holder ("Emmanuel Macron") even when the
|
||
grounding corpus is a fixed ~2010 vintage (where the answer is Sarkozy).
|
||
The model isn't lying — it's reciting its training-data present — but it
|
||
fails to recognise it should defer to the supplied source, or abstain
|
||
when it has none. The matrix isolates **what fixes that**: model scale,
|
||
question framing, retrieval, or reasoning.
|
||
|
||
## Fixture
|
||
|
||
- **386 office-holder questions** (`bench/qa_questions_stale_map.json`),
|
||
each paired with a **fixed corpus-vintage gold article** (~2010-2011,
|
||
verified from the artefact). The gold is the single source of truth a
|
||
grounded answer must match.
|
||
- Adversarial by construction: the corpus answer (Sarkozy) differs from
|
||
the model's training-data present (Macron), so a model reciting its
|
||
prior is measurably *wrong against the source*.
|
||
|
||
## Axes
|
||
|
||
| axis | values | what it isolates |
|
||
|------|--------|------------------|
|
||
| **model** | `hermes` (Hermes-3-8B, vLLM) · `qwen-nothink` (Qwen3.6-27B, llama.cpp, reasoning off) · `qwen-think` (same, reasoning on) | scale (8B vs 27B) and reasoning |
|
||
| **framing** | `plain` ("who is the president of France?") · `source_relative` ("According to the reference knowledge base, who is …") · `as_of_corpus` ("As of 2010, who was …") | whether prompt framing alone fixes the drift |
|
||
| **arm** | `solo` (model alone — measures the parametric prior) · `arborist` (model + retrieval over the 2010 corpus, `answer_mode=claim_lattice`) | whether retrieval fixes it |
|
||
|
||
Full cross product = **3 models × 3 framings × 2 arms = 18 cells**, each
|
||
over up to 386 items. (The `arborist` arm currently runs `hermes` and
|
||
`qwen-nothink`; `qwen-think` is measurable but cost-prohibitive — see
|
||
"Cost dimension".)
|
||
|
||
## Metrics
|
||
|
||
Every (question, answer, gold) triple gets one verdict from a closed
|
||
vocabulary:
|
||
|
||
| verdict | meaning |
|
||
|---------|---------|
|
||
| **CORRECT_GROUNDED** (CG) | answer matches the gold source |
|
||
| **WRONG** (W) | gold states something the answer contradicts |
|
||
| **FABRICATED** (F) | answer asserts specifics absent from gold |
|
||
| **ABSTAINED** (A) | answer honestly declines ("I don't have that") |
|
||
| **JUDGE_ERROR** (JE) | judge could not classify deterministically |
|
||
|
||
Two reads of the same verdicts:
|
||
|
||
- **accuracy** — "did it say the corpus answer?" Penalises a model for
|
||
knowing the *current* office-holder. Reported but flagged as
|
||
misleading.
|
||
- **grounding-fidelity** (the defensible read) — "did the model do the
|
||
right conversational thing: answer when grounded, abstain when not?"
|
||
W + F are failures (confident ungrounded assertion); A is success on
|
||
the questions the model can't ground. **CG% and abstain% are the
|
||
headline numbers.**
|
||
|
||
## The judge
|
||
|
||
Deterministic, no LLM, no quota: `bench/judge_code.py`. Pipeline (first
|
||
hit decides): empty/no-gold guard → explicit-abstention regex →
|
||
short-answer entity-grounding fast path → NLI contradiction (θ=0.85) →
|
||
lexical verifier (quote/span/entity/paraphrase) → WRONG-vs-FABRICATED
|
||
tie-break on subject-in-gold. Claim-lattice JSON envelopes are unwrapped
|
||
to prose before grading so the `arborist` arm grades on equal terms with
|
||
`solo`.
|
||
|
||
The judge was calibrated against an Opus (SOTA) reference judge on the
|
||
records Opus could grade: agreement CG 13→47 %, WRONG 56→89 %, ABSTAINED
|
||
80→95 %. The deterministic verdict is a *proxy* for grounding (lexical +
|
||
NLI), not Opus-grade reading; JUDGE_ERROR residue is the natural input
|
||
to a later LLM-batch pass. NLI runs on a GPU (currently the 4090 `ai`
|
||
box); the rest is pure Python.
|
||
|
||
## Cost dimension — for the constraint optimizer (expanded goal)
|
||
|
||
The next-goal matrix scores **quality AND cost** across the serving
|
||
stack, not just the model. Every cell is a deployable configuration; the
|
||
optimizer trades grounding-fidelity against energy to pick the config to
|
||
ship.
|
||
|
||
**Expanded axes (the full cost/quality matrix):**
|
||
|
||
| axis | values |
|
||
|------|--------|
|
||
| model | qwen-27B · hermes-8B |
|
||
| **engine** | **llama.cpp · vLLM** (where the model/engine combo is supported) |
|
||
| **GPU** | **RTX 3090 · RTX 4090** |
|
||
| arm | solo · arborist (retrieval) |
|
||
|
||
Cross product = 2 model × 2 engine × 2 GPU × 2 arm = up to **16
|
||
quality+cost cells**. Each cell records:
|
||
|
||
| metric | unit | source |
|
||
|--------|------|--------|
|
||
| quality (CG% / abstain%) | rate | code judge |
|
||
| tokens / answer | completion tokens | API usage |
|
||
| latency / answer | seconds | wall-clock |
|
||
| **GPU energy** | **watts mean/peak · joules · J/answer · J/token** | `nvidia-smi power.draw` on the inference GPU |
|
||
| **CPU energy** | **watts mean · joules · J/answer** | Intel RAPL `energy_uj` (package sum) on the box |
|
||
|
||
`bench/watt_bench.py` samples both GPU (`nvidia-smi`) and CPU (RAPL) on
|
||
the inference box while driving the subset. RAPL `energy_uj` is root-only
|
||
by default (PLATYPUS / CVE-2020-8694 mitigation) — pass
|
||
`--cpu-energy-cmd 'sudo cat {path}'` with a sudo rule, or relax the
|
||
sysfs perm, to capture CPU watts; GPU watts need no special perm.
|
||
|
||
**Serving-stack inventory (recon 2026-05-20):**
|
||
|
||
| box | host | GPU | engine present | model present |
|
||
|-----|------|-----|----------------|---------------|
|
||
| 4090 | `ai.foxhop.net:18888` | RTX 4090, i9-14900K (32t) | llama.cpp | qwen-27B gguf |
|
||
| 3090 | `3090-ai.foxhop.net:18888` | RTX 3090, i9-12900K (24t) | vLLM (venv) | hermes-8B |
|
||
|
||
To fill the matrix, the missing serving configs must be stood up:
|
||
vLLM+qwen, llama.cpp+hermes (gguf), and each model on the other GPU.
|
||
That's an infra buildout (install engines, fetch models, manage the
|
||
single-slot live endpoints), to be captured as **salt states in
|
||
`~/git/foxhop-states`** (where `ai.foxhop.net`'s `llama-qwen` state
|
||
already lives; the 3090 box is not yet salt-managed).
|
||
|
||
**Known-good serving invocations (captured 2026-05-20 — the basis for
|
||
the salt states):**
|
||
|
||
- **qwen / llama.cpp / 4090** (live, salt `llama-qwen/init.sls`):
|
||
```
|
||
/mnt/data/llm/llama.cpp/build/bin/llama-server \
|
||
-m /mnt/data/llm/cache/Qwen3.6-27B-UD-Q4_K_XL.gguf \
|
||
--host 0.0.0.0 --port 18888 -ngl 99 -c 65536 -fa on \
|
||
--cache-type-k q8_0 --cache-type-v q8_0 -np 1 \
|
||
--spec-type draft-mtp --spec-draft-n-max 6 \
|
||
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.5
|
||
```
|
||
- **hermes / vLLM / 3090** (live, MANUAL tmux — not yet salt; vLLM
|
||
0.10.1.1 on python3.12):
|
||
```
|
||
python -m vllm.entrypoints.openai.api_server \
|
||
--model adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic \
|
||
--host 0.0.0.0 --port 18888 --max-model-len 82000 \
|
||
--enable-auto-tool-choice --tool-call-parser hermes
|
||
```
|
||
|
||
Per the existing convention, salt manages the **systemd unit**; the big
|
||
artifacts (engine binaries, model files / gguf) live on `/mnt/data` and
|
||
are provisioned manually, documented as state prereqs. Migrating the
|
||
live 3090 hermes from its manual tmux session to a salt systemd unit
|
||
must keep ≥1 hermes online (qwen is the expendable/exploratory
|
||
workload) — a careful cutover, not a blind highstate.
|
||
|
||
**Why cost matters most here:** reasoning chains are the headline cost
|
||
finding — `qwen-think` spends 1300-3300 completion tokens/answer vs
|
||
`qwen-nothink`'s ~50-100 (**20-50× the token cost**), for a workload
|
||
where `arborist + qwen-nothink` already lands **82 % CG**. And hermes-8B
|
||
on a 3090 idles at ~127 W vs qwen-27B on a 4090 at ~20 W — idle floor,
|
||
model size, engine, and GPU tier all move the joules-per-grounded-answer
|
||
the optimizer cares about. The metric is **grounding-fidelity per joule**,
|
||
not per answer.
|
||
|
||
## Results so far (CG%, all arms on the identical calibrated judge)
|
||
|
||
| arm / model | plain | source_relative | as_of_corpus |
|
||
|-------------|-------|-----------------|--------------|
|
||
| solo / hermes | 9 % | 5 % | 18 % |
|
||
| solo / qwen-nothink | 7 % | 0 % | 50 % |
|
||
| solo / qwen-think | 6 % | 5 % | 44 % |
|
||
| arborist / hermes (n=40) | 60 % | 62 % | 25 % |
|
||
| **arborist / qwen-nothink** | **82 %** | **65 %** | 50 % |
|
||
|
||
Headline findings: (1) **retrieval dominates** — no solo config
|
||
approaches the arborist arms; (2) **reasoning doesn't improve raw
|
||
correctness** (qwen-think/as_of 44 % vs nothink 50 %) and **breaks
|
||
honest-abstention** under source_relative framing (nothink abstains
|
||
98 %, think only 62 %); (3) **production answer: arborist + qwen-nothink,
|
||
plain framing, reasoning off — 82 % CG.** The cost axis will confirm
|
||
whether reasoning's energy premium is ever justified.
|
||
|
||
## Reproduce
|
||
|
||
```
|
||
# full matrix cell (one model, all framings, both arms)
|
||
python -m bench.control_sweep --judge code --models qwen-nothink \
|
||
--arborist-ref qwen-nothink --n 386 --arborist-n 386 --max-workers 1
|
||
|
||
# arborist-only (solo data already collected)
|
||
python -m bench.control_sweep ... --skip-solo
|
||
|
||
# wattage subset (run ON the GPU box)
|
||
python -m bench.watt_bench --models qwen-nothink --n 20 --gpu-label 4090
|
||
```
|
||
|
||
Artifacts land in `bench/qa_results/`; scorecards via
|
||
`bench/score_with_code_judge.py`; judge-vs-judge reconciliation via
|
||
`bench/analyze_judge_disagreement.py`.
|