arborist/docs/benchmark-matrix.md

#000057 benchmark test matrix

Shareable spec of the control-experiment benchmark — what we measure, how, and the cells in the matrix. Companion to the rolling results in docs/qa-modes-bench.md (Addendum 8).

The question

A small language model, asked "who is the president of France?", answers with the present-day office-holder ("Emmanuel Macron") even when the grounding corpus is a fixed ~2010 vintage (where the answer is Sarkozy). The model isn't lying — it's reciting its training-data present — but it fails to recognise it should defer to the supplied source, or abstain when it has none. The matrix isolates what fixes that: model scale, question framing, retrieval, or reasoning.

Fixture

• 386 office-holder questions (bench/qa_questions_stale_map.json), each paired with a fixed corpus-vintage gold article (~2010-2011, verified from the artefact). The gold is the single source of truth a grounded answer must match.
• Adversarial by construction: the corpus answer (Sarkozy) differs from the model's training-data present (Macron), so a model reciting its prior is measurably wrong against the source.

Axes

axis	values	what it isolates
model	hermes (Hermes-3-8B, vLLM) · qwen-nothink (Qwen3.6-27B, llama.cpp, reasoning off) · qwen-think (same, reasoning on)	scale (8B vs 27B) and reasoning
framing	plain ("who is the president of France?") · source_relative ("According to the reference knowledge base, who is …") · as_of_corpus ("As of 2010, who was …")	whether prompt framing alone fixes the drift
arm	solo (model alone — measures the parametric prior) · arborist (model + retrieval over the 2010 corpus, answer_mode=claim_lattice)	whether retrieval fixes it

Full cross product = 3 models × 3 framings × 2 arms = 18 cells, each over up to 386 items. (The arborist arm currently runs hermes and qwen-nothink; qwen-think is measurable but cost-prohibitive — see "Cost dimension".)

Metrics

Every (question, answer, gold) triple gets one verdict from a closed vocabulary:

verdict	meaning
CORRECT_GROUNDED (CG)	answer matches the gold source
WRONG (W)	gold states something the answer contradicts
FABRICATED (F)	answer asserts specifics absent from gold
ABSTAINED (A)	answer honestly declines ("I don't have that")
JUDGE_ERROR (JE)	judge could not classify deterministically

Two reads of the same verdicts:

• accuracy — "did it say the corpus answer?" Penalises a model for knowing the current office-holder. Reported but flagged as misleading.
• grounding-fidelity (the defensible read) — "did the model do the right conversational thing: answer when grounded, abstain when not?" W + F are failures (confident ungrounded assertion); A is success on the questions the model can't ground. CG% and abstain% are the headline numbers.

The judge

Deterministic, no LLM, no quota: bench/judge_code.py. Pipeline (first hit decides): empty/no-gold guard → explicit-abstention regex → short-answer entity-grounding fast path → NLI contradiction (θ=0.85) → lexical verifier (quote/span/entity/paraphrase) → WRONG-vs-FABRICATED tie-break on subject-in-gold. Claim-lattice JSON envelopes are unwrapped to prose before grading so the arborist arm grades on equal terms with solo.

The judge was calibrated against an Opus (SOTA) reference judge on the records Opus could grade: agreement CG 13→47 %, WRONG 56→89 %, ABSTAINED 80→95 %. The deterministic verdict is a proxy for grounding (lexical + NLI), not Opus-grade reading; JUDGE_ERROR residue is the natural input to a later LLM-batch pass. NLI runs on a GPU (currently the 4090 ai box); the rest is pure Python.

Cost dimension — for the constraint optimizer (expanded goal)

The next-goal matrix scores quality AND cost across the serving stack, not just the model. Every cell is a deployable configuration; the optimizer trades grounding-fidelity against energy to pick the config to ship.

Expanded axes (the full cost/quality matrix):

axis	values
model	qwen-27B · hermes-8B
engine	llama.cpp · vLLM (where the model/engine combo is supported)
GPU	RTX 3090 · RTX 4090
arm	solo · arborist (retrieval)

Cross product = 2 model × 2 engine × 2 GPU × 2 arm = up to 16 quality+cost cells. Each cell records:

metric	unit	source
quality (CG% / abstain%)	rate	code judge
tokens / answer	completion tokens	API usage
latency / answer	seconds	wall-clock
GPU energy	watts mean/peak · joules · J/answer · J/token	nvidia-smi power.draw on the inference GPU
CPU energy	watts mean · joules · J/answer	Intel RAPL energy_uj (package sum) on the box

bench/watt_bench.py samples both GPU (nvidia-smi) and CPU (RAPL) on the inference box while driving the subset. RAPL energy_uj is root-only by default (PLATYPUS / CVE-2020-8694 mitigation) — pass --cpu-energy-cmd 'sudo cat {path}' with a sudo rule, or relax the sysfs perm, to capture CPU watts; GPU watts need no special perm.

Serving-stack inventory (recon 2026-05-20):

box	host	GPU	engine present	model present
4090	ai.foxhop.net:18888	RTX 4090, i9-14900K (32t)	llama.cpp	qwen-27B gguf
3090	3090-ai.foxhop.net:18888	RTX 3090, i9-12900K (24t)	vLLM (venv)	hermes-8B

To fill the matrix, the missing serving configs must be stood up: vLLM+qwen, llama.cpp+hermes (gguf), and each model on the other GPU. That's an infra buildout (install engines, fetch models, manage the single-slot live endpoints), to be captured as salt states in ~/git/foxhop-states (where ai.foxhop.net's llama-qwen state already lives; the 3090 box is not yet salt-managed).

Known-good serving invocations (captured 2026-05-20 — the basis for the salt states):

• qwen / llama.cpp / 4090 (live, salt llama-qwen/init.sls):

  /mnt/data/llm/llama.cpp/build/bin/llama-server \
    -m /mnt/data/llm/cache/Qwen3.6-27B-UD-Q4_K_XL.gguf \
    --host 0.0.0.0 --port 18888 -ngl 99 -c 65536 -fa on \
    --cache-type-k q8_0 --cache-type-v q8_0 -np 1 \
    --spec-type draft-mtp --spec-draft-n-max 6 \
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.5
  

• hermes / vLLM / 3090 (live, MANUAL tmux — not yet salt; vLLM 0.10.1.1 on python3.12):

  python -m vllm.entrypoints.openai.api_server \
    --model adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic \
    --host 0.0.0.0 --port 18888 --max-model-len 82000 \
    --enable-auto-tool-choice --tool-call-parser hermes
  

Per the existing convention, salt manages the systemd unit; the big artifacts (engine binaries, model files / gguf) live on /mnt/data and are provisioned manually, documented as state prereqs. Migrating the live 3090 hermes from its manual tmux session to a salt systemd unit must keep ≥1 hermes online (qwen is the expendable/exploratory workload) — a careful cutover, not a blind highstate.

Why cost matters most here: reasoning chains are the headline cost finding — qwen-think spends 1300-3300 completion tokens/answer vs qwen-nothink's ~50-100 (20-50× the token cost), for a workload where arborist + qwen-nothink already lands 82 % CG. And hermes-8B on a 3090 idles at ~127 W vs qwen-27B on a 4090 at ~20 W — idle floor, model size, engine, and GPU tier all move the joules-per-grounded-answer the optimizer cares about. The metric is grounding-fidelity per joule, not per answer.

Results so far (CG%, all arms on the identical calibrated judge)

arm / model	plain	source_relative	as_of_corpus
solo / hermes	9 %	5 %	18 %
solo / qwen-nothink	7 %	0 %	50 %
solo / qwen-think	6 %	5 %	44 %
arborist / hermes (n=40)	60 %	62 %	25 %
arborist / qwen-nothink	82 %	65 %	50 %

Headline findings: (1) retrieval dominates — no solo config approaches the arborist arms; (2) reasoning doesn't improve raw correctness (qwen-think/as_of 44 % vs nothink 50 %) and breaks honest-abstention under source_relative framing (nothink abstains 98 %, think only 62 %); (3) production answer: arborist + qwen-nothink, plain framing, reasoning off — 82 % CG. The cost axis will confirm whether reasoning's energy premium is ever justified.

Reproduce

# full matrix cell (one model, all framings, both arms)
python -m bench.control_sweep --judge code --models qwen-nothink \
    --arborist-ref qwen-nothink --n 386 --arborist-n 386 --max-workers 1

# arborist-only (solo data already collected)
python -m bench.control_sweep ... --skip-solo

# wattage subset (run ON the GPU box)
python -m bench.watt_bench --models qwen-nothink --n 20 --gpu-label 4090

Artifacts land in bench/qa_results/; scorecards via bench/score_with_code_judge.py; judge-vs-judge reconciliation via bench/analyze_judge_disagreement.py.