arborist/docs/benchmark-matrix.md

#000057 benchmark test matrix

Shareable spec of the control-experiment benchmark — what we measure, how, and the cells in the matrix. Companion to the rolling results in docs/qa-modes-bench.md (Addendum 8).

The question

A small language model, asked "who is the president of France?", answers with the present-day office-holder ("Emmanuel Macron") even when the grounding corpus is a fixed ~2010 vintage (where the answer is Sarkozy). The model isn't lying — it's reciting its training-data present — but it fails to recognise it should defer to the supplied source, or abstain when it has none. The matrix isolates what fixes that: model scale, question framing, retrieval, or reasoning.

Fixture

• 386 office-holder questions (bench/qa_questions_stale_map.json), each paired with a fixed corpus-vintage gold article (~2010-2011, verified from the artefact). The gold is the single source of truth a grounded answer must match.
• Adversarial by construction: the corpus answer (Sarkozy) differs from the model's training-data present (Macron), so a model reciting its prior is measurably wrong against the source.

Axes

axis	values	what it isolates
model	hermes (Hermes-3-8B, vLLM) · qwen-nothink (Qwen3.6-27B, llama.cpp, reasoning off) · qwen-think (same, reasoning on)	scale (8B vs 27B) and reasoning
framing	plain ("who is the president of France?") · source_relative ("According to the reference knowledge base, who is …") · as_of_corpus ("As of 2010, who was …")	whether prompt framing alone fixes the drift
arm	solo (model alone — measures the parametric prior) · arborist (model + retrieval over the 2010 corpus, answer_mode=claim_lattice)	whether retrieval fixes it

Full cross product = 3 models × 3 framings × 2 arms = 18 cells, each over up to 386 items. (The arborist arm currently runs hermes and qwen-nothink; qwen-think is measurable but cost-prohibitive — see "Cost dimension".)

Metrics

Every (question, answer, gold) triple gets one verdict from a closed vocabulary:

verdict	meaning
CORRECT_GROUNDED (CG)	answer matches the gold source
WRONG (W)	gold states something the answer contradicts
FABRICATED (F)	answer asserts specifics absent from gold
ABSTAINED (A)	answer honestly declines ("I don't have that")
JUDGE_ERROR (JE)	judge could not classify deterministically

Two reads of the same verdicts:

• accuracy — "did it say the corpus answer?" Penalises a model for knowing the current office-holder. Reported but flagged as misleading.
• grounding-fidelity (the defensible read) — "did the model do the right conversational thing: answer when grounded, abstain when not?" W + F are failures (confident ungrounded assertion); A is success on the questions the model can't ground. CG% and abstain% are the headline numbers.

The judge

Deterministic, no LLM, no quota: bench/judge_code.py. Pipeline (first hit decides): empty/no-gold guard → explicit-abstention regex → short-answer entity-grounding fast path → NLI contradiction (θ=0.85) → lexical verifier (quote/span/entity/paraphrase) → WRONG-vs-FABRICATED tie-break on subject-in-gold. Claim-lattice JSON envelopes are unwrapped to prose before grading so the arborist arm grades on equal terms with solo.

The judge was calibrated against an Opus (SOTA) reference judge on the records Opus could grade: agreement CG 13→47 %, WRONG 56→89 %, ABSTAINED 80→95 %. The deterministic verdict is a proxy for grounding (lexical + NLI), not Opus-grade reading; JUDGE_ERROR residue is the natural input to a later LLM-batch pass. NLI runs on a GPU (currently the 4090 ai box); the rest is pure Python.

Cost dimension (new — for the constraint optimizer)

The matrix has been scored on quality (CG%) but not yet on cost. Adding the energy axis so the constraint optimizer can trade grounding fidelity against power:

cost metric	unit	source
tokens / answer	completion tokens	API usage field
latency / answer	seconds	wall-clock per cell
GPU power	watts (mean / peak)	nvidia-smi power.draw, sampled on the inference GPU during the cell
energy / answer	joules	∫ power dt / questions

Measured per GPU (3090 and 4090) so the optimizer knows the energy cost of each config on each hardware tier. The wattage harness (bench/watt_bench.py) samples nvidia-smi on the GPU box while driving a small representative subset of the matrix.

Why it matters: reasoning chains are the headline cost finding — qwen-think spends 1300-3300 completion tokens/answer vs qwen-nothink's ~50-100 (20-50× the token cost), for a workload where arborist + qwen-nothink already lands 82 % CG. The optimizer should weight that: grounding-fidelity per joule, not per answer.

Results so far (CG%, all arms on the identical calibrated judge)

arm / model	plain	source_relative	as_of_corpus
solo / hermes	9 %	5 %	18 %
solo / qwen-nothink	7 %	0 %	50 %
solo / qwen-think	6 %	5 %	44 %
arborist / hermes (n=40)	60 %	62 %	25 %
arborist / qwen-nothink	82 %	65 %	50 %

Headline findings: (1) retrieval dominates — no solo config approaches the arborist arms; (2) reasoning doesn't improve raw correctness (qwen-think/as_of 44 % vs nothink 50 %) and breaks honest-abstention under source_relative framing (nothink abstains 98 %, think only 62 %); (3) production answer: arborist + qwen-nothink, plain framing, reasoning off — 82 % CG. The cost axis will confirm whether reasoning's energy premium is ever justified.

Reproduce

# full matrix cell (one model, all framings, both arms)
python -m bench.control_sweep --judge code --models qwen-nothink \
    --arborist-ref qwen-nothink --n 386 --arborist-n 386 --max-workers 1

# arborist-only (solo data already collected)
python -m bench.control_sweep ... --skip-solo

# wattage subset (run ON the GPU box)
python -m bench.watt_bench --models qwen-nothink --n 20 --gpu-label 4090

Artifacts land in bench/qa_results/; scorecards via bench/score_with_code_judge.py; judge-vs-judge reconciliation via bench/analyze_judge_disagreement.py.