arborist/docs/benchmarks.md

# Benchmarks — orientation, harnesses, fixtures, discipline

Arborist ships two bench harnesses, four question fixtures, and a
signal-floor discipline pinned across the substrate. This doc is
the canonical entry point. Three companion docs cover specifics:

| File                                   | Purpose                                    |
|----------------------------------------|--------------------------------------------|
| `docs/bench-maxing.md`                 | Discipline rules (5pp signal floor, etc.)  |
| `docs/qa-modes-bench.md`               | Historical journal — running addendum log  |
| `docs/bench-emergent-design.md`        | Random-word stress-test design rationale   |
| `docs/qa-modes-bench-2026-04-30.md`    | Frozen point-in-time bench (Apr-30 baseline) |
| `docs/energy-cogs-benchmark.md`        | **qwen vs hermes** — GPU energy COGS + quality-per-dollar (Dav1d-audience report) |
| `docs/benchmark-matrix.md`             | Control-arm (solo vs substrate) A/B matrix + the deterministic code judge |

Read those when you need the why or the long-form data. This file
covers the what, where, and how.

## 1. Three harnesses, three purposes

### 1.1 `bench/qa_sweep.py` — curated QA-quality bench

Runs a fixture file × answer modes × n samples. Persists a JSONL
of per-row results plus a markdown summary. Used for:

- **Regression checks** before merging substrate changes.
- **A/B cycles** on policy knobs (cap on/off, reminder on/off,
  preflight enabled/disabled).
- **Per-mode strict-rate tracking** over time.

Run via `make bench-qa` or `bench/qa_sweep.py` directly. Bench
rows carry ~25 fields per cell, including the four-rung label
(`audit_mode`), the verifier-method, the violation kinds, the
preflight QuestionState projection (`preflight_logical_statuses`,
`preflight_question_shape`, etc.), the quantifier classifier
output (`quantifier_intensity`, `scope_bound_hint`,
`claim_cap_applied`), the preflight stage hash 12-char prefix
(#000009 §7.2), and a per-stage timing breakdown
(`preflight_ms`, `soft_preflight_ms`, `search_ms`, `context_ms`,
`llm_ms`, `persist_ms`, `total_ms`).

### 1.2 `scripts/bench_emergent.py` — random-word stress test

Picks three random words from `/usr/share/dict/words`, asks
Hermes (at temp 0.8) to weave them into a creative question
paragraph, then sends that paragraph to arborist. Logs every cycle
to `bench/emergent_log.jsonl` as one JSONL line. Designed for
**blue-moon cadence**, not every-commit benching — the
combinatoric word space surfaces failure shapes the curated bench
doesn't reach.

The teacher review step is intentionally **not automated.** Fox
brings interesting log entries to a teacher model (Claude Opus
4.7 in the current setup) and asks for guidance: did the answer
match the question? Novelty class
(known_truth_grounding / emergent_synthesis / novel_claim /
no_signal)? Which hyperparam to tune next?

Run via `make bench-emergent` (default `EMERGENT_N=10`) or
`make bench-emergent EMERGENT_N=100` for a longer cycle. See
`docs/bench-emergent-design.md` for the design rationale.

### 1.3 `bench/control_ab.py` + `watt_*` — cross-model cost & quality-per-dollar

This is how we benchmark **qwen vs hermes with arborist**, on two axes
that multiply into **quality-per-dollar**. Full living report (and the
Dav1d-facing writeup) in `docs/energy-cogs-benchmark.md`; this is the
orientation.

**Two rigs, run on the same harness** (the GPU work happens on the
producer box, never in arborist's `python+sqlite3` core):

| | rig A | rig B |
|---|---|---|
| model | `Qwen3.6-27B` UD-Q4_K_XL (llama.cpp) | `Hermes-3-8B` FP8 (vLLM) |
| GPU | RTX 4090, `ai.foxhop.net` (isolated for the bench) | RTX 3090, `3090-ai.foxhop.net` (live/public) |

**Value axis — `bench/control_ab.py`.** Runs the *same* model two ways:
**solo** (question only, no retrieval/verifier) vs **substrate-ON** (full
`query()` — 4-route FTS5 → context → LLM → verifier), n=30 per fixture,
grades each answer with the judge (§12). Headline finding: the substrate
lifts both models 2–13× and they **nearly converge** (an 8B and a 27B
reach similar grounded quality — grounding comes from retrieval+verify,
not parametric size).

**Cost axis — `bench/watt_bench.py` / `watt_calibrate.py` / `watt_probe.py`.**
Measures **GPU joules per token** by slope calibration (prefill J/input-tok
and decode J/output-tok measured separately — they differ ~35–40×), times
the *real* token mix per query, at a site `--price-per-kwh`. Power states
are measured at runtime, never hardcoded; cache-miss is forced so we time
generation not a SQLite lookup. Result: a grounded answer costs
**~$0.07–0.16 per 1,000 queries** of GPU electricity (hermes-8B cheaper
than qwen-27B — but that's a 4-way confound: params × quant × engine ×
card, not "8B vs 27B" alone). The dominant cost is **prefilling the
retrieved context** (~⅔ of substrate GPU energy), not generating the
answer.

Run the value axis via `make control-ab` (`N=…  FIXTURE=…`); the watt
instruments run directly — `bench/watt_calibrate.py` / `watt_bench.py`,
with `make rapl-access` first for CPU-package power. Invocations and raw
artifacts are in `docs/energy-cogs-benchmark.md` §8. **All value-axis
grading is done by the deterministic code judge (§12)** — an Opus
LLM-judge was attempted for #000057 and abandoned (too expensive per
call, ~79.5 % `JUDGE_ERROR` rate-limited, burned coding quota); it never
produced usable verdicts at scale and is gated off so it can't re-burn.

## 2. Four question fixtures

```
bench/qa_questions.txt                                — 75 questions
bench/qa_questions_smoke.txt                          —  smoke test
bench/qa_questions_quantifier_subset.txt              —  9 questions
bench/qa_questions_quantifier_baseline.txt            —  1 question
bench/qa_questions_metacog_subset.txt                 — 28 questions
```

| Fixture                              | Use case                                                                                    | Cell size at n=3 × 3 modes |
|--------------------------------------|---------------------------------------------------------------------------------------------|----------------------------|
| `qa_questions.txt`                   | Full regression bench. Run before merging substrate changes. Wide coverage + corner cases. | 675 runs, ~3-3.5h          |
| `qa_questions_smoke.txt`             | Smoke test on 1-2 questions. Use to confirm a CLI flag wires through correctly.            | 3-6 runs, ~30s             |
| `qa_questions_quantifier_subset.txt` | 9 broad-quantifier questions (7 unbounded + 2 bounded). Use for A/B cycles on #000008 knobs. | 81 runs, ~7-15min          |
| `qa_questions_quantifier_baseline.txt` | Single broad question for narrow n=3 baseline measurement.                              | 9 runs, ~2-4min            |
| `qa_questions_metacog_subset.txt`    | 28 questions targeting each #000010 detector (temporal, contradiction, false-premise, out-of-corpus) plus mixed multi-trigger + well-formed controls. | 252 runs, ~25-35min |

When choosing a fixture: ask whether you're measuring the
**substrate as a whole** (full bench), a **specific feature**
(broad-quantifier subset, metacog subset), or just **wiring
correctness** (smoke).

## 3. The signal floor — n=3 × 9 = 27 sample minimum, 5pp delta floor

Per `docs/bench-maxing.md`:

- **5pp floor** for STRICT-rate deltas at n=3 × any-fixture sample
  size. Differences below this are Hermes nondeterminism, not
  signal. Quote-mode often jitters ±10pp on 27-sample cells.
- **Mean-ratio + UNGROUNDED-rate** are tighter signals at the same
  sample size — they aggregate per-row values rather than
  counting discrete verdicts.
- **n=5 verification** when an A/B cell sits within the noise
  band (e.g. 0.30 vs 0.33 STRICT-rate is 3pp — within noise; n=5
  on the closer cell tightens the variance).
- **vLLM concurrency saturates at c=3-4.** Going higher slows
  per-call latency without parallelizing harder. Default `c=4`.

Don't tune to a single bench cycle. Always have a baseline +
an A/B cell + (sometimes) a verification cell.

## 4. Make targets cheat sheet

```
# Curated QA bench (default: bench/qa_questions.txt × 3 modes × n=3)
make bench-qa

# Same but with limit / different file
.venv/bin/python bench/qa_sweep.py \
    --questions bench/qa_questions_quantifier_subset.txt \
    --shards-dir ~/.arborist/shards --n 3 --concurrency 4 \
    --out-dir bench/qa_results

# A/B cycle: flip a policy knob for one bench cell
.venv/bin/python bench/qa_sweep.py \
    --questions bench/qa_questions_quantifier_subset.txt \
    --shards-dir ~/.arborist/shards --n 3 --concurrency 4 \
    --policy quantifier_guard_apply_caps=true \
    --policy quantifier_reminder_enabled=true

# Resume an interrupted bench (same --seed, append to existing JSONL)
.venv/bin/python bench/qa_sweep.py \
    --resume bench/qa_results/2026-05-03T19-30-13Z.jsonl \
    --seed 0

# Smoke test — quick sanity of the wiring
.venv/bin/python bench/qa_sweep.py \
    --questions bench/qa_questions_smoke.txt --n 1 --concurrency 1

# Random-word stress test (blue-moon cadence)
make bench-emergent              # n=10 default
make bench-emergent EMERGENT_N=100
```

## 5. The bench-row schema

Every row in `bench/qa_results/<stamp>.jsonl` carries (current
shape — fields are additive across versions):

**Identity & verdict:**
- `question`, `answer_mode`, `audit_mode`, `verifier_method`
- `n_quotes`, `n_verified`, `ratio`
- `cache_key` (12-char prefix)
- `preflight_hash` (12-char prefix; see #000009 §7.2)
- `model_profile_id`

**Diagnostics:**
- `format_collapsed` (pointer-mode only signal; #000008)
- `violation_kinds` (sorted list of unique kinds)
- `lazy_anchor_ratio`, `pointer_id_distribution`
- `answer_brackets`, `answer_pointer_count`,
  `answer_chars_with_brackets`, `raw_meaningful_line_count`
- `deflection_kind`, `subject_anchor`, `subject_in_answer`
- `metaphor_deflection_kind`, `metaphor_cue_count`,
  `metaphor_overlap_count`

**Preflight projection (#000010):**
- `preflight_logical_statuses` (list)
- `preflight_question_shape`, `preflight_result`
- `preflight_temporal_sensitivity`
- `preflight_has_false_premise`, `preflight_has_contradiction`
- `preflight_corpus_requirement`

**Quantifier classifier (#000008):**
- `quantifier_intensity`, `quantifier_matched_token`
- `scope_bound_hint`, `quantifier_explicit_count`
- `claim_cap_applied`

**Capacity:**
- `prompt_chars_total`, `prompt_chars_evidence`,
  `prompt_chars_system`, `prompt_chars_question`
- `answer_chars`

**Directive compliance** (per-row pass/fail for the seven-point
program directives observable from a single bench row):
- `directive_compliance` dict keyed on D2/D3/D4/D6/D7

**Time:**
- `elapsed_s`, `iso_ts`, `error`

The markdown summary alongside the JSONL renders aggregate views
(per-mode strict-rate, format-collapse + violation kinds table,
strict-rate-by-prompt-size buckets, directive coverage).

## 6. The journal — where headlines live

`docs/qa-modes-bench.md` is the **rolling research log** for the
QA bench. Each substantial bench cycle adds a new addendum (or
refines an existing one). The 2026-05-02 baseline is the frozen
authoritative state of the substrate before #000008/9/10/11
landed; subsequent addenda compare against it.

Current addenda:
- **2026-05-02 baseline** — pre-preflight authoritative state.
- **Addendum 1** (broad-quantifier A/B from #000008 §12.6).
- **Addendum 2** (preflight on vs off, 9-question broad subset).
- **Addendum 3** (full 75-question regression check post-flip).
- **Addendum 4** (metacog-trigger detector validation, 6 → 28
  question fixture).

When a new bench cycle produces a substantive finding, add an
addendum or refine an existing one. Bench artifacts (JSONL +
markdown) live in `bench/qa_results/` and are gitignored — only
headlines + cross-references go in the journal.

Per-ticket bench data lives in the relevant ticket file's `§12`
or `§13` (e.g. `docs/tickets/ticket-000008-...md` §12 carries the
four-cell A/B; #000010 §13 has three sub-validations; #000006 has
the rolling-amend pattern across multiple cycle counts).

## 7. The bench-emergent log

`bench/emergent_log.jsonl` is a long-running append-only log of
random-word stress cycles. Each entry has:

```json
{
  "ts": <unix>, "iso_ts": "2026-05-04T...",
  "words": ["dismally", "heats", "Jamaicans"],
  "question": "<model-generated question paragraph>",
  "answer": "<arborist's answer>",
  "audit_mode": "STRICT|HYBRID|UNGROUNDED",
  "verifier_method": "claim_lattice",
  "n_quotes": 3, "n_verified": 0,
  "violation_kinds": ["CITATION_MISMATCH", "TOO_MANY_EVIDENCE_IDS"],
  "metaphor_deflection_kind": "no_signal",
  "metaphor_cue_count": 2, "metaphor_overlap_count": 2,
  "sources": [{"title": "...", "uri": "...", "used": false}, ...],
  "answer_seconds": <float>, "total_seconds": <float>,
  "teacher": <reserved for human review>
}
```

Findings live in `docs/tickets/ticket-000006-bench-emergent-findings.md`
as rolling amends — each new cycle batch produces a new amend with
the verdict shift, violation profile, and tuning candidates.

## 8. How to run a focused A/B

Pattern from #000008 §12 (broad-quantifier four-cell A/B):

1. **Pick a fixture** matching the feature under test
   (broad-subset for quantifier work; metacog-subset for
   metacognition work; smoke for wiring sanity).
2. **Run the baseline** — same fixture, default policy.
3. **Run the cell** — same fixture, one policy knob flipped via
   `--policy KEY=VALUE`. Repeat for each independent knob.
4. **Compare** — STRICT-rate, mean-ratio, UNGROUNDED-rate,
   FORMAT_COLLAPSED rate, dominant violation kinds. Apply the
   5pp floor to STRICT-rate; mean-ratio and UNGROUNDED tend to
   be tighter.
5. **Verify with n=5** if a cell sits within the noise band.
6. **Document** — add an addendum to `docs/qa-modes-bench.md` or
   a new `§12.x` / `§13.x` to the relevant ticket file.

The four-cell A/B in #000008 (preflight off / reminder only / cap
only / cap+reminder) is the canonical pattern for measuring
multi-knob features.

## 9. How to interpret results

**STRICT-rate** = the fraction of cells where the verifier could
prove every claim grounded against cited evidence. Most direct
quality metric, but noisy at small sample sizes.

**HYBRID-rate** = mixed verdicts (some claims grounded, some
not). Often more informative than STRICT-rate when a feature's
job is to rescue UNGROUNDED → HYBRID rather than HYBRID → STRICT.

**UNGROUNDED-rate** = honest "the verifier could not ground."
Lower is generally better, but UNGROUNDED on questions the corpus
genuinely can't answer is the *correct* outcome — the substrate
preferring UNGROUNDED to fabricated STRICT is the architectural
win named in `docs/seven-point-program.md` D7.

**Mean ratio** = mean of `n_verified / n_quotes` across rows.
Aggregates per-row values rather than counting verdicts; tighter
at small sample sizes than STRICT-rate.

**FORMAT_COLLAPSED rate** = pointer-mode-specific gauge of
whether the model is following the `[E\d+]` citation protocol.
Zero is the goal; non-zero means the model emitted free prose
without pointer tags.

**Violation kind distribution** = per-mode tallies of which hard
checks fired most often. Gives the operator a per-failure-mode
view of the substrate's weaknesses.

**Directive compliance** = per-row pass/fail for the
seven-point-program directives (D2/D3/D4/D6/D7). Substrate-level
hygiene check; should be ~100% on lattice modes.

**Audit-line tails** (rendered by
`arborist/cli.py:_render_warrant_tail`):
- Hard tails: `· warrant missing`, `· title mismatch`,
  `· format collapsed`, `· broad cap N`, `· broad rejected`,
  `· broad unbounded`, `· broad runaway`, `· false premise`,
  `· contradictory`, `· stale risk`, `· out of corpus`,
  `· frame ambiguous`.
- Soft tails (advisory): `· soft: <label>` (#000011 sidecar
  output; SOFT_DISABLED / SOFT_PARSE_FAIL / SOFT_WELL_FORMED
  suppress).

A row with multiple tails composes them: `EVIDENCE-WARRANTED ·
via claim_lattice · false premise · soft: time sensitive` is a
verdict the substrate reached but with two distinct preflight
warnings worth the operator's attention.

## 10. Operator commands cheat sheet

```bash
# Inspect preflight stage payload from any cache row
arborist providence --show-preflight CACHE_KEY_PREFIX

# Single query with all preflight knobs flipped
arborist query "winners of all major sports?" \
    --apply-quantifier-caps --reject-broad --soft-preflight

# Dry-run — assemble context but skip LLM
arborist query "..." --dry-run

# Force fresh inference (delete prior cache row first)
arborist query "..." --burn

# Show raw JSON record instead of human render
arborist query "..." --json
```

The `Makefile` exposes most of these via shorthand:

```
make query Q="..." BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 \
    BURN=1 JSON=1 ANSWER_MODE=claim_lattice
```

See the `Makefile` directly for the full set.

## 11. Deterministic retrieval instruments (no-LLM)

The bench-maxing rule (CLAUDE.md): when a lever's failure class is
below the n=3 / 5pp noise floor, fix the instrument — mine
ground-truth-carrying questions and grade by deterministic retrieval,
not by the LLM verifier. These instruments take `query --dry-run`
output only: no LLM, no verifier, no judge, no noise floor, scalable
to the corpus. They measure *retrieval surfacing* —
necessary-not-sufficient for STRICT, and they **complement, never
replace**, the curated adversarial set (the verifier-honesty gate).

- **`bench/mine_questions.py`** — mines questions whose target article
  is known by construction (surface-variant of a corpus title).
  Classes: `numeral` (Roman↔ordinal), `accent`, `hyphen`,
  `honorific` (Mt↔Mount), `amp` (&↔and), `brit` (US↔UK spelling),
  `stale` (temporal fabrication-bait, control_ab only).
- **`bench/recall_at_k.py`** — recall@1/@3/@k of the known target on
  a mined fixture. Returns rank, so recall at every k is free from one
  retrieval (a too-lenient k hides a rank-only lift — report @1/@3/@k).
- **`bench/jaggedness.py`** (`make bench-jaggedness`, #000060) — does
  retrieval surface the SAME target for a question's *canonical*
  phrasing (title verbatim) AND its *surface-perturbed* phrasing? A
  non-jagged retriever agrees; a jagged one finds "Henry VIII" but
  misses "Henry the eighth". Reuses `recall_at_k.probe` +
  `mine_questions._surface_variant`.
  - `J_norm` (binary) = fraction of titles where canonical & perturbed
    DISAGREE on surfacing the target @k (XOR). Lower = more non-jagged.
  - graded mean `|Δrank|` over titles where both surfaced — rank
    instability the binary metric can't see.
  - Distinguishes a true recall-miss (both phrasings miss = not
    jagged) from jaggedness (one surfaces, one doesn't). Feeds
    #000012's ForkScore `ΔJaggednessReduction`: a retrieval fold that
    lowers `J_norm` without lowering recall is a non-jagged win.
  - The A-vs-C *answer-quality* jaggedness delta (does the substrate
    make answers more consistent across variants than the bare model?)
    is the LLM+judge version — rides `bench/control_sweep.py` over
    variant groups, gated v2.

## 12. The judge — how bench answers are graded

A grading judge appears **only** in the cross-model value bench (§1.3) —
deciding whether a *model's answer* is correct-and-grounded. It is
external bench science and must not be confused with arborist's in-system
**verifier** (`arborist/qa/verify.py`), which decides `audit_mode` on the
proof path and is bound by the "no LLM-as-judge in the verifier" rule.
The bench judge sits *outside* both arms (solo and substrate) and grades
post-hoc; it never touches either system's internals.

Verdict vocabulary (closed, so aggregation is deterministic):
`CORRECT_GROUNDED` / `WRONG` / `ABSTAINED` / `JUDGE_ERROR`.

**The judge is the deterministic code judge.** Every value-axis number
in `docs/energy-cogs-benchmark.md` was graded by it. There is no
LLM-judge running on this path.

### 12.1 Code judge — `bench/judge_code.py`

Deterministic, no LLM, no quota. **Five verdicts:**
`CORRECT_GROUNDED` / `WRONG` / `FABRICATED` / `ABSTAINED` /
`JUDGE_ERROR`. The `WRONG`-vs-`FABRICATED` split is what powers the
bare-model failure-style finding in energy-cogs §5.5 (bare qwen
fabricates, bare hermes abstains; the substrate fixes both).

Authoritative source: `bench/judge_code.py` `judge()` at line 493 — its
own docstring (line 501-528) is the truth on rule order. Pipeline, first
hit decides:

1. **Empty / no-gold guards.** Empty answer → `ABSTAINED`; empty gold →
   `JUDGE_ERROR`.
2. **Explicit abstention phrases** — broad lexical regex including the
   "I do not have access to information / context" Hermes family that
   the original narrow patterns missed. Match → `ABSTAINED`.
3. **Short-answer entity-grounding fast path.** When the answer is
   short, asserts at least one specific, every specific is in gold, AND
   the question's subject anchor is in gold → `CORRECT_GROUNDED`. Runs
   **before** NLI (since the 2026-05-19 Poland-Tusk smoke): NLI's
   clause-level aggregation can mis-fire as contradiction on multi-PM
   lists even when the answer is right; positive lexical evidence wins.
4. **Strong NLI contradiction** — `arborist.qa.nli.shadow_check`,
   `θ_contra = 0.85` (code-judge-pinned `_CODE_JUDGE_THETA_CONTRA`,
   **raised** from the manifest's 0.5 default after measuring 114 FPs
   in the 0.5-0.75 band). `max_contradiction ≥ 0.85` → `WRONG`.
5. **Lexical verifier vs gold** — `arborist.qa.verify.verify_quotes`
   (the existing quote → span → entity → paraphrase ladder), then map:
   - `STRICT` → `CORRECT_GROUNDED`
   - `HYBRID` + NLI entail ≥ 0.55 (`_CODE_JUDGE_THETA_ENTAIL_CORROBORATE`) → `CORRECT_GROUNDED`
   - `HYBRID` + subject-anchor-in-gold + every specific in gold → `CORRECT_GROUNDED` (entity-grounding rescue)
   - `HYBRID` + ≥1 verbatim quote verified + on-topic + **no unsourced
     numeric** specific + not NLI-contradicted → `CORRECT_GROUNDED`
     (2026-05-21 relaxed rescue, `2d31866`: unsourced proper-nouns
     treated as aliases / paraphrase, but a wrong date/count is held
     back)
   - `HYBRID` otherwise → `JUDGE_ERROR` (`"code judge ambiguous … residue
     for LLM judge"`)
   - `UNGROUNDED` + specifics-not-in-gold + subject-**in**-gold →
     `WRONG` (source has the topic but a different value)
   - `UNGROUNDED` + specifics-not-in-gold + subject-**not**-in-gold →
     `FABRICATED` (source silent on the topic)
   - `UNGROUNDED` + no asserted specifics → `ABSTAINED` (thin /
     non-answer, not a fabrication)

Claim-lattice JSON envelopes are unwrapped to prose
(`_unwrap_claim_lattice_json`) before any rule runs, so substrate-CL
grades on equal terms with solo-prose. Each `Verdict` carries a
`decision` trace (rules_fired, NLI probabilities, subject anchor,
specifics, audit_mode) — fully replayable. `JUDGE_ERROR` is reserved
*only* for the `HYBRID`-without-corroboration class (the ~10 % residue
after the `2d31866` HYBRID-rescue strengthening; energy-cogs §5.5);
everything else lands on a concrete verdict.

Self-test: `bench/judge_code.py:self_test()` (`make judge-self-test`),
gating before any control-ab run. The pipeline cross-reference (older,
shorter) is `docs/benchmark-matrix.md` "## The judge"; used in context
in `docs/energy-cogs-benchmark.md` §5.5.

### 12.2 Why there is no LLM judge in active use — `bench/judge.py` (abandoned)

An Opus headless judge (`claude -p`) was attempted for #000057 as
"external bench science." It did not work in practice:

- **~79.5 % of its verdicts came back `JUDGE_ERROR`** — rate-limited by
  the Anthropic API on a huge-N sweep, per `docs/qa-modes-bench.md`.
- **Each call was expensive enough** that completing a real sweep would
  have consumed the coding-agent's quota for unrelated work; the
  experiment burned Opus quota and wasted dev time wiring it up
  (`f63b00d` → `9dc02e4`, the wakeup call for the CLAUDE.md "Budget
  discipline" rule).
- It is now **fail-closed gated off** by an env flag (`ARBORIST_JUDGE_ENABLE=1`)
  so a stray re-run can't re-burn the budget — a guardrail, not a
  feature flag a reviewer should treat as available.

The intended methodological hygiene (hermetic / blinded / grounded /
structured / audited; stated same-family threat-to-validity) is preserved
in the `bench/judge.py` module docstring as a record of what *would* be
required of any future external SOTA judge — it is **not** a description
of an active grading path. Re-introducing it needs a different-family
SOTA model (per the same-family threat note) AND fox's explicit go on
spend.

**To forward to a reviewer (Dav1d):** `docs/energy-cogs-benchmark.md`
(qwen-vs-hermes, already Dav1d-audience — shows the code judge grading
in §5.5/§8) + this §12. The separate
`docs/relevance-and-veto-synthesis-for-dav1d.md` covers the in-system
*verifier / veto* layer (proof-path; a different thing from bench
grading) and should not be confused with this section.