feat(#000057): fail-closed Opus judge gate — ARBORIST_JUDGE_ENABLE=1 to run

2026-05-19: huge-N #000057 control sweep (f63b00d9dc02e4) burned
our Opus quota. Disable judge.py by default so a stray re-run can't
re-burn — every call short-circuits to JUDGE_ERROR with rationale
'disabled — set ARBORIST_JUDGE_ENABLE=1 ...' and zero subprocess
spawn (0ms in the disabled path, smoke-tested).

Why a gate, not a model swap:
- judge.py uses Opus deliberately as EXTERNAL SOTA outside both arms;
  swapping the judge to Hermes/Qwen would corrupt the experiment
  (Hermes is itself an arm under test). The hygiene comment at
  judge.py:26-29 already names same-family-judging as the live threat
  to validity at Opus level; downgrading further changes the science.
- Gating instead preserves the science when fox re-enables, and gives
  us the data-first workflow he asked for: deterministic tool
  pre-filters (verifier / NLI / recall@k) up front, judge only on
  residue worth Opus tokens, with explicit go.

Behaviour:
- control_sweep.py + control_ab.py already treat JUDGE_ERROR
  non-fatally (counted as JE in _bucket); disabled runs degrade to
  100% JE in the tally and surface the disable reason in rationale —
  the loudest possible 'judge did not run here' signal.
- Re-enable per-run: ARBORIST_JUDGE_ENABLE=1 python -m bench.control_sweep ...
- self_test() will report 4× JUDGE_ERROR when gated — intentional;
  if the instrument is off, the self-test must NOT silently pass.

Smoke-test (without flag): label='JUDGE_ERROR'  rationale='disabled — ...'
dt=0.0ms · no claude subprocess spawned.

Cross-referenced from CLAUDE.md '## Live endpoints' /
'Budget discipline' subsection added in 2365bd1.
This commit is contained in:
russell@unturf.com 2026-05-19 17:26:06 -04:00
parent 2365bd1da4
commit 1cabfe6850
No known key found for this signature in database

View file

@ -1,6 +1,16 @@
#!/usr/bin/env python3
"""Hermetic external judge for the #000057 control experiment.
**Disabled by default as of 2026-05-19.** The huge-N control sweep
(`f63b00d` `9dc02e4`) burned our Opus quota; per CLAUDE.md
"Budget discipline" we no longer call Opus on autopilot. `judge()`
now fails-closed with a `JUDGE_ERROR` verdict ("disabled — set
ARBORIST_JUDGE_ENABLE=1") unless that env flag is set, so a stray
re-run cannot re-burn the budget. Tool-based pre-filters
(deterministic verifier, NLI, lexical / recall@k scoring) front-run
the judge: only the residue actually worth Opus tokens gets passed
through later, with fox's explicit go.
The judge is a SOTA model (Opus via `claude -p`, headless) used as
EXTERNAL SCIENCE it sits outside both arms (Hermes-solo, Arborist),
scores outputs post-hoc, and touches neither system's internals. This
@ -39,6 +49,19 @@ from dataclasses import dataclass
JUDGE_PROMPT_ID = "ctrl-judge-v1"
JUDGE_MODEL = "opus (claude -p)"
# Fail-closed gate (2026-05-19): the huge-N #000057 sweep burned our Opus
# quota. `judge()` short-circuits with a JUDGE_ERROR verdict unless
# ARBORIST_JUDGE_ENABLE=1 is exported. Existing callers (control_sweep,
# control_ab) already handle JUDGE_ERROR non-fatally, so disabled runs
# degrade cleanly — every record returns "disabled" in rationale, the
# tally goes 100% JE, and nothing fires `claude -p`. Re-enable per run:
# ARBORIST_JUDGE_ENABLE=1 python -m bench.control_sweep ...
_JUDGE_ENABLE_ENV = "ARBORIST_JUDGE_ENABLE"
_DISABLED_RATIONALE = (
f"disabled — set {_JUDGE_ENABLE_ENV}=1 to enable Opus judge calls "
"(per CLAUDE.md 'Budget discipline')"
)
# Pinned, hermetic. The judge sees ONLY these three fields. It is NOT
# told either system exists. Verdict vocabulary is closed so scoring
# is deterministic.
@ -95,7 +118,14 @@ class Verdict:
def judge(question: str, answer: str, gold_source: str,
*, timeout: int = 180, gold_cap: int = 6000) -> Verdict:
"""One hermetic blinded reference-grounded verdict. `answer` MUST
already be arm-blinded by the caller."""
already be arm-blinded by the caller.
Fail-closed: returns `JUDGE_ERROR` immediately unless
`ARBORIST_JUDGE_ENABLE=1` is exported (see module docstring &
`_JUDGE_ENABLE_ENV`). No `claude -p` subprocess is spawned in the
disabled path zero Opus tokens cost when the gate is closed."""
if os.environ.get(_JUDGE_ENABLE_ENV) != "1":
return Verdict("JUDGE_ERROR", _DISABLED_RATIONALE, "")
prompt = _PROMPT.format(
q=question.strip(),
gold=(gold_source or "").strip()[:gold_cap],