Was a stale hardcoded 'same Hermes; judge=Opus hermetic' label that
misreported any run with --model/--judge overrides (e.g. qwen + code
judge). Now reflects the real config — honest header for the artifact.
fox 2026-05-21: characterize substrate-ON under BOTH answer shapes, so
answer_mode is a swept axis, not a single pinned value.
stock_v1.py now exposes STOCK_V1_POLICIES{quote,claim_lattice} +
STOCK_V1_GOVERNANCE_HASHES (quote 5b6ca4c5..., claim_lattice 036a4c79...),
policy_for(mode), and assert_not_drifted(mode). Shared pins (crosslang
OFF, repair OFF, quantifier dry-run, metacognition label-only,
soft-preflight OFF, claim cap 12, v2-acronym-aware) are frozen
identically across modes.
Wire the treatment arms to the pin (the consumer-side step that makes
the freeze real):
* control_ab --answer-mode {quote,claim_lattice}
* control_sweep --arborist-answer-mode {quote,claim_lattice}
Both default claim_lattice (prior behavior), call assert_not_drifted on
non-reasoning runs (halts the sweep if DEFAULT_QUERY_POLICY drifts), and
load the frozen policy_for(mode) instead of an inline
dict(DEFAULT_QUERY_POLICY, ...). Reasoning refs (phase 3) keep their
documented JSON overrides and skip the assert by design (different hash).
jaggedness is left standalone — it is a mode-agnostic retrieval
instrument, coupling it to the answer-policy freeze adds friction with no
correctness gain. Full suite 2528 passed.
Wire bench/judge_code.py into the sweep harnesses as the default judge.
Both control_sweep.py and control_ab.py grow a --judge {code,opus} CLI
arg; both share the same Verdict shape so the dispatch is a pointer
assignment + threading the judge_fn through _process_item.
Behaviour:
- DEFAULT = code: zero LLM, zero quota, deterministic. Self-test gate
is the code judge's 4-fixture contract.
- --judge opus: original gated Opus path; needs ARBORIST_JUDGE_ENABLE=1
set per 1cabfe6's fail-closed gate, otherwise every record returns
JUDGE_ERROR with rationale 'disabled — set ARBORIST_JUDGE_ENABLE=1'
and the sweep records that label honestly.
Reporting:
- Header line now records which judge ran ('Judge = code (...)' or
'Judge = opus (...)') so partial-reports & resumes don't lie about
provenance.
- Spend banner shows '0 LLM calls' for the code path so the no-burn
property is visible in the operator output.
Test surface: pytest sweep across tests/ still 136/136 (no regressions);
new --judge flag visible in --help on both harnesses.
Next: bench/score_with_code_judge.py to re-grade existing sweep JSONLs
(written under the gated-Opus run) with the code judge; agreement
matrix surfaces residue size for the eventual LLM-batch needle-haystack.
control_ab.py called client.chat_completion(messages) but model is a
keyword-only REQUIRED arg → TypeError every call → solo answer was
always '[solo-error: TypeError]', which the hermetic judge correctly
classified ABSTAINED. Every prior 'Hermes-solo abstained' smoke
(incl. the n=2 the ticket cites as a case against the treatment) was
a broken-control-arm artefact, not a measurement. Fix: pass
model=a.model (symmetry with the Arborist arm). Error token now
carries the exception message so the next failure self-describes.
Also adds the #000057 'stale' fabrication-bait mining class
(time-varying executive offices; corpus = ~2010-2011 Wikipedia
VERIFIED from the artifact, NOT the 2003 dump CLAUDE.md names —
flagged) and restores the numeral fixture deterministically.
bench/control_ab.py + `make control-ab` (gated on judge-self-test
as a make dependency — instrument gate cannot be skipped). Same
model both arms; gold = target-article text by target_root; Arborist
[E…] scaffolding stripped (blinding — format can't betray the arm);
Arborist UNGROUNDED credited as honest abstention; hermetic Opus
judge; deterministic aggregate; self-auditing JSONL; threats-to-
validity printed in the report.
N=2 smoke: clean end-to-end, 0 JUDGE_ERROR — and already surfaced a
case AGAINST the treatment (solo correctly ABSTAINED; Arborist
HYBRID-WRONG). The instrument can falsify the Arborist value claim;
that is the point. n=2 proves nothing (report says so) — verdict
needs a real N.
