engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(#000057): hermetic external judge instrument — built + verified 4/4 (make judge-self-test)

Browse source 
fox ruled judge = Opus via `claude -p`. bench/judge.py:
hermetic (`env -u CLAUDECODE claude -p`, fresh process, context =
only (Q, answer, gold) — no arm label, no Arborist context, no
session), blinded-by-caller, reference-grounded against the fixed
gold (ignore parametric knowledge), structured via FINAL_VERDICT=
sentinel parsed LAST-match.

Instrument-before-experiment gate worked: first cut parsed
first-match over the model's chain-of-thought → 0/3 self-test. The
judge REASONED correctly; the parser was the defect (+ two bad test
fixtures, my error). Hardened (sentinel contract + fixed fixtures),
re-verified: `make judge-self-test` = 4/4 on known-verdict triples
via real claude -p. The make target is the precondition gate; no
control run trusts the judge until it passes.

Threat to validity recorded, not hidden: same model family judging;
mitigated (blind + no-stake + reference-grounded) not eliminated —
different-family SOTA cross-check is the only full removal.

Next: bench/control_ab.py + `make control-ab` (Hermes-solo vs
Arborist, gold=target-article text, blinded, judged) — NOT yet
built; no broken make target shipped for it.
This commit is contained in:
russell@unturf.com 2026-05-19 08:44:47 -04:00
parent fa81b97c5c
commit 65fd9fad5d
No known key found for this signature in database
3 changed files with 211 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

28
docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md
View file

@ -154,6 +154,34 @@ to what it was given). (ii) is Arborist-arm-only and measures
faithfulness, not correctness-vs-truth — explicitly labelled so it
is never read as the headline.
## 4c. Judge instrument — BUILT + VERIFIED (2026-05-19)
fox ruled the judge = Opus via `claude -p`. `bench/judge.py` built
to the §4b hygiene: hermetic (`env -u CLAUDECODE claude -p`, fresh
process, context = only (Q, answer, gold)), blinded-by-caller,
reference-grounded, **structured via a `FINAL_VERDICT=<TOKEN>`
sentinel parsed last-match** (the first cut parsed first-match over
the model's chain-of-thought → a 0/3 self-test; the judge reasoned
correctly, the *parser* was the defect — instrument-before-
experiment gate working). Hardened + re-verified: `make
judge-self-test` = **4/4** on known-verdict triples
(CORRECT_GROUNDED / WRONG / FABRICATED / ABSTAINED) via real
`claude -p`. This make target IS the precondition gate; no control
run may trust the judge until it passes.
Threat to validity (recorded, not hidden): same model *family*
judging (this repo's agent is also Opus); blinding + no-stake +
reference-grounding mitigate, do not eliminate — a different-family
SOTA cross-check is the only full removal; stated as a known limit.
**Next (the experiment, on the now-trusted instrument):**
`bench/control_ab.py` + `make control-ab` — Hermes-solo arm
(`OpenAICompatibleClient`, question only, no Arborist) vs Arborist
arm (`query()`), gold = target-article text fetched by
`target_root`, blinded+shuffled, judged by `bench/judge.py`,
deterministic aggregation → fabrication-vs-abstention delta. Not yet
built; no broken make target shipped for it.
## 5. Decision status & the smallest-proof decomposition
fox's control-arm framing (§4b) reorders the experiment honestly: