engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(#000057): control experiment harness — Hermes-solo vs Arborist, blinded Opus judge (smoke-verified)

Browse source 
bench/control_ab.py + `make control-ab` (gated on judge-self-test
as a make dependency — instrument gate cannot be skipped). Same
model both arms; gold = target-article text by target_root; Arborist
[E…] scaffolding stripped (blinding — format can't betray the arm);
Arborist UNGROUNDED credited as honest abstention; hermetic Opus
judge; deterministic aggregate; self-auditing JSONL; threats-to-
validity printed in the report.

N=2 smoke: clean end-to-end, 0 JUDGE_ERROR — and already surfaced a
case AGAINST the treatment (solo correctly ABSTAINED; Arborist
HYBRID-WRONG). The instrument can falsify the Arborist value claim;
that is the point. n=2 proves nothing (report says so) — verdict
needs a real N.
This commit is contained in:
russell@unturf.com 2026-05-19 08:58:38 -04:00
parent 65fd9fad5d
commit 1356459091
No known key found for this signature in database
3 changed files with 254 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

16
Makefile
View file

@ -37,7 +37,7 @@ SEARCH_Q ?= computer
prometheus-trigger-probe bench-5f-threshold-calibration \
bench-5f-selfmodel-snapshot bench-5f-finetuning-shardchain \
bench-5f-falsification-hard bench-fork-baseline-hard bench-5f-formulate-hard \
bootstrap-math bootstrap-nli bootstrap-nli-only bench-nli-shadow export-nli-onnx judge-self-test clean clean-db clean-data help \
bootstrap-math bootstrap-nli bootstrap-nli-only bench-nli-shadow export-nli-onnx judge-self-test control-ab clean clean-db clean-data help \
textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \
crawl-textbooks crawl-textbooks-stats textbook textbook-list
@ -219,6 +219,20 @@ bench-qa-smoke: bootstrap ## quick 5-question smoke (all anchor classes; ~30s)
judge-self-test: ## verify the #000057 external judge on known-verdict triples (gate before control-ab)
$(PY) bench/judge.py --self-test
# #000057 v1 control experiment: Hermes-solo vs Arborist, same model,
# blinded Opus judge vs fixed gold. Depends on judge-self-test so the
# instrument-before-experiment gate CANNOT be skipped (make enforces
# it). Bounded spend: N items -> 2N Hermes + 2N `claude -p`.
# make control-ab N=12 FIXTURE=bench/qa_questions_numeral_map.json
CONTROL_AB_FIXTURE ?= bench/qa_questions_numeral_map.json
CONTROL_AB_N ?= 12
control-ab: judge-self-test ## #000057 v1: Hermes-solo vs Arborist, blinded Opus judge [N=12 FIXTURE=...]
$(PY) bench/control_ab.py \
--fixture $(CONTROL_AB_FIXTURE) \
--n $(CONTROL_AB_N) \
--shards-dir $(SHARDS_DIR) \
--out-dir $(BENCH_QA_OUT)
# Progressive-AND / DF-filter fixture: 9 questions chosen to exercise
# the OR-fallback and progressive-AND drop paths. Use this for any
# retrieval-side A/B (alternative search backends, synonym/rerank