feat(#000057): control-arm characterization sweep — model × framing

fox ruling: 'we both do not know which framing is right, measure all
benchmarks and bring results forward for review'; 'we have qwen with
and without reasoning to use'.

bench/control_sweep.py: sweeps the CONTROL (solo) arm across
{Hermes-3-8B, Qwen3.6-27B reasoning, Qwen3.6-27B no-reasoning} ×
{plain, source-relative, as-of-corpus-era} questions, judged vs the
fixed corpus-vintage gold; Arborist-Hermes treatment reference run
alongside at the same N. Presents the SAME judge verdicts under both
the accuracy framing (the naive read fox flagged as unfair-as-truth,
shown for contrast) and the grounding-fidelity framing (the
defensible read), plus an explicit note on the faithfulness-ablation
framing + Arborist×Qwen — both deliberately NOT run (proof-path
surgery / different instrument, not a relabelling).

Answers fox's open question empirically: does a 27B *reasoning*
control honestly ABSTAIN where the 8B fabricates (gap = weak-small-
model artefact) or still confidently assert post-corpus (gap = real,
scale-independent)? as_of_corpus separates 'can't recall the era'
from 'won't constrain to a source'.

Qwen toggles probed live: reasoning answer in message.content (CoT in
separate reasoning_content, not surfaced — logged limitation);
no-reasoning via chat_template_kwargs{enable_thinking:false}. Gate =
in-script judge self_test (aborts on fail) — stronger than a make
edge, no double self-test spend.

Makefile

@@ -37,7 +37,7 @@ SEARCH_Q     ?= computer
         prometheus-trigger-probe bench-5f-threshold-calibration \
         bench-5f-selfmodel-snapshot bench-5f-finetuning-shardchain \
         bench-5f-falsification-hard bench-fork-baseline-hard bench-5f-formulate-hard \
-        bootstrap-math bootstrap-nli bootstrap-nli-only bench-nli-shadow export-nli-onnx bench-nli-backends judge-self-test control-ab clean clean-db clean-data help \
+        bootstrap-math bootstrap-nli bootstrap-nli-only bench-nli-shadow export-nli-onnx bench-nli-backends judge-self-test control-ab control-sweep clean clean-db clean-data help \
         textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \
         crawl-textbooks crawl-textbooks-stats textbook textbook-list
 
@@ -233,6 +233,23 @@ control-ab: judge-self-test ## #000057 v1: Hermes-solo vs Arborist, blinded Opus
 	    --shards-dir $(SHARDS_DIR) \
 	    --out-dir $(BENCH_QA_OUT)
 
+# #000057 control-arm characterization sweep: model × question-framing
+# matrix (fox: "measure all benchmarks, bring forward for review";
+# "qwen with and without reasoning"). NO judge-self-test make dep on
+# purpose — control_sweep.py runs the gate ONCE in-script and ABORTS
+# the run on failure (a stronger guarantee than a build-graph edge:
+# it gates cell execution, not just the target, and avoids paying the
+# 4-call Opus self-test twice per invocation).
+#   make control-sweep CONTROL_SWEEP_N=5
+CONTROL_SWEEP_FIXTURE ?= bench/qa_questions_stale_map.json
+CONTROL_SWEEP_N       ?= 3
+control-sweep: ## #000057: model×framing control sweep, review table [CONTROL_SWEEP_N=3 ...]
+	$(PY) bench/control_sweep.py \
+	    --fixture $(CONTROL_SWEEP_FIXTURE) \
+	    --n $(CONTROL_SWEEP_N) \
+	    --shards-dir $(SHARDS_DIR) \
+	    --out-dir $(BENCH_QA_OUT)
+
 # Progressive-AND / DF-filter fixture: 9 questions chosen to exercise
 # the OR-fallback and progressive-AND drop paths. Use this for any
 # retrieval-side A/B (alternative search backends, synonym/rerank

bench/control_sweep.py

@@ -0,0 +1,347 @@
+#!/usr/bin/env python3
+"""#000057 — control-arm characterization sweep (fox: "we both do not
+know which framing is right, measure all benchmarks and bring the
+results forward for review"; "we have qwen with and without reasoning
+to use").
+
+This is NOT a verdict generator. It is the wide sweep the bench-maxing
+discipline demands when a config/framing decision is genuinely open:
+sweep model × question-framing, log everything, present the SAME judge
+verdicts under BOTH narrative framings, and hand the table to review.
+
+WHY this sweep (the scientific question fox is probing):
+  Is "Hermes-solo confidently fabricates the current officeholder" a
+  real capability gap, or an artefact of (a) an 8B model being weak
+  and (b) an unfair *truth* framing (penalising a model for knowing
+  post-corpus reality)?  The control arm answered across
+    {Hermes-3-8B, Qwen3.6-27B reasoning, Qwen3.6-27B no-reasoning}
+  × {plain, source-relative, as-of-corpus-era} questions, judged vs
+  the FIXED corpus-vintage gold, decides it empirically:
+    * if Qwen-27B-reasoning honestly ABSTAINS ("I cannot verify the
+      current officeholder") where Hermes-8B fabricates → the gap is
+      largely a weak-small-model artefact; the Arborist value claim
+      weakens and review must hear that;
+    * if even Qwen-27B-reasoning confidently asserts a post-corpus
+      answer → the gap is real and scale-independent (no bare
+      parametric model can be source-faithful) — the necessary-
+      substrate claim strengthens;
+    * the `as_of_corpus` variant ("As of 2010, who was…") separates
+      "model can't recall the corpus era" from "model won't constrain
+      to a source" — the exact ambiguity fox flagged.
+
+FRAMINGS, measured not chosen (all three from the AskUserQuestion):
+  * accuracy        — WRONG+FABRICATED read as "model is wrong"
+                      (the naive read fox correctly called unfair as a
+                      *truth* claim — included precisely so review can
+                      see why it misleads);
+  * grounding       — the identical verdicts relabelled: WRONG/
+                      FABRICATED = "ungrounded confident assertion",
+                      ABSTAINED = "honest about ungroundedness" (good),
+                      CORRECT_GROUNDED = "matched the designated
+                      source". Deterministic relabelling — zero extra
+                      judge spend.
+  * faithfulness    — judge each answer vs the context IT was given
+                      (Arborist-arm-only, different judge call). Ticket
+                      §4b scopes this v2/ext-ii; NOT run here (it is a
+                      different instrument, not a relabelling) — listed
+                      so the omission is explicit, not silent.
+
+Arborist arm: the Hermes treatment reference is run alongside for
+direct same-N comparison. Arborist×Qwen needs a guided_json+extra_body
+merge inside the proof path (runner.py) — a distinct, careful piece of
+work; flagged in the report as the next step, deliberately NOT done
+here (no proof-path surgery in a measurement harness).
+
+Gate: judge-self-test must be 4/4 before any cell is trusted (run
+once up front here, not per-cell — the gate, not 9× the gate).
+
+Spend is bounded and printed up front; N is a parameter so review can
+scale it. Hermetic Opus judge, blinded, fixed gold — same hygiene as
+control_ab.py (whose helpers this reuses; DRY).
+"""
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import sys
+import time
+from collections import Counter
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+from bench.control_ab import _SOLO_SYS, _descaffold, _gold  # noqa: E402
+from bench.judge import judge, self_test  # noqa: E402
+
+# Model arms. `extra` merges into the chat payload root (the client
+# forwards it); unrecognised keys are silently dropped server-side.
+# qwen-think keeps reasoning ON (Qwen3.6 puts the clean answer in
+# message.content and the chain-of-thought in a separate
+# reasoning_content field the client does not surface — logged as a
+# known limitation); its token budget is raised so the answer survives
+# after the thinking span. qwen-nothink disables thinking via the
+# chat-template kwarg probed live on the deployment.
+MODELS: dict[str, dict] = {
+    "hermes": dict(
+        endpoint="https://hermes.ai.unturf.com/v1",
+        model="adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic",
+        extra=None, max_tokens=512),
+    "qwen-think": dict(
+        endpoint="https://qwen.ai.unturf.com/v1",
+        model="Qwen3.6-27B-UD-Q4_K_XL.gguf",
+        extra=None, max_tokens=1024),
+    "qwen-nothink": dict(
+        endpoint="https://qwen.ai.unturf.com/v1",
+        model="Qwen3.6-27B-UD-Q4_K_XL.gguf",
+        extra={"chat_template_kwargs": {"enable_thinking": False}},
+        max_tokens=512),
+}
+
+
+def _lower_first(s: str) -> str:
+    return s[:1].lower() + s[1:] if s else s
+
+
+def _v_plain(q: str) -> str:
+    return q
+
+
+def _v_source_relative(q: str) -> str:
+    # The FAIR framing: explicitly ask for a source-grounded answer.
+    # A bare model has no such source — honoring this means abstaining
+    # ("I have no reference knowledge base"); still asserting a
+    # remembered name is ungrounded-confident-assertion, not an unfair
+    # "you're out of date" penalty. Both arms get the identical text.
+    return "According to the reference knowledge base, " + _lower_first(q)
+
+
+def _v_as_of_corpus(q: str) -> str:
+    # Separates "can't recall the corpus era" from "won't constrain to
+    # a source": names the snapshot explicitly. If the model CAN say
+    # the 2010 holder when asked "as of 2010", the gap is snapshot
+    # ambiguity; if it still can't, parametric memory genuinely cannot
+    # serve the corpus era. (Corpus vintage ~2010-2011, verified from
+    # the artefact — see mine_questions.py; NOT the 2003 dump CLAUDE.md
+    # names.)
+    q2 = re.sub(r"^who is\b", "As of 2010, who was", q, flags=re.I)
+    return q2 if q2 != q else f"As of 2010: {q}"
+
+
+VARIANTS = {
+    "plain": _v_plain,
+    "source_relative": _v_source_relative,
+    "as_of_corpus": _v_as_of_corpus,
+}
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--fixture",
+                    default="bench/qa_questions_stale_map.json")
+    ap.add_argument("--n", type=int, default=3)
+    ap.add_argument("--shards-dir",
+                    default=str(Path.home() / ".arborist" / "shards"))
+    ap.add_argument("--models", default="hermes,qwen-think,qwen-nothink")
+    ap.add_argument("--variants",
+                    default="plain,source_relative,as_of_corpus")
+    ap.add_argument("--arborist-ref", default="hermes",
+                    help="model key for the Arborist treatment "
+                         "reference column (proof path = Hermes; "
+                         "Arborist×Qwen is the flagged follow-up)")
+    ap.add_argument("--out-dir", default="bench/qa_results")
+    ap.add_argument("--skip-self-test", action="store_true",
+                    help="DANGER: bypasses the instrument gate; only "
+                         "for offline harness development")
+    a = ap.parse_args()
+
+    models = [m for m in a.models.split(",") if m in MODELS]
+    variants = [v for v in a.variants.split(",") if v in VARIANTS]
+    items = json.loads(Path(a.fixture).read_text())[:a.n]
+
+    n_solo = len(models) * len(variants) * len(items)
+    n_arb = len(variants) * len(items)
+    print(f"#000057 control sweep — fixture={a.fixture} N={len(items)}")
+    print(f"  models={models}  variants={variants}")
+    print(f"  bounded spend: {n_solo} solo LLM + {n_arb} Arborist "
+          f"query + ~{n_solo + n_arb} claude -p judge + 4 self-test")
+
+    # The instrument gate — once, up front. Not 9× the gate.
+    if a.skip_self_test:
+        print("  !! self-test SKIPPED (--skip-self-test) — verdicts "
+              "are UNTRUSTED")
+    else:
+        print("  running judge self-test (instrument gate) …")
+        if self_test() != 0:
+            print("ABORT: judge unreliable — no cell may be trusted")
+            return 1
+
+    from arborist.qa.client import OpenAICompatibleClient
+    from arborist.qa.query import DEFAULT_QUERY_POLICY, query
+
+    ts = time.strftime("%Y-%m-%dT%H-%M-%SZ", time.gmtime())
+    outp = Path(a.out_dir) / f"control_sweep_{ts}.jsonl"
+    outp.parent.mkdir(parents=True, exist_ok=True)
+    shards_dir = Path(a.shards_dir)
+    qa_db = Path("/tmp") / f"control_sweep_{ts}.db"
+
+    clients: dict[str, OpenAICompatibleClient] = {}
+
+    def client_for(mkey: str) -> OpenAICompatibleClient:
+        if mkey not in clients:
+            clients[mkey] = OpenAICompatibleClient(
+                base_url=MODELS[mkey]["endpoint"],
+                api_key=os.environ.get("ARBORIST_LLM_API_KEY"))
+        return clients[mkey]
+
+    # tally[(arm_label, variant)] -> Counter(verdict)
+    tally: dict[tuple, Counter] = {}
+    log = open(outp, "w")
+
+    def record(arm, mkey, variant, i, q_orig, q_asked, answer,
+               verdict, judge_raw, audit_mode=None, eff=None):
+        key = (arm if arm == "solo" else f"arborist", mkey, variant)
+        tally.setdefault(key, Counter())[eff or verdict] += 1
+        rec = {"arm": arm, "model": mkey, "variant": variant, "i": i,
+               "question_orig": q_orig, "question_asked": q_asked,
+               "answer": (answer or "")[:1500], "verdict": verdict,
+               "verdict_eff": eff or verdict,
+               "arb_audit_mode": audit_mode,
+               "judge_raw": (judge_raw or "")[:700]}
+        log.write(json.dumps(rec, ensure_ascii=False) + "\n")
+        log.flush()
+
+    for i, it in enumerate(items, 1):
+        q0 = it["question"]
+        gold = _gold(shards_dir, it.get("shard", ""), it["target_root"])
+        if not gold:
+            print(f"  [{i}] SKIP (no gold): {q0[:50]}")
+            continue
+        for variant in variants:
+            q_asked = VARIANTS[variant](q0)
+            # ---- control arms: each model, solo ----
+            for mkey in models:
+                cfg = MODELS[mkey]
+                cl = client_for(mkey)
+                try:
+                    ans = cl.chat_completion(
+                        [{"role": "system", "content": _SOLO_SYS},
+                         {"role": "user", "content": q_asked}],
+                        model=cfg["model"], max_tokens=cfg["max_tokens"],
+                        extra_body=cfg["extra"])
+                except Exception as e:  # noqa: BLE001
+                    ans = f"[solo-error: {type(e).__name__}: {e}]"
+                v = judge(q_asked, _descaffold(ans), gold)
+                record("solo", mkey, variant, i, q0, q_asked, ans,
+                       v.label, v.raw)
+                print(f"  [{i}] solo/{mkey}/{variant} -> {v.label}")
+            # ---- treatment reference: Arborist on the proof-path
+            #      model (Hermes) ----
+            ak = a.arborist_ref
+            try:
+                r = query(question=q_asked, qa_db=qa_db,
+                          chat_client=client_for(ak),
+                          model_id=MODELS[ak]["model"],
+                          shards_dir=shards_dir,
+                          policy=dict(DEFAULT_QUERY_POLICY,
+                                      answer_mode="claim_lattice"))
+                arb_raw = r.get("raw_answer") or r.get("answer_text") or ""
+                arb_mode = r.get("audit_mode")
+            except Exception as e:  # noqa: BLE001
+                arb_raw = f"[arborist-error: {type(e).__name__}: {e}]"
+                arb_mode = "ERROR"
+            va = judge(q_asked, _descaffold(arb_raw), gold)
+            eff = ("ABSTAINED" if arb_mode == "UNGROUNDED"
+                   and va.label in ("WRONG", "FABRICATED")
+                   else va.label)
+            record("arborist", ak, variant, i, q0, q_asked, arb_raw,
+                   va.label, va.raw, audit_mode=arb_mode, eff=eff)
+            print(f"  [{i}] arborist/{ak}/{variant} -> {va.label}"
+                  f" (mode={arb_mode}, eff={eff})")
+
+    log.close()
+    for c in clients.values():
+        try:
+            c.close()
+        except Exception:  # noqa: BLE001
+            pass
+
+    md = _report(tally, items, models, variants, a, outp, ts)
+    mdp = Path(a.out_dir) / f"control_sweep_{ts}.md"
+    mdp.write_text(md)
+    print("\n" + md)
+    print(f"\nreview deliverables:\n  {outp}\n  {mdp}")
+    return 0
+
+
+def _bucket(c: Counter) -> str:
+    tot = sum(c.values()) or 1
+    return (f"CG={c['CORRECT_GROUNDED']} W={c['WRONG']} "
+            f"F={c['FABRICATED']} A={c['ABSTAINED']} "
+            f"JE={c['JUDGE_ERROR']}  (n={tot})")
+
+
+def _report(tally, items, models, variants, a, outp, ts) -> str:
+    n = len(items)
+    L = [f"# #000057 control sweep — {ts}",
+         "",
+         f"Fixture `{a.fixture}` · N={n} · models={models} · "
+         f"variants={variants}",
+         f"Raw per-item JSONL: `{outp}`",
+         "",
+         "**This is review input, not a verdict.** fox: measure all "
+         "benchmarks, bring forward for review.",
+         "",
+         "Verdict codes: CG=CORRECT_GROUNDED W=WRONG F=FABRICATED "
+         "A=ABSTAINED JE=JUDGE_ERROR. Gold = fixed corpus-vintage "
+         "article (~2010-2011, verified from the artefact, NOT the "
+         "2003 dump CLAUDE.md names). Judge = hermetic blinded Opus.",
+         "",
+         "## Raw verdicts (framing-neutral)",
+         "",
+         "| arm | model | variant | verdicts |",
+         "|-----|-------|---------|----------|"]
+    for (arm, mkey, variant), c in sorted(tally.items()):
+        L.append(f"| {arm} | {mkey} | {variant} | {_bucket(c)} |")
+    L += ["",
+          "## Framing A — accuracy (the naive read; shown so review "
+          "sees why it misleads)",
+          "",
+          "WRONG+FABRICATED counted as \"model is wrong\". This "
+          "penalises a model for knowing post-corpus reality "
+          "(\"Macron\" is world-true); fox correctly flagged it is "
+          "NOT a fair *truth* claim. Reported only for contrast.",
+          "",
+          "## Framing B — grounding-fidelity (the defensible read)",
+          "",
+          "Identical verdicts, relabelled: CG = matched the "
+          "designated source; A = honest about ungroundedness "
+          "(GOOD — the capability bare models lack); W+F = "
+          "ungrounded confident assertion (the failure that is "
+          "structurally impossible to avoid without a retrieval+"
+          "verify substrate). The headline question for review: does "
+          "a 27B *reasoning* control ABSTAIN where the 8B fabricates?",
+          "",
+          "## Framing C — faithfulness ablation (NOT run here)",
+          "",
+          "Judge each answer vs the context IT was given (Arborist-"
+          "arm-only, a different judge call, not a relabelling). "
+          "Ticket §4b scopes it v2/ext-ii. Listed so the omission is "
+          "explicit. Arborist×Qwen (proof-path guided_json+extra_body "
+          "merge) is the coupled follow-up — also deliberately not "
+          "done in a measurement harness.",
+          "",
+          "## Open for fox / review",
+          "",
+          "- bump `--n` (smallest-proof N here; denominators tiny);",
+          "- Arborist×Qwen arm (proof-path surgery — separate);",
+          "- which framing becomes the whitepaper headline;",
+          "- Qwen reasoning_content is not logged (client doesn't "
+          "surface it) — add if review wants the CoT audited."]
+    return "\n".join(L) + "\n"
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())