diff --git a/Makefile b/Makefile index 0fb6873..2cea8ed 100644 --- a/Makefile +++ b/Makefile @@ -37,7 +37,7 @@ SEARCH_Q ?= computer prometheus-trigger-probe bench-5f-threshold-calibration \ bench-5f-selfmodel-snapshot bench-5f-finetuning-shardchain \ bench-5f-falsification-hard bench-fork-baseline-hard bench-5f-formulate-hard \ - bootstrap-math bootstrap-nli bootstrap-nli-only bench-nli-shadow export-nli-onnx bench-nli-backends judge-self-test control-ab clean clean-db clean-data help \ + bootstrap-math bootstrap-nli bootstrap-nli-only bench-nli-shadow export-nli-onnx bench-nli-backends judge-self-test control-ab control-sweep clean clean-db clean-data help \ textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \ crawl-textbooks crawl-textbooks-stats textbook textbook-list @@ -233,6 +233,23 @@ control-ab: judge-self-test ## #000057 v1: Hermes-solo vs Arborist, blinded Opus --shards-dir $(SHARDS_DIR) \ --out-dir $(BENCH_QA_OUT) +# #000057 control-arm characterization sweep: model × question-framing +# matrix (fox: "measure all benchmarks, bring forward for review"; +# "qwen with and without reasoning"). NO judge-self-test make dep on +# purpose — control_sweep.py runs the gate ONCE in-script and ABORTS +# the run on failure (a stronger guarantee than a build-graph edge: +# it gates cell execution, not just the target, and avoids paying the +# 4-call Opus self-test twice per invocation). +# make control-sweep CONTROL_SWEEP_N=5 +CONTROL_SWEEP_FIXTURE ?= bench/qa_questions_stale_map.json +CONTROL_SWEEP_N ?= 3 +control-sweep: ## #000057: model×framing control sweep, review table [CONTROL_SWEEP_N=3 ...] + $(PY) bench/control_sweep.py \ + --fixture $(CONTROL_SWEEP_FIXTURE) \ + --n $(CONTROL_SWEEP_N) \ + --shards-dir $(SHARDS_DIR) \ + --out-dir $(BENCH_QA_OUT) + # Progressive-AND / DF-filter fixture: 9 questions chosen to exercise # the OR-fallback and progressive-AND drop paths. Use this for any # retrieval-side A/B (alternative search backends, synonym/rerank diff --git a/bench/control_sweep.py b/bench/control_sweep.py new file mode 100644 index 0000000..1e0d04e --- /dev/null +++ b/bench/control_sweep.py @@ -0,0 +1,347 @@ +#!/usr/bin/env python3 +"""#000057 — control-arm characterization sweep (fox: "we both do not +know which framing is right, measure all benchmarks and bring the +results forward for review"; "we have qwen with and without reasoning +to use"). + +This is NOT a verdict generator. It is the wide sweep the bench-maxing +discipline demands when a config/framing decision is genuinely open: +sweep model × question-framing, log everything, present the SAME judge +verdicts under BOTH narrative framings, and hand the table to review. + +WHY this sweep (the scientific question fox is probing): + Is "Hermes-solo confidently fabricates the current officeholder" a + real capability gap, or an artefact of (a) an 8B model being weak + and (b) an unfair *truth* framing (penalising a model for knowing + post-corpus reality)? The control arm answered across + {Hermes-3-8B, Qwen3.6-27B reasoning, Qwen3.6-27B no-reasoning} + × {plain, source-relative, as-of-corpus-era} questions, judged vs + the FIXED corpus-vintage gold, decides it empirically: + * if Qwen-27B-reasoning honestly ABSTAINS ("I cannot verify the + current officeholder") where Hermes-8B fabricates → the gap is + largely a weak-small-model artefact; the Arborist value claim + weakens and review must hear that; + * if even Qwen-27B-reasoning confidently asserts a post-corpus + answer → the gap is real and scale-independent (no bare + parametric model can be source-faithful) — the necessary- + substrate claim strengthens; + * the `as_of_corpus` variant ("As of 2010, who was…") separates + "model can't recall the corpus era" from "model won't constrain + to a source" — the exact ambiguity fox flagged. + +FRAMINGS, measured not chosen (all three from the AskUserQuestion): + * accuracy — WRONG+FABRICATED read as "model is wrong" + (the naive read fox correctly called unfair as a + *truth* claim — included precisely so review can + see why it misleads); + * grounding — the identical verdicts relabelled: WRONG/ + FABRICATED = "ungrounded confident assertion", + ABSTAINED = "honest about ungroundedness" (good), + CORRECT_GROUNDED = "matched the designated + source". Deterministic relabelling — zero extra + judge spend. + * faithfulness — judge each answer vs the context IT was given + (Arborist-arm-only, different judge call). Ticket + §4b scopes this v2/ext-ii; NOT run here (it is a + different instrument, not a relabelling) — listed + so the omission is explicit, not silent. + +Arborist arm: the Hermes treatment reference is run alongside for +direct same-N comparison. Arborist×Qwen needs a guided_json+extra_body +merge inside the proof path (runner.py) — a distinct, careful piece of +work; flagged in the report as the next step, deliberately NOT done +here (no proof-path surgery in a measurement harness). + +Gate: judge-self-test must be 4/4 before any cell is trusted (run +once up front here, not per-cell — the gate, not 9× the gate). + +Spend is bounded and printed up front; N is a parameter so review can +scale it. Hermetic Opus judge, blinded, fixed gold — same hygiene as +control_ab.py (whose helpers this reuses; DRY). +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import sys +import time +from collections import Counter +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) +from bench.control_ab import _SOLO_SYS, _descaffold, _gold # noqa: E402 +from bench.judge import judge, self_test # noqa: E402 + +# Model arms. `extra` merges into the chat payload root (the client +# forwards it); unrecognised keys are silently dropped server-side. +# qwen-think keeps reasoning ON (Qwen3.6 puts the clean answer in +# message.content and the chain-of-thought in a separate +# reasoning_content field the client does not surface — logged as a +# known limitation); its token budget is raised so the answer survives +# after the thinking span. qwen-nothink disables thinking via the +# chat-template kwarg probed live on the deployment. +MODELS: dict[str, dict] = { + "hermes": dict( + endpoint="https://hermes.ai.unturf.com/v1", + model="adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic", + extra=None, max_tokens=512), + "qwen-think": dict( + endpoint="https://qwen.ai.unturf.com/v1", + model="Qwen3.6-27B-UD-Q4_K_XL.gguf", + extra=None, max_tokens=1024), + "qwen-nothink": dict( + endpoint="https://qwen.ai.unturf.com/v1", + model="Qwen3.6-27B-UD-Q4_K_XL.gguf", + extra={"chat_template_kwargs": {"enable_thinking": False}}, + max_tokens=512), +} + + +def _lower_first(s: str) -> str: + return s[:1].lower() + s[1:] if s else s + + +def _v_plain(q: str) -> str: + return q + + +def _v_source_relative(q: str) -> str: + # The FAIR framing: explicitly ask for a source-grounded answer. + # A bare model has no such source — honoring this means abstaining + # ("I have no reference knowledge base"); still asserting a + # remembered name is ungrounded-confident-assertion, not an unfair + # "you're out of date" penalty. Both arms get the identical text. + return "According to the reference knowledge base, " + _lower_first(q) + + +def _v_as_of_corpus(q: str) -> str: + # Separates "can't recall the corpus era" from "won't constrain to + # a source": names the snapshot explicitly. If the model CAN say + # the 2010 holder when asked "as of 2010", the gap is snapshot + # ambiguity; if it still can't, parametric memory genuinely cannot + # serve the corpus era. (Corpus vintage ~2010-2011, verified from + # the artefact — see mine_questions.py; NOT the 2003 dump CLAUDE.md + # names.) + q2 = re.sub(r"^who is\b", "As of 2010, who was", q, flags=re.I) + return q2 if q2 != q else f"As of 2010: {q}" + + +VARIANTS = { + "plain": _v_plain, + "source_relative": _v_source_relative, + "as_of_corpus": _v_as_of_corpus, +} + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--fixture", + default="bench/qa_questions_stale_map.json") + ap.add_argument("--n", type=int, default=3) + ap.add_argument("--shards-dir", + default=str(Path.home() / ".arborist" / "shards")) + ap.add_argument("--models", default="hermes,qwen-think,qwen-nothink") + ap.add_argument("--variants", + default="plain,source_relative,as_of_corpus") + ap.add_argument("--arborist-ref", default="hermes", + help="model key for the Arborist treatment " + "reference column (proof path = Hermes; " + "Arborist×Qwen is the flagged follow-up)") + ap.add_argument("--out-dir", default="bench/qa_results") + ap.add_argument("--skip-self-test", action="store_true", + help="DANGER: bypasses the instrument gate; only " + "for offline harness development") + a = ap.parse_args() + + models = [m for m in a.models.split(",") if m in MODELS] + variants = [v for v in a.variants.split(",") if v in VARIANTS] + items = json.loads(Path(a.fixture).read_text())[:a.n] + + n_solo = len(models) * len(variants) * len(items) + n_arb = len(variants) * len(items) + print(f"#000057 control sweep — fixture={a.fixture} N={len(items)}") + print(f" models={models} variants={variants}") + print(f" bounded spend: {n_solo} solo LLM + {n_arb} Arborist " + f"query + ~{n_solo + n_arb} claude -p judge + 4 self-test") + + # The instrument gate — once, up front. Not 9× the gate. + if a.skip_self_test: + print(" !! self-test SKIPPED (--skip-self-test) — verdicts " + "are UNTRUSTED") + else: + print(" running judge self-test (instrument gate) …") + if self_test() != 0: + print("ABORT: judge unreliable — no cell may be trusted") + return 1 + + from arborist.qa.client import OpenAICompatibleClient + from arborist.qa.query import DEFAULT_QUERY_POLICY, query + + ts = time.strftime("%Y-%m-%dT%H-%M-%SZ", time.gmtime()) + outp = Path(a.out_dir) / f"control_sweep_{ts}.jsonl" + outp.parent.mkdir(parents=True, exist_ok=True) + shards_dir = Path(a.shards_dir) + qa_db = Path("/tmp") / f"control_sweep_{ts}.db" + + clients: dict[str, OpenAICompatibleClient] = {} + + def client_for(mkey: str) -> OpenAICompatibleClient: + if mkey not in clients: + clients[mkey] = OpenAICompatibleClient( + base_url=MODELS[mkey]["endpoint"], + api_key=os.environ.get("ARBORIST_LLM_API_KEY")) + return clients[mkey] + + # tally[(arm_label, variant)] -> Counter(verdict) + tally: dict[tuple, Counter] = {} + log = open(outp, "w") + + def record(arm, mkey, variant, i, q_orig, q_asked, answer, + verdict, judge_raw, audit_mode=None, eff=None): + key = (arm if arm == "solo" else f"arborist", mkey, variant) + tally.setdefault(key, Counter())[eff or verdict] += 1 + rec = {"arm": arm, "model": mkey, "variant": variant, "i": i, + "question_orig": q_orig, "question_asked": q_asked, + "answer": (answer or "")[:1500], "verdict": verdict, + "verdict_eff": eff or verdict, + "arb_audit_mode": audit_mode, + "judge_raw": (judge_raw or "")[:700]} + log.write(json.dumps(rec, ensure_ascii=False) + "\n") + log.flush() + + for i, it in enumerate(items, 1): + q0 = it["question"] + gold = _gold(shards_dir, it.get("shard", ""), it["target_root"]) + if not gold: + print(f" [{i}] SKIP (no gold): {q0[:50]}") + continue + for variant in variants: + q_asked = VARIANTS[variant](q0) + # ---- control arms: each model, solo ---- + for mkey in models: + cfg = MODELS[mkey] + cl = client_for(mkey) + try: + ans = cl.chat_completion( + [{"role": "system", "content": _SOLO_SYS}, + {"role": "user", "content": q_asked}], + model=cfg["model"], max_tokens=cfg["max_tokens"], + extra_body=cfg["extra"]) + except Exception as e: # noqa: BLE001 + ans = f"[solo-error: {type(e).__name__}: {e}]" + v = judge(q_asked, _descaffold(ans), gold) + record("solo", mkey, variant, i, q0, q_asked, ans, + v.label, v.raw) + print(f" [{i}] solo/{mkey}/{variant} -> {v.label}") + # ---- treatment reference: Arborist on the proof-path + # model (Hermes) ---- + ak = a.arborist_ref + try: + r = query(question=q_asked, qa_db=qa_db, + chat_client=client_for(ak), + model_id=MODELS[ak]["model"], + shards_dir=shards_dir, + policy=dict(DEFAULT_QUERY_POLICY, + answer_mode="claim_lattice")) + arb_raw = r.get("raw_answer") or r.get("answer_text") or "" + arb_mode = r.get("audit_mode") + except Exception as e: # noqa: BLE001 + arb_raw = f"[arborist-error: {type(e).__name__}: {e}]" + arb_mode = "ERROR" + va = judge(q_asked, _descaffold(arb_raw), gold) + eff = ("ABSTAINED" if arb_mode == "UNGROUNDED" + and va.label in ("WRONG", "FABRICATED") + else va.label) + record("arborist", ak, variant, i, q0, q_asked, arb_raw, + va.label, va.raw, audit_mode=arb_mode, eff=eff) + print(f" [{i}] arborist/{ak}/{variant} -> {va.label}" + f" (mode={arb_mode}, eff={eff})") + + log.close() + for c in clients.values(): + try: + c.close() + except Exception: # noqa: BLE001 + pass + + md = _report(tally, items, models, variants, a, outp, ts) + mdp = Path(a.out_dir) / f"control_sweep_{ts}.md" + mdp.write_text(md) + print("\n" + md) + print(f"\nreview deliverables:\n {outp}\n {mdp}") + return 0 + + +def _bucket(c: Counter) -> str: + tot = sum(c.values()) or 1 + return (f"CG={c['CORRECT_GROUNDED']} W={c['WRONG']} " + f"F={c['FABRICATED']} A={c['ABSTAINED']} " + f"JE={c['JUDGE_ERROR']} (n={tot})") + + +def _report(tally, items, models, variants, a, outp, ts) -> str: + n = len(items) + L = [f"# #000057 control sweep — {ts}", + "", + f"Fixture `{a.fixture}` · N={n} · models={models} · " + f"variants={variants}", + f"Raw per-item JSONL: `{outp}`", + "", + "**This is review input, not a verdict.** fox: measure all " + "benchmarks, bring forward for review.", + "", + "Verdict codes: CG=CORRECT_GROUNDED W=WRONG F=FABRICATED " + "A=ABSTAINED JE=JUDGE_ERROR. Gold = fixed corpus-vintage " + "article (~2010-2011, verified from the artefact, NOT the " + "2003 dump CLAUDE.md names). Judge = hermetic blinded Opus.", + "", + "## Raw verdicts (framing-neutral)", + "", + "| arm | model | variant | verdicts |", + "|-----|-------|---------|----------|"] + for (arm, mkey, variant), c in sorted(tally.items()): + L.append(f"| {arm} | {mkey} | {variant} | {_bucket(c)} |") + L += ["", + "## Framing A — accuracy (the naive read; shown so review " + "sees why it misleads)", + "", + "WRONG+FABRICATED counted as \"model is wrong\". This " + "penalises a model for knowing post-corpus reality " + "(\"Macron\" is world-true); fox correctly flagged it is " + "NOT a fair *truth* claim. Reported only for contrast.", + "", + "## Framing B — grounding-fidelity (the defensible read)", + "", + "Identical verdicts, relabelled: CG = matched the " + "designated source; A = honest about ungroundedness " + "(GOOD — the capability bare models lack); W+F = " + "ungrounded confident assertion (the failure that is " + "structurally impossible to avoid without a retrieval+" + "verify substrate). The headline question for review: does " + "a 27B *reasoning* control ABSTAIN where the 8B fabricates?", + "", + "## Framing C — faithfulness ablation (NOT run here)", + "", + "Judge each answer vs the context IT was given (Arborist-" + "arm-only, a different judge call, not a relabelling). " + "Ticket §4b scopes it v2/ext-ii. Listed so the omission is " + "explicit. Arborist×Qwen (proof-path guided_json+extra_body " + "merge) is the coupled follow-up — also deliberately not " + "done in a measurement harness.", + "", + "## Open for fox / review", + "", + "- bump `--n` (smallest-proof N here; denominators tiny);", + "- Arborist×Qwen arm (proof-path surgery — separate);", + "- which framing becomes the whitepaper headline;", + "- Qwen reasoning_content is not logged (client doesn't " + "surface it) — add if review wants the CoT audited."] + return "\n".join(L) + "\n" + + +if __name__ == "__main__": + raise SystemExit(main())