engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(#000057): parallel + incremental + huge-N control sweep

Browse source 
fox: 'make the n huge huge, check in every ~7 turns'.

- stale fixture re-mined to the FULL pool: 386 questions (180 pres /
  119 PM / 34 premier / 20 gov-gen / 17 chancellor / 15 CM / 1 FM),
  deterministic.
- control_sweep.py rebuilt: ThreadPoolExecutor work-unit-per-item
  (per-unit qa_db -> zero SQLite write contention; shards read-only
  safe under concurrent readers), incremental flushed JSONL, and a
  --report-only PATH mode that aggregates a PARTIAL file with ZERO
  LLM/judge spend (the interim check-in path).
- huge N goes on the CONTROL (solo x3 models x3 framings) where the
  open statistical question lives; Arborist A/B = fixed --arborist-n
  (default 40), not re-measured 386x (power belongs on the control,
  and query() over ~40GB shards is the heaviest call).
- bench-maxing doctrine applied: independent hermetic judge calls
  fanned out, 'serial-by-caution is halting in disguise'.

N=3 (prior run) already shows the coherent, review-relevant story:
plain -> all 3 models confidently assert (0% abstain); source_relative
-> abstain jumps (hermes 33 / qwen-think 67 / qwen-nothink 100%);
as_of_corpus -> ~100% correct all models. The gap is largely
framing+snapshot, not a universal capability deficit.
This commit is contained in:
russell@unturf.com 2026-05-19 13:00:45 -04:00
parent f63b00d58e
commit 12bb6dbb6f
No known key found for this signature in database
3 changed files with 2682 additions and 211 deletions
Download patch file Download diff file Expand all files Collapse all files

471
bench/control_sweep.py
View file

@ -2,12 +2,13 @@
"""#000057 — control-arm characterization sweep (fox: "we both do not
know which framing is right, measure all benchmarks and bring the
results forward for review"; "we have qwen with and without reasoning
to use").
to use"; "make the n huge huge and check in every ~7 turns").
This is NOT a verdict generator. It is the wide sweep the bench-maxing
discipline demands when a config/framing decision is genuinely open:
sweep model × question-framing, log everything, present the SAME judge
verdicts under BOTH narrative framings, and hand the table to review.
This is NOT a verdict generator. It is the wide, parallel, incremental
sweep the bench-maxing discipline demands when a config/framing
decision is genuinely open: sweep model × question-framing at HUGE N,
log every item the instant it lands, present the SAME judge verdicts
under BOTH narrative framings, and hand the table to review.
WHY this sweep (the scientific question fox is probing):
Is "Hermes-solo confidently fabricates the current officeholder" a
@ -17,58 +18,57 @@ WHY this sweep (the scientific question fox is probing):
{Hermes-3-8B, Qwen3.6-27B reasoning, Qwen3.6-27B no-reasoning}
× {plain, source-relative, as-of-corpus-era} questions, judged vs
the FIXED corpus-vintage gold, decides it empirically:
* if Qwen-27B-reasoning honestly ABSTAINS ("I cannot verify the
current officeholder") where Hermes-8B fabricates → the gap is
largely a weak-small-model artefact; the Arborist value claim
weakens and review must hear that;
* if even Qwen-27B-reasoning confidently asserts a post-corpus
answer → the gap is real and scale-independent (no bare
parametric model can be source-faithful) — the necessary-
substrate claim strengthens;
* the `as_of_corpus` variant ("As of 2010, who was…") separates
"model can't recall the corpus era" from "model won't constrain
to a source" — the exact ambiguity fox flagged.
* if Qwen-27B-reasoning honestly ABSTAINS where Hermes-8B
fabricates → the gap is largely a weak-small-model artefact;
* if even Qwen-27B-reasoning confidently asserts post-corpus →
the gap is real and scale-independent;
* `as_of_corpus` ("As of 2010, who was…") separates "can't recall
the corpus era" from "won't constrain to a source" — the exact
ambiguity fox flagged. (Corpus ~2010-2011, verified from the
artefact — see mine_questions.py — NOT the 2003 dump CLAUDE.md
names.)
DESIGN for huge N:
* the huge N goes on the CONTROL (solo × 3 models × 3 framings) —
that is where fox's open statistical question lives. The Arborist
treatment arm is a FIXED smaller A/B reference (--arborist-n,
default 40): re-measuring Arborist 386× does not buy power on the
control question, and it is the heaviest call (retrieval over
~40GB shards + proof path). Power belongs where the question is.
* parallel: independent hermetic judge calls + independent solo HTTP
calls fanned across a bounded worker pool (CLAUDE.md bench-maxing:
"serial-by-caution is halting in disguise"). Work unit = one
fixture item (all its variants/models) so each Arborist call gets
a per-unit qa_db — zero SQLite write contention; shards are
read-only and safe under concurrent readers.
* incremental: every record is appended+flushed the instant it
lands. `--report-only PATH.jsonl` aggregates a PARTIAL file with
NO LLM/judge spend — that is how the loop checks in every ~7
turns while the sweep is still running.
FRAMINGS, measured not chosen (all three from the AskUserQuestion):
* accuracy — WRONG+FABRICATED read as "model is wrong"
(the naive read fox correctly called unfair as a
*truth* claim — included precisely so review can
see why it misleads);
* grounding — the identical verdicts relabelled: WRONG/
FABRICATED = "ungrounded confident assertion",
ABSTAINED = "honest about ungroundedness" (good),
CORRECT_GROUNDED = "matched the designated
source". Deterministic relabelling — zero extra
judge spend.
* faithfulness — judge each answer vs the context IT was given
(Arborist-arm-only, different judge call). Ticket
§4b scopes this v2/ext-ii; NOT run here (it is a
different instrument, not a relabelling) — listed
so the omission is explicit, not silent.
accuracy / grounding = two deterministic relabellings of the SAME
judge verdicts (zero extra spend). faithfulness-ablation = a
different instrument (judge vs the context each arm was given,
Arborist-only); ticket §4b scopes it v2/ext-ii — NOT run here, so
the omission is explicit not silent. Arborist×Qwen needs proof-path
guided_json+extra_body surgery — the coupled follow-up, also not
done in a measurement harness.
Arborist arm: the Hermes treatment reference is run alongside for
direct same-N comparison. Arborist×Qwen needs a guided_json+extra_body
merge inside the proof path (runner.py) — a distinct, careful piece of
work; flagged in the report as the next step, deliberately NOT done
here (no proof-path surgery in a measurement harness).
Gate: judge-self-test must be 4/4 before any cell is trusted (run
once up front here, not per-cell — the gate, not 9× the gate).
Spend is bounded and printed up front; N is a parameter so review can
scale it. Hermetic Opus judge, blinded, fixed gold — same hygiene as
control_ab.py (whose helpers this reuses; DRY).
Gate: judge-self-test 4/4 in-script, once, before any cell (aborts on
fail). `--report-only` skips it (read-only aggregation, no judge).
"""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import re
import sys
import threading
import time
from collections import Counter
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
@ -76,14 +76,6 @@ sys.path.insert(0, str(ROOT))
from bench.control_ab import _SOLO_SYS, _descaffold, _gold # noqa: E402
from bench.judge import judge, self_test # noqa: E402
# Model arms. `extra` merges into the chat payload root (the client
# forwards it); unrecognised keys are silently dropped server-side.
# qwen-think keeps reasoning ON (Qwen3.6 puts the clean answer in
# message.content and the chain-of-thought in a separate
# reasoning_content field the client does not surface — logged as a
# known limitation); its token budget is raised so the answer survives
# after the thinking span. qwen-nothink disables thinking via the
# chat-template kwarg probed live on the deployment.
MODELS: dict[str, dict] = {
"hermes": dict(
endpoint="https://hermes.ai.unturf.com/v1",
@ -110,22 +102,10 @@ def _v_plain(q: str) -> str:
def _v_source_relative(q: str) -> str:
# The FAIR framing: explicitly ask for a source-grounded answer.
# A bare model has no such source — honoring this means abstaining
# ("I have no reference knowledge base"); still asserting a
# remembered name is ungrounded-confident-assertion, not an unfair
# "you're out of date" penalty. Both arms get the identical text.
return "According to the reference knowledge base, " + _lower_first(q)
def _v_as_of_corpus(q: str) -> str:
# Separates "can't recall the corpus era" from "won't constrain to
# a source": names the snapshot explicitly. If the model CAN say
# the 2010 holder when asked "as of 2010", the gap is snapshot
# ambiguity; if it still can't, parametric memory genuinely cannot
# serve the corpus era. (Corpus vintage ~2010-2011, verified from
# the artefact — see mine_questions.py; NOT the 2003 dump CLAUDE.md
# names.)
q2 = re.sub(r"^who is\b", "As of 2010, who was", q, flags=re.I)
return q2 if q2 != q else f"As of 2010: {q}"
@ -137,144 +117,98 @@ VARIANTS = {
}
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--fixture",
default="bench/qa_questions_stale_map.json")
ap.add_argument("--n", type=int, default=3)
ap.add_argument("--shards-dir",
default=str(Path.home() / ".arborist" / "shards"))
ap.add_argument("--models", default="hermes,qwen-think,qwen-nothink")
ap.add_argument("--variants",
default="plain,source_relative,as_of_corpus")
ap.add_argument("--arborist-ref", default="hermes",
help="model key for the Arborist treatment "
"reference column (proof path = Hermes; "
"Arborist×Qwen is the flagged follow-up)")
ap.add_argument("--out-dir", default="bench/qa_results")
ap.add_argument("--skip-self-test", action="store_true",
help="DANGER: bypasses the instrument gate; only "
"for offline harness development")
a = ap.parse_args()
models = [m for m in a.models.split(",") if m in MODELS]
variants = [v for v in a.variants.split(",") if v in VARIANTS]
items = json.loads(Path(a.fixture).read_text())[:a.n]
n_solo = len(models) * len(variants) * len(items)
n_arb = len(variants) * len(items)
print(f"#000057 control sweep — fixture={a.fixture} N={len(items)}")
print(f" models={models} variants={variants}")
print(f" bounded spend: {n_solo} solo LLM + {n_arb} Arborist "
f"query + ~{n_solo + n_arb} claude -p judge + 4 self-test")
# The instrument gate — once, up front. Not 9× the gate.
if a.skip_self_test:
print(" !! self-test SKIPPED (--skip-self-test) — verdicts "
"are UNTRUSTED")
else:
print(" running judge self-test (instrument gate) …")
if self_test() != 0:
print("ABORT: judge unreliable — no cell may be trusted")
return 1
# ---------------------------------------------------------------- work
def _process_item(idx: int, it: dict, variants: list[str],
models: list[str], shards_dir: Path,
arborist_on: bool, arborist_ref: str,
ts: str) -> list[dict]:
"""All variants × models for ONE fixture item. Self-contained:
its own qa_db, its own clients — safe to run concurrently."""
from arborist.qa.client import OpenAICompatibleClient
from arborist.qa.query import DEFAULT_QUERY_POLICY, query
ts = time.strftime("%Y-%m-%dT%H-%M-%SZ", time.gmtime())
outp = Path(a.out_dir) / f"control_sweep_{ts}.jsonl"
outp.parent.mkdir(parents=True, exist_ok=True)
shards_dir = Path(a.shards_dir)
qa_db = Path("/tmp") / f"control_sweep_{ts}.db"
out: list[dict] = []
q0 = it["question"]
gold = _gold(shards_dir, it.get("shard", ""), it["target_root"])
if not gold:
return [{"arm": "skip", "i": idx, "question_orig": q0,
"reason": "no gold"}]
clients: dict[str, OpenAICompatibleClient] = {}
def client_for(mkey: str) -> OpenAICompatibleClient:
def cl(mkey: str) -> OpenAICompatibleClient:
if mkey not in clients:
clients[mkey] = OpenAICompatibleClient(
base_url=MODELS[mkey]["endpoint"],
api_key=os.environ.get("ARBORIST_LLM_API_KEY"))
return clients[mkey]
# tally[(arm_label, variant)] -> Counter(verdict)
tally: dict[tuple, Counter] = {}
log = open(outp, "w")
def record(arm, mkey, variant, i, q_orig, q_asked, answer,
verdict, judge_raw, audit_mode=None, eff=None):
key = (arm if arm == "solo" else f"arborist", mkey, variant)
tally.setdefault(key, Counter())[eff or verdict] += 1
rec = {"arm": arm, "model": mkey, "variant": variant, "i": i,
"question_orig": q_orig, "question_asked": q_asked,
"answer": (answer or "")[:1500], "verdict": verdict,
"verdict_eff": eff or verdict,
"arb_audit_mode": audit_mode,
"judge_raw": (judge_raw or "")[:700]}
log.write(json.dumps(rec, ensure_ascii=False) + "\n")
log.flush()
for i, it in enumerate(items, 1):
q0 = it["question"]
gold = _gold(shards_dir, it.get("shard", ""), it["target_root"])
if not gold:
print(f" [{i}] SKIP (no gold): {q0[:50]}")
continue
qa_db = Path("/tmp") / f"control_sweep_{ts}_{idx}.db"
try:
for variant in variants:
q_asked = VARIANTS[variant](q0)
# ---- control arms: each model, solo ----
for mkey in models:
cfg = MODELS[mkey]
cl = client_for(mkey)
try:
ans = cl.chat_completion(
ans = cl(mkey).chat_completion(
[{"role": "system", "content": _SOLO_SYS},
{"role": "user", "content": q_asked}],
model=cfg["model"], max_tokens=cfg["max_tokens"],
model=cfg["model"],
max_tokens=cfg["max_tokens"],
extra_body=cfg["extra"])
except Exception as e: # noqa: BLE001
ans = f"[solo-error: {type(e).__name__}: {e}]"
v = judge(q_asked, _descaffold(ans), gold)
record("solo", mkey, variant, i, q0, q_asked, ans,
v.label, v.raw)
print(f" [{i}] solo/{mkey}/{variant} -> {v.label}")
# ---- treatment reference: Arborist on the proof-path
# model (Hermes) ----
ak = a.arborist_ref
out.append({"arm": "solo", "model": mkey,
"variant": variant, "i": idx,
"question_orig": q0,
"question_asked": q_asked,
"answer": (ans or "")[:1500],
"verdict": v.label,
"verdict_eff": v.label,
"arb_audit_mode": None,
"judge_raw": (v.raw or "")[:700]})
if arborist_on:
try:
r = query(question=q_asked, qa_db=qa_db,
chat_client=cl(arborist_ref),
model_id=MODELS[arborist_ref]["model"],
shards_dir=shards_dir,
policy=dict(DEFAULT_QUERY_POLICY,
answer_mode="claim_lattice"))
arb_raw = (r.get("raw_answer")
or r.get("answer_text") or "")
arb_mode = r.get("audit_mode")
except Exception as e: # noqa: BLE001
arb_raw = f"[arborist-error: {type(e).__name__}: {e}]"
arb_mode = "ERROR"
va = judge(q_asked, _descaffold(arb_raw), gold)
eff = ("ABSTAINED" if arb_mode == "UNGROUNDED"
and va.label in ("WRONG", "FABRICATED")
else va.label)
out.append({"arm": "arborist", "model": arborist_ref,
"variant": variant, "i": idx,
"question_orig": q0,
"question_asked": q_asked,
"answer": (arb_raw or "")[:1500],
"verdict": va.label, "verdict_eff": eff,
"arb_audit_mode": arb_mode,
"judge_raw": (va.raw or "")[:700]})
finally:
for c in clients.values():
try:
r = query(question=q_asked, qa_db=qa_db,
chat_client=client_for(ak),
model_id=MODELS[ak]["model"],
shards_dir=shards_dir,
policy=dict(DEFAULT_QUERY_POLICY,
answer_mode="claim_lattice"))
arb_raw = r.get("raw_answer") or r.get("answer_text") or ""
arb_mode = r.get("audit_mode")
except Exception as e: # noqa: BLE001
arb_raw = f"[arborist-error: {type(e).__name__}: {e}]"
arb_mode = "ERROR"
va = judge(q_asked, _descaffold(arb_raw), gold)
eff = ("ABSTAINED" if arb_mode == "UNGROUNDED"
and va.label in ("WRONG", "FABRICATED")
else va.label)
record("arborist", ak, variant, i, q0, q_asked, arb_raw,
va.label, va.raw, audit_mode=arb_mode, eff=eff)
print(f" [{i}] arborist/{ak}/{variant} -> {va.label}"
f" (mode={arb_mode}, eff={eff})")
log.close()
for c in clients.values():
c.close()
except Exception: # noqa: BLE001
pass
try:
c.close()
qa_db.unlink(missing_ok=True)
except Exception: # noqa: BLE001
pass
return out
md = _report(tally, items, models, variants, a, outp, ts)
mdp = Path(a.out_dir) / f"control_sweep_{ts}.md"
mdp.write_text(md)
print("\n" + md)
print(f"\nreview deliverables:\n {outp}\n {mdp}")
return 0
# -------------------------------------------------------------- report
def _bucket(c: Counter) -> str:
tot = sum(c.values()) or 1
@ -283,18 +217,36 @@ def _bucket(c: Counter) -> str:
f"JE={c['JUDGE_ERROR']} (n={tot})")
def _report(tally, items, models, variants, a, outp, ts) -> str:
n = len(items)
L = [f"# #000057 control sweep — {ts}",
def _aggregate(jsonl_path: Path):
tally: dict[tuple, Counter] = {}
seen_items: set = set()
n_recs = 0
for ln in jsonl_path.read_text().splitlines():
if not ln.strip():
continue
r = json.loads(ln)
if r.get("arm") == "skip":
continue
n_recs += 1
seen_items.add(r["i"])
key = (r["arm"], r["model"], r["variant"])
tally.setdefault(key, Counter())[r["verdict_eff"]] += 1
return tally, len(seen_items), n_recs
def _report(tally, n_items, n_recs, args, jsonl_path, ts,
done: bool) -> str:
L = [f"# #000057 control sweep — {ts}"
f"{'' if done else ' (INTERIM — sweep still running)'}",
"",
f"Fixture `{a.fixture}` · N={n} · models={models} · "
f"variants={variants}",
f"Raw per-item JSONL: `{outp}`",
f"Fixture `{args.fixture}` · items scored so far={n_items} · "
f"records={n_recs}",
f"Raw per-item JSONL: `{jsonl_path}`",
"",
"**This is review input, not a verdict.** fox: measure all "
"**Review input, not a verdict.** fox: measure all "
"benchmarks, bring forward for review.",
"",
"Verdict codes: CG=CORRECT_GROUNDED W=WRONG F=FABRICATED "
"Codes: CG=CORRECT_GROUNDED W=WRONG F=FABRICATED "
"A=ABSTAINED JE=JUDGE_ERROR. Gold = fixed corpus-vintage "
"article (~2010-2011, verified from the artefact, NOT the "
"2003 dump CLAUDE.md names). Judge = hermetic blinded Opus.",
@ -305,43 +257,140 @@ def _report(tally, items, models, variants, a, outp, ts) -> str:
"|-----|-------|---------|----------|"]
for (arm, mkey, variant), c in sorted(tally.items()):
L.append(f"| {arm} | {mkey} | {variant} | {_bucket(c)} |")
# Framing B headline numbers, computed deterministically.
L += ["",
"## Framing A — accuracy (the naive read; shown so review "
"sees why it misleads)",
"",
"WRONG+FABRICATED counted as \"model is wrong\". This "
"penalises a model for knowing post-corpus reality "
"(\"Macron\" is world-true); fox correctly flagged it is "
"NOT a fair *truth* claim. Reported only for contrast.",
"",
"## Framing B — grounding-fidelity (the defensible read)",
"",
"Identical verdicts, relabelled: CG = matched the "
"designated source; A = honest about ungroundedness "
"(GOOD — the capability bare models lack); W+F = "
"ungrounded confident assertion (the failure that is "
"structurally impossible to avoid without a retrieval+"
"verify substrate). The headline question for review: does "
"a 27B *reasoning* control ABSTAIN where the 8B fabricates?",
"A = honest about ungroundedness (GOOD); W+F = ungrounded "
"confident assertion; CG = matched the designated source. "
"Solo abstention-rate per (model,variant):",
""]
for (arm, mkey, variant), c in sorted(tally.items()):
if arm != "solo":
continue
tot = sum(c.values()) or 1
a = c["ABSTAINED"]
wf = c["WRONG"] + c["FABRICATED"]
L.append(f"- `{mkey}/{variant}`: abstain "
f"{a}/{tot} ({a/tot:.0%}) · ungrounded-assert "
f"{wf}/{tot} ({wf/tot:.0%})")
L += ["",
"## Framing A — accuracy (shown only so review sees why it "
"misleads: it penalises a model for knowing post-corpus "
"reality; \"Macron\" is world-true).",
"",
"## Framing C — faithfulness ablation (NOT run here)",
"",
"Judge each answer vs the context IT was given (Arborist-"
"arm-only, a different judge call, not a relabelling). "
"Ticket §4b scopes it v2/ext-ii. Listed so the omission is "
"explicit. Arborist×Qwen (proof-path guided_json+extra_body "
"merge) is the coupled follow-up — also deliberately not "
"done in a measurement harness.",
"",
"## Open for fox / review",
"",
"- bump `--n` (smallest-proof N here; denominators tiny);",
"- Arborist×Qwen arm (proof-path surgery — separate);",
"- which framing becomes the whitepaper headline;",
"- Qwen reasoning_content is not logged (client doesn't "
"surface it) — add if review wants the CoT audited."]
"## Not run here (explicit, not silent)",
"- faithfulness-ablation (judge vs each arm's own context) "
"— different instrument, ticket §4b v2/ext-ii;",
"- Arborist×Qwen — proof-path guided_json+extra_body merge, "
"coupled follow-up.",
""]
return "\n".join(L) + "\n"
def _emit_report(jsonl_path: Path, args, ts, done: bool) -> Path:
tally, n_items, n_recs = _aggregate(jsonl_path)
md = _report(tally, n_items, n_recs, args, jsonl_path, ts, done)
mdp = jsonl_path.with_suffix(".md")
mdp.write_text(md)
print(md)
return mdp
# ----------------------------------------------------------------- main
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--fixture",
default="bench/qa_questions_stale_map.json")
ap.add_argument("--n", type=int, default=386,
help="solo (control) item count — the HUGE axis")
ap.add_argument("--arborist-n", type=int, default=40,
help="Arborist A/B reference item count (fixed "
"small — power belongs on the control)")
ap.add_argument("--max-workers", type=int, default=6)
ap.add_argument("--shards-dir",
default=str(Path.home() / ".arborist" / "shards"))
ap.add_argument("--models", default="hermes,qwen-think,qwen-nothink")
ap.add_argument("--variants",
default="plain,source_relative,as_of_corpus")
ap.add_argument("--arborist-ref", default="hermes")
ap.add_argument("--out-dir", default="bench/qa_results")
ap.add_argument("--report-only", default="",
help="aggregate a (partial) JSONL with NO spend "
"and exit — the interim check-in path")
ap.add_argument("--skip-self-test", action="store_true")
a = ap.parse_args()
if a.report_only:
jp = Path(a.report_only)
ts = jp.stem.replace("control_sweep_", "")
done = not any( # crude: running if a sweep proc still alive
True for _ in [])
_emit_report(jp, a, ts, done=False)
return 0
models = [m for m in a.models.split(",") if m in MODELS]
variants = [v for v in a.variants.split(",") if v in VARIANTS]
items = json.loads(Path(a.fixture).read_text())[:a.n]
n_solo = len(models) * len(variants) * len(items)
arb_units = min(a.arborist_n, len(items))
n_arb = len(variants) * arb_units
print(f"#000057 control sweep — fixture={a.fixture}")
print(f" solo N={len(items)} arborist-ref N={arb_units} "
f"models={models} variants={variants} "
f"workers={a.max_workers}")
print(f" bounded spend: {n_solo} solo + {n_arb} Arborist + "
f"~{n_solo + n_arb} claude -p judge + 4 self-test")
if a.skip_self_test:
print(" !! self-test SKIPPED — verdicts UNTRUSTED")
else:
print(" judge self-test (instrument gate) …")
if self_test() != 0:
print("ABORT: judge unreliable")
return 1
ts = time.strftime("%Y-%m-%dT%H-%M-%SZ", time.gmtime())
outp = Path(a.out_dir) / f"control_sweep_{ts}.jsonl"
outp.parent.mkdir(parents=True, exist_ok=True)
shards_dir = Path(a.shards_dir)
lock = threading.Lock()
t0 = time.time()
done_units = 0
with open(outp, "w") as log, ThreadPoolExecutor(
max_workers=a.max_workers) as ex:
futs = {
ex.submit(_process_item, i, it, variants, models,
shards_dir, i <= arb_units, a.arborist_ref,
ts): i
for i, it in enumerate(items, 1)
}
for fut in as_completed(futs):
i = futs[fut]
try:
recs = fut.result()
except Exception as e: # noqa: BLE001
recs = [{"arm": "skip", "i": i,
"reason": f"unit-crash {type(e).__name__}: {e}"}]
with lock:
for rec in recs:
log.write(json.dumps(rec, ensure_ascii=False) + "\n")
log.flush()
done_units += 1
el = time.time() - t0
rate = done_units / el if el else 0
eta = (len(items) - done_units) / rate / 60 if rate else 0
print(f" unit {done_units}/{len(items)} (item {i}) "
f"· {el/60:.1f}m elapsed · ETA {eta:.0f}m")
print(f"\nsweep complete in {(time.time()-t0)/60:.1f}m")
mdp = _emit_report(outp, a, ts, done=True)
print(f"review deliverables:\n {outp}\n {mdp}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

346
bench/qa_questions_stale.txt
View file

@ -41,3 +41,349 @@ who is the governor-general of Fiji?
who is the premier of the Northwest Territories?
who is the president of Suriname?
who is the president of Cuba?
who is the president of Portugal?
who is the prime minister of the Faroe Islands?
who is the prime minister of Angola?
who is the prime minister of Turkey?
who is the prime minister of Afghanistan?
who is the prime minister of Yemen?
who is the prime minister of Tuvalu?
who is the prime minister of Yemen Arab Republic?
who is the governor-general of Tuvalu?
who is the prime minister of Zambia?
who is the president of Serbia?
who is the president of Serbia and Montenegro?
who is the chancellor of Justice?
who is the president of Spain?
who is the prime minister of Iran?
who is the prime minister of Uzbekistan?
who is the premier of the Cayman Islands?
who is the chancellor of Uppsala University?
who is the prime minister of Kosovo?
who is the president of Hawaii?
who is the president of Mongolia?
who is the prime minister of Mauritius?
who is the prime minister of Somalia?
who is the president of the European Central Bank?
who is the prime minister of the Palestinian National Authority?
who is the president of the Indian National Congress?
who is the president of the Sahrawi Arab Democratic Republic?
who is the prime minister of the Philippines?
who is the president of the Maldives?
who is the chief minister of the Isle of Man?
who is the premier of the Northern Cape?
who is the prime minister of Azerbaijan?
who is the prime minister of Cambodia?
who is the prime minister of Kazakhstan?
who is the governor-general of the West Indies Federation?
who is the premier of the Turks and Caicos Islands?
who is the president of Palau?
who is the prime minister of Solomon Islands?
who is the president of Abkhazia?
who is the chancellor of the Exchequer of Ireland?
who is the prime minister of Yugoslavia?
who is the president of the Canary Islands?
who is the president of the University of Southern California?
who is the governor-general of the Federation of Rhodesia and Nyasaland?
who is the prime minister of Zimbabwe?
who is the president of the Chechen Republic?
who is the prime minister of Crimea?
who is the prime minister of Saudi Arabia?
who is the president of Chile?
who is the prime minister of China?
who is the president of French Polynesia?
who is the president of the Associação Brasileira de Estatística?
who is the prime minister of Serbia and Montenegro?
who is the president of the Federation of Bosnia and Herzegovina?
who is the chief minister of Singapore?
who is the president of the University of Richmond?
who is the chief minister of Sindh?
who is the president of the Autonomous Region of Bougainville?
who is the prime minister of Seychelles?
who is the president of Ingushetia?
who is the president of the Komi Republic?
who is the president of the Sakha Republic?
who is the president of the Republic of Buryatia?
who is the president of Dagestan?
who is the president of Karachay-Cherkessia?
who is the prime minister of the West Indies Federation?
who is the president of the National Congress of Honduras?
who is the president of Lombardy?
who is the prime minister of Iraqi Kurdistan?
who is the chancellor of the University of Cape Town?
who is the prime minister of Lower Saxony?
who is the governor-general of Australia?
who is the president of the United States?
who is the prime minister of India?
who is the premier of Western Australia?
who is the prime minister of Japan?
who is the president of Poland?
who is the chancellor of the Exchequer?
who is the president of the People's Republic of China?
who is the prime minister of Northern Ireland?
who is the prime minister of France?
who is the prime minister of the United States?
who is the president of the Republic?
who is the governor-general of India?
who is the first minister of Scotland?
who is the prime minister of Greece?
who is the prime minister of Italy?
who is the president of Nauru?
who is the president of Croatia?
who is the premier of Ontario?
who is the president of Malta?
who is the president of Georgia?
who is the president of Ghana?
who is the president of Bangladesh?
who is the prime minister of Vietnam?
who is the prime minister of Mongolia?
who is the president of Uganda?
who is the prime minister of Antigua and Barbuda?
who is the governor-general of Belize?
who is the prime minister of Lithuania?
who is the premier of Manitoba?
who is the premier of Prince Edward Island?
who is the prime minister of Fiji?
who is the prime minister of Armenia?
who is the governor-general of the Philippines?
who is the prime minister of Burma?
who is the president of Haiti?
who is the president of North Korea?
who is the president of Rhodesia?
who is the prime minister of Dominica?
who is the prime minister of the Sahrawi Arab Democratic Republic?
who is the prime minister of Cameroon?
who is the president of the Government?
who is the president of Harvard University?
who is the prime minister of Estonia?
who is the president of Azerbaijan?
who is the chancellor of Florence?
who is the prime minister of Egypt?
who is the president of Moldova?
who is the president of the Palestinian National Authority?
who is the president of the German Bundesrat?
who is the president of Kyrgyzstan?
who is the prime minister of Kyrgyzstan?
who is the president of Zanzibar?
who is the prime minister of Rwanda?
who is the president of Turkmenistan?
who is the president of Belarus?
who is the prime minister of Ghana?
who is the president of the Institutional Revolutionary Party?
who is the president of Italy?
who is the president of Kiribati?
who is the prime minister of Samoa?
who is the president of Nagorno-Karabakh Republic?
who is the president of Uzbekistan?
who is the president of the RSUA?
who is the chancellor of the University of St Andrews?
who is the president of Northern Cyprus?
who is the chief minister of Khyber Pakhtunkhwa?
who is the president of the United Arab Emirates?
who is the president of Burma?
who is the president of Madagascar's Small Grants Scheme?
who is the president of Savannah State University?
who is the chancellor of Keele University?
who is the prime minister of the Republic of Adygea?
who is the prime minister of Zimbabwe Rhodesia?
who is the premier of Niue?
who is the president of Veneto?
who is the president of Grinnell College?
who is the prime minister of South Ossetia?
who is the premier of Dominica?
who is the president of the Dewan Negara?
who is the chief minister of Sri Lanka?
who is the chancellor of the University of Glasgow?
who is the president of Bashkortostan?
who is the president of the Marshall Islands?
who is the president of Iraqi Kurdistan?
who is the prime minister of Slovenia?
who is the prime minister of Curaçao?
who is the prime minister of Sint Maarten?
who is the chief minister of the Northern Territory?
who is the premier of Victoria?
who is the president of Ireland?
who is the president of South Korea?
who is the president of the Irish Republic?
who is the president of Israel?
who is the prime minister of Iceland?
who is the president of Tajikistan?
who is the president of Bolivia?
who is the prime minister of Belgium?
who is the prime minister of Finland?
who is the president of Iran?
who is the president of Russia?
who is the president of the Philippines?
who is the prime minister of Thailand?
who is the prime minister of the Netherlands?
who is the president of Nicaragua?
who is the prime minister of Jamaica?
who is the president of South Africa?
who is the president of Sri Lanka?
who is the premier of China?
who is the president of Sierra Leone?
who is the prime minister of Pakistan?
who is the president of Fiji?
who is the premier of Nunavut?
who is the prime minister of Latvia?
who is the president of Trinidad and Tobago?
who is the prime minister of Georgia?
who is the prime minister of South Korea?
who is the president of Madrid?
who is the president of the Confederate States of America?
who is the prime minister of Rhodesia?
who is the president of the Federated States of Micronesia?
who is the president of the European Parliament?
who is the president of Kosovo?
who is the governor-general of Taiwan?
who is the president of East Timor?
who is the prime minister of East Timor?
who is the president of South Ossetia?
who is the president of Princeton University?
who is the president of Vanuatu?
who is the prime minister of Moldova?
who is the premier of the Eastern Cape?
who is the premier of Mpumalanga?
who is the prime minister of Bahrain?
who is the prime minister of Kenya?
who is the chancellor of the Order of the Thistle?
who is the president of Tynwald?
who is the president of the Municipal Chamber?
who is the prime minister of Brunei?
who is the prime minister of Vanuatu?
who is the president of the Gaelic Athletic Association?
who is the president of Montenegro?
who is the president of the Family Division?
who is the prime minister of Cuba?
who is the president of the Geological Society of London?
who is the premier of Nevis?
who is the prime minister of Tajikistan?
who is the chancellor of the Tang Dynasty?
who is the chancellor of Justice of Finland?
who is the president of Crimea?
who is the president of the Republic of Adygea?
who is the president of the Chamber of Deputies of Romania?
who is the president of Nepal?
who is the president of the Republic of Sierra Leone?
who is the president of the Government of Vojvodina?
who is the president of Ukraine?
who is the chief minister of the Australian Capital Territory?
who is the president of Mexico?
who is the premier of Gauteng?
who is the premier of KwaZulu-Natal?
who is the president of the Parliament of Montenegro?
who is the president of Arellano University?
who is the prime minister of Canada?
who is the prime minister of the United Kingdom?
who is the prime minister of Israel?
who is the president of Germany?
who is the chancellor of Austria?
who is the president of the Republic of China?
who is the prime minister of New Zealand?
who is the governor-general of the Irish Free State?
who is the premier of the People's Republic of China?
who is the president of Turkey?
who is the prime minister of Sweden?
who is the prime minister of Spain?
who is the prime minister of Norway?
who is the president of Argentina?
who is the president of Colombia?
who is the prime minister of Romania?
who is the president of Brazil?
who is the president of Romania?
who is the governor-general of Finland?
who is the prime minister of Papua New Guinea?
who is the prime minister of South Africa?
who is the president of Slovenia?
who is the premier of British Columbia?
who is the president of Egypt?
who is the president of the Continental Congress?
who is the president of Singapore?
who is the prime minister of Sri Lanka?
who is the governor-general of Papua New Guinea?
who is the premier of Quebec?
who is the president of Angola?
who is the governor-general of Kenya?
who is the prime minister of Algeria?
who is the president of Pakistan?
who is the president of Mauritius?
who is the president of El Salvador?
who is the president of Vietnam?
who is the premier of Nova Scotia?
who is the premier of Saskatchewan?
who is the premier of Newfoundland and Labrador?
who is the premier of Yukon?
who is the governor-general of Pakistan?
who is the prime minister of Malaysia?
who is the president of Greece?
who is the president of Nigeria?
who is the prime minister of Abkhazia?
who is the prime minister of Serbia?
who is the president of Paraguay?
who is the governor-general of the Dutch East Indies?
who is the governor-general of Solomon Islands?
who is the prime minister of Uganda?
who is the prime minister of Ukraine?
who is the president of the Republic of Macedonia?
who is the president of Estonia?
who is the prime minister of Namibia?
who is the president of Cyprus?
who is the prime minister of Senegal?
who is the premier of the Kingdom of Viti?
who is the president of Ichkeria?
who is the prime minister of Madagascar?
who is the governor-general of Ceylon?
who is the president of Laos?
who is the prime minister of Tunisia?
who is the prime minister of Aruba?
who is the chief minister of Anguilla?
who is the chief minister of Jersey?
who is the premier of North West?
who is the chief minister of Papua and New Guinea?
who is the prime minister of Laos?
who is the prime minister of Tonga?
who is the chief minister of Montserrat?
who is the prime minister of the Republic of Macedonia?
who is the governor-general of New Zealand?
who is the prime minister of Nagorno-Karabakh Republic?
who is the president of Transnistria?
who is the chief minister of Balochistan?
who is the prime minister of Montenegro?
who is the president of the Legislative Yuan?
who is the president of Bucknell University?
who is the president of the Queen's Bench Division?
who is the chief minister of Guernsey?
who is the prime minister of Northern Cyprus?
who is the president of Grande Comore?
who is the president of Anjouan?
who is the president of Mohéli?
who is the president of the Bundestag?
who is the president of the Moldovan Parliament?
who is the chief minister of Punjab?
who is the president of Indonesia?
who is the president of Cantabria?
who is the president of the University of Michigan?
who is the chancellor of the University of Edinburgh?
who is the prime minister of the Federation of Rhodesia and Nyasaland?
who is the president of the Congress of Colombia?
who is the premier of the Western Cape?
who is the president of Congress?
who is the president of Zimbabwe Rhodesia?
who is the president of the World Esperanto Association?
who is the prime minister of Turkmenistan?
who is the chancellor of Germany?
who is the president of the States of Alderney?
who is the president of the Government of New Caledonia?
who is the chancellor of the University of Dallas?
who is the president of the Naval War College?
who is the president of the University of Southern Maine?
who is the president of Kabardino-Balkaria?
who is the president of the University of the Philippines?
who is the president of Honduras?
who is the premier of the Free State?
who is the premier of Limpopo?
who is the president of Uruguay?
who is the president of the Xunta of Galicia?
who is the president of Umbria?
who is the prime minister of Hyderabad?
who is the premier of Montserrat?

2076
bench/qa_questions_stale_map.json
View file

File diff suppressed because it is too large Load diff