engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
arborist/bench
History
russell@unturf.com 12bb6dbb6f
feat(#000057): parallel + incremental + huge-N control sweep 
fox: 'make the n huge huge, check in every ~7 turns'.

- stale fixture re-mined to the FULL pool: 386 questions (180 pres /
  119 PM / 34 premier / 20 gov-gen / 17 chancellor / 15 CM / 1 FM),
  deterministic.
- control_sweep.py rebuilt: ThreadPoolExecutor work-unit-per-item
  (per-unit qa_db -> zero SQLite write contention; shards read-only
  safe under concurrent readers), incremental flushed JSONL, and a
  --report-only PATH mode that aggregates a PARTIAL file with ZERO
  LLM/judge spend (the interim check-in path).
- huge N goes on the CONTROL (solo x3 models x3 framings) where the
  open statistical question lives; Arborist A/B = fixed --arborist-n
  (default 40), not re-measured 386x (power belongs on the control,
  and query() over ~40GB shards is the heaviest call).
- bench-maxing doctrine applied: independent hermetic judge calls
  fanned out, 'serial-by-caution is halting in disguise'.

N=3 (prior run) already shows the coherent, review-relevant story:
plain -> all 3 models confidently assert (0% abstain); source_relative
-> abstain jumps (hermes 33 / qwen-think 67 / qwen-nothink 100%);
as_of_corpus -> ~100% correct all models. The gap is largely
framing+snapshot, not a universal capability deficit.
2026-05-19 13:00:45 -04:00
..
batteries #000025 §10.11 + §10.13 + §10.14 — close the 5F battery 2026-05-11 07:41:37 -04:00
fixtures #000052 §3.2.2 step 3 part B: real-context MIS-CITE fixtures (n=20) — bge-large catches 100% of mis-cite at strictly 0% real-STRICT FP (the Zionist failure mode is fully covered with margin) 2026-05-13 15:27:58 -04:00
results #000052 §3.2.2 step 3 part C: multi-signal combination experiment — does NOT lift precision/recall on §3.2's failure shapes; clean architectural finding that lexical sidecars + relevance reranker cover NON-OVERLAPPING failure modes 2026-05-13 15:30:39 -04:00
scripts feat(#000049 §7 #28): tinygrad NLI backend + deterministic engine-agreement A/B; ONNX-immunity rationale 2026-05-19 12:34:04 -04:00
control_ab.py fix(#000057): solo arm never ran — chat_completion(model=) was omitted 2026-05-19 11:49:30 -04:00
control_sweep.py feat(#000057): parallel + incremental + huge-N control sweep 2026-05-19 13:00:45 -04:00
emergent_log.jsonl #000006 — +30 emergent cycles (2026-05-12); verifier-ladder health re-confirmed 2026-05-12 11:28:57 -04:00
es_delta.py feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
es_join_patterns.py feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
es_roundtrip_analysis.py feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
judge.py feat(#000057): hermetic external judge instrument — built + verified 4/4 (make judge-self-test) 2026-05-19 08:44:47 -04:00
make_es_questions.py feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
make_lang_questions.py feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
mine_questions.py fix(#000057): solo arm never ran — chat_completion(model=) was omitted 2026-05-19 11:49:30 -04:00
prometheus_sigma_trigger_probe.py #000012 Phase 1c follow-through: wire #000037 §12 Trigger 1 probe to fork_score_branches 2026-05-11 06:56:09 -04:00
qa_questions.txt aborist/arborist 2026-05-07 09:31:49 -04:00
qa_questions_accent.txt feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_accent_map.json feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_amp.txt feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_amp_map.json feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_brit.txt feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_brit_map.json feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_canonical_witness_npower.txt three-thread session output: stale TODOs, N-power probe, ForkScore Phase 1c 2026-05-10 07:46:35 -04:00
qa_questions_es.txt feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
qa_questions_es_map.json feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
qa_questions_fr.txt feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
qa_questions_fr_map.json feat: cross-language Q&A (Operation Sandwich) + Windows quickstart — all default-OFF 2026-05-18 12:12:23 -04:00
qa_questions_honorific.txt feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_honorific_map.json feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_hyphen.txt feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_hyphen_map.json feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
qa_questions_metacog_subset.txt qa(#000011 + 4 more): SOFT_PREFLIGHT_HINT impl + 5-task fan-out 2026-05-03 23:00:56 -04:00
qa_questions_numeral.txt fix(#000057): restore numeral fixture clobbered by b573c59 (40 entries, not 5) 2026-05-19 11:12:12 -04:00
qa_questions_numeral_map.json fix(#000057): restore numeral fixture clobbered by b573c59 (40 entries, not 5) 2026-05-19 11:12:12 -04:00
qa_questions_progressive_and.txt bench: progressive-AND fixture + 2026-05-09 A/B baseline report 2026-05-10 06:35:18 -04:00
qa_questions_quantifier_baseline.txt bench(#000008): harness extension — FC rate, violation kinds, raw brackets 2026-05-02 18:35:08 -04:00
qa_questions_quantifier_subset.txt ticket(#000008): §12 dry-run bench findings + --policy harness flag 2026-05-03 08:39:20 -04:00
qa_questions_smoke.txt speed: pytest-xdist, bench smoke, concurrency default; UTF surrogate fix 2026-05-02 09:29:40 -04:00
qa_questions_stale.txt feat(#000057): parallel + incremental + huge-N control sweep 2026-05-19 13:00:45 -04:00
qa_questions_stale_map.json feat(#000057): parallel + incremental + huge-N control sweep 2026-05-19 13:00:45 -04:00
qa_questions_warrant_chain_aggressive.txt bench: aggressive warrant fixture confirms Phase 3 is rescue-only, not default-path 2026-05-10 09:53:21 -04:00
qa_questions_warrant_chain_paraphrase.txt bench: Phase 3 paraphrase fixture investigation — empirically dormant on current corpus 2026-05-10 10:06:23 -04:00
qa_questions_warrant_chain_probe.txt bench: #000031 Phase 3 A/B finds mechanism dormant on warrant-targeted fixture 2026-05-10 09:37:27 -04:00
qa_sweep.py bench/qa_sweep: scrub lone surrogates from the NLI-shadow answer_text/context fields before json.dumps 2026-05-12 17:04:33 -04:00
recall_at_k.py feat(retrieval): accent-fold (+30pp recall@1) + fold-search factory hardening 2026-05-18 19:23:22 -04:00
run.sh aborist/arborist 2026-05-07 09:31:49 -04:00