russell@unturf.com
43c97a03e7
Adds bench/cross_model_selfplay.py + `make bench-cross-model` target.
For each question in a fixture, runs `arborist query` once per
configured model (default: Hermes + Qwen) and tabulates:
* audit_mode per model (EVIDENCE-WARRANTED → POINTER-LINKED → UNGROUNDED)
* agreement on primary source URI
* grounding rate per model
* estimated $/grounded-answer (per-call prices configurable)
* cheap-first cascade analysis (try cheapest, escalate on UNGROUNDED)
This is the "ask twice for two options" pattern from the agent
perspective — bakes it in as a benchmark so we can measure whether
the cascade beats always-using-the-stronger-model on $/grounded.
First live run (2 questions × 2 models, $0.41):
- Hermes: 1/1 grounded (1 timeout — operational issue)
- Qwen: 2/2 grounded STRICT
- Cascade: 2/2 grounded for $0.25 — beats always-Qwen ($0.32)
when Hermes succeeds on its first call.
Output: bench/cross_model_results/<utc-iso>.{jsonl,md} (gitignored).
28 lines
493 B
Text
28 lines
493 B
Text
__pycache__/
|
|
*.py[cod]
|
|
*.egg-info/
|
|
.venv/
|
|
.pytest_cache/
|
|
.coverage
|
|
*.swp
|
|
_build/
|
|
|
|
# data + caches stay out of git
|
|
data/
|
|
*.db
|
|
*.db-journal
|
|
*.db-wal
|
|
*.db-shm
|
|
|
|
# bench artifacts
|
|
bench/qa_results/
|
|
bench/cross_model_results/
|
|
|
|
# Claude Code session-local artifacts (worktrees, transient state)
|
|
.claude/
|
|
|
|
# Sphinx build artifacts (generated; RTD builds on its servers)
|
|
docs/_source/_build/
|
|
|
|
# Auto-generated by docs/_source/_ext/makefile_targets.py at Sphinx build time
|
|
docs/_source/api/makefile.rst
|