russell@unturf.com
2601575f15
Adds bench/qa_sweep.py + bench/qa_questions.txt + `make bench-qa` target.
Sweeps a fixed question set through quote and claim_lattice_pointer modes,
N samples per cell (default 3), each sample burns the cached record so
Hermes nondeterminism becomes the variance source.
Outputs:
bench/qa_results/<utc>.jsonl one row per (mode, question, sample)
bench/qa_results/<utc>.md summary + per-question vote counts
Question set spans the failure-mode shapes the verifier needs to handle:
narrow factoid, broad descriptive ("tell me about X"), entity list,
relationship, and out-of-corpus (should land UNGROUNDED honestly).
Methodology gap this closes: at n=1 the strict-rate swings ±20pp on a
fixed prompt purely from sampling noise. n=3 makes 5pp deltas legible.
17 lines
185 B
Text
17 lines
185 B
Text
__pycache__/
|
|
*.py[cod]
|
|
*.egg-info/
|
|
.venv/
|
|
.pytest_cache/
|
|
.coverage
|
|
*.swp
|
|
|
|
# data + caches stay out of git
|
|
data/
|
|
*.db
|
|
*.db-journal
|
|
*.db-wal
|
|
*.db-shm
|
|
|
|
# bench artifacts
|
|
bench/qa_results/
|