Adds --resume <jsonl-path>: read the existing JSONL, build a set
of (question, mode, sample_idx) tasks already done, skip those
in the shuffled task list, and append fresh rows to the same
file. The markdown rollup uses the union of pre-existing + new
rows. Same --seed required for the order to align across the
resumed run.
Why: a 60-90 min full bench is annoying to re-run from zero when
something interrupts (network blip, kill, kernel panic). The
JSONL has been the durable artifact for ages; the bench just
didn't know how to read it.
Implementation:
- args.resume: Path | None
- if given: parse JSONL line-by-line, populate done_tasks set,
reuse the resume_path's stem as the run stamp, set jsonl_path
+ md_path to the resumed paths
- file_mode = 'a' if resuming else 'w'
- tasks filter: drop entries already in done_tasks
- rows initialized with existing_rows so the markdown summary
sees the union
- done counter starts at len(existing_rows) so the print
countdown reflects total progress
Empirical context: the c=6 bench killed at 142/639 left a
recoverable JSONL. Future kills can resume via:
.venv/bin/python bench/qa_sweep.py \
--resume bench/qa_results/<stamp>.jsonl \
--seed 0 --concurrency 4
(Same seed reproduces the shuffle so the remaining tasks come
out in the original order.)
Concurrency sweep on the smoke (15 tasks at each c):
c=3 102s (peak throughput)
c=4 106s (4% slower; chosen for full bench — more
forgiving on single-call hiccups)
c=5 119s (12% slower than c=3)
c=6 185s (45% slower; vLLM batching ceiling)
Lesson: more concurrent ≠ faster. vLLM continuous batching has
a sweet spot around c=3-4 for this endpoint.
russell@unturf.com
780f5afd69