russell@unturf.com
a64d941528
Per fox's "partial punt on larger ones" — ships the bench/ skeleton +
small seed fixture sets so future v8/v7-W/SelfModel work can cite a
real fitness target. Full Phase 1 (50-200 fixtures per sub-battery)
and Phases 2-3 stay open in the ticket.
Phase 1a delivers:
- bench/batteries/{base,b_5s,b_5t,runner}.py — Battery protocol,
BatteryResult, fixture-digest helpers, CLI runner.
- Seed fixtures:
- bench/fixtures/5s/syntax-v1.jsonl — 10 tasks against
wikitext-base@v1 and claim-lattice@v1
- bench/fixtures/5s/semantics-v1.jsonl — 8 equivalence tasks
- bench/fixtures/5t/transfer-v1.jsonl — 4 paraphrase-invariance
tasks
- Runners for 5S Syntax, 5S Semantics, 5T Transfer. Other 5S/5T
sub-batteries are stubs returning zero-task results.
- Makefile targets: bench-5s, bench-5t, bench-5s5t.
- runtime_digest field captures the active π* registry fingerprint
so a registry change surfaces in bench results.
Tests: tests/test_bench_batteries.py (17 cases). Full suite:
1076 passed, 36 skipped. `make bench-5s5t` runs end-to-end and
emits JSON results.
Ticket #000021 status: in progress · Phase 1a landed; Phase 1b/2/3
remain open.
13 lines
426 B
Python
13 lines
426 B
Python
"""5S/5T/5R battery harness (ticket #000021).
|
|
|
|
Phase 1a (this landing): 5S Syntax + 5S Semantics + 5T Transfer
|
|
runners with seed fixtures. Phase 1b (full fixture authoring) and
|
|
Phase 2 (5R battery, depends on tickets #000014 + #000017) land
|
|
in follow-up tickets.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from bench.batteries.base import Battery, BatteryResult, TaskResult
|
|
|
|
__all__ = ["Battery", "BatteryResult", "TaskResult"]
|