# Ticket #000021 — 5S/5T/5R benchmark fixtures + harness extension

**Status:** closed · Phase 1a + Phase 2 landed 2026-05-08; Phase 3 (external corpora) deferred
**Opened:** 2026-05-07
**Closed:** 2026-05-08
**Scope:** Implement the SQD whitepaper's named bench batteries
(5S Syntax/Semantics/Synthesis/Syllogism/Semiotics, 5T Transfer/
Triangulate/Timing/Transitivity/Truth, 5R React/Rearrange/Restore/
Replicate/Resonate) as concrete fixtures + harness targets so future
tickets can cite measured numbers instead of theoretical bench shapes.
Code + fixtures + Makefile target.
**Audience:** fox + future bench authors + future v8 / v7-W /
SelfModel work that needs concrete fitness targets.
**Hard constraint:** fixtures land deterministically (committed JSONL,
versioned). Bench-maxing discipline (5pp signal floor, n≥3 runs)
applies. Each battery is independently runnable.

---

## 1. Problem statement

SQD §9 names the bench shapes:

```text
5S = Syntax · Semantics · Synthesis · Syllogism · Semiotics
5T = Transfer · Triangulate · Timing · Transitivity · Truth
5R = React · Rearrange · Restore · Replicate · Resonate
```

These are abstract. arborist has bench harness targets
(`make bench-qa`) that test arborist's *own* QA stack on
question-answering, not the abstract 5S/5T/5R shapes.

Without concrete fixtures:

| Failure | What happens |
|---|---|
| v8 selection protocol can't reference a fitness target | Proof-of-Upgrade has no canonical floor to non-regress on. |
| Cross-validator fitness comparison has no schema | No common bench → no cross-validator agreement. |
| New verifier-strategy work has no transfer test | Strategy A wins on training corpus, fails silently on novel domain. |
| Self-improvement loop has no objective | Recursive update can't be scored. |

This ticket converts the SQD names into actual fixtures + a runner.

### 1.1 What a fixture looks like

A fixture is a JSONL file with one task per line:

```jsonl
{"id": "5s-syntax-001", "battery": "syntax", "kind": "parse-pass",
 "input": "...", "expected": "pass" }
{"id": "5s-semantics-014", "battery": "semantics", "kind": "paraphrase-stability",
 "input_a": "...", "input_b": "...", "expected_invariant": true }
{"id": "5t-transfer-002", "battery": "transfer", "kind": "domain-shift",
 "train_corpus": "...", "eval_corpus": "...", "metric": "strict_rate" }
```

The harness reads a fixture file, runs the prescribed test, computes
the metric, compares to threshold.

---

## 2. Design choices

### 2.1 Fixture authority: where does each fixture come from?

**A. Hand-curated from existing arborist bench data.** Cheapest;
preserves continuity with `bench/qa_sweep.py`.

**B. Generated synthetically (e.g., template-based for syntax tests).**
Cheap; reproducibility tax (template + seed must be committed).

**C. External corpora (e.g., Wikipedia, code corpora) downscoped to
canonical subsets.** Most realistic; license/distribution complexity.

**Recommendation:** A for first round (re-use what's there), with
B for syntax/semantics where templates are natural, and C as a
downstream expansion when interop with external benchmarks
matters.

### 2.2 Fixture versioning

Each fixture file commits to a version pin:

```jsonl
{"_meta": {"battery": "syntax", "version": "v1",
           "fixture_digest": "...", "task_count": 142}}
```

Version pin enters `governance_policy_hash` if the bench is gated
on policy. Otherwise sibling reference.

### 2.3 Battery-runner shape

Each battery runs through a common interface:

```python
class Battery(Protocol):
    name: str
    def run(self, fixtures_path: Path, runtime_config) -> BatteryResult:
        ...

class BatteryResult:
    pass_count: int
    fail_count: int
    metrics: dict[str, float]   # per-battery metrics (e.g., strict_rate)
    per_task: list[TaskResult]
    fixture_digest: str
    runtime_digest: str
    timestamp: str
```

The runner emits `BatteryResult` as JSON on stdout; CI ingests it.

### 2.4 5R running on workspace state

The 5R battery is structurally different — it tests workspace
operators (React/Rearrange/Restore/Replicate/Resonate) over a
running agent's state. Implementation needs:

- A workspace abstraction the battery can poke (probably mocking
  arborist's existing providence + audit chain as the workspace).
- Each operator (React, etc.) is a verb the battery applies to
  workspace state and observes the result.

Recommend implementing 5S and 5T first (they're per-task fixtures);
defer 5R until SelfModel (#000014) and memory_root (#000017) land,
since 5R tests workspace operators those tickets define.

---

## 3. Recommendation

**Phase 1 (this ticket):** 5S + 5T fixture sets + runner.
~50-200 fixtures per battery, hand-curated from existing arborist
bench corpora plus a small number of synthetic templates.
Makefile target `make bench-5s5t`.

**Phase 2 (follow-up ticket):** 5R battery, gated on #000014 +
#000017.

**Phase 3 (follow-up ticket):** External-corpus expansion (option
C above) for cross-substrate transfer testing.

---

## 4. Implementation sketch

### 4.1 Repo layout

```
bench/fixtures/
├── 5s/
│   ├── syntax-v1.jsonl
│   ├── semantics-v1.jsonl
│   ├── synthesis-v1.jsonl
│   ├── syllogism-v1.jsonl
│   └── semiotics-v1.jsonl
├── 5t/
│   ├── transfer-v1.jsonl
│   ├── triangulate-v1.jsonl
│   ├── timing-v1.jsonl
│   ├── transitivity-v1.jsonl
│   └── truth-v1.jsonl
└── README.md       # fixture authoring + version pinning policy

bench/batteries/
├── __init__.py
├── base.py         # Battery protocol + BatteryResult
├── b_5s.py         # 5S battery runner
├── b_5t.py         # 5T battery runner
└── runner.py       # CLI entry point
```

### 4.2 Battery semantics (initial cuts)

**5S — Syntax.** Each task: a string + a parser ID + expected
pass/fail. Metric: parse-pass-rate. Tests arborist's
`pi_star.text.canonicalize` (and other π*'s) at the parser layer.

**5S — Semantics.** Each task: two strings + an expected
equivalence-class flag (same/different). Metric: equivalence-
recovery-rate. Tests claim-lattice equivalence.

**5S — Synthesis.** Each task: a goal description + a fact set +
expected derivation pass/fail. Metric: derivation-pass-rate. Tests
end-to-end `arborist ask` with a constrained source set.

**5S — Syllogism.** Each task: a proof chain + a step index +
expected step-valid pass/fail. Metric: step-validity-rate.

**5S — Semiotics.** Each task: a controlled label-swap + invariant
expected. Metric: invariance-under-swap.

**5T — Transfer.** Each task: train-on-A, eval-on-B; metric is
domain-shift drop. Cross-corpus.

**5T — Triangulate.** Each task: a set of independent verifiers;
metric is conjunction-failure-rate vs single-verifier rate.

**5T — Timing.** Each task: a chronological sequence with a
reorder; metric is reorder-detection-rate.

**5T — Transitivity.** Each task: a multi-step entailment; metric
is full-chain-pass-rate.

**5T — Truth.** Each task: a verifier-acceptance recorded against
disagreement log; metric is verifier-truth-rate.

### 4.3 Fixture sourcing (Phase 1)

- 5S Syntax: synthetic templates via `bench/fixtures/_templates/`
  with committed seed.
- 5S Semantics: hand-curated from arborist's existing claim-lattice
  test fixtures.
- 5S Synthesis: hand-curated from `bench/qa_sweep.py` corpora.
- 5S Syllogism: hand-curated from `arborist/qa/parse_claims.py`
  test cases.
- 5S Semiotics: hand-curated label-swap pairs.
- 5T Transfer: cross-shard splits using existing wiki shards.
- 5T Triangulate: existing verifier-strategy outputs replayed.
- 5T Timing: small hand-curated chronology set.
- 5T Transitivity: hand-curated multi-step claims.
- 5T Truth: replays of existing audit_events with declared
  verdict.

### 4.4 Makefile

```makefile
bench-5s: ## Run 5S battery
	$(VENV)/bin/python -m bench.batteries.runner \
	    --battery 5s \
	    --fixtures bench/fixtures/5s \
	    --shards-dir $(SHARDS_DIR) \
	    --out bench/results/5s-$(shell date -u +%Y%m%dT%H%M%SZ).json

bench-5t: ## Run 5T battery
	$(VENV)/bin/python -m bench.batteries.runner \
	    --battery 5t \
	    --fixtures bench/fixtures/5t \
	    --shards-dir $(SHARDS_DIR) \
	    --out bench/results/5t-$(shell date -u +%Y%m%dT%H%M%SZ).json

bench-5s5t: bench-5s bench-5t
```

### 4.5 Tests

- Unit: each battery runner accepts a tiny fixture set and produces
  `BatteryResult`.
- Integration: `make bench-5s` against checked-in fixtures runs end
  to end and writes a result file.
- Determinism: same fixture set + same shard set + same seed →
  identical metric values across runs.
- Bench discipline: 5pp signal floor honored; n≥3 runs averaged
  before declaring a metric.

### 4.6 Bumps

- New repo subtree (`bench/fixtures/`, `bench/batteries/`).
- New Makefile targets.
- No schema changes.
- No `governance_policy_hash` changes (battery results are sibling
  artifacts; folding fixture digests into governance is a v8
  concern — ticket #000012).

---

## 5. Out of scope

- 5R battery. Deferred to Phase 2 (depends on SelfModel + memory_root).
- External-corpus expansion. Deferred to Phase 3.
- Continuous-integration scheduling. CI cron is its own concern.
- Cross-validator fitness comparison protocol. Part of v8 (ticket
  #000012).

---

## 6. Risks & open questions

- **Fixture quality.** Hand-curated fixtures bias toward existing
  bench corpora. Mitigation: explicit version pinning; v2 of each
  battery can land later with broader sourcing.
- **Coverage gaps.** 5S/5T as named in SQD aren't exhaustive of
  what AGI evaluation needs. Treat as foundation, not ceiling.
- **Determinism vs realism.** Hand-curated fixtures may overfit to
  arborist's current behavior. Cross-validator regression on novel
  fixtures (Phase 3) is the real test.

---

## 7. Status

**In progress · Phase 1a landed 2026-05-07.** Skeleton + seed
fixtures ship; full Phase 1 fixture authoring + Phase 2 (5R) +
Phase 3 (external corpora) deferred to follow-up tickets.

### Phase 1a (landed)

- `bench/batteries/{base,b_5s,b_5t,runner}.py` — Battery protocol,
  BatteryResult, fixture digest helpers, runner CLI.
- `bench/fixtures/5s/syntax-v1.jsonl` — 10 seed tasks against
  `wikitext-base@v1` and `claim-lattice@v1`. All pass.
- `bench/fixtures/5s/semantics-v1.jsonl` — 8 seed tasks
  (equivalence assertions). All pass.
- `bench/fixtures/5t/transfer-v1.jsonl` — 4 seed tasks
  (paraphrase-invariance). All pass.
- Battery runners for **5S Syntax**, **5S Semantics**, and **5T
  Transfer** are functional. Other 5S/5T sub-batteries are stubs
  returning zero-task BatteryResults.
- Makefile targets `bench-5s`, `bench-5t`, `bench-5s5t`.
- Tests: `tests/test_bench_batteries.py` — 17 cases. Full suite:
  1076 passed, 36 skipped.

### Phase 1b (landed via #000023 + #000024, 2026-05-08)

- Expand seed fixtures to 50-200 per sub-battery so signal floor
  is meaningful (5pp tolerance over n≥3 runs).
- Implement Synthesis, Syllogism, Semiotics under 5S — landed in
  ticket **#000023** (closed 2026-05-08).
- Implement Triangulate, Timing, Transitivity, Truth under 5T —
  landed in ticket **#000024** (closed 2026-05-08).

The Phase 1b work was scope-split into the per-sub-battery
tickets above for atomic landing; the Closure section below
reflects the resulting 5S × 108 / 5T × 154 fixture totals.

### Phase 2 (landed 2026-05-08 — depended on #000014 + #000017)

- `bench/batteries/b_5r.py` — React / Rearrange / Restore /
  Replicate / Resonate runners. All deterministic, no LLM-as-judge.
- 30 fixtures × 5 sub-batteries = 150 new fixtures.
- React: incorporate new facts/constraints. Workspace = (snapshot_t0,
  snapshot_t1, expected_delta).
- Rearrange: re-canonicalize different surface forms through a
  named π*; pass when bytes match `expected_equivalent`. Tests the
  order-invariance contracts in SelfModel + Memory.
- Restore: retrieve a `prior_fact` from current OR historical
  snapshots. Workspace = `(history[], current_facts[])`.
- Replicate: same input through a π* must yield byte-equal output
  across N replicas. Tests determinism contract.
- Resonate: variance check across N runs; deterministic π*'s
  yield distinct=1. Tests stability.
- `make bench-5r` + `make bench-suite` (5S+5T+5F+5R aggregate).
- Tests: 6 new in `tests/test_bench_batteries.py`. Full suite:
  1192 passed.

### Phase 3 (deferred)

- External-corpus expansion. Cross-substrate transfer testing.
  Phase 1b/2 fixtures hand-curated; Phase 3 pulls from external
  corpora to broaden signal floor. Not gated on any other ticket.

### Closure (Phase 1a + Phase 2)

`make bench-suite` runs the **complete 21-sub-battery
Dav1DPrometheus surface** end-to-end:

- 5S: Syntax · Semantics · Syllogism · Synthesis · Semiotics (108)
- 5T: Transfer · Transfer Learning · Triangulation · Truthtables ·
  Transitivity · Time (154 incl. legacy transfer)
- 5F: Function · Finetuning · Falsification · Formulate ·
  Feedback Loop (50)
- 5R: React · Rearrange · Restore · Replicate · Resonate (150)

**462 deterministic fixtures, all passing.** Phase 3 stays open
under this ticket but does not gate closure.
