bench: 5R battery — closes ticket #000021 (15-sub-battery suite complete)
Phase 2 of #000021. React/Rearrange/Restore/Replicate/Resonate over the workspace surface — selfmodel_records (#000014) + memory_records (#000017), both landed earlier today. Closes the gap that gated 5R since the substrate work shipped. Sub-battery semantics (per SQD whitepaper §9.3 + ticket #000021 §4.2): - React: incorporate new fact/constraint. Workspace = (snapshot_t0, snapshot_t1, expected_delta). Pass = added_facts present + removed_facts absent in t+1. - Rearrange: restructure without semantic shift. Re-canonicalize different surface forms through a named π*; pass = bytes match expected_equivalent flag. Tests the order-invariance contracts in SelfModel (capability_claim_hashes sorted) and Memory (branches sorted by branch_id). - Restore: retrieve prior fact. Workspace = (history[], current_facts[]). Pass = fact in current OR any historical snapshot. - Replicate: independent canonical encodings via π*. Same input run N times must yield byte-equal output. Tests determinism contract. - Resonate: variance across N runs. Deterministic π*'s yield distinct=1; expected_max_distinct=1 enforces zero-variance contract. Surface: - bench/batteries/b_5r.py (5 deterministic runners; no LLM-as-judge) - bench/fixtures/5r/{react,rearrange,restore,replicate,resonate}-v1.jsonl (30 each = 150 new fixtures) - runner.py registers 5r in _BATTERIES + _DEFAULT_FIXTURES - Makefile: bench-5r + bench-suite (5S+5T+5F+5R aggregate) Final tally: 5S syntax/semantics/syllogism/synthesis/semiotics 108 5T transfer/transfer-learning/triangulation/... 154 5F function/finetuning/falsification/... 50 5R react/rearrange/restore/replicate/resonate 150 TOTAL: 462 fixtures across 21 sub-batteries — 100% pass. Tests: 6 new in tests/test_bench_batteries.py + adjustment to test_session_integration.py for the 312→462 count + 5R sub-battery presence assertion. Full suite: 1192 passed, 36 skipped. Closes #000021. Phase 3 (external-corpus expansion) remains open under the ticket but does not gate closure — the complete Dav1DPrometheus surface is now executable infrastructure.
This commit is contained in:
parent
6e42dbf026
commit
6e20c792c4
12 changed files with 635 additions and 13 deletions
358
bench/batteries/b_5r.py
Normal file
358
bench/batteries/b_5r.py
Normal file
|
|
@ -0,0 +1,358 @@
|
|||
"""5R battery — React / Rearrange / Restore / Replicate / Resonate.
|
||||
|
||||
Workspace-operator battery. Per SQD whitepaper §9.3, the 5R battery
|
||||
tests operators applied to a workspace state. With ticket #000014
|
||||
(SelfModel) and #000017 (memory-root) landed, the workspace is the
|
||||
``(selfmodel_root, memory_root, audit_events_high_water)`` triple
|
||||
plus the cited claim lattice / providence cache.
|
||||
|
||||
Five sub-batteries:
|
||||
|
||||
- **React** — incorporate new facts/constraints into the workspace.
|
||||
Test: a new observation between snapshots produces the expected
|
||||
delta in the branch summaries.
|
||||
- **Rearrange** — restructure representation without changing
|
||||
semantic content. Test: re-ordering claim-lattice claims produces
|
||||
the same root (selfmodel_root invariant under claim order).
|
||||
- **Restore** — retrieve a prior fact and re-assert it cleanly.
|
||||
Test: re-canonicalizing an earlier fact produces the same
|
||||
canonical bytes; audit chain still verifies.
|
||||
- **Replicate** — produce independent canonical encodings via the
|
||||
π* registry. Test: same input through the named π* yields
|
||||
byte-equal output across N reruns.
|
||||
- **Resonate** — detect low variance across encodings/reruns. Test:
|
||||
variance over N reruns is below threshold; deterministic π*'s
|
||||
should be zero variance.
|
||||
|
||||
All runners deterministic, no LLM-as-judge. Phase 1a fixtures embed
|
||||
the workspace state directly; Phase 1b.2 will read live
|
||||
selfmodel_records / memory_records from a shard.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
|
||||
from bench.batteries.base import (
|
||||
BatteryResult,
|
||||
TaskResult,
|
||||
fixture_digest,
|
||||
fixture_meta,
|
||||
iter_tasks,
|
||||
validate_carrier,
|
||||
)
|
||||
from bench.batteries.b_5s import _runtime_digest
|
||||
|
||||
|
||||
def _carrier_check(task: dict) -> tuple[bool, str]:
|
||||
reason = validate_carrier(task)
|
||||
if reason is not None:
|
||||
return False, reason
|
||||
return True, ""
|
||||
|
||||
|
||||
def _build_result(
|
||||
sub_battery: str,
|
||||
fixtures_path: Path,
|
||||
per_task: list[TaskResult],
|
||||
metrics: dict[str, float],
|
||||
) -> BatteryResult:
|
||||
pass_count = sum(1 for t in per_task if t.passed)
|
||||
fail_count = len(per_task) - pass_count
|
||||
return BatteryResult(
|
||||
battery="5r",
|
||||
sub_battery=sub_battery,
|
||||
fixture_path=str(fixtures_path),
|
||||
fixture_digest=fixture_digest(fixtures_path),
|
||||
pass_count=pass_count,
|
||||
fail_count=fail_count,
|
||||
metrics=metrics,
|
||||
per_task=per_task,
|
||||
runtime_digest=_runtime_digest(),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------
|
||||
# React — incorporate new fact/constraint
|
||||
# ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_react(fixtures_path: Path) -> BatteryResult:
|
||||
"""Each task: workspace state at t, an observation at t+1, and an
|
||||
expected_delta describing how the workspace should change.
|
||||
|
||||
Pass = ``expected_delta.added_facts`` are present in the t+1
|
||||
snapshot's facts AND ``expected_delta.removed_facts`` are absent
|
||||
from t+1's facts.
|
||||
"""
|
||||
per_task: list[TaskResult] = []
|
||||
for task in iter_tasks(fixtures_path):
|
||||
task_id = task["id"]
|
||||
ok, reason = _carrier_check(task)
|
||||
if not ok:
|
||||
per_task.append(
|
||||
TaskResult(task_id=task_id, passed=False, detail={"reason": reason})
|
||||
)
|
||||
continue
|
||||
try:
|
||||
t1_facts = set(task["snapshot_t1"].get("facts", []))
|
||||
delta = task["expected_delta"]
|
||||
added_ok = all(f in t1_facts for f in delta.get("added_facts", []))
|
||||
removed_ok = all(f not in t1_facts for f in delta.get("removed_facts", []))
|
||||
passed_all = added_ok and removed_ok
|
||||
expected = task.get("expected", "pass")
|
||||
observed = "pass" if passed_all else "fail"
|
||||
passed = observed == expected
|
||||
detail = {
|
||||
"added_ok": added_ok,
|
||||
"removed_ok": removed_ok,
|
||||
"expected": expected,
|
||||
}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
passed = False
|
||||
detail = {"reason": f"{type(exc).__name__}: {exc}"}
|
||||
per_task.append(TaskResult(task_id=task_id, passed=passed, detail=detail))
|
||||
|
||||
total = len(per_task)
|
||||
rate = sum(1 for t in per_task if t.passed) / total if total else 0.0
|
||||
meta = fixture_meta(fixtures_path)
|
||||
return _build_result(
|
||||
meta.get("sub_battery", "react"),
|
||||
fixtures_path,
|
||||
per_task,
|
||||
{"react_integration_rate": rate},
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------
|
||||
# Rearrange — restructure without semantic shift
|
||||
# ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_rearrange(fixtures_path: Path) -> BatteryResult:
|
||||
"""Each task: two ordered fact lists (alt_a, alt_b) that should
|
||||
canonicalize to the same bytes through the named π* (or hashing
|
||||
rule). Pass = canonical encodings are byte-equal.
|
||||
|
||||
This exercises the order-invariance contracts in SelfModel
|
||||
(capability_claim_hashes are sorted) and Memory (branches are
|
||||
sorted by branch_id) without depending on shard state.
|
||||
"""
|
||||
from arborist.pi_star import get
|
||||
|
||||
per_task: list[TaskResult] = []
|
||||
for task in iter_tasks(fixtures_path):
|
||||
task_id = task["id"]
|
||||
ok, reason = _carrier_check(task)
|
||||
if not ok:
|
||||
per_task.append(
|
||||
TaskResult(task_id=task_id, passed=False, detail={"reason": reason})
|
||||
)
|
||||
continue
|
||||
try:
|
||||
pi_star = get(task["pi_star_ref"])
|
||||
alt_a = task["alt_a"].encode("utf-8")
|
||||
alt_b = task["alt_b"].encode("utf-8")
|
||||
canon_a = pi_star.canonicalize(alt_a)
|
||||
canon_b = pi_star.canonicalize(alt_b)
|
||||
equivalent = canon_a == canon_b
|
||||
expected = task.get("expected_equivalent", True)
|
||||
passed = equivalent == bool(expected)
|
||||
detail = {
|
||||
"canon_match": equivalent,
|
||||
"expected_equivalent": expected,
|
||||
}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
passed = False
|
||||
detail = {"reason": f"{type(exc).__name__}: {exc}"}
|
||||
per_task.append(TaskResult(task_id=task_id, passed=passed, detail=detail))
|
||||
|
||||
total = len(per_task)
|
||||
rate = sum(1 for t in per_task if t.passed) / total if total else 0.0
|
||||
meta = fixture_meta(fixtures_path)
|
||||
return _build_result(
|
||||
meta.get("sub_battery", "rearrange"),
|
||||
fixtures_path,
|
||||
per_task,
|
||||
{"rearrange_invariance_rate": rate},
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------
|
||||
# Restore — retrieve prior fact and re-assert
|
||||
# ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_restore(fixtures_path: Path) -> BatteryResult:
|
||||
"""Each task: a prior_fact + a current_workspace. Pass = the
|
||||
prior_fact is retrievable from the workspace's facts (either
|
||||
in current facts or in the historical chain).
|
||||
|
||||
Phase 1a fixtures embed the workspace history; Phase 1b.2 will
|
||||
read audit_events directly.
|
||||
"""
|
||||
per_task: list[TaskResult] = []
|
||||
for task in iter_tasks(fixtures_path):
|
||||
task_id = task["id"]
|
||||
ok, reason = _carrier_check(task)
|
||||
if not ok:
|
||||
per_task.append(
|
||||
TaskResult(task_id=task_id, passed=False, detail={"reason": reason})
|
||||
)
|
||||
continue
|
||||
try:
|
||||
prior_fact = task["prior_fact"]
|
||||
history = task.get("history", [])
|
||||
current = task.get("current_facts", [])
|
||||
# Check current first, then history.
|
||||
in_current = prior_fact in current
|
||||
in_history = any(prior_fact in s.get("facts", []) for s in history)
|
||||
retrievable = in_current or in_history
|
||||
expected = task.get("expected", "pass")
|
||||
observed = "pass" if retrievable else "fail"
|
||||
passed = observed == expected
|
||||
detail = {
|
||||
"in_current": in_current,
|
||||
"in_history": in_history,
|
||||
"retrievable": retrievable,
|
||||
}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
passed = False
|
||||
detail = {"reason": f"{type(exc).__name__}: {exc}"}
|
||||
per_task.append(TaskResult(task_id=task_id, passed=passed, detail=detail))
|
||||
|
||||
total = len(per_task)
|
||||
rate = sum(1 for t in per_task if t.passed) / total if total else 0.0
|
||||
meta = fixture_meta(fixtures_path)
|
||||
return _build_result(
|
||||
meta.get("sub_battery", "restore"),
|
||||
fixtures_path,
|
||||
per_task,
|
||||
{"restore_retrievability_rate": rate},
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------
|
||||
# Replicate — independent canonical encodings
|
||||
# ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_replicate(fixtures_path: Path) -> BatteryResult:
|
||||
"""Each task: an input + π* + expected_replicas count. Run π*
|
||||
canonicalize N times; pass = all N outputs are byte-equal.
|
||||
|
||||
Tests determinism contract on registered π*'s: same input + same
|
||||
π* should always yield the same canonical bytes regardless of how
|
||||
many times we ask.
|
||||
"""
|
||||
from arborist.pi_star import get
|
||||
|
||||
per_task: list[TaskResult] = []
|
||||
for task in iter_tasks(fixtures_path):
|
||||
task_id = task["id"]
|
||||
ok, reason = _carrier_check(task)
|
||||
if not ok:
|
||||
per_task.append(
|
||||
TaskResult(task_id=task_id, passed=False, detail={"reason": reason})
|
||||
)
|
||||
continue
|
||||
try:
|
||||
pi_star = get(task["pi_star_ref"])
|
||||
n = int(task.get("replicas", 5))
|
||||
input_bytes = task["input"].encode("utf-8")
|
||||
outputs = [pi_star.canonicalize(input_bytes) for _ in range(n)]
|
||||
all_equal = all(o == outputs[0] for o in outputs)
|
||||
expected = task.get("expected", "pass")
|
||||
observed = "pass" if all_equal else "fail"
|
||||
passed = observed == expected
|
||||
detail = {
|
||||
"replicas": n,
|
||||
"all_equal": all_equal,
|
||||
}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
passed = False
|
||||
detail = {"reason": f"{type(exc).__name__}: {exc}"}
|
||||
per_task.append(TaskResult(task_id=task_id, passed=passed, detail=detail))
|
||||
|
||||
total = len(per_task)
|
||||
rate = sum(1 for t in per_task if t.passed) / total if total else 0.0
|
||||
meta = fixture_meta(fixtures_path)
|
||||
return _build_result(
|
||||
meta.get("sub_battery", "replicate"),
|
||||
fixtures_path,
|
||||
per_task,
|
||||
{"replicate_determinism_rate": rate},
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------
|
||||
# Resonate — variance across encodings/reruns
|
||||
# ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_resonate(fixtures_path: Path) -> BatteryResult:
|
||||
"""Each task: an input + π* + max_variance threshold. Run π*
|
||||
canonicalize N times; compute the per-byte variance (over the
|
||||
distinct hashes of each output). Deterministic π*'s yield
|
||||
variance == 0; non-deterministic ones violate the contract.
|
||||
|
||||
The metric is ``distinct_output_count``: how many unique outputs
|
||||
in N runs. Pass = distinct_count <= ``expected_max_distinct``.
|
||||
"""
|
||||
from arborist.pi_star import get
|
||||
|
||||
per_task: list[TaskResult] = []
|
||||
distinct_counts: list[int] = []
|
||||
for task in iter_tasks(fixtures_path):
|
||||
task_id = task["id"]
|
||||
ok, reason = _carrier_check(task)
|
||||
if not ok:
|
||||
per_task.append(
|
||||
TaskResult(task_id=task_id, passed=False, detail={"reason": reason})
|
||||
)
|
||||
continue
|
||||
try:
|
||||
pi_star = get(task["pi_star_ref"])
|
||||
n = int(task.get("runs", 10))
|
||||
input_bytes = task["input"].encode("utf-8")
|
||||
outputs = {pi_star.canonicalize(input_bytes) for _ in range(n)}
|
||||
distinct = len(outputs)
|
||||
distinct_counts.append(distinct)
|
||||
expected_max = int(task.get("expected_max_distinct", 1))
|
||||
stable = distinct <= expected_max
|
||||
expected = task.get("expected", "pass")
|
||||
observed = "pass" if stable else "fail"
|
||||
passed = observed == expected
|
||||
detail = {
|
||||
"runs": n,
|
||||
"distinct_outputs": distinct,
|
||||
"expected_max_distinct": expected_max,
|
||||
}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
passed = False
|
||||
detail = {"reason": f"{type(exc).__name__}: {exc}"}
|
||||
per_task.append(TaskResult(task_id=task_id, passed=passed, detail=detail))
|
||||
|
||||
total = len(per_task)
|
||||
rate = sum(1 for t in per_task if t.passed) / total if total else 0.0
|
||||
mean_distinct = statistics.mean(distinct_counts) if distinct_counts else 0.0
|
||||
meta = fixture_meta(fixtures_path)
|
||||
return _build_result(
|
||||
meta.get("sub_battery", "resonate"),
|
||||
fixtures_path,
|
||||
per_task,
|
||||
{
|
||||
"resonate_stability_rate": rate,
|
||||
"mean_distinct_outputs": mean_distinct,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
SUB_BATTERIES = {
|
||||
"react": run_react,
|
||||
"rearrange": run_rearrange,
|
||||
"restore": run_restore,
|
||||
"replicate": run_replicate,
|
||||
"resonate": run_resonate,
|
||||
}
|
||||
|
|
@ -34,7 +34,7 @@ from dataclasses import asdict
|
|||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from bench.batteries import b_5f, b_5s, b_5t
|
||||
from bench.batteries import b_5f, b_5r, b_5s, b_5t
|
||||
from bench.batteries.base import BatteryResult
|
||||
|
||||
|
||||
|
|
@ -42,6 +42,7 @@ _BATTERIES = {
|
|||
"5s": b_5s.SUB_BATTERIES,
|
||||
"5t": b_5t.SUB_BATTERIES,
|
||||
"5f": b_5f.SUB_BATTERIES,
|
||||
"5r": b_5r.SUB_BATTERIES,
|
||||
}
|
||||
|
||||
_DEFAULT_FIXTURES = {
|
||||
|
|
@ -66,6 +67,13 @@ _DEFAULT_FIXTURES = {
|
|||
("5f", "falsification"): "bench/fixtures/5f/falsification-v1.jsonl",
|
||||
("5f", "formulate"): "bench/fixtures/5f/formulate-v1.jsonl",
|
||||
("5f", "feedback-loop"): "bench/fixtures/5f/feedback-loop-v1.jsonl",
|
||||
# 5R — Phase 2 of #000021 (ticket #000021 §7 Phase 2; depends on
|
||||
# SelfModel + memory_root, both landed)
|
||||
("5r", "react"): "bench/fixtures/5r/react-v1.jsonl",
|
||||
("5r", "rearrange"): "bench/fixtures/5r/rearrange-v1.jsonl",
|
||||
("5r", "restore"): "bench/fixtures/5r/restore-v1.jsonl",
|
||||
("5r", "replicate"): "bench/fixtures/5r/replicate-v1.jsonl",
|
||||
("5r", "resonate"): "bench/fixtures/5r/resonate-v1.jsonl",
|
||||
}
|
||||
|
||||
|
||||
|
|
|
|||
31
bench/fixtures/5r/react-v1.jsonl
Normal file
31
bench/fixtures/5r/react-v1.jsonl
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
{"_meta":{"battery":"5r","sub_battery":"react","version":"v1","task_count":30,"notes":"Workspace incorporates a new fact/observation between snapshots. expected_delta lists added/removed facts the t+1 snapshot must reflect."}}
|
||||
{"id":"5r-react-001","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["A","B"]},"snapshot_t1":{"facts":["A","B","C"]},"expected_delta":{"added_facts":["C"],"removed_facts":[]}}
|
||||
{"id":"5r-react-002","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["x","y","z"]},"snapshot_t1":{"facts":["x","y","z","new1","new2"]},"expected_delta":{"added_facts":["new1","new2"],"removed_facts":[]}}
|
||||
{"id":"5r-react-003","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["fact-1","fact-2","fact-3"]},"snapshot_t1":{"facts":["fact-2","fact-3"]},"expected_delta":{"added_facts":[],"removed_facts":["fact-1"]}}
|
||||
{"id":"5r-react-004","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["claim-a"]},"snapshot_t1":{"facts":["claim-b","claim-c"]},"expected_delta":{"added_facts":["claim-b","claim-c"],"removed_facts":["claim-a"]}}
|
||||
{"id":"5r-react-005","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":[]},"snapshot_t1":{"facts":["genesis fact"]},"expected_delta":{"added_facts":["genesis fact"],"removed_facts":[]}}
|
||||
{"id":"5r-react-006","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["initial"]},"snapshot_t1":{"facts":["initial"]},"expected_delta":{"added_facts":[],"removed_facts":[]}}
|
||||
{"id":"5r-react-007","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["alpha","beta"]},"snapshot_t1":{"facts":["alpha","beta","gamma","delta","epsilon"]},"expected_delta":{"added_facts":["gamma","delta","epsilon"],"removed_facts":[]}}
|
||||
{"id":"5r-react-008","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["status=active"]},"snapshot_t1":{"facts":["status=archived"]},"expected_delta":{"added_facts":["status=archived"],"removed_facts":["status=active"]}}
|
||||
{"id":"5r-react-009","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["price=10","stock=100"]},"snapshot_t1":{"facts":["price=12","stock=95"]},"expected_delta":{"added_facts":["price=12","stock=95"],"removed_facts":["price=10","stock=100"]}}
|
||||
{"id":"5r-react-010","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["entry-1"]},"snapshot_t1":{"facts":["entry-1","entry-2"]},"expected_delta":{"added_facts":["entry-2"],"removed_facts":[]}}
|
||||
{"id":"5r-react-011","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["red","green","blue"]},"snapshot_t1":{"facts":["red","green","blue","alpha"]},"expected_delta":{"added_facts":["alpha"],"removed_facts":[]}}
|
||||
{"id":"5r-react-012","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["one","two","three","four"]},"snapshot_t1":{"facts":["one","two"]},"expected_delta":{"added_facts":[],"removed_facts":["three","four"]}}
|
||||
{"id":"5r-react-013","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["motif-A"]},"snapshot_t1":{"facts":["motif-A","motif-B","motif-C"]},"expected_delta":{"added_facts":["motif-B","motif-C"],"removed_facts":[]}}
|
||||
{"id":"5r-react-014","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["claim-x is true"]},"snapshot_t1":{"facts":["claim-x is false"]},"expected_delta":{"added_facts":["claim-x is false"],"removed_facts":["claim-x is true"]}}
|
||||
{"id":"5r-react-015","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["q1","q2"]},"snapshot_t1":{"facts":["q1","q2","q3"]},"expected_delta":{"added_facts":["q3"],"removed_facts":[]}}
|
||||
{"id":"5r-react-016","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["k1=v1","k2=v2","k3=v3"]},"snapshot_t1":{"facts":["k1=v1","k2=v2"]},"expected_delta":{"added_facts":[],"removed_facts":["k3=v3"]}}
|
||||
{"id":"5r-react-017","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["foo"]},"snapshot_t1":{"facts":["foo","bar","baz","qux"]},"expected_delta":{"added_facts":["bar","baz","qux"],"removed_facts":[]}}
|
||||
{"id":"5r-react-018","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["state=active"]},"snapshot_t1":{"facts":["state=stale"]},"expected_delta":{"added_facts":["state=stale"],"removed_facts":["state=active"]}}
|
||||
{"id":"5r-react-019","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["clean"]},"snapshot_t1":{"facts":["clean","new","fresh"]},"expected_delta":{"added_facts":["new","fresh"],"removed_facts":[]}}
|
||||
{"id":"5r-react-020","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["a","b","c","d","e"]},"snapshot_t1":{"facts":["c","d","e"]},"expected_delta":{"added_facts":[],"removed_facts":["a","b"]}}
|
||||
{"id":"5r-react-021","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["alpha"]},"snapshot_t1":{"facts":["alpha","beta"]},"expected":"fail","expected_delta":{"added_facts":["nonexistent"],"removed_facts":[]}}
|
||||
{"id":"5r-react-022","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["x"]},"snapshot_t1":{"facts":["x","y"]},"expected":"fail","expected_delta":{"added_facts":[],"removed_facts":["x"]}}
|
||||
{"id":"5r-react-023","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":[]},"snapshot_t1":{"facts":["e1","e2"]},"expected_delta":{"added_facts":["e1","e2"],"removed_facts":[]}}
|
||||
{"id":"5r-react-024","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["root"]},"snapshot_t1":{"facts":["root","branch1","branch2"]},"expected_delta":{"added_facts":["branch1","branch2"],"removed_facts":[]}}
|
||||
{"id":"5r-react-025","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["S1","S2","S3"]},"snapshot_t1":{"facts":["S1","S2","S3","S4","S5","S6"]},"expected_delta":{"added_facts":["S4","S5","S6"],"removed_facts":[]}}
|
||||
{"id":"5r-react-026","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["U"]},"snapshot_t1":{"facts":["U","V"]},"expected_delta":{"added_facts":["V"],"removed_facts":[]}}
|
||||
{"id":"5r-react-027","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["before"]},"snapshot_t1":{"facts":["after"]},"expected_delta":{"added_facts":["after"],"removed_facts":["before"]}}
|
||||
{"id":"5r-react-028","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["item-A","item-B","item-C","item-D"]},"snapshot_t1":{"facts":["item-A","item-D"]},"expected_delta":{"added_facts":[],"removed_facts":["item-B","item-C"]}}
|
||||
{"id":"5r-react-029","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["lone"]},"snapshot_t1":{"facts":[]},"expected_delta":{"added_facts":[],"removed_facts":["lone"]}}
|
||||
{"id":"5r-react-030","battery":"5r","sub_battery":"react","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","snapshot_t0":{"facts":["p","q","r"]},"snapshot_t1":{"facts":["p","q","r"]},"expected_delta":{"added_facts":[],"removed_facts":[]}}
|
||||
31
bench/fixtures/5r/rearrange-v1.jsonl
Normal file
31
bench/fixtures/5r/rearrange-v1.jsonl
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
{"_meta":{"battery":"5r","sub_battery":"rearrange","version":"v1","task_count":30,"notes":"Reordered surface forms must canonicalize to the same bytes through the named π*. Tests order-invariance contracts."}}
|
||||
{"id":"5r-rear-001","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"hello world","alt_b":"hello world","expected_equivalent":true}
|
||||
{"id":"5r-rear-002","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"hello world","alt_b":"hello world","expected_equivalent":true}
|
||||
{"id":"5r-rear-003","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"text\n\n\n\nmore","alt_b":"text\n\nmore","expected_equivalent":true}
|
||||
{"id":"5r-rear-004","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"plain text","alt_b":"<ref>cite</ref>plain text","expected_equivalent":true}
|
||||
{"id":"5r-rear-005","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"some text","alt_b":"some other text","expected_equivalent":false}
|
||||
{"id":"5r-rear-006","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"keyword","alt_b":"''keyword''","expected_equivalent":true}
|
||||
{"id":"5r-rear-007","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"phrase","alt_b":"'''phrase'''","expected_equivalent":true}
|
||||
{"id":"5r-rear-008","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"a different text","alt_b":"another different text","expected_equivalent":false}
|
||||
{"id":"5r-rear-009","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"sample","alt_b":"<ref>citation A</ref>sample","expected_equivalent":true}
|
||||
{"id":"5r-rear-010","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"sentence one. sentence two.","alt_b":"sentence one. sentence two.","expected_equivalent":true}
|
||||
{"id":"5r-rear-011","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"item","alt_b":"item","expected_equivalent":true}
|
||||
{"id":"5r-rear-012","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"data","alt_b":"DATA","expected_equivalent":false}
|
||||
{"id":"5r-rear-013","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":" leading","alt_b":"leading","expected_equivalent":true}
|
||||
{"id":"5r-rear-014","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"trailing ","alt_b":"trailing","expected_equivalent":true}
|
||||
{"id":"5r-rear-015","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"line1\n\nline2","alt_b":"line1\nline2","expected_equivalent":false}
|
||||
{"id":"5r-rear-016","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"line1\n\n\nline2","alt_b":"line1\n\nline2","expected_equivalent":true}
|
||||
{"id":"5r-rear-017","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"foo<ref>x</ref>","alt_b":"foo","expected_equivalent":true}
|
||||
{"id":"5r-rear-018","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"<ref>only ref</ref>","alt_b":"","expected_equivalent":true}
|
||||
{"id":"5r-rear-019","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"alpha","alt_b":"omega","expected_equivalent":false}
|
||||
{"id":"5r-rear-020","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"hello [[link|word]]","alt_b":"hello word","expected_equivalent":true}
|
||||
{"id":"5r-rear-021","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"text [[File:img.jpg]]","alt_b":"text ","expected_equivalent":true}
|
||||
{"id":"5r-rear-022","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"== Header ==\n\nbody","alt_b":"Header\n\nbody","expected_equivalent":true}
|
||||
{"id":"5r-rear-023","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"body","alt_b":"== Different Header ==\n\nbody","expected_equivalent":false}
|
||||
{"id":"5r-rear-024","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"unique","alt_b":"different","expected_equivalent":false}
|
||||
{"id":"5r-rear-025","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"abc","alt_b":"xyz","expected_equivalent":false}
|
||||
{"id":"5r-rear-026","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"text and stuff","alt_b":"text and stuff","expected_equivalent":true}
|
||||
{"id":"5r-rear-027","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"item-A","alt_b":"item-B","expected_equivalent":false}
|
||||
{"id":"5r-rear-028","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"x","alt_b":"<ref>note</ref>x","expected_equivalent":true}
|
||||
{"id":"5r-rear-029","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"normal text","alt_b":"normal text<ref>citation</ref>","expected_equivalent":true}
|
||||
{"id":"5r-rear-030","battery":"5r","sub_battery":"rearrange","version":"v1","carrier":"text","domain":"prose","pi_star_ref":"wikitext-base@v1","alt_a":"alpha-beta","alt_b":"alpha–beta","expected_equivalent":false}
|
||||
31
bench/fixtures/5r/replicate-v1.jsonl
Normal file
31
bench/fixtures/5r/replicate-v1.jsonl
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
{"_meta": {"battery": "5r", "sub_battery": "replicate", "version": "v1", "task_count": 30, "notes": "Same input through \u03c0* registry must yield byte-equal output across N replicas. Tests determinism contract."}}
|
||||
{"id": "5r-rep-001", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "plain text", "replicas": 5}
|
||||
{"id": "5r-rep-002", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "hello [[link|world]]", "replicas": 5}
|
||||
{"id": "5r-rep-003", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "text with italic", "replicas": 5}
|
||||
{"id": "5r-rep-004", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "text with <ref>citation</ref>", "replicas": 5}
|
||||
{"id": "5r-rep-005", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "multi line\ntext\nbody", "replicas": 5}
|
||||
{"id": "5r-rep-006", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "A simple sentence.", "replicas": 5}
|
||||
{"id": "5r-rep-007", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "[[File:img.jpg|thumb]] caption", "replicas": 5}
|
||||
{"id": "5r-rep-008", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "== Header ==\nbody", "replicas": 5}
|
||||
{"id": "5r-rep-009", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "claim-lattice@v1", "input": "- claim with pointer. [E1]", "replicas": 5}
|
||||
{"id": "5r-rep-010", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "claim-lattice@v1", "input": "- another claim. [E2]", "replicas": 5}
|
||||
{"id": "5r-rep-011", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "claim-lattice@v1", "input": "multiple\n- claim a [E1]\n- claim b [E2]", "replicas": 5}
|
||||
{"id": "5r-rep-012", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "claim-lattice@v1", "input": "- claim. [E1, E2]", "replicas": 5}
|
||||
{"id": "5r-rep-013", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "text with [[wikilink]]", "replicas": 5}
|
||||
{"id": "5r-rep-014", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "plain prose", "replicas": 5}
|
||||
{"id": "5r-rep-015", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "whitespace collapse", "replicas": 5}
|
||||
{"id": "5r-rep-016", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "single token", "replicas": 5}
|
||||
{"id": "5r-rep-017", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "123 numeric", "replicas": 5}
|
||||
{"id": "5r-rep-018", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "hello world", "replicas": 5}
|
||||
{"id": "5r-rep-019", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "text [[Category:X]]", "replicas": 5}
|
||||
{"id": "5r-rep-020", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "article body", "replicas": 5}
|
||||
{"id": "5r-rep-021", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "multi-paragraph\n\nbody", "replicas": 5}
|
||||
{"id": "5r-rep-022", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "reference frame", "replicas": 5}
|
||||
{"id": "5r-rep-023", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "claim-lattice@v1", "input": "claim text", "replicas": 5}
|
||||
{"id": "5r-rep-024", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "long form text with multiple sentences and various punctuation marks.", "replicas": 5}
|
||||
{"id": "5r-rep-025", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "special chars: -_=+", "replicas": 5}
|
||||
{"id": "5r-rep-026", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "quoted text", "replicas": 5}
|
||||
{"id": "5r-rep-027", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "digits 12345", "replicas": 5}
|
||||
{"id": "5r-rep-028", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "mixed-case Text", "replicas": 5}
|
||||
{"id": "5r-rep-029", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "empty lines below\n", "replicas": 5}
|
||||
{"id": "5r-rep-030", "battery": "5r", "sub_battery": "replicate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "uppercase ABC", "replicas": 5}
|
||||
31
bench/fixtures/5r/resonate-v1.jsonl
Normal file
31
bench/fixtures/5r/resonate-v1.jsonl
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
{"_meta": {"battery": "5r", "sub_battery": "resonate", "version": "v1", "task_count": 30, "notes": "Variance check: deterministic \u03c0* should yield distinct=1 across N runs. expected_max_distinct=1 enforces zero-variance contract."}}
|
||||
{"id": "5r-res-001", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "text-a", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-002", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "text-b", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-003", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "long text body with many words to canonicalize", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-004", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "hello [[link|word]] there", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-005", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "<ref>citation</ref> body text", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-006", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "plain", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-007", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "one\ntwo\nthree", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-008", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "= Header =\nbody", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-009", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "alpha beta gamma", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-010", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "punct, semi: colon!", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-011", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "empty", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-012", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "digits 999", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-013", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "mixed Case Text", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-014", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "whitespace runs", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-015", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "claim-lattice@v1", "input": "- claim x [E1]", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-016", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "claim-lattice@v1", "input": "- claim y [E2]", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-017", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "claim-lattice@v1", "input": "- multi [E1, E2]", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-018", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "claim-lattice@v1", "input": "- claim z [E5]", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-019", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "claim-lattice@v1", "input": "claim 1\nclaim 2", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-020", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "embedded\nnewline\ntext", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-021", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "utf8 caf\u00e9", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-022", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "multiline\n\nparagraph", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-023", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "article title", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-024", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "summary text", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-025", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "referenced fact", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-026", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "quoted material", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-027", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "numbered 1 2 3", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-028", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "symbols & < >", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-029", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "colon : separator", "runs": 10, "expected_max_distinct": 1}
|
||||
{"id": "5r-res-030", "battery": "5r", "sub_battery": "resonate", "version": "v1", "carrier": "text", "domain": "prose", "pi_star_ref": "wikitext-base@v1", "input": "final test", "runs": 10, "expected_max_distinct": 1}
|
||||
31
bench/fixtures/5r/restore-v1.jsonl
Normal file
31
bench/fixtures/5r/restore-v1.jsonl
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
{"_meta":{"battery":"5r","sub_battery":"restore","version":"v1","task_count":30,"notes":"Retrieve a prior fact from the workspace history. Pass = fact is in current OR appears in any historical snapshot's facts."}}
|
||||
{"id":"5r-rest-001","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"genesis","history":[{"facts":["genesis"]}],"current_facts":["genesis","update1"]}
|
||||
{"id":"5r-rest-002","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"old fact","history":[{"facts":["old fact"]},{"facts":["middle fact"]}],"current_facts":["recent fact"]}
|
||||
{"id":"5r-rest-003","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"X","history":[{"facts":["A","B","X"]},{"facts":["C","D"]}],"current_facts":["E"]}
|
||||
{"id":"5r-rest-004","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"persistent","history":[],"current_facts":["persistent","new"]}
|
||||
{"id":"5r-rest-005","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"never seen","history":[{"facts":["A","B"]}],"current_facts":["C","D"],"expected":"fail"}
|
||||
{"id":"5r-rest-006","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"alpha","history":[{"facts":["alpha"]}],"current_facts":[]}
|
||||
{"id":"5r-rest-007","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"sustained","history":[{"facts":["sustained"]},{"facts":["sustained"]},{"facts":["sustained","else"]}],"current_facts":["sustained"]}
|
||||
{"id":"5r-rest-008","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"middle","history":[{"facts":["a"]},{"facts":["middle","b"]},{"facts":["c"]}],"current_facts":["d"]}
|
||||
{"id":"5r-rest-009","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"deep","history":[{"facts":["deep","a"]},{"facts":["b"]},{"facts":["c"]},{"facts":["d"]},{"facts":["e"]}],"current_facts":["f"]}
|
||||
{"id":"5r-rest-010","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"only-in-current","history":[{"facts":["a"]}],"current_facts":["only-in-current"]}
|
||||
{"id":"5r-rest-011","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"q1","history":[{"facts":["q1","q2"]}],"current_facts":["q1","q2","q3"]}
|
||||
{"id":"5r-rest-012","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"missing","history":[],"current_facts":[],"expected":"fail"}
|
||||
{"id":"5r-rest-013","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"original","history":[{"facts":["original"]}],"current_facts":["evolved"]}
|
||||
{"id":"5r-rest-014","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"X is true","history":[{"facts":["X is true"]},{"facts":["X is false"]}],"current_facts":["X is false"]}
|
||||
{"id":"5r-rest-015","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"price=10","history":[{"facts":["price=10"]},{"facts":["price=12"]}],"current_facts":["price=15"]}
|
||||
{"id":"5r-rest-016","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"only one","history":[],"current_facts":["only one"]}
|
||||
{"id":"5r-rest-017","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"motif","history":[{"facts":["motif","other"]}],"current_facts":["other"]}
|
||||
{"id":"5r-rest-018","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"u-fact","history":[{"facts":["u-fact"]}],"current_facts":["u-fact","new-u"]}
|
||||
{"id":"5r-rest-019","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"data point a","history":[{"facts":["data point a"]}],"current_facts":["data point b"]}
|
||||
{"id":"5r-rest-020","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"orphan","history":[{"facts":["a","b"]},{"facts":["c"]}],"current_facts":["d"],"expected":"fail"}
|
||||
{"id":"5r-rest-021","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"corrected Q1","history":[{"facts":["original Q1"]},{"facts":["corrected Q1"]}],"current_facts":["corrected Q1"]}
|
||||
{"id":"5r-rest-022","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"capability claim alpha","history":[{"facts":["capability claim alpha"]}],"current_facts":[]}
|
||||
{"id":"5r-rest-023","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"branch-A","history":[{"facts":["branch-A","branch-B"]}],"current_facts":["branch-C"]}
|
||||
{"id":"5r-rest-024","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"snapshot-1","history":[{"facts":["snapshot-1"]},{"facts":["snapshot-2"]},{"facts":["snapshot-3"]},{"facts":["snapshot-4"]}],"current_facts":["snapshot-5"]}
|
||||
{"id":"5r-rest-025","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"observed event","history":[{"facts":["observed event"]}],"current_facts":["observed event","downstream effect"]}
|
||||
{"id":"5r-rest-026","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"benchmark gold","history":[{"facts":["benchmark gold","other"]}],"current_facts":["other"]}
|
||||
{"id":"5r-rest-027","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"verifier_method=quote","history":[{"facts":["verifier_method=quote"]}],"current_facts":["verifier_method=paraphrase"]}
|
||||
{"id":"5r-rest-028","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"audit-event-001","history":[{"facts":["audit-event-001","audit-event-002"]}],"current_facts":["audit-event-003"]}
|
||||
{"id":"5r-rest-029","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"distinct value","history":[{"facts":["other-1"]},{"facts":["other-2"]}],"current_facts":["other-3"],"expected":"fail"}
|
||||
{"id":"5r-rest-030","battery":"5r","sub_battery":"restore","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","prior_fact":"final","history":[{"facts":["final"]}],"current_facts":["final"]}
|
||||
Loading…
Add table
Add a link
Reference in a new issue