russell@unturf.com
1f4c8b93c8
Four-item fan-out per fox's 1/2/4/5 directive on the open menu.
(1) code-py-ast@v1 graduates from stub
First non-text canonicalizer. Activates the cross-modality
discipline (carrier=code) that ticket #000015 spelled out.
Algorithm: ast.parse → walk node._fields in lexical order →
emit deterministic S-expression. Source positions skipped
naturally (not in _fields).
Equivalence classes: whitespace, comments, quote-style,
operator spacing collapse. Identifier names, operator types,
argument order remain distinct.
Projective (canonical = S-expression text, NOT Python).
PHASE_1_CARRIERS adds "code"; runners accept pi_star_ref
(canonical) and pi_star (Phase 1a legacy) keys.
Tests: 11 new in test_pi_star.py.
Bench fixtures: bench/fixtures/5s/{syntax,semantics}-code-v1.jsonl
(10 + 12 = 22 code-carrier fixtures, all pass).
Makefile: bench-5s-code target.
(2) CI gate via .gitlab-ci.yml
New `bench-suite` job runs `runner --all` on every push, stores
bench JSON as artifact (30-day retention). Fails the pipeline
if any fixture fails.
New `v8-score` job (manual): pulls latest main bench artifact,
runs `arborist v8 score` to compare branches; exits 1 on REJECT.
Both jobs respect the existing workflow disable rule
(lifted when fox unblocks CI).
(4) 5R Phase 1b.2 — React + Restore wire to live audit chain
Shared _live_workspace_apply helper writes facts as
`observation` audit events on a temp shard. React live mode:
snapshot_t1.facts written + audit bodies queried for
expected_delta substrings. Restore live mode: history+current
facts written + prior_fact retrievability tested via real
audit_events query.
12 react-live + 12 restore-live fixtures = 24 new live fixtures.
Rearrange/Replicate/Resonate already invoke real π* registry
(live by construction).
(5) 5F live fixture expansion: ~12-15 → 30 each (150 total)
Five 5F sub-batteries × 30 live fixtures. Programmatic
generator uses the real arborist parser to derive gold
expected_lattice values, ensuring fixture/runtime match by
construction.
Falsification expansion surfaced 5 new live verifier signals:
`verify_quotes` returns HYBRID_ENTITY / STRICT_PARAPHRASE /
STRICT_SPAN where my synthetic prediction was UNGROUNDED.
Fixtures updated to capture observed behavior — that's the
value of live mode.
Surface delta:
- arborist/pi_star/code.py — full impl, replaces stub
- arborist/pi_star (no other changes; code.py is the action)
- bench/batteries/b_5s.py — pi_star_ref/pi_star backward-compat
- bench/batteries/b_5r.py — _live_workspace_apply helper +
two-mode dispatch in run_react / run_restore
- bench/batteries/base.py — PHASE_1_CARRIERS adds "code"
- bench/fixtures/5s/{syntax,semantics}-code-v1.jsonl (new)
- bench/fixtures/5r/{react,restore}-live-v1.jsonl (new)
- bench/fixtures/5f/*-live-v1.jsonl (expanded to 30 each)
- .gitlab-ci.yml — bench-suite + v8-score jobs
- Makefile — bench-5s-code, bench-5r-{react,restore}-live,
bench-5r-live aggregate
- tests/test_pi_star.py — 11 new (code-py-ast graduation)
- tests/test_bench_batteries.py — 5 new (5R live)
Full suite: 1226 passed, 36 skipped.
Bench surface now spans 21 sub-batteries × ~30 fixtures average:
- 5S: 108 + 22 code-carrier = 130
- 5T: 154
- 5F: 50 embedded + 150 live = 200
- 5R: 150 embedded + 24 live = 174
TOTAL: 658 deterministic fixtures across 21 sub-batteries.
31 lines
13 KiB
JSON
31 lines
13 KiB
JSON
{"_meta": {"battery": "5f", "sub_battery": "formulate", "version": "v1", "task_count": 30, "notes": "Phase 1b.2 live-derived fixtures: input_text routes through arborist.qa.parse_claims.parse_pointer_claims; expected_lattice is the gold the live parser must produce. No produced_lattice embedded \u2014 runner derives it."}}
|
|
{"id":"5f-form-live-001","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- One claim. [E1]\n- Two claim. [E2]","expected_lattice":{"claim_count":2,"claims":[{"claim_text":"One claim.","pointer_ids":["E1"]},{"claim_text":"Two claim.","pointer_ids":["E2"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-002","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- Multi pointer claim. [E1, E2]","expected_lattice":{"claim_count":1,"claims":[{"claim_text":"Multi pointer claim.","pointer_ids":["E1","E2"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-003","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- A claim. [E1]\n- Another. [E2]\n- Third. [E3]","expected_lattice":{"claim_count":3,"claims":[{"claim_text":"A claim.","pointer_ids":["E1"]},{"claim_text":"Another.","pointer_ids":["E2"]},{"claim_text":"Third.","pointer_ids":["E3"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-004","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"The system has one claim. [E1]","expected_lattice":{"claim_count":1,"claims":[{"claim_text":"The system has one claim.","pointer_ids":["E1"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-005","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- claim alpha. [E1]\n- claim beta. [E2]\n- claim gamma. [E1, E3]","expected_lattice":{"claim_count":3,"claims":[{"claim_text":"claim alpha.","pointer_ids":["E1"]},{"claim_text":"claim beta.","pointer_ids":["E2"]},{"claim_text":"claim gamma.","pointer_ids":["E1","E3"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-006","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- The release date was July 3 1985. [E1]","expected_lattice":{"claim_count":1,"claims":[{"claim_text":"The release date was July 3 1985.","pointer_ids":["E1"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-007","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- Penicillin was discovered in 1928. [E1]\n- Alexander Fleming discovered it. [E1, E2]","expected_lattice":{"claim_count":2,"claims":[{"claim_text":"Penicillin was discovered in 1928.","pointer_ids":["E1"]},{"claim_text":"Alexander Fleming discovered it.","pointer_ids":["E1","E2"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-008","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- Mount Everest is the tallest mountain. [E5]","expected_lattice":{"claim_count":1,"claims":[{"claim_text":"Mount Everest is the tallest mountain.","pointer_ids":["E5"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-009","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- Water freezes at zero degrees Celsius. [E1]\n- The boiling point at sea level is one hundred. [E1]","expected_lattice":{"claim_count":2,"claims":[{"claim_text":"Water freezes at zero degrees Celsius.","pointer_ids":["E1"]},{"claim_text":"The boiling point at sea level is one hundred.","pointer_ids":["E1"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-010","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- Lone claim with two pointers. [E1, E2]","expected_lattice":{"claim_count":1,"claims":[{"claim_text":"Lone claim with two pointers.","pointer_ids":["E1","E2"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-011","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- DNA has a double helix structure. [E1]","expected_lattice":{"claim_count":1,"claims":[{"claim_text":"DNA has a double helix structure.","pointer_ids":["E1"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-012","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- Hydrogen has atomic number 1. [E1]\n- Helium has atomic number 2. [E2]\n- Lithium has atomic number 3. [E3]\n- Beryllium has atomic number 4. [E4]","expected_lattice":{"claim_count":4,"claims":[{"claim_text":"Hydrogen has atomic number 1.","pointer_ids":["E1"]},{"claim_text":"Helium has atomic number 2.","pointer_ids":["E2"]},{"claim_text":"Lithium has atomic number 3.","pointer_ids":["E3"]},{"claim_text":"Beryllium has atomic number 4.","pointer_ids":["E4"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-013","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- The Pacific Ocean is the largest ocean. [E10]","expected_lattice":{"claim_count":1,"claims":[{"claim_text":"The Pacific Ocean is the largest ocean.","pointer_ids":["E10"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-014","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- Goal-supporting claim. [E1]\n- Counter-evidence claim. [E2]","expected_lattice":{"claim_count":2,"claims":[{"claim_text":"Goal-supporting claim.","pointer_ids":["E1"]},{"claim_text":"Counter-evidence claim.","pointer_ids":["E2"]}]},"expected":"pass"}
|
|
{"id":"5f-form-live-015","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"- Claim with multiple pointers. [E1, E2, E3, E4]","expected_lattice":{"claim_count":1,"claims":[{"claim_text":"Claim with multiple pointers.","pointer_ids":["E1","E2","E3","E4"]}]},"expected":"pass"}
|
|
{"id": "5f-form-live-016", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- Snapshot at t1 contains the new fact. [E1]", "expected_lattice": {"claim_count": 1, "claims": [{"claim_text": "Snapshot at t1 contains the new fact.", "pointer_ids": ["E1"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-017", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- The audit chain stays intact. [E1]\n- The verifier returns STRICT. [E2]", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "The audit chain stays intact.", "pointer_ids": ["E1"]}, {"claim_text": "The verifier returns STRICT.", "pointer_ids": ["E2"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-018", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- Goldens are dogs. [E1]\n- Dogs are mammals. [E2]\n- Mammals are animals. [E3]\n- Therefore goldens are animals. [E1, E2, E3]", "expected_lattice": {"claim_count": 4, "claims": [{"claim_text": "Goldens are dogs.", "pointer_ids": ["E1"]}, {"claim_text": "Dogs are mammals.", "pointer_ids": ["E2"]}, {"claim_text": "Mammals are animals.", "pointer_ids": ["E3"]}, {"claim_text": "Therefore goldens are animals.", "pointer_ids": ["E1", "E2", "E3"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-019", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- Falsification flips state from live. [E1]\n- The audit event chains. [E2]", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Falsification flips state from live.", "pointer_ids": ["E1"]}, {"claim_text": "The audit event chains.", "pointer_ids": ["E2"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-020", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- Memory snapshot reflects added fact. [E5]", "expected_lattice": {"claim_count": 1, "claims": [{"claim_text": "Memory snapshot reflects added fact.", "pointer_ids": ["E5"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-021", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- Capital ledger row writes on ingest. [E1]\n- The chain stays clean. [E2]", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Capital ledger row writes on ingest.", "pointer_ids": ["E1"]}, {"claim_text": "The chain stays clean.", "pointer_ids": ["E2"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-022", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- \u03c0* registry is read-only after import. [E10]", "expected_lattice": {"claim_count": 1, "claims": [{"claim_text": "\u03c0* registry is read-only after import.", "pointer_ids": ["E10"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-023", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- ForkScore returns ACCEPT for improvement. [E1]\n- ForkScore returns REJECT for regression. [E2]\n- ForkScore returns MARGINAL for noise. [E3]", "expected_lattice": {"claim_count": 3, "claims": [{"claim_text": "ForkScore returns ACCEPT for improvement.", "pointer_ids": ["E1"]}, {"claim_text": "ForkScore returns REJECT for regression.", "pointer_ids": ["E2"]}, {"claim_text": "ForkScore returns MARGINAL for noise.", "pointer_ids": ["E3"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-024", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- code-py-ast canonicalizes whitespace. [E1]\n- code-py-ast canonicalizes comments. [E2]", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "code-py-ast canonicalizes whitespace.", "pointer_ids": ["E1"]}, {"claim_text": "code-py-ast canonicalizes comments.", "pointer_ids": ["E2"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-025", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- 5R React applies observations live. [E1]", "expected_lattice": {"claim_count": 1, "claims": [{"claim_text": "5R React applies observations live.", "pointer_ids": ["E1"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-026", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- 5R Restore queries audit chain. [E1]\n- 5R Restore returns retrievability. [E1, E2]", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "5R Restore queries audit chain.", "pointer_ids": ["E1"]}, {"claim_text": "5R Restore returns retrievability.", "pointer_ids": ["E1", "E2"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-027", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- Bench fixture digest is stable. [E1]\n- Bench runtime digest reflects \u03c0* registry. [E2]", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Bench fixture digest is stable.", "pointer_ids": ["E1"]}, {"claim_text": "Bench runtime digest reflects \u03c0* registry.", "pointer_ids": ["E2"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-028", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- Hard regression flag triggers REJECT. [E1]", "expected_lattice": {"claim_count": 1, "claims": [{"claim_text": "Hard regression flag triggers REJECT.", "pointer_ids": ["E1"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-029", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- neg_inf efficiency triggers REJECT. [E1]\n- inf efficiency adds capped bonus. [E2]", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "neg_inf efficiency triggers REJECT.", "pointer_ids": ["E1"]}, {"claim_text": "inf efficiency adds capped bonus.", "pointer_ids": ["E2"]}]}, "expected": "pass"}
|
|
{"id": "5f-form-live-030", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- 5F bridges synthetic to live across all five sub-batteries. [E1]", "expected_lattice": {"claim_count": 1, "claims": [{"claim_text": "5F bridges synthetic to live across all five sub-batteries.", "pointer_ids": ["E1"]}]}, "expected": "pass"}
|