russell@unturf.com
2af28e66af
Implements three coupled tickets in one push: complete the 5S battery, align 5T to Dav1DPrometheus vocabulary + complete it, open 5F battery. All runners deterministic, no LLM-as-judge anywhere. #000023 — 5S Phase 1b (closed) run_syllogism + run_synthesis + run_semiotics replace stubs. 30 deterministic fixtures each (90 total, all passing). Carrier metadata mandatory; unsupported carriers fail explicitly. Syllogism kernel handles categorical_transitivity, chain_3, invalid_converse, missing_premise. Order-agnostic over premise permutations. Synthesis uses content-token subset check (stopwords removed) — catches single-token entity swaps that lax overlap missed. Semiotics validates synonym-swap invariance via π* canonicalize + re-substitution. Hidden-channel work stays defensive only. #000024 — 5T Phase 1b (closed) Vocabulary aligned: Transfer→transfer-learning, Truth→truthtables, Timing→time. Legacy transfer-v1.jsonl + run_transfer kept intact. run_transfer_learning, run_triangulation, run_truthtables, run_transitivity, run_time replace stubs. 30 fixtures each × 5 = 150 total (155 with legacy transfer). Triangulation runs 4 strategies (substring, token_subset, token_overlap, entity_match) and gates on agreement threshold. Truthtables: deterministic propositional evaluator with recursive-descent parser; supports AND/OR/NOT/XOR/IMPL/IFF; capped at N=4 variables. Transitivity: typed-relation whitelist (implies, subset_of, ancestor_of, before, less_than). BFS path walk; mixed/unknown relations fail by construction. Time: synthetic memory-snapshot chains test preservation, stale-marking, current_root tracking. Phase 1b.2 will read real memory_records. #000025 — 5F Phase 1a (in progress; Phase 1b expansion still open) New bench/batteries/b_5f.py with five sub-batteries: - Function (shape_match / pointer_set_match / threshold_on_metric) - Finetuning (parent→child SelfModel improvement check) - Falsification (planted-error detection rate, verifier_method_root pinned) - Formulate (structural lattice match: claim count + sorted-approx text + exact pointer-ID-set; not exact-string) - Feedback Loop (operation/observation chain → expected_delta in aggregated observation feed) 10 deterministic seed fixtures per sub-battery (50 total). feedback_efficiency / adaptation_efficiency hooks stub the capital- ledger integration for v8 fork-choice. Cross-cutting: - bench/batteries/base.py: PHASE_1_CARRIERS whitelist + validate_carrier helper. Backward-compatible with Phase 1a fixtures that lack carrier field (defaults to "text"). - runner.py registers all 16 sub-batteries in _DEFAULT_FIXTURES so `--all` runs the entire Dav1DPrometheus suite. - Makefile: bench-5s / bench-5t / bench-5f / bench-5s5t5f targets. bench-5t-legacy preserves Phase 1a access. - tests/test_bench_batteries.py: 29 tests (was 17). Phase 1a digest stability checks, sub-battery smoke tests, carrier rejection tests, transitivity-whitelist test, truthtables N>4 cap test. Bench summary across the full suite: 5s syntax/semantics/syllogism/synthesis/semiotics 108/108 pass 5t transfer/transfer-learning/triangulation/truthtables/transitivity/time 154/154 pass 5f function/finetuning/falsification/formulate/feedback-loop 50/50 pass TOTAL: 312 fixtures across 16 sub-batteries — 100% pass. Full test suite: 1103 passed, 36 skipped. Source: Legally Unprecedented Dav1DPrometheus (BasementAGI host). Honoring his framework. The complete state-space synthesis (SQD + v7 + 5S/5T/5F) is now executable infrastructure, not metaphor.
11 lines
4.9 KiB
JSON
11 lines
4.9 KiB
JSON
{"_meta":{"battery":"5f","sub_battery":"formulate","version":"v1","task_count":10,"notes":"Phase 1a fixtures embed produced + expected lattices for structural matching. Phase 1b.2 will route through arborist.qa.parse_claims."}}
|
|
{"id":"5f-form-001","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"The system saved one claim with one pointer.","produced_lattice":{"claims":[{"claim_text":"The system saved one claim with one pointer.","pointer_ids":["E1"]}]},"expected_lattice":{"claim_count":1,"claims":[{"claim_text":"The system saved one claim with one pointer.","pointer_ids":["E1"]}]},"expected":"pass"}
|
|
{"id":"5f-form-002","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"Two claims with two pointers each.","produced_lattice":{"claims":[{"claim_text":"first claim with pointers","pointer_ids":["E1","E2"]},{"claim_text":"second claim with pointers","pointer_ids":["E3","E4"]}]},"expected_lattice":{"claim_count":2,"claims":[{"claim_text":"first claim with pointers","pointer_ids":["E1","E2"]},{"claim_text":"second claim with pointers","pointer_ids":["E3","E4"]}]},"expected":"pass"}
|
|
{"id":"5f-form-003","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"Detect missing pointer claim.","produced_lattice":{"claims":[{"claim_text":"a claim","pointer_ids":["E1"]}]},"expected_lattice":{"claim_count":1,"claims":[{"claim_text":"a claim","pointer_ids":["E1","E2"]}]},"expected":"fail"}
|
|
{"id":"5f-form-004","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"Detect wrong claim count.","produced_lattice":{"claims":[{"claim_text":"a","pointer_ids":["E1"]}]},"expected_lattice":{"claim_count":2,"claims":[{"claim_text":"a","pointer_ids":["E1"]},{"claim_text":"b","pointer_ids":["E2"]}]},"expected":"fail"}
|
|
{"id":"5f-form-005","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"Three sequential claims preserved.","produced_lattice":{"claims":[{"claim_text":"alpha beta gamma","pointer_ids":["E1"]},{"claim_text":"delta epsilon zeta","pointer_ids":["E2"]},{"claim_text":"eta theta iota","pointer_ids":["E3"]}]},"expected_lattice":{"claim_count":3,"claims":[{"claim_text":"alpha beta gamma","pointer_ids":["E1"]},{"claim_text":"delta epsilon zeta","pointer_ids":["E2"]},{"claim_text":"eta theta iota","pointer_ids":["E3"]}]},"expected":"pass"}
|
|
{"id":"5f-form-006","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"Sorted-approx text match works under reordered tokens.","produced_lattice":{"claims":[{"claim_text":"world hello there","pointer_ids":["E1"]}]},"expected_lattice":{"claim_count":1,"claims":[{"claim_text":"hello world there","pointer_ids":["E1"]}]},"expected":"pass"}
|
|
{"id":"5f-form-007","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"Detect text drift below 70% overlap.","produced_lattice":{"claims":[{"claim_text":"completely different sentence","pointer_ids":["E1"]}]},"expected_lattice":{"claim_count":1,"claims":[{"claim_text":"the original gold reference text","pointer_ids":["E1"]}]},"expected":"fail"}
|
|
{"id":"5f-form-008","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"Two claims with shared pointer.","produced_lattice":{"claims":[{"claim_text":"first observation","pointer_ids":["E1"]},{"claim_text":"second observation","pointer_ids":["E1"]}]},"expected_lattice":{"claim_count":2,"claims":[{"claim_text":"first observation","pointer_ids":["E1"]},{"claim_text":"second observation","pointer_ids":["E1"]}]},"expected":"pass"}
|
|
{"id":"5f-form-009","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"Detect pointer-id mismatch between produced and expected.","produced_lattice":{"claims":[{"claim_text":"sentence text","pointer_ids":["E5"]}]},"expected_lattice":{"claim_count":1,"claims":[{"claim_text":"sentence text","pointer_ids":["E1"]}]},"expected":"fail"}
|
|
{"id":"5f-form-010","battery":"5f","sub_battery":"formulate","version":"v1","carrier":"text","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","input_text":"Multi-pointer claim with full pointer set match.","produced_lattice":{"claims":[{"claim_text":"complex multi-source claim","pointer_ids":["E1","E2","E3"]}]},"expected_lattice":{"claim_count":1,"claims":[{"claim_text":"complex multi-source claim","pointer_ids":["E1","E2","E3"]}]},"expected":"pass"}
|