russell@unturf.com
bc77f961f3
Three small streams: #3 — close #000030 properly ============================ All 7 phases + Phase 1b landed across two commits (`04f3f5d`, `abe5988`). Status header updated; ticket body now carries a phase landing table with commit refs: Phase 1 algebra-symbolic@v104f3f5dPhase 1b algebra-symbolic-simplified@v104f3f5dPhase 2 calculus-derivative@v104f3f5dPhase 3 calculus-integral@v1 fox-direct Phase 4 calculus-limit@v1abe5988Phase 5 calculus-series@v1abe5988Phase 6 linear-algebra@v1abe5988Phase 7 function-sampled@v1abe5988Plus tabular-pinned@v1 (last reserved stub) graduated inabe5988closes the registry chapter — 15 concrete π*'s, no remaining reserved stubs. Index updated. #5 — composition fixtures across new SymPy π*'s ================================================ 12 new tests in tests/test_pi_star_compositions.py covering pairs that compose naturally: - algebra-symbolic ∘ algebra-symbolic — idempotency check (running expand twice equals expand once for any expression). - algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`). - Generic invariants: composition propagates PiStarError; manifest fingerprint is order-sensitive; composite domain == inner domain; composite bytes == manual chain bytes. Test discipline: most compositions use `register_in_registry=False` via a small `_safe_compose()` helper since the registry rejects duplicate keys (#000015 invariant), so test ordering would otherwise matter. Only the registration-test path uses real compose(). #4 — end-to-end witness sweep against real shards + Hermes =========================================================== New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape questions (3 arithmetic + 3 logic + 2 algebra) through query() with `canonical_witness_enabled=True`, against ~/.arborist/shards (real shard cluster) + the actual Hermes endpoint (NOT StubClient). Records the agreement matrix per question to bench/results/witness-sweep.json. `make bench-witness-sweep` Makefile target. Honors `ARBORIST_SHARDS_DIR`. First real sweep (this commit, against Hermes-3-8B): agreement label count rate KERNEL-LLM-DIVERGED 5 62.5% KERNEL-LLM-AGREE 3 37.5% ─────────────────────────────────────────── divergence_count 5 62.5% wall median / max 130 ms / 1.1 s Hermes diverged on 5/8 of the canonical-shape questions: - said `1/10` for `0.1 + 0.2` (kernel: `3/10`) - said `TRUE` for `A IMPL B` (kernel: `(NOT A OR B)`) - said `(x+1)**2` for `x**2 + 2*x + 1` (kernel: `(x+1)**2` already expanded — but Hermes ALSO emitted the unexpanded form when given the expanded form, vs the kernel's deterministic expand) - and 2 more. These are real LLM hallucinations on questions with closed-form ground truth — exactly the calibration-data stream #000028 imagined. Pipeline validated end-to-end against actual hardware. Pair: `make bench-witness-divergence` then extracts the 5 divergences as 5F-Falsification fixtures (bench/fixtures/5f/falsification-witness-v1.jsonl, also committed). Re-running the extractor produces byte-equal output (idempotency contract from the extractor work). Tests ===== Full suite: 1636 passed, 37 skipped (was 1624; +12 composition tests). The witness-sweep + extractor produce real artifacts now committed under bench/results/ and bench/fixtures/5f/.
6 lines
3.1 KiB
JSON
6 lines
3.1 KiB
JSON
{"_meta": {"battery": "5f", "sub_battery": "falsification", "version": "v1", "task_count": 5, "notes": "Witness-divergence-extracted falsification fixtures. Auto-generated by `make bench-witness-divergence` from providence_canonical_witness audit events. Each row pairs an LLM raw answer with the kernel's canonical answer; verify_quotes is expected to return UNGROUNDED because the LLM prose doesn't substring-match the kernel's terse canonical form. _witness_meta carries the original agreement_label + audit_seq for traceability."}}
|
|
{"id": "5f-fal-witness-011631", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "arithmetic@v1", "answer_text": "1/10", "context": "canonical_kernel_answer=3/10\nquestion=0.1 + 0.2", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11631, "pi_star_ref": "arithmetic@v1"}}
|
|
{"id": "5f-fal-witness-011637", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "logic-kernel@v1", "answer_text": "TRUE", "context": "canonical_kernel_answer=(NOT A OR B)\nquestion=A IMPL B", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11637, "pi_star_ref": "logic-kernel@v1"}}
|
|
{"id": "5f-fal-witness-011639", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "logic-kernel@v1", "answer_text": "TRUE", "context": "canonical_kernel_answer=(NOT A OR B)\nquestion=(NOT B) IMPL (NOT A)", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11639, "pi_star_ref": "logic-kernel@v1"}}
|
|
{"id": "5f-fal-witness-011643", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "algebra-symbolic@v1", "answer_text": "1", "context": "canonical_kernel_answer=Add(Pow(Symbol('x'), Integer(2)), Mul(Integer(2), Symbol('x')), Integer(1))\nquestion=(x+1)**2", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11643, "pi_star_ref": "algebra-symbolic@v1"}}
|
|
{"id": "5f-fal-witness-011645", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "algebra-symbolic@v1", "answer_text": "1", "context": "canonical_kernel_answer=Add(Pow(Symbol('x'), Integer(2)), Mul(Integer(2), Symbol('x')), Integer(1))\nquestion=x**2 + 2*x + 1", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11645, "pi_star_ref": "algebra-symbolic@v1"}}
|