Three small streams: #3 — close #000030 properly ============================ All 7 phases + Phase 1b landed across two commits (`04f3f5d`, `abe5988`). Status header updated; ticket body now carries a phase landing table with commit refs: Phase 1 algebra-symbolic@v104f3f5dPhase 1b algebra-symbolic-simplified@v104f3f5dPhase 2 calculus-derivative@v104f3f5dPhase 3 calculus-integral@v1 fox-direct Phase 4 calculus-limit@v1abe5988Phase 5 calculus-series@v1abe5988Phase 6 linear-algebra@v1abe5988Phase 7 function-sampled@v1abe5988Plus tabular-pinned@v1 (last reserved stub) graduated inabe5988closes the registry chapter — 15 concrete π*'s, no remaining reserved stubs. Index updated. #5 — composition fixtures across new SymPy π*'s ================================================ 12 new tests in tests/test_pi_star_compositions.py covering pairs that compose naturally: - algebra-symbolic ∘ algebra-symbolic — idempotency check (running expand twice equals expand once for any expression). - algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`). - Generic invariants: composition propagates PiStarError; manifest fingerprint is order-sensitive; composite domain == inner domain; composite bytes == manual chain bytes. Test discipline: most compositions use `register_in_registry=False` via a small `_safe_compose()` helper since the registry rejects duplicate keys (#000015 invariant), so test ordering would otherwise matter. Only the registration-test path uses real compose(). #4 — end-to-end witness sweep against real shards + Hermes =========================================================== New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape questions (3 arithmetic + 3 logic + 2 algebra) through query() with `canonical_witness_enabled=True`, against ~/.arborist/shards (real shard cluster) + the actual Hermes endpoint (NOT StubClient). Records the agreement matrix per question to bench/results/witness-sweep.json. `make bench-witness-sweep` Makefile target. Honors `ARBORIST_SHARDS_DIR`. First real sweep (this commit, against Hermes-3-8B): agreement label count rate KERNEL-LLM-DIVERGED 5 62.5% KERNEL-LLM-AGREE 3 37.5% ─────────────────────────────────────────── divergence_count 5 62.5% wall median / max 130 ms / 1.1 s Hermes diverged on 5/8 of the canonical-shape questions: - said `1/10` for `0.1 + 0.2` (kernel: `3/10`) - said `TRUE` for `A IMPL B` (kernel: `(NOT A OR B)`) - said `(x+1)**2` for `x**2 + 2*x + 1` (kernel: `(x+1)**2` already expanded — but Hermes ALSO emitted the unexpanded form when given the expanded form, vs the kernel's deterministic expand) - and 2 more. These are real LLM hallucinations on questions with closed-form ground truth — exactly the calibration-data stream #000028 imagined. Pipeline validated end-to-end against actual hardware. Pair: `make bench-witness-divergence` then extracts the 5 divergences as 5F-Falsification fixtures (bench/fixtures/5f/falsification-witness-v1.jsonl, also committed). Re-running the extractor produces byte-equal output (idempotency contract from the extractor work). Tests ===== Full suite: 1636 passed, 37 skipped (was 1624; +12 composition tests). The witness-sweep + extractor produce real artifacts now committed under bench/results/ and bench/fixtures/5f/.
206 lines
7.7 KiB
Python
206 lines
7.7 KiB
Python
"""Tests for π* compositions across the SymPy substrate (#000030).
|
|
|
|
The compose API auto-registers composite kernels under the key
|
|
``<inner-name>-then-<outer-name>@v1``. These tests exercise pairs
|
|
that compose naturally — kernels that emit canonical bytes the
|
|
next kernel can re-parse.
|
|
|
|
What composes:
|
|
|
|
- ``algebra-symbolic@v1 ∘ algebra-symbolic@v1`` — idempotency.
|
|
Inner produces ``sp.expand(sp.sympify(text))`` srepr; outer
|
|
sympifies the srepr back, expands again. Should equal the
|
|
one-pass output.
|
|
|
|
- ``algebra-symbolic@v1 ∘ algebra-symbolic-simplified@v1`` —
|
|
expand then simplify. Catches polynomial identity AND trig /
|
|
exp / log identity that single-pass `expand` misses.
|
|
|
|
- ``calculus-derivative@v1 ∘ algebra-symbolic@v1`` — derivative
|
|
composed with re-canonicalizer. Output of derivative is already
|
|
re-canonicalized through algebra-symbolic internally, so the
|
|
composite is idempotent over the derivative's output.
|
|
|
|
What doesn't compose (documented gaps):
|
|
|
|
- Most cross-π* pairs require shape-compatible round-trip; e.g.
|
|
``arithmetic@v1 ∘ algebra-symbolic@v1`` would require arithmetic
|
|
to accept symbolic-srepr bytes, which it doesn't. Compose
|
|
raises a clean error path on incompatible bytes — see
|
|
``test_composition_propagates_pi_star_error``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
|
|
sympy = pytest.importorskip("sympy")
|
|
|
|
|
|
def _safe_compose(inner_key: str, outer_key: str):
|
|
"""Compose without re-registering. The registry rejects dup
|
|
keys (#000015 invariant), so test ordering would otherwise
|
|
matter. Using register_in_registry=False sidesteps the
|
|
invariant for tests that only need .canonicalize() access."""
|
|
from arborist.pi_star import compose
|
|
return compose(inner_key, outer_key, register_in_registry=False)
|
|
|
|
|
|
def test_compose_auto_registers_composite_key():
|
|
"""Calling compose() registers the result under
|
|
``<inner>-then-<outer>@v1`` so subsequent get() calls work.
|
|
Re-registration is rejected by the registry — operators that
|
|
just want to construct the composite (without re-registering)
|
|
pass ``register_in_registry=False``."""
|
|
from arborist.pi_star import compose, get, list_keys
|
|
key = "algebra-symbolic-then-algebra-symbolic@v1"
|
|
if key not in list_keys():
|
|
composite = compose(
|
|
"algebra-symbolic@v1", "algebra-symbolic@v1",
|
|
)
|
|
else:
|
|
# Already registered by an earlier test; build without
|
|
# re-registering, then look up the registered one.
|
|
composite = compose(
|
|
"algebra-symbolic@v1", "algebra-symbolic@v1",
|
|
register_in_registry=False,
|
|
)
|
|
assert composite.name == "algebra-symbolic-then-algebra-symbolic"
|
|
|
|
# Lookup via get() works after registration.
|
|
looked_up = get(key)
|
|
assert looked_up.name == composite.name
|
|
|
|
|
|
def test_algebra_symbolic_idempotent_under_self_composition():
|
|
"""Running expand twice should equal expand once. The composite
|
|
key bytes must equal the one-pass key bytes for any expression."""
|
|
from arborist.pi_star import compose, get
|
|
one_pass = get("algebra-symbolic@v1")
|
|
twice = _safe_compose("algebra-symbolic@v1", "algebra-symbolic@v1")
|
|
|
|
for expr in (b"(x+1)**2", b"x**2 + 2*x + 1", b"a*b - c", b"sin(x)"):
|
|
single = one_pass.canonicalize(expr)
|
|
composed = twice.canonicalize(expr)
|
|
assert single == composed, f"divergence on {expr!r}"
|
|
|
|
|
|
def test_simplify_composition_collapses_trig_identity():
|
|
"""`expand` alone leaves sin²+cos² untouched; simplify catches it.
|
|
Composing expand-then-simplify should produce ``Integer(1)`` for
|
|
the Pythagorean identity."""
|
|
from arborist.pi_star import compose
|
|
composite = _safe_compose(
|
|
"algebra-symbolic@v1", "algebra-symbolic-simplified@v1",
|
|
)
|
|
out = composite.canonicalize(b"sin(x)**2 + cos(x)**2")
|
|
# SymPy's srepr of Integer(1) is exactly "Integer(1)"
|
|
assert out == b"Integer(1)"
|
|
|
|
|
|
def test_simplify_composition_does_not_break_polynomial_identity():
|
|
"""Expand-then-simplify on a polynomial that's already in
|
|
expanded form should round-trip stably."""
|
|
from arborist.pi_star import compose
|
|
composite = _safe_compose(
|
|
"algebra-symbolic@v1", "algebra-symbolic-simplified@v1",
|
|
)
|
|
a = composite.canonicalize(b"(x+1)**2")
|
|
b = composite.canonicalize(b"x**2 + 2*x + 1")
|
|
# Both should collapse to the same canonical bytes.
|
|
assert a == b
|
|
|
|
|
|
def test_composition_propagates_pi_star_error():
|
|
"""When the inner kernel raises PiStarError, the composite
|
|
surfaces it rather than swallowing or converting."""
|
|
from arborist.pi_star import PiStarError, compose
|
|
composite = _safe_compose(
|
|
"algebra-symbolic@v1", "algebra-symbolic@v1",
|
|
)
|
|
with pytest.raises(PiStarError):
|
|
composite.canonicalize(b"this is not !#$ a valid !#$ sympy expression")
|
|
|
|
|
|
def test_composition_is_listed_in_registry_after_compose():
|
|
"""``list_keys()`` shows composite kernels alongside the rest of
|
|
the registry once they've been compose()'d. Auto-registered
|
|
composites have keys with ``-then-`` infix."""
|
|
from arborist.pi_star import compose, list_keys
|
|
compose("algebra-symbolic@v1", "algebra-symbolic-simplified@v1")
|
|
keys = list_keys()
|
|
assert any(
|
|
k.startswith("algebra-symbolic-then-") for k in keys
|
|
)
|
|
|
|
|
|
def test_canonical_composition_id_is_stable():
|
|
"""The composition manifest fingerprint depends only on the
|
|
(inner, outer) keys; same pair → same id."""
|
|
from arborist.pi_star import canonical_composition_id
|
|
a = canonical_composition_id(
|
|
"algebra-symbolic@v1", "algebra-symbolic-simplified@v1",
|
|
)
|
|
b = canonical_composition_id(
|
|
"algebra-symbolic@v1", "algebra-symbolic-simplified@v1",
|
|
)
|
|
assert a == b
|
|
assert len(a) == 64 # sha256 hex
|
|
|
|
|
|
def test_canonical_composition_id_distinguishes_order():
|
|
"""Order matters in composition: A∘B ≠ B∘A, so the manifest id
|
|
must distinguish them."""
|
|
from arborist.pi_star import canonical_composition_id
|
|
forward = canonical_composition_id(
|
|
"algebra-symbolic@v1", "algebra-symbolic-simplified@v1",
|
|
)
|
|
reversed_ = canonical_composition_id(
|
|
"algebra-symbolic-simplified@v1", "algebra-symbolic@v1",
|
|
)
|
|
assert forward != reversed_
|
|
|
|
|
|
def test_compose_unknown_inner_raises_keyerror():
|
|
"""compose() guards both ends — an unknown inner π* fails
|
|
cleanly with KeyError."""
|
|
from arborist.pi_star import compose
|
|
with pytest.raises(KeyError):
|
|
compose("does-not-exist@v1", "algebra-symbolic@v1")
|
|
|
|
|
|
def test_compose_unknown_outer_raises_keyerror():
|
|
from arborist.pi_star import compose
|
|
with pytest.raises(KeyError):
|
|
compose("algebra-symbolic@v1", "does-not-exist@v1")
|
|
|
|
|
|
def test_composite_domain_is_inner_domain():
|
|
"""Composition's domain is the inner π*'s domain (the composite
|
|
accepts what the inner accepts)."""
|
|
from arborist.pi_star import compose
|
|
composite = _safe_compose(
|
|
"algebra-symbolic@v1", "algebra-symbolic@v1",
|
|
)
|
|
assert composite.domain == "symbolic-algebra"
|
|
|
|
|
|
def test_composition_bytes_match_manual_application():
|
|
"""A composite kernel must produce byte-equal output to a manual
|
|
inner.canonicalize() → outer.canonicalize() chain."""
|
|
from arborist.pi_star import compose, get
|
|
inner = get("algebra-symbolic@v1")
|
|
outer = get("algebra-symbolic-simplified@v1")
|
|
composite = _safe_compose(
|
|
"algebra-symbolic@v1", "algebra-symbolic-simplified@v1",
|
|
)
|
|
for expr in (
|
|
b"sin(x)**2 + cos(x)**2",
|
|
b"(a + b)**2",
|
|
b"x**3 - x**3 + 2*x",
|
|
):
|
|
manual = outer.canonicalize(inner.canonicalize(expr))
|
|
assert composite.canonicalize(expr) == manual, (
|
|
f"composite ≠ manual chain on {expr!r}"
|
|
)
|