arborist/docs/tickets/ticket-000030-math-pi-star-expansion-sympy-substrate.md

Ticket #000030 — Math π* expansion: SymPy substrate

Status: closed · all 7 phases + Phase 1b landed 2026-05-09 (registry-closure fan-out, commit abe5988) Opened: 2026-05-09 Scope: Extend the π* registry with SymPy-backed canonicalizers covering symbolic algebra, calculus (derivatives, integrals, limits, series), linear algebra, and a function-sampled bridge to time-series-quantized@v1. Multi-phase; each phase is a single new π* in the registry. Audience: maintainers of arborist's π* layer + downstream consumers (canonical-projection witness, math/logic preflight, benchmark batteries, opencompletion math activities). Hard constraint: SymPy is an optional dependency ([math] extra). Each new π* registers ONLY when import sympy succeeds at package load — same defensive-import pattern as the existing html / wikitext / crawler extras. A fresh checkout without SymPy must continue to pass the full test suite (graceful skip).

---

1. Problem statement

Arborist's math substrate today (post-#000027 + #000028):

π* (math)	scope
arithmetic@v1	Closed-form rational arithmetic — 0.1+0.2 → 3/10. SQD §14.1.
logic-kernel@v1	Propositional Boolean → CNF. SQD §14.3.
time-series-quantized@v1	Sample-array (dt, dv, [(t, v),…]) → quantized integer vector. SQD §13.5.

What's missing — the symbolic algebra & calculus layer that the opencompletion activity24-math-plot.yaml activity exercises (SymPy

• numpy + matplotlib for parsing → simplification → sampling → plotting). Two consequences of the gap:

1. Question shapes the canonical-projection preflight can't answer. "Simplify (x+1)**2", "what's the derivative of x**3 + 2*x", "integrate sin(x) from 0 to π" all fall through to RAG today, even though they have closed-form deterministic answers. The math witness (#000028) STRICT warrant is therefore unreachable for symbolic questions.
2. 5T (Truthtables) and 5F (Falsification) bench batteries cover propositional logic but not symbolic algebra/calculus. No supervised calibration data for symbolic LLM hallucinations.

The opencompletion activity uses SymPy already. Pulling that substrate into arborist closes both gaps with one optional dependency.

2. Design choices

2.1 Dependency strategy

A. Optional [math] extra (RECOMMENDED). SymPy is ~30 MB installed; pulling it into core deps would inflate every fresh checkout. Mirror the existing pattern:

pip install 'arborist[math]'    # installs sympy

Each new π* module guards registration with try: import sympy except ImportError: SYMPY = None and skips register(...) when absent. Tests use pytest.importorskip("sympy"). Real-shard / bench paths fail explicitly with unsupported_pi_star rather than crashing.

B. Bundle SymPy into core deps. Simplifies "does it work?" but doubles the install footprint and pulls in mpmath as a transitive. Not worth it for users who never touch math.

C. Vendor a tiny subset. SymPy is too large to vendor; the algebra/calculus paths use deep symbolic infrastructure.

→ A. Matches the html / wikitext / crawler extras already in pyproject.toml. Test suite stays green either way.

2.2 Which kernels in scope

Phase-by-phase rollout. Each phase = one π* in the registry.

Phase 1 — algebra-symbolic@v1

• Parse expression via sp.sympify.
• Canonicalize via sp.expand + lexical ordering of free symbols + canonical printing (sp.srepr for proof-grade bytes; str(expr) for human-readable).
• (x+1)**2 and x**2 + 2*x + 1 → same canonical bytes.
• Equivalence-class-preserving for polynomial identities; trigonometric identities (sin(x)**2 + cos(x)**2 = 1) NOT guaranteed unless the caller invokes a simplify overlay variant (Phase 1b: algebra-symbolic-simplified@v1).
• Errors: undefined symbol → PiStarError; non-expression input → PiStarError.

Phase 2 — calculus-derivative@v1

• Input: JSON {"f": "<expr>", "x": "<var>", "n"?: <order>}.
• Output: canonical bytes from sp.diff(f, x, n) re-canonicalized through algebra-symbolic@v1.
• d/dx(x**2) → 2*x. d²/dx²(x**3) → 6*x.
• Composition note: this π* COMPOSES with algebra-symbolic@v1 via the existing arborist.pi_star.compose registry.

Phase 3 — calculus-integral@v1

• Input: JSON {"f": "<expr>", "x": "<var>", "limits"?: [a, b]}.
• Indefinite: SymPy sp.integrate(f, x). Definite: with limits.
• Output: canonical bytes if closed form exists. If SymPy returns Integral(...) unevaluated → emit explicit sentinel unevaluated:Integral(<original>) rather than raising. Lets the caller distinguish "no closed form" from "input invalid."
• ∫x dx → x**2/2; ∫(0..π) sin(x) dx → 2.

Phase 4 — calculus-limit@v1

• sp.limit(f, x, point, dir?).
• Input JSON: {"f", "x", "point", "dir"?: "+/-"}.
• Common cases: lim(sin(x)/x, x, 0) → 1; lim(1/x, x, 0, "+") → oo. Infinity sentinel: inf / -inf / complex_inf.

Phase 5 — calculus-series@v1

• Taylor / Maclaurin series to N terms via sp.series(f, x, x0, n).
• Input JSON: {"f", "x", "x0", "n"}.
• Output: canonical truncated polynomial through algebra-symbolic@v1. The O(x**n) remainder is dropped (the truncation is what makes this finite-byte canonical).

Phase 6 — linear-algebra@v1

• Input JSON: {"op": "rref|det|eigenvalues|eigenvectors|inverse", "matrix": [[…], […], …]}.
• Output: canonical matrix bytes (rational entries via arithmetic@v1) for RREF / inverse; rational scalar for det; sorted list of eigenvalues for eigenvalues; etc.
• One π* covers the whole linear-algebra surface; the op field in input switches behavior. Cleaner than 4 separate π*'s.

Phase 7 — function-sampled@v1

• Bridge to time-series-quantized@v1. Input JSON: {"f": "<expr>", "x": "<var>", "x_min": …, "x_max": …, "n_samples": …, "dv": …}.
• Lambdifies f, samples on the linspace grid, quantizes via the same dt/dv canonicalization as time-series-quantized@v1.
• Output: identical format to time-series-quantized@v1 (dt=…;dv=…;n=…;t0=…:v0|v1|…).
• This is what plotting CAN become in π* terms: two functions that render identically (within sample tolerance) collapse to the same canonical bytes.
• The PNG visualization is an output adapter — see §4.

2.3 Ordering & normalization choices (per Phase)

SymPy doesn't guarantee a single canonical form; different expressions can be mathematically equivalent but textually distinct. Each phase pins its normalization recipe:

Phase	Normalization recipe
algebra-symbolic@v1	sp.expand only (fast; preserves polynomial identity but NOT trig identity).
algebra-symbolic-simplified@v1 (1b — optional)	sp.simplify — slower; catches sin²+cos²=1 etc.
calculus-derivative@v1	derivative → algebra-symbolic@v1.
calculus-integral@v1	integral → algebra-symbolic@v1; sentinel for unevaluated.
calculus-limit@v1	limit → printed canonical form (+oo, -oo, etc.).
calculus-series@v1	series truncated → algebra-symbolic@v1.
linear-algebra@v1	matrix entries via arithmetic@v1; ordered rows.
function-sampled@v1	sample grid → time-series-quantized@v1 format.

The simplify variant lives behind a separate version because its CPU cost is unbounded (SymPy simplify can be exponential); operators opt in when they need it.

2.4 Composition with existing π*

Two natural composites land for free via arborist.pi_star.compose:

from arborist.pi_star import compose

# Symbolic differentiation ending in a numeric evaluation:
deriv_then_arith = compose("calculus-derivative@v1", "arithmetic@v1")

# Function plotting end-to-end signature:
expr_then_sample = compose("algebra-symbolic@v1", "function-sampled@v1")

Auto-registers as <a>-then-<b>@v1 per the existing composition algebra (see docs/pi-star-composition.md).

3. Recommendation

Phase 1 first. algebra-symbolic@v1 alone unlocks the biggest single class of new questions (algebraic simplification, identity-checking) and demonstrates the SymPy-extra pattern. Phase 2 (calculus-derivative@v1) follows naturally — same parsing, same printing, just one extra sp.diff call.

After Phases 1+2 land and pass real-world bench fixtures, fox decides which of Phases 3-7 to pull next. None depends on the others except function-sampled@v1 (which composes with Phase 1 but doesn't structurally require it).

4. Implementation sketch

4.1 Dependency wiring

In pyproject.toml:

[project.optional-dependencies]
math = [
  "sympy>=1.13",
]

make bootstrap pattern continues to work; users who want math run pip install '.[math]' separately, or we add a bootstrap-math Makefile target.

4.2 New module: arborist/pi_star/algebra_symbolic.py

try:
    import sympy as sp
except ImportError:  # pragma: no cover — optional extra
    sp = None


@dataclass
class AlgebraSymbolicV1:
    name: str = "algebra-symbolic"
    version: str = "v1"
    domain: str = "symbolic-algebra"

    def canonicalize(self, raw: bytes) -> bytes:
        if sp is None:
            raise PiStarError(
                "algebra-symbolic@v1 requires sympy; install with "
                "`pip install 'arborist[math]'`"
            )
        text = raw.decode("utf-8", errors="surrogatepass")
        try:
            expr = sp.sympify(text)
        except (sp.SympifyError, SyntaxError) as exc:
            raise PiStarError(
                f"algebra-symbolic@v1 cannot parse {text!r}: {exc}"
            ) from exc
        canonical = sp.expand(expr)
        # Stable canonical form: srepr (S-expression representation)
        # is more deterministic than str() across SymPy versions.
        return sp.srepr(canonical).encode("utf-8")


if sp is not None:
    register(AlgebraSymbolicV1())

4.3 New module: arborist/pi_star/calculus_derivative.py

@dataclass
class CalculusDerivativeV1:
    name: str = "calculus-derivative"
    version: str = "v1"
    domain: str = "calculus"

    def canonicalize(self, raw: bytes) -> bytes:
        if sp is None:
            raise PiStarError("requires sympy; pip install 'arborist[math]'")
        try:
            obj = json.loads(raw.decode("utf-8", errors="surrogatepass"))
        except json.JSONDecodeError as exc:
            raise PiStarError(f"input must be JSON object: {exc}") from exc
        for k in ("f", "x"):
            if k not in obj:
                raise PiStarError(f"missing required field {k!r}")
        n = obj.get("n", 1)
        if not isinstance(n, int) or n < 0:
            raise PiStarError("'n' must be non-negative integer")
        x_sym = sp.Symbol(obj["x"])
        try:
            f_expr = sp.sympify(obj["f"], locals={obj["x"]: x_sym})
            deriv = sp.diff(f_expr, x_sym, n)
            canonical = sp.expand(deriv)
        except (sp.SympifyError, SyntaxError, ValueError) as exc:
            raise PiStarError(f"derivative failed: {exc}") from exc
        return sp.srepr(canonical).encode("utf-8")

4.4 Test layout

tests/test_pi_star_algebra.py:

sympy = pytest.importorskip("sympy")  # entire file skipped if absent

def test_polynomial_expansion_identity():
    ps = get("algebra-symbolic@v1")
    assert ps.canonicalize(b"(x+1)**2") == ps.canonicalize(b"x**2 + 2*x + 1")

def test_distinct_polynomials_distinct():
    ps = get("algebra-symbolic@v1")
    assert ps.canonicalize(b"x**2") != ps.canonicalize(b"x**3")

def test_undefined_input_raises():
    ps = get("algebra-symbolic@v1")
    with pytest.raises(PiStarError):
        ps.canonicalize(b"not a valid expression $$$")

Same shape for tests/test_pi_star_calculus_derivative.py.

4.5 Bench fixtures (Phase 1 only — others follow when those land)

bench/fixtures/5s/syntax-algebra-symbolic-v1.jsonl — 10 fixtures of valid expressions through sp.sympify + sp.expand.

bench/fixtures/5s/semantics-algebra-symbolic-v1.jsonl — equivalence-class fixtures:

• (x+1)**2 ≡ x**2 + 2*x + 1 ≡ x*x + 2*x + 1
• (a+b)*(a-b) ≡ a**2 - b**2
• 2*x + 3*x ≡ 5*x
• distinct: x**2 + 1 vs x**2

4.6 Math/logic preflight extension

In arborist/qa/query.py:_canonical_projection_preflight, expand the sniff to include "looks like an algebraic expression with letters" — currently the regex rejects letters (math route) or requires uppercase atoms (logic route). New route: lowercase letters + arithmetic ops → try algebra-symbolic@v1.

# Algebra route — lowercase letters present, but otherwise
# expression-shaped (parens, ops, digits, letters, whitespace).
if _ALGEBRA_RE.fullmatch(q):
    try:
        out = get("algebra-symbolic@v1").canonicalize(q.encode("utf-8"))
        return ("algebra-symbolic@v1", out)
    except PiStarError:
        return None

Order: arithmetic → algebra → logic. Arithmetic claims no-letters first; algebra picks up the letter-containing remainder; logic claims uppercase-only.

4.7 Documentation updates

• docs/_source/pi-star.rst: add Algebra / Calculus section alongside the existing Arithmetic + Logic descriptions.
• bench/batteries/base.py PHASE_1_CARRIERS: add symbolic_algebra (and others as phases land).
• New Makefile target: bench-5s-algebra mirroring bench-5s-math.

5. Out of scope

• Plotting / image rendering. The PNG output the opencompletion activity produces is a visualization, not a canonical projection. function-sampled@v1 (Phase 7) gives the canonical signature; rendering the PNG from that signature is an output adapter that lives outside the π* registry. If needed, ships as a sibling utility (arborist viz CLI?).
• Numerical solvers (scipy.optimize, root-finding, numerical integration with scipy.integrate.quad). These are approximate, not canonical. Different solver tolerances produce different bytes — bad fit for π*.
• Probabilistic / statistical kernels. Distributions, MCMC, Bayesian inference — non-deterministic on purpose.
• 3D / surface plotting. Same reasoning as 2D plotting.
• LaTeX rendering. Display layer.
• Cross-modality witness for symbolic answers. Lands as a follow-up once Phase 1 has bench fixtures (#000028 pattern inherits).

6. Risks / things to watch

• SymPy canonicalization is not unique. sp.expand covers polynomial identity but not trigonometric / exponential identities. Two mathematically equivalent expressions can produce different canonical bytes. The algebra-symbolic@v1 contract is "expanded polynomial form" — explicit in the module docstring. The simplified variant (Phase 1b) handles the trig / exp cases at unbounded CPU cost.
• sp.srepr vs str(expr) for canonical bytes. srepr is more verbose but more deterministic across SymPy versions (str ordering can change). Picking srepr pins the canonical form to SymPy's internal AST representation; bumping SymPy major version may require algebra-symbolic@v2.
• Integration timeouts. sp.integrate can hang on pathological expressions. Wrap with a soft timeout (separate thread + future.result(timeout=…)). On timeout: emit unevaluated:Integral(<original>) sentinel, same as the no-closed-form case.
• Symbol-name collision. A user passing f="x" + "x" (literal Python string concat) vs f="2*x" should not produce a false equivalence-class match. SymPy's sympify handles this; tests pin the boundary.
• opencompletion activity uses different normalization. That code uses numpy.lambdify for plotting, not canonical bytes. function-sampled@v1 (Phase 7) bridges by quantizing to integer grid — the SymPy expression is the source, the sampled bytes are the proof. opencompletion's PNG output is unaffected.

7. Test plan

Each phase ships with:

• 8-10 syntax fixtures (does it canonicalize without error?).
• 12-15 semantics fixtures (equivalence-class pairs that should collapse + distinct pairs that should not).
• Unit tests covering the kernel directly.
• Optional-import skip behavior (suite stays green when sympy absent).
• Composition test if the kernel composes naturally (Phase 2 via compose("calculus-derivative@v1", "arithmetic@v1") for numeric eval at a point).

Phase 1 + 2 acceptance: ≥40 test cases between them; full suite stays at 0 regressions; bench-5s-algebra Makefile target passes 100%.

8. Implementation order

Step 1 — pyproject.toml [math] extra + Makefile bootstrap-math.
Step 2 — Phase 1 (algebra-symbolic@v1) — module + tests + fixtures.
Step 3 — Math preflight in query.py learns the algebra route.
Step 4 — Phase 2 (calculus-derivative@v1) — module + tests + fixtures.
Step 5 — Bench fixtures graduate to a real bench-5s-algebra target.
Step 6 — Decision point: continue with Phases 3-7 or pause for
         downstream consumers (witness fixtures, opencompletion
         activity integration).

Phases 1+2 fit one commit (~400 LOC total + tests). Phases 3-7 each warrant their own commit but share infrastructure.

9. Forward links

• Source paper / motivation: ~/git/opencompletion/research/activity24-math-plot.yaml (SymPy + numpy + matplotlib pipeline; the π* expansion makes the same substrate available inside arborist's audit chain).
• Composition algebra: docs/pi-star-composition.md.
• #000028 (multi-modality witness): symbolic LLM outputs become witnessable once Phase 1 lands; expect divergence events for SymPy-class hallucinations.
• 5T Truthtables / 5F Falsification: new fixture surface for symbolic algebra LLM calibration.
• #000027 (canonical persistence): algebra/calculus answers inherit the persistence + audit chain for free; no extra wiring needed.

Status

All 7 phases + Phase 1b landed 2026-05-09 across two commits. Registry-closure complete; the math substrate is real end-to-end.

Phase	Landed	Commit
1 — algebra-symbolic@v1	2026-05-09	04f3f5d
1b — algebra-symbolic-simplified@v1	2026-05-09	04f3f5d
2 — calculus-derivative@v1	2026-05-09	04f3f5d
3 — calculus-integral@v1	2026-05-09	(fox-direct)
4 — calculus-limit@v1	2026-05-09	abe5988
5 — calculus-series@v1	2026-05-09	abe5988
6 — linear-algebra@v1	2026-05-09	abe5988
7 — function-sampled@v1	2026-05-09	abe5988

Plus the tabular-pinned@v1 last-stub graduation in abe5988 closes the registry chapter — every modality the substrate paper reserved (text, claim_lattice, code, arithmetic, logic, time-series, tabular) plus the math substrate (algebra-symbolic + 4 calculus + linear-algebra + function-sampled) is real. 15 concrete π's in the registry; no remaining reserved stubs.*

Follow-up landed in 70ffc01: make demo-plot script closes the opencompletion activity24-math-plot.yaml loop end-to-end (SymPy expression → canonical bytes via function-sampled@v1 → optional matplotlib PNG).

Landed scope:

• pyproject.toml — [math] extra (sympy>=1.13); folded into [dev] so make bootstrap pulls it transitively.
• Makefile — bootstrap-math explicit target + bench-5s-algebra bench target.
• arborist/pi_star/algebra_symbolic.py — Phase 1 module (AlgebraSymbolicV1). sp.expand → sp.srepr canonical bytes. Polynomial AND exponential identity collapse; trigonometric identity does not (deferred to a future algebra-symbolic-simplified@v1 Phase 1b ticket).
• arborist/pi_star/calculus_derivative.py — Phase 2 module (CalculusDerivativeV1). JSON-shaped {f, x, n} input contract → SymPy diff + expand → srepr bytes (output is itself a valid algebra-symbolic@v1 input).
• arborist/pi_star/__init__.py — registers both side-effect modules at package load; both self-guard via defensive import sympy so a fresh checkout without [math] still loads.
• arborist/qa/query.py:_canonical_projection_preflight — third route between arithmetic and logic. Charset regex (_CANONICAL_ALGEBRA_RE) + leading-natural-language-verb reject (_CANONICAL_ALGEBRA_NL_LEAD_RE, 4-letter minimum so x, xy, sin, cos, pi survive) + at-least-one-letter requirement (else arithmetic wins). PiStarError + KeyError both fall through cleanly.
• bench/fixtures/5s/{syntax,semantics}-algebra-symbolic-v1.jsonl — 10 syntax fixtures (parse-pass) + 13 semantics fixtures (equivalence classes including the documented trig non-collapse
  • exp collapse). bench-5s-algebra target reports 100% pass.
• bench/batteries/base.py PHASE_1_CARRIERS — symbolic_algebra added to the carrier whitelist so the harness accepts the new fixtures.
• tests/test_pi_star_algebra_symbolic.py (18 tests) + tests/test_pi_star_calculus_derivative.py (38 tests) + preflight algebra-route tests in tests/test_canonical_projection.py. All gate on pytest.importorskip("sympy") so a sympy-less checkout stays green.
• Full test suite passes (1369 passed / 27 skipped).

Estimated-size predictions held: Phase 1 ~130 LOC + tests, Phase 2 ~140 LOC + tests. Total commit ~1100 LOC including tests, fixtures, docs, and the preflight wiring.

Phase 3-7 follow-on size estimate (~80 LOC each) carries forward; each gets its own future ticket when an actual consumer surfaces the need.

---

Phase 1b — algebra-symbolic-simplified@v1 (landed 2026-05-09)

arborist/pi_star/algebra_symbolic_simplified.py — full-simplify variant. Recipe: sp.expand(sp.simplify(expr)) → srepr. The follow-up expand after simplify enforces a uniform polynomial canonical form; without it, (x+1)**2 and x**2 + 2*x + 1 collapse under simplify to DIFFERENT shapes (factored vs expanded) and break the equivalence-class invariant. The composed recipe collapses both polynomial + trig identity:

• sin(x)**2 + cos(x)**2 ≡ 1
• tan(x)*cos(x) ≡ sin(x)
• (x+1)**2 ≡ x**2 + 2*x + 1 (still)

Cost: 1-360 ms typical; pathological inputs unbounded. No in-π* timeout (the calling pipeline owns that budget). Operators opt in by registry key — the fast Phase-1 sibling stays the default for callers that only need polynomial canonicalization.

22 unit tests, all gating on pytest.importorskip("sympy").

Phase 3 — calculus-integral@v1 (landed 2026-05-09)

arborist/pi_star/calculus_integral.py — symbolic integration with thread-timeout fallback. JSON-shaped {f, x, limits?, timeout_seconds?} input. Two output paths:

1. Closed form: sp.srepr(sp.expand(integrate_result)) — same recipe as algebra-symbolic@v1, so the output is itself a valid algebra-symbolic input and composes naturally.
2. Unevaluated (SymPy returns Integral(...), OR the timeout fires): b"unevaluated:" + sp.srepr(<Integral>). The unevaluated: prefix lets callers tell "no closed form" from "input invalid" without re-parsing the canonical form.

Timeout discipline: concurrent.futures.ThreadPoolExecutor with future.result(timeout=…). On timeout, synthesize the same unevaluated sentinel SymPy itself would emit, so timeout + no-closed-form converge to the same bytes for the same input. Default 30 s; per-call override via timeout_seconds. Python threads cannot be killed cleanly — a timed-out worker leaks until SymPy returns; document this as the cost of the discipline.

Coverage:

• Indefinite integrals (∫x dx = x²/2, ∫sin(x) dx = -cos(x), …).
• Definite integrals with finite + infinite limits (Gaussian ∫_{-∞}^{∞} exp(-x²) dx = √π).
• Unevaluated sentinel for exp(x)/log(x) and timeout-induced unevaluated paths.
• 31 unit tests including a monkeypatch deterministic timeout test (sleep-mocked SymPy so the timeout path doesn't depend on any specific input being slow on every CI runner).