docs/tickets: open #000030 — math π* expansion (SymPy substrate)

Brain-stormed off opencompletion's activity24-math-plot.yaml
(SymPy + numpy + matplotlib pipeline). Three things in there
map to π* shapes; one doesn't.

Maps:
- algebra-symbolic@v1   — symbolic expression → expand + canonical
                          ordering. Closes the (x+1)**2 ≡ x**2+2x+1
                          equivalence class.
- calculus-derivative@v1 — d/dx(f) via sp.diff, re-canonicalized
                           through algebra-symbolic@v1.
- calculus-integral@v1  — ∫f dx via sp.integrate; sentinel for
                          unevaluated cases.
- calculus-limit@v1, calculus-series@v1, linear-algebra@v1 —
                          additional SymPy-friendly phases.
- function-sampled@v1   — bridges symbolic expressions to the
                          existing time-series-quantized@v1
                          format. Two functions that render
                          identically (within sample tolerance)
                          collapse to same canonical bytes. This
                          is what plotting CAN become in π* terms.

Doesn't map:
- PNG plot rendering — different DPIs / fonts / palettes all
  valid; image bytes aren't canonical. Stays as an output adapter
  that COMPOSES with function-sampled@v1.

Hard constraint: SymPy is an OPTIONAL dependency ([math] extra).
Each new π* registers only when sympy is importable, mirroring
the existing html / wikitext / crawler pattern. Fresh checkout
without sympy keeps passing the full test suite (graceful skip).

Multi-phase rollout. Recommendation: Phase 1 (algebra-symbolic@v1)
+ Phase 2 (calculus-derivative@v1) in one commit (~400 LOC total
+ tests + 5S fixtures). Subsequent phases (integral, limit,
series, linalg, function-sampled) each their own commit.

Forward links in the ticket: #000027 (canonical persistence —
algebra/calculus answers inherit audit chain for free), #000028
(witness — symbolic answers become witnessable), 5T/5F (new
fixture surface for symbolic LLM calibration), composition algebra
(deriv+arith, expr+sample compose naturally).

Index updated. Next ID 000031.
This commit is contained in:
russell@unturf.com 2026-05-09 11:45:39 -04:00
parent b38f4b8b59
commit 62d9c7a440
No known key found for this signature in database
2 changed files with 465 additions and 1 deletions

View file

@ -61,6 +61,7 @@ Newest first. Update on every open/close.
| ID | Title | Status | Opened | Directive |
|----------|------------------------------------------------|-----------------------|------------|-----------|
| #000030 | Math π* expansion: SymPy substrate (algebra · calculus · linalg) | open · awaiting go/no-go | 2026-05-09 | — |
| #000029 | Claim-pack source (axiom/theorem JSON bundles) | closed · landed 2026-05-09 | 2026-05-09 | — |
| #000028 | Multi-modality witness for canonical shapes | closed · landed 2026-05-09 (STRICT-WITNESSED reachable post-#000027) | 2026-05-08 | — |
| #000027 | Canonical projections persist to providence_cache | closed · landed 2026-05-09 | 2026-05-08 | — |
@ -93,4 +94,4 @@ Newest first. Update on every open/close.
## Next ID
`000030`
`000031`

View file

@ -0,0 +1,463 @@
# Ticket #000030 — Math π* expansion: SymPy substrate
**Status:** open · awaiting go/no-go
**Opened:** 2026-05-09
**Scope:** Extend the π* registry with SymPy-backed canonicalizers
covering symbolic algebra, calculus (derivatives, integrals,
limits, series), linear algebra, and a function-sampled bridge to
`time-series-quantized@v1`. Multi-phase; each phase is a single
new π* in the registry.
**Audience:** maintainers of arborist's π* layer + downstream
consumers (canonical-projection witness, math/logic preflight,
benchmark batteries, opencompletion math activities).
**Hard constraint:** SymPy is an optional dependency (`[math]`
extra). Each new π* registers ONLY when `import sympy` succeeds at
package load — same defensive-import pattern as the existing
`html` / `wikitext` / `crawler` extras. A fresh checkout without
SymPy must continue to pass the full test suite (graceful skip).
---
## 1. Problem statement
Arborist's math substrate today (post-#000027 + #000028):
| π* (math) | scope |
|-------------------------|--------------------------------------|
| `arithmetic@v1` | Closed-form rational arithmetic — `0.1+0.2` → `3/10`. SQD §14.1. |
| `logic-kernel@v1` | Propositional Boolean → CNF. SQD §14.3. |
| `time-series-quantized@v1` | Sample-array (dt, dv, [(t, v),…]) → quantized integer vector. SQD §13.5. |
What's missing — the symbolic algebra & calculus layer that the
opencompletion `activity24-math-plot.yaml` activity exercises (SymPy
+ numpy + matplotlib for parsing → simplification → sampling →
plotting). Two consequences of the gap:
1. **Question shapes the canonical-projection preflight can't
answer.** "Simplify `(x+1)**2`", "what's the derivative of
`x**3 + 2*x`", "integrate `sin(x)` from 0 to π" all fall
through to RAG today, even though they have closed-form
deterministic answers. The math witness (#000028) STRICT
warrant is therefore unreachable for symbolic questions.
2. **5T (Truthtables) and 5F (Falsification) bench batteries
cover propositional logic but not symbolic algebra/calculus.**
No supervised calibration data for symbolic LLM hallucinations.
The opencompletion activity uses SymPy already. Pulling that
substrate into arborist closes both gaps with one optional
dependency.
## 2. Design choices
### 2.1 Dependency strategy
**A. Optional `[math]` extra (RECOMMENDED).** SymPy is ~30 MB
installed; pulling it into core deps would inflate every fresh
checkout. Mirror the existing pattern:
```
pip install 'arborist[math]' # installs sympy
```
Each new π* module guards registration with `try: import sympy
except ImportError: SYMPY = None` and skips `register(...)` when
absent. Tests use `pytest.importorskip("sympy")`. Real-shard /
bench paths fail explicitly with `unsupported_pi_star` rather
than crashing.
**B. Bundle SymPy into core deps.** Simplifies "does it work?"
but doubles the install footprint and pulls in mpmath as a
transitive. Not worth it for users who never touch math.
**C. Vendor a tiny subset.** SymPy is too large to vendor; the
algebra/calculus paths use deep symbolic infrastructure.
→ **A.** Matches the `html` / `wikitext` / `crawler` extras
already in `pyproject.toml`. Test suite stays green either way.
### 2.2 Which kernels in scope
Phase-by-phase rollout. Each phase = one π* in the registry.
**Phase 1 — algebra-symbolic@v1**
- Parse expression via `sp.sympify`.
- Canonicalize via `sp.expand` + lexical ordering of free
symbols + canonical printing (`sp.srepr` for proof-grade
bytes; `str(expr)` for human-readable).
- `(x+1)**2` and `x**2 + 2*x + 1` → same canonical bytes.
- Equivalence-class-preserving for polynomial identities;
trigonometric identities (`sin(x)**2 + cos(x)**2 = 1`) NOT
guaranteed unless the caller invokes a `simplify` overlay
variant (Phase 1b: `algebra-symbolic-simplified@v1`).
- Errors: undefined symbol → `PiStarError`; non-expression
input → `PiStarError`.
**Phase 2 — calculus-derivative@v1**
- Input: JSON `{"f": "<expr>", "x": "<var>", "n"?: <order>}`.
- Output: canonical bytes from `sp.diff(f, x, n)` re-canonicalized
through `algebra-symbolic@v1`.
- `d/dx(x**2)` → `2*x`. `d²/dx²(x**3)` → `6*x`.
- Composition note: this π* COMPOSES with `algebra-symbolic@v1`
via the existing `arborist.pi_star.compose` registry.
**Phase 3 — calculus-integral@v1**
- Input: JSON `{"f": "<expr>", "x": "<var>", "limits"?: [a, b]}`.
- Indefinite: SymPy `sp.integrate(f, x)`. Definite: with limits.
- Output: canonical bytes if closed form exists. If SymPy returns
`Integral(...)` unevaluated → emit explicit sentinel
`unevaluated:Integral(<original>)` rather than raising. Lets
the caller distinguish "no closed form" from "input invalid."
- `∫x dx` → `x**2/2`; `∫(0..π) sin(x) dx` → `2`.
**Phase 4 — calculus-limit@v1**
- `sp.limit(f, x, point, dir?)`.
- Input JSON: `{"f", "x", "point", "dir"?: "+/-"}`.
- Common cases: `lim(sin(x)/x, x, 0)` → `1`; `lim(1/x, x, 0, "+")`
→ `oo`. Infinity sentinel: `inf` / `-inf` / `complex_inf`.
**Phase 5 — calculus-series@v1**
- Taylor / Maclaurin series to N terms via `sp.series(f, x, x0, n)`.
- Input JSON: `{"f", "x", "x0", "n"}`.
- Output: canonical truncated polynomial through
`algebra-symbolic@v1`. The `O(x**n)` remainder is dropped (the
truncation is what makes this finite-byte canonical).
**Phase 6 — linear-algebra@v1**
- Input JSON: `{"op": "rref|det|eigenvalues|eigenvectors|inverse",
"matrix": [[…], […], …]}`.
- Output: canonical matrix bytes (rational entries via
arithmetic@v1) for RREF / inverse; rational scalar for det;
sorted list of eigenvalues for eigenvalues; etc.
- One π* covers the whole linear-algebra surface; the `op` field
in input switches behavior. Cleaner than 4 separate π*'s.
**Phase 7 — function-sampled@v1**
- Bridge to `time-series-quantized@v1`. Input JSON:
`{"f": "<expr>", "x": "<var>", "x_min": …, "x_max": …,
"n_samples": …, "dv": …}`.
- Lambdifies `f`, samples on the linspace grid, quantizes via
the same dt/dv canonicalization as `time-series-quantized@v1`.
- Output: identical format to `time-series-quantized@v1`
(`dt=…;dv=…;n=…;t0=…:v0|v1|…`).
- This is what plotting CAN become in π* terms: two functions
that render identically (within sample tolerance) collapse to
the same canonical bytes.
- The PNG visualization is an output adapter — see §4.
### 2.3 Ordering & normalization choices (per Phase)
SymPy doesn't guarantee a single canonical form; different
expressions can be mathematically equivalent but textually
distinct. Each phase pins its normalization recipe:
| Phase | Normalization recipe |
|--------------------------|-----------------------------------------------------|
| `algebra-symbolic@v1` | `sp.expand` only (fast; preserves polynomial identity but NOT trig identity). |
| `algebra-symbolic-simplified@v1` (1b — optional) | `sp.simplify` — slower; catches `sin²+cos²=1` etc. |
| `calculus-derivative@v1` | derivative → `algebra-symbolic@v1`. |
| `calculus-integral@v1` | integral → `algebra-symbolic@v1`; sentinel for unevaluated. |
| `calculus-limit@v1` | limit → printed canonical form (`+oo`, `-oo`, etc.). |
| `calculus-series@v1` | series truncated → `algebra-symbolic@v1`. |
| `linear-algebra@v1` | matrix entries via `arithmetic@v1`; ordered rows. |
| `function-sampled@v1` | sample grid → `time-series-quantized@v1` format. |
The `simplify` variant lives behind a separate version because
its CPU cost is unbounded (SymPy `simplify` can be exponential);
operators opt in when they need it.
### 2.4 Composition with existing π*
Two natural composites land for free via
`arborist.pi_star.compose`:
```python
from arborist.pi_star import compose
# Symbolic differentiation ending in a numeric evaluation:
deriv_then_arith = compose("calculus-derivative@v1", "arithmetic@v1")
# Function plotting end-to-end signature:
expr_then_sample = compose("algebra-symbolic@v1", "function-sampled@v1")
```
Auto-registers as `<a>-then-<b>@v1` per the existing composition
algebra (see `docs/pi-star-composition.md`).
## 3. Recommendation
**Phase 1 first.** `algebra-symbolic@v1` alone unlocks the
biggest single class of new questions (algebraic simplification,
identity-checking) and demonstrates the SymPy-extra pattern.
Phase 2 (`calculus-derivative@v1`) follows naturally — same
parsing, same printing, just one extra `sp.diff` call.
After Phases 1+2 land and pass real-world bench fixtures, fox
decides which of Phases 3-7 to pull next. None depends on the
others except `function-sampled@v1` (which composes with Phase 1
but doesn't structurally require it).
## 4. Implementation sketch
### 4.1 Dependency wiring
In `pyproject.toml`:
```toml
[project.optional-dependencies]
math = [
"sympy>=1.13",
]
```
`make bootstrap` pattern continues to work; users who want math
run `pip install '.[math]'` separately, or we add a
`bootstrap-math` Makefile target.
### 4.2 New module: `arborist/pi_star/algebra_symbolic.py`
```python
try:
import sympy as sp
except ImportError: # pragma: no cover — optional extra
sp = None
@dataclass
class AlgebraSymbolicV1:
name: str = "algebra-symbolic"
version: str = "v1"
domain: str = "symbolic-algebra"
def canonicalize(self, raw: bytes) -> bytes:
if sp is None:
raise PiStarError(
"algebra-symbolic@v1 requires sympy; install with "
"`pip install 'arborist[math]'`"
)
text = raw.decode("utf-8", errors="surrogatepass")
try:
expr = sp.sympify(text)
except (sp.SympifyError, SyntaxError) as exc:
raise PiStarError(
f"algebra-symbolic@v1 cannot parse {text!r}: {exc}"
) from exc
canonical = sp.expand(expr)
# Stable canonical form: srepr (S-expression representation)
# is more deterministic than str() across SymPy versions.
return sp.srepr(canonical).encode("utf-8")
if sp is not None:
register(AlgebraSymbolicV1())
```
### 4.3 New module: `arborist/pi_star/calculus_derivative.py`
```python
@dataclass
class CalculusDerivativeV1:
name: str = "calculus-derivative"
version: str = "v1"
domain: str = "calculus"
def canonicalize(self, raw: bytes) -> bytes:
if sp is None:
raise PiStarError("requires sympy; pip install 'arborist[math]'")
try:
obj = json.loads(raw.decode("utf-8", errors="surrogatepass"))
except json.JSONDecodeError as exc:
raise PiStarError(f"input must be JSON object: {exc}") from exc
for k in ("f", "x"):
if k not in obj:
raise PiStarError(f"missing required field {k!r}")
n = obj.get("n", 1)
if not isinstance(n, int) or n < 0:
raise PiStarError("'n' must be non-negative integer")
x_sym = sp.Symbol(obj["x"])
try:
f_expr = sp.sympify(obj["f"], locals={obj["x"]: x_sym})
deriv = sp.diff(f_expr, x_sym, n)
canonical = sp.expand(deriv)
except (sp.SympifyError, SyntaxError, ValueError) as exc:
raise PiStarError(f"derivative failed: {exc}") from exc
return sp.srepr(canonical).encode("utf-8")
```
### 4.4 Test layout
`tests/test_pi_star_algebra.py`:
```python
sympy = pytest.importorskip("sympy") # entire file skipped if absent
def test_polynomial_expansion_identity():
ps = get("algebra-symbolic@v1")
assert ps.canonicalize(b"(x+1)**2") == ps.canonicalize(b"x**2 + 2*x + 1")
def test_distinct_polynomials_distinct():
ps = get("algebra-symbolic@v1")
assert ps.canonicalize(b"x**2") != ps.canonicalize(b"x**3")
def test_undefined_input_raises():
ps = get("algebra-symbolic@v1")
with pytest.raises(PiStarError):
ps.canonicalize(b"not a valid expression $$$")
```
Same shape for `tests/test_pi_star_calculus_derivative.py`.
### 4.5 Bench fixtures (Phase 1 only — others follow when those land)
`bench/fixtures/5s/syntax-algebra-symbolic-v1.jsonl` — 10
fixtures of valid expressions through `sp.sympify` + `sp.expand`.
`bench/fixtures/5s/semantics-algebra-symbolic-v1.jsonl` —
equivalence-class fixtures:
- `(x+1)**2` ≡ `x**2 + 2*x + 1` ≡ `x*x + 2*x + 1`
- `(a+b)*(a-b)` ≡ `a**2 - b**2`
- `2*x + 3*x` ≡ `5*x`
- distinct: `x**2 + 1` vs `x**2`
### 4.6 Math/logic preflight extension
In `arborist/qa/query.py:_canonical_projection_preflight`,
expand the sniff to include "looks like an algebraic expression
with letters" — currently the regex rejects letters (math route)
or requires uppercase atoms (logic route). New route: lowercase
letters + arithmetic ops → try `algebra-symbolic@v1`.
```python
# Algebra route — lowercase letters present, but otherwise
# expression-shaped (parens, ops, digits, letters, whitespace).
if _ALGEBRA_RE.fullmatch(q):
try:
out = get("algebra-symbolic@v1").canonicalize(q.encode("utf-8"))
return ("algebra-symbolic@v1", out)
except PiStarError:
return None
```
Order: arithmetic → algebra → logic. Arithmetic claims
no-letters first; algebra picks up the letter-containing
remainder; logic claims uppercase-only.
### 4.7 Documentation updates
- `docs/_source/pi-star.rst`: add Algebra / Calculus section
alongside the existing Arithmetic + Logic descriptions.
- `bench/batteries/base.py PHASE_1_CARRIERS`: add
`symbolic_algebra` (and others as phases land).
- New Makefile target: `bench-5s-algebra` mirroring
`bench-5s-math`.
## 5. Out of scope
- **Plotting / image rendering.** The PNG output the
opencompletion activity produces is a visualization, not a
canonical projection. `function-sampled@v1` (Phase 7) gives
the canonical signature; rendering the PNG from that signature
is an output adapter that lives outside the π* registry. If
needed, ships as a sibling utility (`arborist viz` CLI?).
- **Numerical solvers** (`scipy.optimize`, root-finding,
numerical integration with `scipy.integrate.quad`). These are
approximate, not canonical. Different solver tolerances
produce different bytes — bad fit for π*.
- **Probabilistic / statistical kernels.** Distributions, MCMC,
Bayesian inference — non-deterministic on purpose.
- **3D / surface plotting.** Same reasoning as 2D plotting.
- **LaTeX rendering.** Display layer.
- **Cross-modality witness for symbolic answers.** Lands as a
follow-up once Phase 1 has bench fixtures (#000028 pattern
inherits).
## 6. Risks / things to watch
- **SymPy canonicalization is not unique.** `sp.expand` covers
polynomial identity but not trigonometric / exponential
identities. Two mathematically equivalent expressions can
produce different canonical bytes. The `algebra-symbolic@v1`
contract is "expanded polynomial form" — explicit in the
module docstring. The `simplified` variant (Phase 1b) handles
the trig / exp cases at unbounded CPU cost.
- **`sp.srepr` vs `str(expr)`** for canonical bytes. `srepr` is
more verbose but more deterministic across SymPy versions
(str ordering can change). Picking `srepr` pins the canonical
form to SymPy's internal AST representation; bumping SymPy
major version may require `algebra-symbolic@v2`.
- **Integration timeouts.** `sp.integrate` can hang on
pathological expressions. Wrap with a soft timeout (separate
thread + future.result(timeout=…)). On timeout: emit
`unevaluated:Integral(<original>)` sentinel, same as the
no-closed-form case.
- **Symbol-name collision.** A user passing `f="x" + "x"`
(literal Python string concat) vs `f="2*x"` should not
produce a false equivalence-class match. SymPy's `sympify`
handles this; tests pin the boundary.
- **`opencompletion` activity uses different normalization.**
That code uses `numpy.lambdify` for plotting, not canonical
bytes. `function-sampled@v1` (Phase 7) bridges by quantizing
to integer grid — the SymPy expression is the source, the
sampled bytes are the proof. opencompletion's PNG output is
unaffected.
## 7. Test plan
Each phase ships with:
- 8-10 syntax fixtures (does it canonicalize without error?).
- 12-15 semantics fixtures (equivalence-class pairs that should
collapse + distinct pairs that should not).
- Unit tests covering the kernel directly.
- Optional-import skip behavior (suite stays green when sympy
absent).
- Composition test if the kernel composes naturally (Phase 2
via `compose("calculus-derivative@v1", "arithmetic@v1")` for
numeric eval at a point).
Phase 1 + 2 acceptance: ≥40 test cases between them; full suite
stays at 0 regressions; `bench-5s-algebra` Makefile target
passes 100%.
## 8. Implementation order
```
Step 1 — pyproject.toml [math] extra + Makefile bootstrap-math.
Step 2 — Phase 1 (algebra-symbolic@v1) — module + tests + fixtures.
Step 3 — Math preflight in query.py learns the algebra route.
Step 4 — Phase 2 (calculus-derivative@v1) — module + tests + fixtures.
Step 5 — Bench fixtures graduate to a real bench-5s-algebra target.
Step 6 — Decision point: continue with Phases 3-7 or pause for
downstream consumers (witness fixtures, opencompletion
activity integration).
```
Phases 1+2 fit one commit (~400 LOC total + tests). Phases 3-7
each warrant their own commit but share infrastructure.
## 9. Forward links
- **Source paper / motivation:**
`~/git/opencompletion/research/activity24-math-plot.yaml`
(SymPy + numpy + matplotlib pipeline; the π* expansion makes
the same substrate available inside arborist's audit chain).
- **Composition algebra:** `docs/pi-star-composition.md`.
- **#000028 (multi-modality witness):** symbolic LLM outputs
become witnessable once Phase 1 lands; expect divergence
events for SymPy-class hallucinations.
- **5T Truthtables / 5F Falsification:** new fixture surface
for symbolic algebra LLM calibration.
- **#000027 (canonical persistence):** algebra/calculus answers
inherit the persistence + audit chain for free; no extra
wiring needed.
## Status
Open · awaiting go/no-go.
Estimated size:
- Phase 1: ~120 LOC module + ~80 LOC tests + ~25 fixtures.
- Phase 2: ~80 LOC module + ~60 LOC tests + ~20 fixtures.
- Phases 3-7: ~80 LOC each (similar shape).
Phase 1+2 single commit feasible. Subsequent phases roll one at a
time as fox prioritizes.