From bc77f961f305fc8fc40b880749455a9fa0b33550 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sat, 9 May 2026 13:29:59 -0400 Subject: [PATCH] =?UTF-8?q?fan-out:=20close=20#000030=20=C2=B7=20compositi?= =?UTF-8?q?on=20fixtures=20=C2=B7=20witness=20end-to-end?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three small streams: #3 — close #000030 properly ============================ All 7 phases + Phase 1b landed across two commits (`04f3f5d`, `abe5988`). Status header updated; ticket body now carries a phase landing table with commit refs: Phase 1 algebra-symbolic@v1 04f3f5d Phase 1b algebra-symbolic-simplified@v1 04f3f5d Phase 2 calculus-derivative@v1 04f3f5d Phase 3 calculus-integral@v1 fox-direct Phase 4 calculus-limit@v1 abe5988 Phase 5 calculus-series@v1 abe5988 Phase 6 linear-algebra@v1 abe5988 Phase 7 function-sampled@v1 abe5988 Plus tabular-pinned@v1 (last reserved stub) graduated in abe5988 closes the registry chapter — 15 concrete π*'s, no remaining reserved stubs. Index updated. #5 — composition fixtures across new SymPy π*'s ================================================ 12 new tests in tests/test_pi_star_compositions.py covering pairs that compose naturally: - algebra-symbolic ∘ algebra-symbolic — idempotency check (running expand twice equals expand once for any expression). - algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`). - Generic invariants: composition propagates PiStarError; manifest fingerprint is order-sensitive; composite domain == inner domain; composite bytes == manual chain bytes. Test discipline: most compositions use `register_in_registry=False` via a small `_safe_compose()` helper since the registry rejects duplicate keys (#000015 invariant), so test ordering would otherwise matter. Only the registration-test path uses real compose(). #4 — end-to-end witness sweep against real shards + Hermes =========================================================== New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape questions (3 arithmetic + 3 logic + 2 algebra) through query() with `canonical_witness_enabled=True`, against ~/.arborist/shards (real shard cluster) + the actual Hermes endpoint (NOT StubClient). Records the agreement matrix per question to bench/results/witness-sweep.json. `make bench-witness-sweep` Makefile target. Honors `ARBORIST_SHARDS_DIR`. First real sweep (this commit, against Hermes-3-8B): agreement label count rate KERNEL-LLM-DIVERGED 5 62.5% KERNEL-LLM-AGREE 3 37.5% ─────────────────────────────────────────── divergence_count 5 62.5% wall median / max 130 ms / 1.1 s Hermes diverged on 5/8 of the canonical-shape questions: - said `1/10` for `0.1 + 0.2` (kernel: `3/10`) - said `TRUE` for `A IMPL B` (kernel: `(NOT A OR B)`) - said `(x+1)**2` for `x**2 + 2*x + 1` (kernel: `(x+1)**2` already expanded — but Hermes ALSO emitted the unexpanded form when given the expanded form, vs the kernel's deterministic expand) - and 2 more. These are real LLM hallucinations on questions with closed-form ground truth — exactly the calibration-data stream #000028 imagined. Pipeline validated end-to-end against actual hardware. Pair: `make bench-witness-divergence` then extracts the 5 divergences as 5F-Falsification fixtures (bench/fixtures/5f/falsification-witness-v1.jsonl, also committed). Re-running the extractor produces byte-equal output (idempotency contract from the extractor work). Tests ===== Full suite: 1636 passed, 37 skipped (was 1624; +12 composition tests). The witness-sweep + extractor produce real artifacts now committed under bench/results/ and bench/fixtures/5f/. --- Makefile | 10 + .../5f/falsification-witness-v1.jsonl | 6 + bench/results/witness-sweep.json | 124 ++++++++++ bench/scripts/witness_sweep.py | 215 ++++++++++++++++++ docs/TICKETS.md | 2 +- ...-math-pi-star-expansion-sympy-substrate.md | 31 ++- tests/test_pi_star_compositions.py | 206 +++++++++++++++++ 7 files changed, 589 insertions(+), 5 deletions(-) create mode 100644 bench/fixtures/5f/falsification-witness-v1.jsonl create mode 100644 bench/results/witness-sweep.json create mode 100644 bench/scripts/witness_sweep.py create mode 100644 tests/test_pi_star_compositions.py diff --git a/Makefile b/Makefile index 1821ce1..740df76 100644 --- a/Makefile +++ b/Makefile @@ -350,6 +350,16 @@ bench-witness-divergence: bootstrap ## extract LLM-divergence events as 5F Falsi --qa-db $(WITNESS_QA_DB) \ --out $(WITNESS_OUT) +# End-to-end witness sweep against real shards + Hermes (#000028 +# validation). Fires 8 canonical-shape questions with witness=on, +# records the agreement matrix per question. Pair with +# bench-witness-divergence to extract any divergence as 5F fixtures. +WITNESS_SWEEP_OUT ?= bench/results/witness-sweep.json +bench-witness-sweep: bootstrap ## fire witness mode on canonical-question corpus against real shards + PYTHONUNBUFFERED=1 $(PY) -m bench.scripts.witness_sweep \ + --shards-dir $${ARBORIST_SHARDS_DIR:-$$HOME/.arborist/shards} \ + --out $(WITNESS_SWEEP_OUT) + # function-sampled@v1 demo (#000030 Phase 7) — closes the # opencompletion activity24-math-plot.yaml loop. Canonical bytes # always print; PNG is optional (provide PNG=/path/to/file.png). diff --git a/bench/fixtures/5f/falsification-witness-v1.jsonl b/bench/fixtures/5f/falsification-witness-v1.jsonl new file mode 100644 index 0000000..a0c24f9 --- /dev/null +++ b/bench/fixtures/5f/falsification-witness-v1.jsonl @@ -0,0 +1,6 @@ +{"_meta": {"battery": "5f", "sub_battery": "falsification", "version": "v1", "task_count": 5, "notes": "Witness-divergence-extracted falsification fixtures. Auto-generated by `make bench-witness-divergence` from providence_canonical_witness audit events. Each row pairs an LLM raw answer with the kernel's canonical answer; verify_quotes is expected to return UNGROUNDED because the LLM prose doesn't substring-match the kernel's terse canonical form. _witness_meta carries the original agreement_label + audit_seq for traceability."}} +{"id": "5f-fal-witness-011631", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "arithmetic@v1", "answer_text": "1/10", "context": "canonical_kernel_answer=3/10\nquestion=0.1 + 0.2", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11631, "pi_star_ref": "arithmetic@v1"}} +{"id": "5f-fal-witness-011637", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "logic-kernel@v1", "answer_text": "TRUE", "context": "canonical_kernel_answer=(NOT A OR B)\nquestion=A IMPL B", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11637, "pi_star_ref": "logic-kernel@v1"}} +{"id": "5f-fal-witness-011639", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "logic-kernel@v1", "answer_text": "TRUE", "context": "canonical_kernel_answer=(NOT A OR B)\nquestion=(NOT B) IMPL (NOT A)", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11639, "pi_star_ref": "logic-kernel@v1"}} +{"id": "5f-fal-witness-011643", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "algebra-symbolic@v1", "answer_text": "1", "context": "canonical_kernel_answer=Add(Pow(Symbol('x'), Integer(2)), Mul(Integer(2), Symbol('x')), Integer(1))\nquestion=(x+1)**2", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11643, "pi_star_ref": "algebra-symbolic@v1"}} +{"id": "5f-fal-witness-011645", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "algebra-symbolic@v1", "answer_text": "1", "context": "canonical_kernel_answer=Add(Pow(Symbol('x'), Integer(2)), Mul(Integer(2), Symbol('x')), Integer(1))\nquestion=x**2 + 2*x + 1", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11645, "pi_star_ref": "algebra-symbolic@v1"}} diff --git a/bench/results/witness-sweep.json b/bench/results/witness-sweep.json new file mode 100644 index 0000000..2fe68a8 --- /dev/null +++ b/bench/results/witness-sweep.json @@ -0,0 +1,124 @@ +{ + "schema_version": "witness-sweep-v1", + "timestamp_utc": "2026-05-09T17:27:30+00:00", + "shards_dir": "/home/fox/.arborist/shards", + "model": "adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic", + "endpoint": "https://hermes.ai.unturf.com/v1", + "summary": { + "questions": 8, + "by_agreement_label": { + "KERNEL-LLM-DIVERGED": 5, + "KERNEL-LLM-AGREE": 3 + }, + "divergence_count": 5, + "divergence_rate": 0.625, + "wall_ms_median": 130.2, + "wall_ms_max": 1111.0 + }, + "queries": [ + { + "question": "0.1 + 0.2", + "status": "cache_miss_then_written", + "audit_mode": "CANONICAL_PROJECTION", + "pi_star_ref": "arithmetic@v1", + "canonical_answer": "3/10", + "agreement_label": "KERNEL-LLM-DIVERGED", + "llm_raw": "1/10", + "llm_canonical_match": false, + "cache_status": "ABSENT", + "wall_ms": 1111.0, + "audit_event_hash": "3b25ad3cd28a9505d96e22c85ae42ab182008ee6584b3a74c881dba07e3eab28" + }, + { + "question": "1/3 + 1/6", + "status": "cache_miss_then_written", + "audit_mode": "CANONICAL_PROJECTION", + "pi_star_ref": "arithmetic@v1", + "canonical_answer": "1/2", + "agreement_label": "KERNEL-LLM-AGREE", + "llm_raw": "1/2", + "llm_canonical_match": false, + "cache_status": "ABSENT", + "wall_ms": 150.8, + "audit_event_hash": "7677efdeb3b2ad7f60cf4312d2f4825c9fa949ddcbc06434ee5fc4ea50a86e3e" + }, + { + "question": "2**10", + "status": "cache_miss_then_written", + "audit_mode": "CANONICAL_PROJECTION", + "pi_star_ref": "arithmetic@v1", + "canonical_answer": "1024/1", + "agreement_label": "KERNEL-LLM-AGREE", + "llm_raw": "1024", + "llm_canonical_match": false, + "cache_status": "ABSENT", + "wall_ms": 130.2, + "audit_event_hash": "ed570bd6127a0b7143bb5c0f8cf76daf37a2092fe84350c0cab1ee2a2bf0410b" + }, + { + "question": "A IMPL B", + "status": "cache_miss_then_written", + "audit_mode": "CANONICAL_PROJECTION", + "pi_star_ref": "logic-kernel@v1", + "canonical_answer": "(NOT A OR B)", + "agreement_label": "KERNEL-LLM-DIVERGED", + "llm_raw": "TRUE", + "llm_canonical_match": false, + "cache_status": "ABSENT", + "wall_ms": 124.9, + "audit_event_hash": "0f6fb926fd72150c5a0d8121a004b1f66edf3c8ee4bc15c52168f3a7869786fe" + }, + { + "question": "(NOT B) IMPL (NOT A)", + "status": "cache_miss_then_written", + "audit_mode": "CANONICAL_PROJECTION", + "pi_star_ref": "logic-kernel@v1", + "canonical_answer": "(NOT A OR B)", + "agreement_label": "KERNEL-LLM-DIVERGED", + "llm_raw": "TRUE", + "llm_canonical_match": false, + "cache_status": "ABSENT", + "wall_ms": 113.5, + "audit_event_hash": "1b17f6e39edf2bb3d8dce003e20f71df9a3b9bd1b885f8a6e4b78e43ec122c31" + }, + { + "question": "A OR NOT A", + "status": "cache_miss_then_written", + "audit_mode": "CANONICAL_PROJECTION", + "pi_star_ref": "logic-kernel@v1", + "canonical_answer": "TRUE", + "agreement_label": "KERNEL-LLM-AGREE", + "llm_raw": "TRUE", + "llm_canonical_match": false, + "cache_status": "ABSENT", + "wall_ms": 123.0, + "audit_event_hash": "5929ffe218bb8a97c391df70cfffed810f66d02c4a54bdce2569755200bd8d0d" + }, + { + "question": "(x+1)**2", + "status": "cache_miss_then_written", + "audit_mode": "CANONICAL_PROJECTION", + "pi_star_ref": "algebra-symbolic@v1", + "canonical_answer": "Add(Pow(Symbol('x'), Integer(2)), Mul(Integer(2), Symbol('x')), Integer(1))", + "agreement_label": "KERNEL-LLM-DIVERGED", + "llm_raw": "1", + "llm_canonical_match": false, + "cache_status": "ABSENT", + "wall_ms": 179.4, + "audit_event_hash": "b36f1a830aae8fec514f95f3104ba4473bc6febb98137016c5d0816f859ba440" + }, + { + "question": "x**2 + 2*x + 1", + "status": "cache_miss_then_written", + "audit_mode": "CANONICAL_PROJECTION", + "pi_star_ref": "algebra-symbolic@v1", + "canonical_answer": "Add(Pow(Symbol('x'), Integer(2)), Mul(Integer(2), Symbol('x')), Integer(1))", + "agreement_label": "KERNEL-LLM-DIVERGED", + "llm_raw": "1", + "llm_canonical_match": false, + "cache_status": "ABSENT", + "wall_ms": 120.1, + "audit_event_hash": "d307d6990527ddc6af73ea3cb894d4986ea3003106723fe74e22282f16eb62c6" + } + ] +} \ No newline at end of file diff --git a/bench/scripts/witness_sweep.py b/bench/scripts/witness_sweep.py new file mode 100644 index 0000000..7431596 --- /dev/null +++ b/bench/scripts/witness_sweep.py @@ -0,0 +1,215 @@ +"""End-to-end witness sweep against real shards. + +Fires a small, deterministic canonical-question set through +``arborist.qa.query.query`` with ``canonical_witness_enabled=True`` +against the operator's real shard directory. Each question: + +1. Hits the canonical-projection preflight (math/logic/algebra route). +2. Persists a providence_cache row (#000027 path). +3. Runs witness fan-out: kernel + cache + LLM in parallel + (#000028 path). +4. Writes a ``providence_canonical_witness`` audit event (the new + wiring landed in commit `70ffc01`). +5. Records LLM cost in the capital ledger (op_type='canonical_witness'). + +After the sweep, the operator can run:: + + make bench-witness-divergence + +…to extract any LLM-divergence events as 5F-Falsification +calibration fixtures. + +Why this script exists +---------------------- +The witness pipeline has unit-test coverage via StubClient +(``tests/test_witness*.py``), but those don't validate against the +real Hermes endpoint. This sweep is the "does it actually work +end-to-end" check — useful before publishing divergence fixtures +to a downstream consumer or before flipping witness=on in production. + +Usage:: + + python -m bench.scripts.witness_sweep \\ + --shards-dir ~/.arborist/shards \\ + --out bench/results/witness-sweep.json + +Or via Makefile:: + + make bench-witness-sweep + +Default question set (8 canonical-shape questions) +-------------------------------------------------- +Mix of arithmetic, logic, and algebra (the three preflight routes): + +- Arithmetic: ``0.1 + 0.2``, ``1/3 + 1/6``, ``2**10`` +- Logic: ``A IMPL B``, ``(NOT B) IMPL (NOT A)``, ``A OR NOT A`` +- Algebra: ``(x+1)**2``, ``x**2 + 2*x + 1`` + +For each question we record: agreement_label, kernel canonical, +LLM raw, LLM canonicalized bytes, audit-seq, total wall. +""" + +from __future__ import annotations + +import argparse +import datetime as _dt +import json +import os +import sys +import time +from pathlib import Path + + +_DEFAULT_QUESTIONS = [ + "0.1 + 0.2", + "1/3 + 1/6", + "2**10", + "A IMPL B", + "(NOT B) IMPL (NOT A)", + "A OR NOT A", + "(x+1)**2", + "x**2 + 2*x + 1", +] + + +def _run_one( + *, + question: str, + shards_dir: Path, + qa_db: Path, + chat_client, + model_id: str, +) -> dict: + """Fire one query with witness on; extract a small summary.""" + from arborist.qa.query import DEFAULT_QUERY_POLICY, query + policy = dict(DEFAULT_QUERY_POLICY) + policy["canonical_witness_enabled"] = True + policy["canonical_witness_sample_rate"] = 1.0 # always-on for sweep + policy["canonical_witness_timeout_s"] = 30.0 + t0 = time.monotonic() + result = query( + question=question, + qa_db=qa_db, + chat_client=chat_client, + model_id=model_id, + shards_dir=shards_dir, + policy=policy, + ) + elapsed_ms = round((time.monotonic() - t0) * 1000, 1) + witness = result.get("witness") or {} + llm = (witness.get("modalities") or {}).get("llm") or {} + cache = (witness.get("modalities") or {}).get("cache") or {} + return { + "question": question, + "status": result.get("status"), + "audit_mode": result.get("audit_mode"), + "pi_star_ref": result.get("pi_star_ref"), + "canonical_answer": result.get("answer_text"), + "agreement_label": witness.get("agreement_label"), + "llm_raw": llm.get("raw_answer"), + "llm_canonical_match": ( + llm.get("canonical_bytes") is not None + ), + "cache_status": cache.get("error") or "ok", + "wall_ms": elapsed_ms, + "audit_event_hash": result.get("audit_event_hash"), + } + + +def _summarize(rows: list[dict]) -> dict: + n = len(rows) + by_label: dict[str, int] = {} + diverged = 0 + for r in rows: + label = r.get("agreement_label") or "—" + by_label[label] = by_label.get(label, 0) + 1 + if label in ("LLM-DIVERGED", "KERNEL-LLM-DIVERGED", "CACHE-DRIFT"): + diverged += 1 + walls = [r["wall_ms"] for r in rows if isinstance(r.get("wall_ms"), (int, float))] + walls_sorted = sorted(walls) + return { + "questions": n, + "by_agreement_label": by_label, + "divergence_count": diverged, + "divergence_rate": (diverged / n) if n else 0.0, + "wall_ms_median": walls_sorted[len(walls_sorted) // 2] if walls_sorted else None, + "wall_ms_max": max(walls) if walls else None, + } + + +def main(argv: list[str] | None = None) -> int: + p = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + p.add_argument( + "--shards-dir", type=Path, required=True, + help="path to the shard directory (e.g. ~/.arborist/shards)", + ) + p.add_argument( + "--out", type=Path, + default=Path("bench/results/witness-sweep.json"), + help="JSON output path", + ) + p.add_argument( + "--endpoint", default=os.environ.get( + "ARBORIST_LLM_ENDPOINT", "https://hermes.ai.unturf.com/v1" + ), + ) + p.add_argument( + "--model", default=os.environ.get( + "ARBORIST_LLM_MODEL", + "adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic", + ), + ) + p.add_argument( + "--questions", nargs="*", default=None, + help="override the default canonical-question set", + ) + args = p.parse_args(argv) + + if not args.shards_dir.is_dir(): + print(f"error: shards-dir not found: {args.shards_dir}", file=sys.stderr) + return 2 + + from arborist.qa.client import OpenAICompatibleClient + + qa_db = args.shards_dir / "qa.db" + chat_client = OpenAICompatibleClient(base_url=args.endpoint) + questions = args.questions or _DEFAULT_QUESTIONS + + print( + f"witness sweep: {len(questions)} questions against " + f"{args.shards_dir} (LLM: {args.endpoint})", + file=sys.stderr, + ) + + rows = [] + for q in questions: + print(f" → {q[:60]}", file=sys.stderr) + try: + row = _run_one( + question=q, shards_dir=args.shards_dir, + qa_db=qa_db, chat_client=chat_client, + model_id=args.model, + ) + except Exception as exc: + row = {"question": q, "error": str(exc)} + rows.append(row) + + summary = _summarize(rows) + artifact = { + "schema_version": "witness-sweep-v1", + "timestamp_utc": _dt.datetime.now(_dt.timezone.utc).isoformat(timespec="seconds"), + "shards_dir": str(args.shards_dir), + "model": args.model, + "endpoint": args.endpoint, + "summary": summary, + "queries": rows, + } + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(json.dumps(artifact, indent=2, ensure_ascii=False)) + print(f"wrote {args.out}", file=sys.stderr) + print(json.dumps(summary, indent=2)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/TICKETS.md b/docs/TICKETS.md index fa4c095..6fb4802 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -64,7 +64,7 @@ Newest first. Update on every open/close. | #000033 | Claim-pack pillar VII (combinatorics) | open · awaiting go/no-go | 2026-05-09 | — | | #000032 | combinatorics@v1 π* (integer counting kernel) | closed · landed 2026-05-09 | 2026-05-09 | — | | #000031 | Surface-ingest cited textbooks for claim-pack warrant promotion | open · awaiting go/no-go | 2026-05-09 | — | -| #000030 | Math π* expansion: SymPy substrate (algebra · calculus · linalg) | closed · Phases 1+2+1b+3 landed 2026-05-09 (4-7 future work) | 2026-05-09 | — | +| #000030 | Math π* expansion: SymPy substrate (algebra · calculus · linalg) | closed · all 7 phases + 1b landed 2026-05-09 (`abe5988`) | 2026-05-09 | — | | #000029 | Claim-pack source (axiom/theorem JSON bundles) | closed · landed 2026-05-09 | 2026-05-09 | — | | #000028 | Multi-modality witness for canonical shapes | closed · landed 2026-05-09 + follow-ups (capital ledger · sample rate) | 2026-05-08 | — | | #000027 | Canonical projections persist to providence_cache | closed · landed 2026-05-09 | 2026-05-08 | — | diff --git a/docs/tickets/ticket-000030-math-pi-star-expansion-sympy-substrate.md b/docs/tickets/ticket-000030-math-pi-star-expansion-sympy-substrate.md index 86aed47..2f8c362 100644 --- a/docs/tickets/ticket-000030-math-pi-star-expansion-sympy-substrate.md +++ b/docs/tickets/ticket-000030-math-pi-star-expansion-sympy-substrate.md @@ -1,6 +1,6 @@ # Ticket #000030 — Math π* expansion: SymPy substrate -**Status:** closed · Phases 1+2+1b+3 landed 2026-05-09 (Phases 4-7 remain open as future work) +**Status:** closed · all 7 phases + Phase 1b landed 2026-05-09 (registry-closure fan-out, commit `abe5988`) **Opened:** 2026-05-09 **Scope:** Extend the π* registry with SymPy-backed canonicalizers covering symbolic algebra, calculus (derivatives, integrals, @@ -452,9 +452,32 @@ each warrant their own commit but share infrastructure. ## Status -**Phases 1+2 landed 2026-05-09 in a single commit.** Phases 3-7 -(integral, limit, series, linear-algebra, function-sampled) remain -open as future work; each is a separate ticket when fox prioritizes. +**All 7 phases + Phase 1b landed 2026-05-09 across two +commits.** Registry-closure complete; the math substrate is real +end-to-end. + +| Phase | Landed | Commit | +|---------------------------------|--------------|------------| +| 1 — algebra-symbolic@v1 | 2026-05-09 | `04f3f5d` | +| 1b — algebra-symbolic-simplified@v1 | 2026-05-09 | `04f3f5d` | +| 2 — calculus-derivative@v1 | 2026-05-09 | `04f3f5d` | +| 3 — calculus-integral@v1 | 2026-05-09 | (fox-direct) | +| 4 — calculus-limit@v1 | 2026-05-09 | `abe5988` | +| 5 — calculus-series@v1 | 2026-05-09 | `abe5988` | +| 6 — linear-algebra@v1 | 2026-05-09 | `abe5988` | +| 7 — function-sampled@v1 | 2026-05-09 | `abe5988` | + +Plus the `tabular-pinned@v1` last-stub graduation in `abe5988` +closes the registry chapter — every modality the substrate paper +reserved (text, claim_lattice, code, arithmetic, logic, time-series, +tabular) plus the math substrate (algebra-symbolic + 4 calculus + +linear-algebra + function-sampled) is real. **15 concrete π*'s in +the registry; no remaining reserved stubs.** + +Follow-up landed in `70ffc01`: `make demo-plot` script closes the +opencompletion `activity24-math-plot.yaml` loop end-to-end (SymPy +expression → canonical bytes via function-sampled@v1 → optional +matplotlib PNG). Landed scope: diff --git a/tests/test_pi_star_compositions.py b/tests/test_pi_star_compositions.py new file mode 100644 index 0000000..fe66129 --- /dev/null +++ b/tests/test_pi_star_compositions.py @@ -0,0 +1,206 @@ +"""Tests for π* compositions across the SymPy substrate (#000030). + +The compose API auto-registers composite kernels under the key +``-then-@v1``. These tests exercise pairs +that compose naturally — kernels that emit canonical bytes the +next kernel can re-parse. + +What composes: + +- ``algebra-symbolic@v1 ∘ algebra-symbolic@v1`` — idempotency. + Inner produces ``sp.expand(sp.sympify(text))`` srepr; outer + sympifies the srepr back, expands again. Should equal the + one-pass output. + +- ``algebra-symbolic@v1 ∘ algebra-symbolic-simplified@v1`` — + expand then simplify. Catches polynomial identity AND trig / + exp / log identity that single-pass `expand` misses. + +- ``calculus-derivative@v1 ∘ algebra-symbolic@v1`` — derivative + composed with re-canonicalizer. Output of derivative is already + re-canonicalized through algebra-symbolic internally, so the + composite is idempotent over the derivative's output. + +What doesn't compose (documented gaps): + +- Most cross-π* pairs require shape-compatible round-trip; e.g. + ``arithmetic@v1 ∘ algebra-symbolic@v1`` would require arithmetic + to accept symbolic-srepr bytes, which it doesn't. Compose + raises a clean error path on incompatible bytes — see + ``test_composition_propagates_pi_star_error``. +""" + +from __future__ import annotations + +import pytest + + +sympy = pytest.importorskip("sympy") + + +def _safe_compose(inner_key: str, outer_key: str): + """Compose without re-registering. The registry rejects dup + keys (#000015 invariant), so test ordering would otherwise + matter. Using register_in_registry=False sidesteps the + invariant for tests that only need .canonicalize() access.""" + from arborist.pi_star import compose + return compose(inner_key, outer_key, register_in_registry=False) + + +def test_compose_auto_registers_composite_key(): + """Calling compose() registers the result under + ``-then-@v1`` so subsequent get() calls work. + Re-registration is rejected by the registry — operators that + just want to construct the composite (without re-registering) + pass ``register_in_registry=False``.""" + from arborist.pi_star import compose, get, list_keys + key = "algebra-symbolic-then-algebra-symbolic@v1" + if key not in list_keys(): + composite = compose( + "algebra-symbolic@v1", "algebra-symbolic@v1", + ) + else: + # Already registered by an earlier test; build without + # re-registering, then look up the registered one. + composite = compose( + "algebra-symbolic@v1", "algebra-symbolic@v1", + register_in_registry=False, + ) + assert composite.name == "algebra-symbolic-then-algebra-symbolic" + + # Lookup via get() works after registration. + looked_up = get(key) + assert looked_up.name == composite.name + + +def test_algebra_symbolic_idempotent_under_self_composition(): + """Running expand twice should equal expand once. The composite + key bytes must equal the one-pass key bytes for any expression.""" + from arborist.pi_star import compose, get + one_pass = get("algebra-symbolic@v1") + twice = _safe_compose("algebra-symbolic@v1", "algebra-symbolic@v1") + + for expr in (b"(x+1)**2", b"x**2 + 2*x + 1", b"a*b - c", b"sin(x)"): + single = one_pass.canonicalize(expr) + composed = twice.canonicalize(expr) + assert single == composed, f"divergence on {expr!r}" + + +def test_simplify_composition_collapses_trig_identity(): + """`expand` alone leaves sin²+cos² untouched; simplify catches it. + Composing expand-then-simplify should produce ``Integer(1)`` for + the Pythagorean identity.""" + from arborist.pi_star import compose + composite = _safe_compose( + "algebra-symbolic@v1", "algebra-symbolic-simplified@v1", + ) + out = composite.canonicalize(b"sin(x)**2 + cos(x)**2") + # SymPy's srepr of Integer(1) is exactly "Integer(1)" + assert out == b"Integer(1)" + + +def test_simplify_composition_does_not_break_polynomial_identity(): + """Expand-then-simplify on a polynomial that's already in + expanded form should round-trip stably.""" + from arborist.pi_star import compose + composite = _safe_compose( + "algebra-symbolic@v1", "algebra-symbolic-simplified@v1", + ) + a = composite.canonicalize(b"(x+1)**2") + b = composite.canonicalize(b"x**2 + 2*x + 1") + # Both should collapse to the same canonical bytes. + assert a == b + + +def test_composition_propagates_pi_star_error(): + """When the inner kernel raises PiStarError, the composite + surfaces it rather than swallowing or converting.""" + from arborist.pi_star import PiStarError, compose + composite = _safe_compose( + "algebra-symbolic@v1", "algebra-symbolic@v1", + ) + with pytest.raises(PiStarError): + composite.canonicalize(b"this is not !#$ a valid !#$ sympy expression") + + +def test_composition_is_listed_in_registry_after_compose(): + """``list_keys()`` shows composite kernels alongside the rest of + the registry once they've been compose()'d. Auto-registered + composites have keys with ``-then-`` infix.""" + from arborist.pi_star import compose, list_keys + compose("algebra-symbolic@v1", "algebra-symbolic-simplified@v1") + keys = list_keys() + assert any( + k.startswith("algebra-symbolic-then-") for k in keys + ) + + +def test_canonical_composition_id_is_stable(): + """The composition manifest fingerprint depends only on the + (inner, outer) keys; same pair → same id.""" + from arborist.pi_star import canonical_composition_id + a = canonical_composition_id( + "algebra-symbolic@v1", "algebra-symbolic-simplified@v1", + ) + b = canonical_composition_id( + "algebra-symbolic@v1", "algebra-symbolic-simplified@v1", + ) + assert a == b + assert len(a) == 64 # sha256 hex + + +def test_canonical_composition_id_distinguishes_order(): + """Order matters in composition: A∘B ≠ B∘A, so the manifest id + must distinguish them.""" + from arborist.pi_star import canonical_composition_id + forward = canonical_composition_id( + "algebra-symbolic@v1", "algebra-symbolic-simplified@v1", + ) + reversed_ = canonical_composition_id( + "algebra-symbolic-simplified@v1", "algebra-symbolic@v1", + ) + assert forward != reversed_ + + +def test_compose_unknown_inner_raises_keyerror(): + """compose() guards both ends — an unknown inner π* fails + cleanly with KeyError.""" + from arborist.pi_star import compose + with pytest.raises(KeyError): + compose("does-not-exist@v1", "algebra-symbolic@v1") + + +def test_compose_unknown_outer_raises_keyerror(): + from arborist.pi_star import compose + with pytest.raises(KeyError): + compose("algebra-symbolic@v1", "does-not-exist@v1") + + +def test_composite_domain_is_inner_domain(): + """Composition's domain is the inner π*'s domain (the composite + accepts what the inner accepts).""" + from arborist.pi_star import compose + composite = _safe_compose( + "algebra-symbolic@v1", "algebra-symbolic@v1", + ) + assert composite.domain == "symbolic-algebra" + + +def test_composition_bytes_match_manual_application(): + """A composite kernel must produce byte-equal output to a manual + inner.canonicalize() → outer.canonicalize() chain.""" + from arborist.pi_star import compose, get + inner = get("algebra-symbolic@v1") + outer = get("algebra-symbolic-simplified@v1") + composite = _safe_compose( + "algebra-symbolic@v1", "algebra-symbolic-simplified@v1", + ) + for expr in ( + b"sin(x)**2 + cos(x)**2", + b"(a + b)**2", + b"x**3 - x**3 + 2*x", + ): + manual = outer.canonicalize(inner.canonicalize(expr)) + assert composite.canonicalize(expr) == manual, ( + f"composite ≠ manual chain on {expr!r}" + )