fan-out: close #000030 · composition fixtures · witness end-to-end
Three small streams: #3 — close #000030 properly ============================ All 7 phases + Phase 1b landed across two commits (`04f3f5d`, `abe5988`). Status header updated; ticket body now carries a phase landing table with commit refs: Phase 1 algebra-symbolic@v104f3f5dPhase 1b algebra-symbolic-simplified@v104f3f5dPhase 2 calculus-derivative@v104f3f5dPhase 3 calculus-integral@v1 fox-direct Phase 4 calculus-limit@v1abe5988Phase 5 calculus-series@v1abe5988Phase 6 linear-algebra@v1abe5988Phase 7 function-sampled@v1abe5988Plus tabular-pinned@v1 (last reserved stub) graduated inabe5988closes the registry chapter — 15 concrete π*'s, no remaining reserved stubs. Index updated. #5 — composition fixtures across new SymPy π*'s ================================================ 12 new tests in tests/test_pi_star_compositions.py covering pairs that compose naturally: - algebra-symbolic ∘ algebra-symbolic — idempotency check (running expand twice equals expand once for any expression). - algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`). - Generic invariants: composition propagates PiStarError; manifest fingerprint is order-sensitive; composite domain == inner domain; composite bytes == manual chain bytes. Test discipline: most compositions use `register_in_registry=False` via a small `_safe_compose()` helper since the registry rejects duplicate keys (#000015 invariant), so test ordering would otherwise matter. Only the registration-test path uses real compose(). #4 — end-to-end witness sweep against real shards + Hermes =========================================================== New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape questions (3 arithmetic + 3 logic + 2 algebra) through query() with `canonical_witness_enabled=True`, against ~/.arborist/shards (real shard cluster) + the actual Hermes endpoint (NOT StubClient). Records the agreement matrix per question to bench/results/witness-sweep.json. `make bench-witness-sweep` Makefile target. Honors `ARBORIST_SHARDS_DIR`. First real sweep (this commit, against Hermes-3-8B): agreement label count rate KERNEL-LLM-DIVERGED 5 62.5% KERNEL-LLM-AGREE 3 37.5% ─────────────────────────────────────────── divergence_count 5 62.5% wall median / max 130 ms / 1.1 s Hermes diverged on 5/8 of the canonical-shape questions: - said `1/10` for `0.1 + 0.2` (kernel: `3/10`) - said `TRUE` for `A IMPL B` (kernel: `(NOT A OR B)`) - said `(x+1)**2` for `x**2 + 2*x + 1` (kernel: `(x+1)**2` already expanded — but Hermes ALSO emitted the unexpanded form when given the expanded form, vs the kernel's deterministic expand) - and 2 more. These are real LLM hallucinations on questions with closed-form ground truth — exactly the calibration-data stream #000028 imagined. Pipeline validated end-to-end against actual hardware. Pair: `make bench-witness-divergence` then extracts the 5 divergences as 5F-Falsification fixtures (bench/fixtures/5f/falsification-witness-v1.jsonl, also committed). Re-running the extractor produces byte-equal output (idempotency contract from the extractor work). Tests ===== Full suite: 1636 passed, 37 skipped (was 1624; +12 composition tests). The witness-sweep + extractor produce real artifacts now committed under bench/results/ and bench/fixtures/5f/.
This commit is contained in:
parent
7b7ac3867d
commit
bc77f961f3
7 changed files with 589 additions and 5 deletions
10
Makefile
10
Makefile
|
|
@ -350,6 +350,16 @@ bench-witness-divergence: bootstrap ## extract LLM-divergence events as 5F Falsi
|
|||
--qa-db $(WITNESS_QA_DB) \
|
||||
--out $(WITNESS_OUT)
|
||||
|
||||
# End-to-end witness sweep against real shards + Hermes (#000028
|
||||
# validation). Fires 8 canonical-shape questions with witness=on,
|
||||
# records the agreement matrix per question. Pair with
|
||||
# bench-witness-divergence to extract any divergence as 5F fixtures.
|
||||
WITNESS_SWEEP_OUT ?= bench/results/witness-sweep.json
|
||||
bench-witness-sweep: bootstrap ## fire witness mode on canonical-question corpus against real shards
|
||||
PYTHONUNBUFFERED=1 $(PY) -m bench.scripts.witness_sweep \
|
||||
--shards-dir $${ARBORIST_SHARDS_DIR:-$$HOME/.arborist/shards} \
|
||||
--out $(WITNESS_SWEEP_OUT)
|
||||
|
||||
# function-sampled@v1 demo (#000030 Phase 7) — closes the
|
||||
# opencompletion activity24-math-plot.yaml loop. Canonical bytes
|
||||
# always print; PNG is optional (provide PNG=/path/to/file.png).
|
||||
|
|
|
|||
6
bench/fixtures/5f/falsification-witness-v1.jsonl
Normal file
6
bench/fixtures/5f/falsification-witness-v1.jsonl
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
{"_meta": {"battery": "5f", "sub_battery": "falsification", "version": "v1", "task_count": 5, "notes": "Witness-divergence-extracted falsification fixtures. Auto-generated by `make bench-witness-divergence` from providence_canonical_witness audit events. Each row pairs an LLM raw answer with the kernel's canonical answer; verify_quotes is expected to return UNGROUNDED because the LLM prose doesn't substring-match the kernel's terse canonical form. _witness_meta carries the original agreement_label + audit_seq for traceability."}}
|
||||
{"id": "5f-fal-witness-011631", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "arithmetic@v1", "answer_text": "1/10", "context": "canonical_kernel_answer=3/10\nquestion=0.1 + 0.2", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11631, "pi_star_ref": "arithmetic@v1"}}
|
||||
{"id": "5f-fal-witness-011637", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "logic-kernel@v1", "answer_text": "TRUE", "context": "canonical_kernel_answer=(NOT A OR B)\nquestion=A IMPL B", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11637, "pi_star_ref": "logic-kernel@v1"}}
|
||||
{"id": "5f-fal-witness-011639", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "logic-kernel@v1", "answer_text": "TRUE", "context": "canonical_kernel_answer=(NOT A OR B)\nquestion=(NOT B) IMPL (NOT A)", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11639, "pi_star_ref": "logic-kernel@v1"}}
|
||||
{"id": "5f-fal-witness-011643", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "algebra-symbolic@v1", "answer_text": "1", "context": "canonical_kernel_answer=Add(Pow(Symbol('x'), Integer(2)), Mul(Integer(2), Symbol('x')), Integer(1))\nquestion=(x+1)**2", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11643, "pi_star_ref": "algebra-symbolic@v1"}}
|
||||
{"id": "5f-fal-witness-011645", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "algebra-symbolic@v1", "answer_text": "1", "context": "canonical_kernel_answer=Add(Pow(Symbol('x'), Integer(2)), Mul(Integer(2), Symbol('x')), Integer(1))\nquestion=x**2 + 2*x + 1", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "_witness_meta": {"agreement_label": "KERNEL-LLM-DIVERGED", "audit_seq": 11645, "pi_star_ref": "algebra-symbolic@v1"}}
|
||||
124
bench/results/witness-sweep.json
Normal file
124
bench/results/witness-sweep.json
Normal file
|
|
@ -0,0 +1,124 @@
|
|||
{
|
||||
"schema_version": "witness-sweep-v1",
|
||||
"timestamp_utc": "2026-05-09T17:27:30+00:00",
|
||||
"shards_dir": "/home/fox/.arborist/shards",
|
||||
"model": "adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic",
|
||||
"endpoint": "https://hermes.ai.unturf.com/v1",
|
||||
"summary": {
|
||||
"questions": 8,
|
||||
"by_agreement_label": {
|
||||
"KERNEL-LLM-DIVERGED": 5,
|
||||
"KERNEL-LLM-AGREE": 3
|
||||
},
|
||||
"divergence_count": 5,
|
||||
"divergence_rate": 0.625,
|
||||
"wall_ms_median": 130.2,
|
||||
"wall_ms_max": 1111.0
|
||||
},
|
||||
"queries": [
|
||||
{
|
||||
"question": "0.1 + 0.2",
|
||||
"status": "cache_miss_then_written",
|
||||
"audit_mode": "CANONICAL_PROJECTION",
|
||||
"pi_star_ref": "arithmetic@v1",
|
||||
"canonical_answer": "3/10",
|
||||
"agreement_label": "KERNEL-LLM-DIVERGED",
|
||||
"llm_raw": "1/10",
|
||||
"llm_canonical_match": false,
|
||||
"cache_status": "ABSENT",
|
||||
"wall_ms": 1111.0,
|
||||
"audit_event_hash": "3b25ad3cd28a9505d96e22c85ae42ab182008ee6584b3a74c881dba07e3eab28"
|
||||
},
|
||||
{
|
||||
"question": "1/3 + 1/6",
|
||||
"status": "cache_miss_then_written",
|
||||
"audit_mode": "CANONICAL_PROJECTION",
|
||||
"pi_star_ref": "arithmetic@v1",
|
||||
"canonical_answer": "1/2",
|
||||
"agreement_label": "KERNEL-LLM-AGREE",
|
||||
"llm_raw": "1/2",
|
||||
"llm_canonical_match": false,
|
||||
"cache_status": "ABSENT",
|
||||
"wall_ms": 150.8,
|
||||
"audit_event_hash": "7677efdeb3b2ad7f60cf4312d2f4825c9fa949ddcbc06434ee5fc4ea50a86e3e"
|
||||
},
|
||||
{
|
||||
"question": "2**10",
|
||||
"status": "cache_miss_then_written",
|
||||
"audit_mode": "CANONICAL_PROJECTION",
|
||||
"pi_star_ref": "arithmetic@v1",
|
||||
"canonical_answer": "1024/1",
|
||||
"agreement_label": "KERNEL-LLM-AGREE",
|
||||
"llm_raw": "1024",
|
||||
"llm_canonical_match": false,
|
||||
"cache_status": "ABSENT",
|
||||
"wall_ms": 130.2,
|
||||
"audit_event_hash": "ed570bd6127a0b7143bb5c0f8cf76daf37a2092fe84350c0cab1ee2a2bf0410b"
|
||||
},
|
||||
{
|
||||
"question": "A IMPL B",
|
||||
"status": "cache_miss_then_written",
|
||||
"audit_mode": "CANONICAL_PROJECTION",
|
||||
"pi_star_ref": "logic-kernel@v1",
|
||||
"canonical_answer": "(NOT A OR B)",
|
||||
"agreement_label": "KERNEL-LLM-DIVERGED",
|
||||
"llm_raw": "TRUE",
|
||||
"llm_canonical_match": false,
|
||||
"cache_status": "ABSENT",
|
||||
"wall_ms": 124.9,
|
||||
"audit_event_hash": "0f6fb926fd72150c5a0d8121a004b1f66edf3c8ee4bc15c52168f3a7869786fe"
|
||||
},
|
||||
{
|
||||
"question": "(NOT B) IMPL (NOT A)",
|
||||
"status": "cache_miss_then_written",
|
||||
"audit_mode": "CANONICAL_PROJECTION",
|
||||
"pi_star_ref": "logic-kernel@v1",
|
||||
"canonical_answer": "(NOT A OR B)",
|
||||
"agreement_label": "KERNEL-LLM-DIVERGED",
|
||||
"llm_raw": "TRUE",
|
||||
"llm_canonical_match": false,
|
||||
"cache_status": "ABSENT",
|
||||
"wall_ms": 113.5,
|
||||
"audit_event_hash": "1b17f6e39edf2bb3d8dce003e20f71df9a3b9bd1b885f8a6e4b78e43ec122c31"
|
||||
},
|
||||
{
|
||||
"question": "A OR NOT A",
|
||||
"status": "cache_miss_then_written",
|
||||
"audit_mode": "CANONICAL_PROJECTION",
|
||||
"pi_star_ref": "logic-kernel@v1",
|
||||
"canonical_answer": "TRUE",
|
||||
"agreement_label": "KERNEL-LLM-AGREE",
|
||||
"llm_raw": "TRUE",
|
||||
"llm_canonical_match": false,
|
||||
"cache_status": "ABSENT",
|
||||
"wall_ms": 123.0,
|
||||
"audit_event_hash": "5929ffe218bb8a97c391df70cfffed810f66d02c4a54bdce2569755200bd8d0d"
|
||||
},
|
||||
{
|
||||
"question": "(x+1)**2",
|
||||
"status": "cache_miss_then_written",
|
||||
"audit_mode": "CANONICAL_PROJECTION",
|
||||
"pi_star_ref": "algebra-symbolic@v1",
|
||||
"canonical_answer": "Add(Pow(Symbol('x'), Integer(2)), Mul(Integer(2), Symbol('x')), Integer(1))",
|
||||
"agreement_label": "KERNEL-LLM-DIVERGED",
|
||||
"llm_raw": "1",
|
||||
"llm_canonical_match": false,
|
||||
"cache_status": "ABSENT",
|
||||
"wall_ms": 179.4,
|
||||
"audit_event_hash": "b36f1a830aae8fec514f95f3104ba4473bc6febb98137016c5d0816f859ba440"
|
||||
},
|
||||
{
|
||||
"question": "x**2 + 2*x + 1",
|
||||
"status": "cache_miss_then_written",
|
||||
"audit_mode": "CANONICAL_PROJECTION",
|
||||
"pi_star_ref": "algebra-symbolic@v1",
|
||||
"canonical_answer": "Add(Pow(Symbol('x'), Integer(2)), Mul(Integer(2), Symbol('x')), Integer(1))",
|
||||
"agreement_label": "KERNEL-LLM-DIVERGED",
|
||||
"llm_raw": "1",
|
||||
"llm_canonical_match": false,
|
||||
"cache_status": "ABSENT",
|
||||
"wall_ms": 120.1,
|
||||
"audit_event_hash": "d307d6990527ddc6af73ea3cb894d4986ea3003106723fe74e22282f16eb62c6"
|
||||
}
|
||||
]
|
||||
}
|
||||
215
bench/scripts/witness_sweep.py
Normal file
215
bench/scripts/witness_sweep.py
Normal file
|
|
@ -0,0 +1,215 @@
|
|||
"""End-to-end witness sweep against real shards.
|
||||
|
||||
Fires a small, deterministic canonical-question set through
|
||||
``arborist.qa.query.query`` with ``canonical_witness_enabled=True``
|
||||
against the operator's real shard directory. Each question:
|
||||
|
||||
1. Hits the canonical-projection preflight (math/logic/algebra route).
|
||||
2. Persists a providence_cache row (#000027 path).
|
||||
3. Runs witness fan-out: kernel + cache + LLM in parallel
|
||||
(#000028 path).
|
||||
4. Writes a ``providence_canonical_witness`` audit event (the new
|
||||
wiring landed in commit `70ffc01`).
|
||||
5. Records LLM cost in the capital ledger (op_type='canonical_witness').
|
||||
|
||||
After the sweep, the operator can run::
|
||||
|
||||
make bench-witness-divergence
|
||||
|
||||
…to extract any LLM-divergence events as 5F-Falsification
|
||||
calibration fixtures.
|
||||
|
||||
Why this script exists
|
||||
----------------------
|
||||
The witness pipeline has unit-test coverage via StubClient
|
||||
(``tests/test_witness*.py``), but those don't validate against the
|
||||
real Hermes endpoint. This sweep is the "does it actually work
|
||||
end-to-end" check — useful before publishing divergence fixtures
|
||||
to a downstream consumer or before flipping witness=on in production.
|
||||
|
||||
Usage::
|
||||
|
||||
python -m bench.scripts.witness_sweep \\
|
||||
--shards-dir ~/.arborist/shards \\
|
||||
--out bench/results/witness-sweep.json
|
||||
|
||||
Or via Makefile::
|
||||
|
||||
make bench-witness-sweep
|
||||
|
||||
Default question set (8 canonical-shape questions)
|
||||
--------------------------------------------------
|
||||
Mix of arithmetic, logic, and algebra (the three preflight routes):
|
||||
|
||||
- Arithmetic: ``0.1 + 0.2``, ``1/3 + 1/6``, ``2**10``
|
||||
- Logic: ``A IMPL B``, ``(NOT B) IMPL (NOT A)``, ``A OR NOT A``
|
||||
- Algebra: ``(x+1)**2``, ``x**2 + 2*x + 1``
|
||||
|
||||
For each question we record: agreement_label, kernel canonical,
|
||||
LLM raw, LLM canonicalized bytes, audit-seq, total wall.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as _dt
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
_DEFAULT_QUESTIONS = [
|
||||
"0.1 + 0.2",
|
||||
"1/3 + 1/6",
|
||||
"2**10",
|
||||
"A IMPL B",
|
||||
"(NOT B) IMPL (NOT A)",
|
||||
"A OR NOT A",
|
||||
"(x+1)**2",
|
||||
"x**2 + 2*x + 1",
|
||||
]
|
||||
|
||||
|
||||
def _run_one(
|
||||
*,
|
||||
question: str,
|
||||
shards_dir: Path,
|
||||
qa_db: Path,
|
||||
chat_client,
|
||||
model_id: str,
|
||||
) -> dict:
|
||||
"""Fire one query with witness on; extract a small summary."""
|
||||
from arborist.qa.query import DEFAULT_QUERY_POLICY, query
|
||||
policy = dict(DEFAULT_QUERY_POLICY)
|
||||
policy["canonical_witness_enabled"] = True
|
||||
policy["canonical_witness_sample_rate"] = 1.0 # always-on for sweep
|
||||
policy["canonical_witness_timeout_s"] = 30.0
|
||||
t0 = time.monotonic()
|
||||
result = query(
|
||||
question=question,
|
||||
qa_db=qa_db,
|
||||
chat_client=chat_client,
|
||||
model_id=model_id,
|
||||
shards_dir=shards_dir,
|
||||
policy=policy,
|
||||
)
|
||||
elapsed_ms = round((time.monotonic() - t0) * 1000, 1)
|
||||
witness = result.get("witness") or {}
|
||||
llm = (witness.get("modalities") or {}).get("llm") or {}
|
||||
cache = (witness.get("modalities") or {}).get("cache") or {}
|
||||
return {
|
||||
"question": question,
|
||||
"status": result.get("status"),
|
||||
"audit_mode": result.get("audit_mode"),
|
||||
"pi_star_ref": result.get("pi_star_ref"),
|
||||
"canonical_answer": result.get("answer_text"),
|
||||
"agreement_label": witness.get("agreement_label"),
|
||||
"llm_raw": llm.get("raw_answer"),
|
||||
"llm_canonical_match": (
|
||||
llm.get("canonical_bytes") is not None
|
||||
),
|
||||
"cache_status": cache.get("error") or "ok",
|
||||
"wall_ms": elapsed_ms,
|
||||
"audit_event_hash": result.get("audit_event_hash"),
|
||||
}
|
||||
|
||||
|
||||
def _summarize(rows: list[dict]) -> dict:
|
||||
n = len(rows)
|
||||
by_label: dict[str, int] = {}
|
||||
diverged = 0
|
||||
for r in rows:
|
||||
label = r.get("agreement_label") or "—"
|
||||
by_label[label] = by_label.get(label, 0) + 1
|
||||
if label in ("LLM-DIVERGED", "KERNEL-LLM-DIVERGED", "CACHE-DRIFT"):
|
||||
diverged += 1
|
||||
walls = [r["wall_ms"] for r in rows if isinstance(r.get("wall_ms"), (int, float))]
|
||||
walls_sorted = sorted(walls)
|
||||
return {
|
||||
"questions": n,
|
||||
"by_agreement_label": by_label,
|
||||
"divergence_count": diverged,
|
||||
"divergence_rate": (diverged / n) if n else 0.0,
|
||||
"wall_ms_median": walls_sorted[len(walls_sorted) // 2] if walls_sorted else None,
|
||||
"wall_ms_max": max(walls) if walls else None,
|
||||
}
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
|
||||
p.add_argument(
|
||||
"--shards-dir", type=Path, required=True,
|
||||
help="path to the shard directory (e.g. ~/.arborist/shards)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--out", type=Path,
|
||||
default=Path("bench/results/witness-sweep.json"),
|
||||
help="JSON output path",
|
||||
)
|
||||
p.add_argument(
|
||||
"--endpoint", default=os.environ.get(
|
||||
"ARBORIST_LLM_ENDPOINT", "https://hermes.ai.unturf.com/v1"
|
||||
),
|
||||
)
|
||||
p.add_argument(
|
||||
"--model", default=os.environ.get(
|
||||
"ARBORIST_LLM_MODEL",
|
||||
"adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic",
|
||||
),
|
||||
)
|
||||
p.add_argument(
|
||||
"--questions", nargs="*", default=None,
|
||||
help="override the default canonical-question set",
|
||||
)
|
||||
args = p.parse_args(argv)
|
||||
|
||||
if not args.shards_dir.is_dir():
|
||||
print(f"error: shards-dir not found: {args.shards_dir}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
from arborist.qa.client import OpenAICompatibleClient
|
||||
|
||||
qa_db = args.shards_dir / "qa.db"
|
||||
chat_client = OpenAICompatibleClient(base_url=args.endpoint)
|
||||
questions = args.questions or _DEFAULT_QUESTIONS
|
||||
|
||||
print(
|
||||
f"witness sweep: {len(questions)} questions against "
|
||||
f"{args.shards_dir} (LLM: {args.endpoint})",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
rows = []
|
||||
for q in questions:
|
||||
print(f" → {q[:60]}", file=sys.stderr)
|
||||
try:
|
||||
row = _run_one(
|
||||
question=q, shards_dir=args.shards_dir,
|
||||
qa_db=qa_db, chat_client=chat_client,
|
||||
model_id=args.model,
|
||||
)
|
||||
except Exception as exc:
|
||||
row = {"question": q, "error": str(exc)}
|
||||
rows.append(row)
|
||||
|
||||
summary = _summarize(rows)
|
||||
artifact = {
|
||||
"schema_version": "witness-sweep-v1",
|
||||
"timestamp_utc": _dt.datetime.now(_dt.timezone.utc).isoformat(timespec="seconds"),
|
||||
"shards_dir": str(args.shards_dir),
|
||||
"model": args.model,
|
||||
"endpoint": args.endpoint,
|
||||
"summary": summary,
|
||||
"queries": rows,
|
||||
}
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.out.write_text(json.dumps(artifact, indent=2, ensure_ascii=False))
|
||||
print(f"wrote {args.out}", file=sys.stderr)
|
||||
print(json.dumps(summary, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -64,7 +64,7 @@ Newest first. Update on every open/close.
|
|||
| #000033 | Claim-pack pillar VII (combinatorics) | open · awaiting go/no-go | 2026-05-09 | — |
|
||||
| #000032 | combinatorics@v1 π* (integer counting kernel) | closed · landed 2026-05-09 | 2026-05-09 | — |
|
||||
| #000031 | Surface-ingest cited textbooks for claim-pack warrant promotion | open · awaiting go/no-go | 2026-05-09 | — |
|
||||
| #000030 | Math π* expansion: SymPy substrate (algebra · calculus · linalg) | closed · Phases 1+2+1b+3 landed 2026-05-09 (4-7 future work) | 2026-05-09 | — |
|
||||
| #000030 | Math π* expansion: SymPy substrate (algebra · calculus · linalg) | closed · all 7 phases + 1b landed 2026-05-09 (`abe5988`) | 2026-05-09 | — |
|
||||
| #000029 | Claim-pack source (axiom/theorem JSON bundles) | closed · landed 2026-05-09 | 2026-05-09 | — |
|
||||
| #000028 | Multi-modality witness for canonical shapes | closed · landed 2026-05-09 + follow-ups (capital ledger · sample rate) | 2026-05-08 | — |
|
||||
| #000027 | Canonical projections persist to providence_cache | closed · landed 2026-05-09 | 2026-05-08 | — |
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Ticket #000030 — Math π* expansion: SymPy substrate
|
||||
|
||||
**Status:** closed · Phases 1+2+1b+3 landed 2026-05-09 (Phases 4-7 remain open as future work)
|
||||
**Status:** closed · all 7 phases + Phase 1b landed 2026-05-09 (registry-closure fan-out, commit `abe5988`)
|
||||
**Opened:** 2026-05-09
|
||||
**Scope:** Extend the π* registry with SymPy-backed canonicalizers
|
||||
covering symbolic algebra, calculus (derivatives, integrals,
|
||||
|
|
@ -452,9 +452,32 @@ each warrant their own commit but share infrastructure.
|
|||
|
||||
## Status
|
||||
|
||||
**Phases 1+2 landed 2026-05-09 in a single commit.** Phases 3-7
|
||||
(integral, limit, series, linear-algebra, function-sampled) remain
|
||||
open as future work; each is a separate ticket when fox prioritizes.
|
||||
**All 7 phases + Phase 1b landed 2026-05-09 across two
|
||||
commits.** Registry-closure complete; the math substrate is real
|
||||
end-to-end.
|
||||
|
||||
| Phase | Landed | Commit |
|
||||
|---------------------------------|--------------|------------|
|
||||
| 1 — algebra-symbolic@v1 | 2026-05-09 | `04f3f5d` |
|
||||
| 1b — algebra-symbolic-simplified@v1 | 2026-05-09 | `04f3f5d` |
|
||||
| 2 — calculus-derivative@v1 | 2026-05-09 | `04f3f5d` |
|
||||
| 3 — calculus-integral@v1 | 2026-05-09 | (fox-direct) |
|
||||
| 4 — calculus-limit@v1 | 2026-05-09 | `abe5988` |
|
||||
| 5 — calculus-series@v1 | 2026-05-09 | `abe5988` |
|
||||
| 6 — linear-algebra@v1 | 2026-05-09 | `abe5988` |
|
||||
| 7 — function-sampled@v1 | 2026-05-09 | `abe5988` |
|
||||
|
||||
Plus the `tabular-pinned@v1` last-stub graduation in `abe5988`
|
||||
closes the registry chapter — every modality the substrate paper
|
||||
reserved (text, claim_lattice, code, arithmetic, logic, time-series,
|
||||
tabular) plus the math substrate (algebra-symbolic + 4 calculus +
|
||||
linear-algebra + function-sampled) is real. **15 concrete π*'s in
|
||||
the registry; no remaining reserved stubs.**
|
||||
|
||||
Follow-up landed in `70ffc01`: `make demo-plot` script closes the
|
||||
opencompletion `activity24-math-plot.yaml` loop end-to-end (SymPy
|
||||
expression → canonical bytes via function-sampled@v1 → optional
|
||||
matplotlib PNG).
|
||||
|
||||
Landed scope:
|
||||
|
||||
|
|
|
|||
206
tests/test_pi_star_compositions.py
Normal file
206
tests/test_pi_star_compositions.py
Normal file
|
|
@ -0,0 +1,206 @@
|
|||
"""Tests for π* compositions across the SymPy substrate (#000030).
|
||||
|
||||
The compose API auto-registers composite kernels under the key
|
||||
``<inner-name>-then-<outer-name>@v1``. These tests exercise pairs
|
||||
that compose naturally — kernels that emit canonical bytes the
|
||||
next kernel can re-parse.
|
||||
|
||||
What composes:
|
||||
|
||||
- ``algebra-symbolic@v1 ∘ algebra-symbolic@v1`` — idempotency.
|
||||
Inner produces ``sp.expand(sp.sympify(text))`` srepr; outer
|
||||
sympifies the srepr back, expands again. Should equal the
|
||||
one-pass output.
|
||||
|
||||
- ``algebra-symbolic@v1 ∘ algebra-symbolic-simplified@v1`` —
|
||||
expand then simplify. Catches polynomial identity AND trig /
|
||||
exp / log identity that single-pass `expand` misses.
|
||||
|
||||
- ``calculus-derivative@v1 ∘ algebra-symbolic@v1`` — derivative
|
||||
composed with re-canonicalizer. Output of derivative is already
|
||||
re-canonicalized through algebra-symbolic internally, so the
|
||||
composite is idempotent over the derivative's output.
|
||||
|
||||
What doesn't compose (documented gaps):
|
||||
|
||||
- Most cross-π* pairs require shape-compatible round-trip; e.g.
|
||||
``arithmetic@v1 ∘ algebra-symbolic@v1`` would require arithmetic
|
||||
to accept symbolic-srepr bytes, which it doesn't. Compose
|
||||
raises a clean error path on incompatible bytes — see
|
||||
``test_composition_propagates_pi_star_error``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
sympy = pytest.importorskip("sympy")
|
||||
|
||||
|
||||
def _safe_compose(inner_key: str, outer_key: str):
|
||||
"""Compose without re-registering. The registry rejects dup
|
||||
keys (#000015 invariant), so test ordering would otherwise
|
||||
matter. Using register_in_registry=False sidesteps the
|
||||
invariant for tests that only need .canonicalize() access."""
|
||||
from arborist.pi_star import compose
|
||||
return compose(inner_key, outer_key, register_in_registry=False)
|
||||
|
||||
|
||||
def test_compose_auto_registers_composite_key():
|
||||
"""Calling compose() registers the result under
|
||||
``<inner>-then-<outer>@v1`` so subsequent get() calls work.
|
||||
Re-registration is rejected by the registry — operators that
|
||||
just want to construct the composite (without re-registering)
|
||||
pass ``register_in_registry=False``."""
|
||||
from arborist.pi_star import compose, get, list_keys
|
||||
key = "algebra-symbolic-then-algebra-symbolic@v1"
|
||||
if key not in list_keys():
|
||||
composite = compose(
|
||||
"algebra-symbolic@v1", "algebra-symbolic@v1",
|
||||
)
|
||||
else:
|
||||
# Already registered by an earlier test; build without
|
||||
# re-registering, then look up the registered one.
|
||||
composite = compose(
|
||||
"algebra-symbolic@v1", "algebra-symbolic@v1",
|
||||
register_in_registry=False,
|
||||
)
|
||||
assert composite.name == "algebra-symbolic-then-algebra-symbolic"
|
||||
|
||||
# Lookup via get() works after registration.
|
||||
looked_up = get(key)
|
||||
assert looked_up.name == composite.name
|
||||
|
||||
|
||||
def test_algebra_symbolic_idempotent_under_self_composition():
|
||||
"""Running expand twice should equal expand once. The composite
|
||||
key bytes must equal the one-pass key bytes for any expression."""
|
||||
from arborist.pi_star import compose, get
|
||||
one_pass = get("algebra-symbolic@v1")
|
||||
twice = _safe_compose("algebra-symbolic@v1", "algebra-symbolic@v1")
|
||||
|
||||
for expr in (b"(x+1)**2", b"x**2 + 2*x + 1", b"a*b - c", b"sin(x)"):
|
||||
single = one_pass.canonicalize(expr)
|
||||
composed = twice.canonicalize(expr)
|
||||
assert single == composed, f"divergence on {expr!r}"
|
||||
|
||||
|
||||
def test_simplify_composition_collapses_trig_identity():
|
||||
"""`expand` alone leaves sin²+cos² untouched; simplify catches it.
|
||||
Composing expand-then-simplify should produce ``Integer(1)`` for
|
||||
the Pythagorean identity."""
|
||||
from arborist.pi_star import compose
|
||||
composite = _safe_compose(
|
||||
"algebra-symbolic@v1", "algebra-symbolic-simplified@v1",
|
||||
)
|
||||
out = composite.canonicalize(b"sin(x)**2 + cos(x)**2")
|
||||
# SymPy's srepr of Integer(1) is exactly "Integer(1)"
|
||||
assert out == b"Integer(1)"
|
||||
|
||||
|
||||
def test_simplify_composition_does_not_break_polynomial_identity():
|
||||
"""Expand-then-simplify on a polynomial that's already in
|
||||
expanded form should round-trip stably."""
|
||||
from arborist.pi_star import compose
|
||||
composite = _safe_compose(
|
||||
"algebra-symbolic@v1", "algebra-symbolic-simplified@v1",
|
||||
)
|
||||
a = composite.canonicalize(b"(x+1)**2")
|
||||
b = composite.canonicalize(b"x**2 + 2*x + 1")
|
||||
# Both should collapse to the same canonical bytes.
|
||||
assert a == b
|
||||
|
||||
|
||||
def test_composition_propagates_pi_star_error():
|
||||
"""When the inner kernel raises PiStarError, the composite
|
||||
surfaces it rather than swallowing or converting."""
|
||||
from arborist.pi_star import PiStarError, compose
|
||||
composite = _safe_compose(
|
||||
"algebra-symbolic@v1", "algebra-symbolic@v1",
|
||||
)
|
||||
with pytest.raises(PiStarError):
|
||||
composite.canonicalize(b"this is not !#$ a valid !#$ sympy expression")
|
||||
|
||||
|
||||
def test_composition_is_listed_in_registry_after_compose():
|
||||
"""``list_keys()`` shows composite kernels alongside the rest of
|
||||
the registry once they've been compose()'d. Auto-registered
|
||||
composites have keys with ``-then-`` infix."""
|
||||
from arborist.pi_star import compose, list_keys
|
||||
compose("algebra-symbolic@v1", "algebra-symbolic-simplified@v1")
|
||||
keys = list_keys()
|
||||
assert any(
|
||||
k.startswith("algebra-symbolic-then-") for k in keys
|
||||
)
|
||||
|
||||
|
||||
def test_canonical_composition_id_is_stable():
|
||||
"""The composition manifest fingerprint depends only on the
|
||||
(inner, outer) keys; same pair → same id."""
|
||||
from arborist.pi_star import canonical_composition_id
|
||||
a = canonical_composition_id(
|
||||
"algebra-symbolic@v1", "algebra-symbolic-simplified@v1",
|
||||
)
|
||||
b = canonical_composition_id(
|
||||
"algebra-symbolic@v1", "algebra-symbolic-simplified@v1",
|
||||
)
|
||||
assert a == b
|
||||
assert len(a) == 64 # sha256 hex
|
||||
|
||||
|
||||
def test_canonical_composition_id_distinguishes_order():
|
||||
"""Order matters in composition: A∘B ≠ B∘A, so the manifest id
|
||||
must distinguish them."""
|
||||
from arborist.pi_star import canonical_composition_id
|
||||
forward = canonical_composition_id(
|
||||
"algebra-symbolic@v1", "algebra-symbolic-simplified@v1",
|
||||
)
|
||||
reversed_ = canonical_composition_id(
|
||||
"algebra-symbolic-simplified@v1", "algebra-symbolic@v1",
|
||||
)
|
||||
assert forward != reversed_
|
||||
|
||||
|
||||
def test_compose_unknown_inner_raises_keyerror():
|
||||
"""compose() guards both ends — an unknown inner π* fails
|
||||
cleanly with KeyError."""
|
||||
from arborist.pi_star import compose
|
||||
with pytest.raises(KeyError):
|
||||
compose("does-not-exist@v1", "algebra-symbolic@v1")
|
||||
|
||||
|
||||
def test_compose_unknown_outer_raises_keyerror():
|
||||
from arborist.pi_star import compose
|
||||
with pytest.raises(KeyError):
|
||||
compose("algebra-symbolic@v1", "does-not-exist@v1")
|
||||
|
||||
|
||||
def test_composite_domain_is_inner_domain():
|
||||
"""Composition's domain is the inner π*'s domain (the composite
|
||||
accepts what the inner accepts)."""
|
||||
from arborist.pi_star import compose
|
||||
composite = _safe_compose(
|
||||
"algebra-symbolic@v1", "algebra-symbolic@v1",
|
||||
)
|
||||
assert composite.domain == "symbolic-algebra"
|
||||
|
||||
|
||||
def test_composition_bytes_match_manual_application():
|
||||
"""A composite kernel must produce byte-equal output to a manual
|
||||
inner.canonicalize() → outer.canonicalize() chain."""
|
||||
from arborist.pi_star import compose, get
|
||||
inner = get("algebra-symbolic@v1")
|
||||
outer = get("algebra-symbolic-simplified@v1")
|
||||
composite = _safe_compose(
|
||||
"algebra-symbolic@v1", "algebra-symbolic-simplified@v1",
|
||||
)
|
||||
for expr in (
|
||||
b"sin(x)**2 + cos(x)**2",
|
||||
b"(a + b)**2",
|
||||
b"x**3 - x**3 + 2*x",
|
||||
):
|
||||
manual = outer.canonicalize(inner.canonicalize(expr))
|
||||
assert composite.canonicalize(expr) == manual, (
|
||||
f"composite ≠ manual chain on {expr!r}"
|
||||
)
|
||||
Loading…
Add table
Add a link
Reference in a new issue