russell@unturf.com
bc77f961f3
Three small streams: #3 — close #000030 properly ============================ All 7 phases + Phase 1b landed across two commits (`04f3f5d`, `abe5988`). Status header updated; ticket body now carries a phase landing table with commit refs: Phase 1 algebra-symbolic@v104f3f5dPhase 1b algebra-symbolic-simplified@v104f3f5dPhase 2 calculus-derivative@v104f3f5dPhase 3 calculus-integral@v1 fox-direct Phase 4 calculus-limit@v1abe5988Phase 5 calculus-series@v1abe5988Phase 6 linear-algebra@v1abe5988Phase 7 function-sampled@v1abe5988Plus tabular-pinned@v1 (last reserved stub) graduated inabe5988closes the registry chapter — 15 concrete π*'s, no remaining reserved stubs. Index updated. #5 — composition fixtures across new SymPy π*'s ================================================ 12 new tests in tests/test_pi_star_compositions.py covering pairs that compose naturally: - algebra-symbolic ∘ algebra-symbolic — idempotency check (running expand twice equals expand once for any expression). - algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`). - Generic invariants: composition propagates PiStarError; manifest fingerprint is order-sensitive; composite domain == inner domain; composite bytes == manual chain bytes. Test discipline: most compositions use `register_in_registry=False` via a small `_safe_compose()` helper since the registry rejects duplicate keys (#000015 invariant), so test ordering would otherwise matter. Only the registration-test path uses real compose(). #4 — end-to-end witness sweep against real shards + Hermes =========================================================== New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape questions (3 arithmetic + 3 logic + 2 algebra) through query() with `canonical_witness_enabled=True`, against ~/.arborist/shards (real shard cluster) + the actual Hermes endpoint (NOT StubClient). Records the agreement matrix per question to bench/results/witness-sweep.json. `make bench-witness-sweep` Makefile target. Honors `ARBORIST_SHARDS_DIR`. First real sweep (this commit, against Hermes-3-8B): agreement label count rate KERNEL-LLM-DIVERGED 5 62.5% KERNEL-LLM-AGREE 3 37.5% ─────────────────────────────────────────── divergence_count 5 62.5% wall median / max 130 ms / 1.1 s Hermes diverged on 5/8 of the canonical-shape questions: - said `1/10` for `0.1 + 0.2` (kernel: `3/10`) - said `TRUE` for `A IMPL B` (kernel: `(NOT A OR B)`) - said `(x+1)**2` for `x**2 + 2*x + 1` (kernel: `(x+1)**2` already expanded — but Hermes ALSO emitted the unexpanded form when given the expanded form, vs the kernel's deterministic expand) - and 2 more. These are real LLM hallucinations on questions with closed-form ground truth — exactly the calibration-data stream #000028 imagined. Pipeline validated end-to-end against actual hardware. Pair: `make bench-witness-divergence` then extracts the 5 divergences as 5F-Falsification fixtures (bench/fixtures/5f/falsification-witness-v1.jsonl, also committed). Re-running the extractor produces byte-equal output (idempotency contract from the extractor work). Tests ===== Full suite: 1636 passed, 37 skipped (was 1624; +12 composition tests). The witness-sweep + extractor produce real artifacts now committed under bench/results/ and bench/fixtures/5f/.
124 lines
No EOL
4.3 KiB
JSON
124 lines
No EOL
4.3 KiB
JSON
{
|
|
"schema_version": "witness-sweep-v1",
|
|
"timestamp_utc": "2026-05-09T17:27:30+00:00",
|
|
"shards_dir": "/home/fox/.arborist/shards",
|
|
"model": "adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic",
|
|
"endpoint": "https://hermes.ai.unturf.com/v1",
|
|
"summary": {
|
|
"questions": 8,
|
|
"by_agreement_label": {
|
|
"KERNEL-LLM-DIVERGED": 5,
|
|
"KERNEL-LLM-AGREE": 3
|
|
},
|
|
"divergence_count": 5,
|
|
"divergence_rate": 0.625,
|
|
"wall_ms_median": 130.2,
|
|
"wall_ms_max": 1111.0
|
|
},
|
|
"queries": [
|
|
{
|
|
"question": "0.1 + 0.2",
|
|
"status": "cache_miss_then_written",
|
|
"audit_mode": "CANONICAL_PROJECTION",
|
|
"pi_star_ref": "arithmetic@v1",
|
|
"canonical_answer": "3/10",
|
|
"agreement_label": "KERNEL-LLM-DIVERGED",
|
|
"llm_raw": "1/10",
|
|
"llm_canonical_match": false,
|
|
"cache_status": "ABSENT",
|
|
"wall_ms": 1111.0,
|
|
"audit_event_hash": "3b25ad3cd28a9505d96e22c85ae42ab182008ee6584b3a74c881dba07e3eab28"
|
|
},
|
|
{
|
|
"question": "1/3 + 1/6",
|
|
"status": "cache_miss_then_written",
|
|
"audit_mode": "CANONICAL_PROJECTION",
|
|
"pi_star_ref": "arithmetic@v1",
|
|
"canonical_answer": "1/2",
|
|
"agreement_label": "KERNEL-LLM-AGREE",
|
|
"llm_raw": "1/2",
|
|
"llm_canonical_match": false,
|
|
"cache_status": "ABSENT",
|
|
"wall_ms": 150.8,
|
|
"audit_event_hash": "7677efdeb3b2ad7f60cf4312d2f4825c9fa949ddcbc06434ee5fc4ea50a86e3e"
|
|
},
|
|
{
|
|
"question": "2**10",
|
|
"status": "cache_miss_then_written",
|
|
"audit_mode": "CANONICAL_PROJECTION",
|
|
"pi_star_ref": "arithmetic@v1",
|
|
"canonical_answer": "1024/1",
|
|
"agreement_label": "KERNEL-LLM-AGREE",
|
|
"llm_raw": "1024",
|
|
"llm_canonical_match": false,
|
|
"cache_status": "ABSENT",
|
|
"wall_ms": 130.2,
|
|
"audit_event_hash": "ed570bd6127a0b7143bb5c0f8cf76daf37a2092fe84350c0cab1ee2a2bf0410b"
|
|
},
|
|
{
|
|
"question": "A IMPL B",
|
|
"status": "cache_miss_then_written",
|
|
"audit_mode": "CANONICAL_PROJECTION",
|
|
"pi_star_ref": "logic-kernel@v1",
|
|
"canonical_answer": "(NOT A OR B)",
|
|
"agreement_label": "KERNEL-LLM-DIVERGED",
|
|
"llm_raw": "TRUE",
|
|
"llm_canonical_match": false,
|
|
"cache_status": "ABSENT",
|
|
"wall_ms": 124.9,
|
|
"audit_event_hash": "0f6fb926fd72150c5a0d8121a004b1f66edf3c8ee4bc15c52168f3a7869786fe"
|
|
},
|
|
{
|
|
"question": "(NOT B) IMPL (NOT A)",
|
|
"status": "cache_miss_then_written",
|
|
"audit_mode": "CANONICAL_PROJECTION",
|
|
"pi_star_ref": "logic-kernel@v1",
|
|
"canonical_answer": "(NOT A OR B)",
|
|
"agreement_label": "KERNEL-LLM-DIVERGED",
|
|
"llm_raw": "TRUE",
|
|
"llm_canonical_match": false,
|
|
"cache_status": "ABSENT",
|
|
"wall_ms": 113.5,
|
|
"audit_event_hash": "1b17f6e39edf2bb3d8dce003e20f71df9a3b9bd1b885f8a6e4b78e43ec122c31"
|
|
},
|
|
{
|
|
"question": "A OR NOT A",
|
|
"status": "cache_miss_then_written",
|
|
"audit_mode": "CANONICAL_PROJECTION",
|
|
"pi_star_ref": "logic-kernel@v1",
|
|
"canonical_answer": "TRUE",
|
|
"agreement_label": "KERNEL-LLM-AGREE",
|
|
"llm_raw": "TRUE",
|
|
"llm_canonical_match": false,
|
|
"cache_status": "ABSENT",
|
|
"wall_ms": 123.0,
|
|
"audit_event_hash": "5929ffe218bb8a97c391df70cfffed810f66d02c4a54bdce2569755200bd8d0d"
|
|
},
|
|
{
|
|
"question": "(x+1)**2",
|
|
"status": "cache_miss_then_written",
|
|
"audit_mode": "CANONICAL_PROJECTION",
|
|
"pi_star_ref": "algebra-symbolic@v1",
|
|
"canonical_answer": "Add(Pow(Symbol('x'), Integer(2)), Mul(Integer(2), Symbol('x')), Integer(1))",
|
|
"agreement_label": "KERNEL-LLM-DIVERGED",
|
|
"llm_raw": "1",
|
|
"llm_canonical_match": false,
|
|
"cache_status": "ABSENT",
|
|
"wall_ms": 179.4,
|
|
"audit_event_hash": "b36f1a830aae8fec514f95f3104ba4473bc6febb98137016c5d0816f859ba440"
|
|
},
|
|
{
|
|
"question": "x**2 + 2*x + 1",
|
|
"status": "cache_miss_then_written",
|
|
"audit_mode": "CANONICAL_PROJECTION",
|
|
"pi_star_ref": "algebra-symbolic@v1",
|
|
"canonical_answer": "Add(Pow(Symbol('x'), Integer(2)), Mul(Integer(2), Symbol('x')), Integer(1))",
|
|
"agreement_label": "KERNEL-LLM-DIVERGED",
|
|
"llm_raw": "1",
|
|
"llm_canonical_match": false,
|
|
"cache_status": "ABSENT",
|
|
"wall_ms": 120.1,
|
|
"audit_event_hash": "d307d6990527ddc6af73ea3cb894d4986ea3003106723fe74e22282f16eb62c6"
|
|
}
|
|
]
|
|
} |