engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
arborist/docs
History
russell@unturf.com bc77f961f3
fan-out: close #000030 · composition fixtures · witness end-to-end 
Three small streams:

#3 — close #000030 properly
============================

All 7 phases + Phase 1b landed across two commits (`04f3f5d`,
`abe5988`). Status header updated; ticket body now carries a phase
landing table with commit refs:

  Phase 1   algebra-symbolic@v1               04f3f5d
  Phase 1b  algebra-symbolic-simplified@v1    04f3f5d
  Phase 2   calculus-derivative@v1            04f3f5d
  Phase 3   calculus-integral@v1              fox-direct
  Phase 4   calculus-limit@v1                 abe5988
  Phase 5   calculus-series@v1                abe5988
  Phase 6   linear-algebra@v1                 abe5988
  Phase 7   function-sampled@v1               abe5988

Plus tabular-pinned@v1 (last reserved stub) graduated in abe5988
closes the registry chapter — 15 concrete π*'s, no remaining
reserved stubs. Index updated.

#5 — composition fixtures across new SymPy π*'s
================================================

12 new tests in tests/test_pi_star_compositions.py covering pairs
that compose naturally:

- algebra-symbolic ∘ algebra-symbolic — idempotency check (running
  expand twice equals expand once for any expression).
- algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean
  identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`).
- Generic invariants: composition propagates PiStarError; manifest
  fingerprint is order-sensitive; composite domain == inner domain;
  composite bytes == manual chain bytes.

Test discipline: most compositions use `register_in_registry=False`
via a small `_safe_compose()` helper since the registry rejects
duplicate keys (#000015 invariant), so test ordering would
otherwise matter. Only the registration-test path uses real
compose().

#4 — end-to-end witness sweep against real shards + Hermes
===========================================================

New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape
questions (3 arithmetic + 3 logic + 2 algebra) through query() with
`canonical_witness_enabled=True`, against ~/.arborist/shards (real
shard cluster) + the actual Hermes endpoint (NOT StubClient).
Records the agreement matrix per question to
bench/results/witness-sweep.json.

`make bench-witness-sweep` Makefile target. Honors
`ARBORIST_SHARDS_DIR`.

First real sweep (this commit, against Hermes-3-8B):

  agreement label             count    rate
  KERNEL-LLM-DIVERGED         5        62.5%
  KERNEL-LLM-AGREE            3        37.5%
  ───────────────────────────────────────────
  divergence_count            5        62.5%
  wall median / max           130 ms / 1.1 s

Hermes diverged on 5/8 of the canonical-shape questions:

- said `1/10` for `0.1 + 0.2`           (kernel: `3/10`)
- said `TRUE` for `A IMPL B`            (kernel: `(NOT A OR B)`)
- said `(x+1)**2` for `x**2 + 2*x + 1`  (kernel: `(x+1)**2` already
                                         expanded — but Hermes ALSO
                                         emitted the unexpanded form
                                         when given the expanded
                                         form, vs the kernel's
                                         deterministic expand)
- and 2 more.

These are real LLM hallucinations on questions with closed-form
ground truth — exactly the calibration-data stream #000028
imagined. Pipeline validated end-to-end against actual hardware.

Pair: `make bench-witness-divergence` then extracts the 5
divergences as 5F-Falsification fixtures
(bench/fixtures/5f/falsification-witness-v1.jsonl, also committed).
Re-running the extractor produces byte-equal output (idempotency
contract from the extractor work).

Tests
=====
Full suite: 1636 passed, 37 skipped (was 1624; +12 composition
tests). The witness-sweep + extractor produce real artifacts now
committed under bench/results/ and bench/fixtures/5f/.
2026-05-09 13:29:59 -04:00
..
_source fan-out: witness audit + 5F extractor + function-sampled demo + docs 2026-05-09 13:19:30 -04:00
diagrams docs/diagrams: rename aborist-modules → arborist-modules (post-rename cleanup) 2026-05-08 07:50:28 -04:00
tickets fan-out: close #000030 · composition fixtures · witness end-to-end 2026-05-09 13:29:59 -04:00
bench-maxing.md aborist/arborist 2026-05-07 09:31:49 -04:00
benchmarks.md aborist/arborist 2026-05-07 09:31:49 -04:00
cti-architecture.md aborist/arborist 2026-05-07 09:31:49 -04:00
mesh.md aborist/arborist 2026-05-07 09:31:49 -04:00
pi-star-composition.md pi_star: land ticket #000015 (π* domain library + composition algebra) 2026-05-07 16:51:33 -04:00
qa-modes-bench.md aborist/arborist 2026-05-07 09:31:49 -04:00
seven-point-program.md aborist/arborist 2026-05-07 09:31:49 -04:00
spec-methodology.md docs: land ticket #000019 (spec methodology for π*, V, policy fields) 2026-05-07 16:53:28 -04:00
TICKETS.md fan-out: close #000030 · composition fixtures · witness end-to-end 2026-05-09 13:29:59 -04:00
tool-action-dag-design.md docs: add tool-action-dag-design.md research path (pre-ticket) 2026-05-07 19:47:50 -04:00
v8-fork-score.md v8: ticket #000012 Phase 1a — ForkScore consumes the new bench substrate 2026-05-08 07:18:22 -04:00