russell@unturf.com
70ffc01ce4
Three small streams in one commit:
#000028 follow-up — witness divergence → 5F fixtures
=====================================================
Witness fan-out now writes a `providence_canonical_witness` audit
event when it fires (next to the capital-ledger record landed in
708aa45). Body carries pi_star_ref, question_text, agreement_label,
canonical_answer_text, llm_raw_text, llm_canonical_bytes,
cache_status. Best-effort write — chain failure never fails the
query.
New extractor `bench/scripts/witness_to_5f.py` reads those events
from a qa.db and writes them out as 5F-Falsification fixtures
matching the existing `falsification-live-v1` schema. Filtering
includes only divergence labels (LLM-DIVERGED / KERNEL-LLM-DIVERGED
/ CACHE-DRIFT); skips KERNEL-LLM-AGREE / STRICT-WITNESSED (no
calibration signal) and KERNEL-ONLY (LLM unparseable, not a
supervised-correction sample).
Idempotent: sorted by audit-event seq, so re-running against the
same qa.db produces byte-equal fixture files. The existing
fixture-digest discipline stays valid.
Makefile: `make bench-witness-divergence` (override default
qa.db / output path via WITNESS_QA_DB / WITNESS_OUT env-vars).
Closes the divergence → calibration data loop the witness ticket
imagined: every LLM hallucination on a canonical-shape question
becomes a supervised-correction fixture downstream prompt
improvements can grade against.
#000030 Phase 7 demo — function-sampled@v1 end-to-end
======================================================
`bench/scripts/demo_plot.py` — closes the loop on opencompletion's
activity24-math-plot.yaml. SymPy expression → quantized
integer-vector signature (canonical bytes) → optional matplotlib
PNG. Canonical bytes are the proof; PNG is just a downstream view
of the same evidence.
$ make demo-plot Q='sin(x)' PNG=/tmp/sin.png
Output JSON contains canonical_bytes_sha256 + canonical_bytes_preview
+ canonical_bytes_total_chars + grid metadata + the optional png_path.
matplotlib is gated — when absent, --png prints a warning to stderr
and skips the render; the canonical bytes still print. Tests skip
the PNG-presence assertion via `pytest.importorskip("matplotlib")`.
Public docs polish (#7)
========================
- docs/_source/bench.rst: updated fixture-count narrative (~660 →
662 default tasks + ~110 math π* fixtures); `make` quick-reference
now lists all per-π* 5S targets (tabular, calculus-limit/series,
linear-algebra, function-sampled) plus bench-real-shard,
bench-fork-baseline/score, bench-witness-divergence.
- docs/_source/v8-fork-score.rst: CLI section gained --out flag
documentation + a Make-harness sub-section covering
bench-fork-baseline / bench-fork-score / FORK_PARENT/CHILD/REPORT
env-vars.
Tests
=====
- tests/test_witness_to_5f.py — 8 new tests covering the audit-event
write (3) + extractor logic (5).
- tests/test_demo_plot.py — 6 new tests covering canonical-bytes
determinism + equivalence-class collapse + matplotlib gating.
Full suite: 1624 passed, 37 skipped (was 1568; +56).
154 lines
6.4 KiB
ReStructuredText
154 lines
6.4 KiB
ReStructuredText
Benchmark surface
|
|
=================
|
|
|
|
Arborist ships the complete **Dav1DPrometheus 5S/5T/5F/5R** evaluation
|
|
suite — 21 sub-batteries, **662+ deterministic fixtures** in the
|
|
default runner (Phase 1d expansion 2026-05-09) plus ~110 additional
|
|
math π* fixtures — as first-class infrastructure. Every benchmark is
|
|
reproducible, no LLM-as-judge, and many sub-batteries route through
|
|
the actual arborist surface (parser, verifier, audit chain, π*
|
|
registry) rather than synthetic gold output.
|
|
|
|
Quick reference
|
|
---------------
|
|
|
|
.. code-block:: bash
|
|
|
|
make bench-suite # complete 5S + 5T + 5F + 5R (662 tasks)
|
|
make bench-5s # representational discipline
|
|
make bench-5t # temporal / cross-reasoning
|
|
make bench-5f # operational quality (synthetic + live)
|
|
make bench-5r # workspace operators
|
|
# Per-π* 5S targets (#000030 SymPy substrate + the registry's text/
|
|
# arithmetic/logic core + the last-stub graduation):
|
|
make bench-5s-math # arithmetic@v1 + logic-kernel@v1
|
|
make bench-5s-code # code-py-ast@v1
|
|
make bench-5s-time-series # time-series-quantized@v1
|
|
make bench-5s-tabular # tabular-pinned@v1 (last reserved stub)
|
|
make bench-5s-algebra # algebra-symbolic@v1
|
|
make bench-5s-calculus-limit # calculus-limit@v1
|
|
make bench-5s-calculus-series # calculus-series@v1
|
|
make bench-5s-linear-algebra # linear-algebra@v1
|
|
make bench-5s-function-sampled # function-sampled@v1 (SymPy → time-series)
|
|
# Real-shard + selection + witness-divergence harness:
|
|
make bench-real-shard # #000026 — real-shard latency / audit baseline
|
|
make bench-fork-baseline # pin current bench output as ForkScore parent
|
|
make bench-fork-score # score child vs pinned parent (CI-gateable)
|
|
make bench-witness-divergence # extract LLM-divergence as 5F fixtures
|
|
|
|
Each invocation emits a JSON :class:`bench.batteries.base.BatteryResult`
|
|
with per-task pass/fail, fixture digest, runtime digest, and
|
|
sub-battery-specific metrics.
|
|
|
|
The four batteries
|
|
------------------
|
|
|
|
5S — representation discipline
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Five sub-batteries testing what the system understands at the
|
|
sign / meaning / derivation level:
|
|
|
|
- **Syntax** — does the named π* parse the input without raising?
|
|
- **Semantics** — do two surface forms canonicalize to the same
|
|
bytes when they should (and not when they shouldn't)?
|
|
- **Syllogism** — does each step in a deductive chain validly
|
|
follow under the named rule (categorical_transitivity, chain_3,
|
|
invalid_converse, missing_premise)?
|
|
- **Synthesis** — does the system assemble cited facts into a
|
|
coherent derivation supported by the fact set?
|
|
- **Semiotics** — is meaning preserved under controlled label
|
|
swaps?
|
|
|
|
5T — temporal / cross-reasoning discipline
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Six sub-batteries (legacy ``transfer`` plus the canonical
|
|
Dav1DPrometheus five):
|
|
|
|
- **Transfer Learning** — does a learned pattern carry across
|
|
task / domain / carrier?
|
|
- **Triangulation** — do independent strategies (substring,
|
|
token_subset, token_overlap, entity_match) agree at threshold?
|
|
- **Truthtables** — exhaustive propositional coverage at N=2..4
|
|
variables.
|
|
- **Transitivity** — typed-relation chains under whitelist
|
|
(``implies``, ``subset_of``, ``ancestor_of``, ``before``,
|
|
``less_than``).
|
|
- **Time** — temporal-context preservation across memory_root
|
|
snapshots; integrates with #000017 surface.
|
|
|
|
5F — operational quality
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Five sub-batteries, all with **embedded** (Phase 1a synthetic) AND
|
|
**live** (Phase 1b.2, routes through real arborist surfaces) modes:
|
|
|
|
================ =============================================
|
|
Sub-battery Live surface
|
|
================ =============================================
|
|
Function ``arborist.qa.parse_claims.parse_pointer_claims``
|
|
Finetuning ``arborist.selfmodel.store_snapshot`` round-trip
|
|
Falsification ``arborist.qa.verify.verify_quotes``
|
|
Formulate ``arborist.qa.parse_claims.parse_pointer_claims``
|
|
Feedback Loop ``arborist.store.append_audit`` + ``memory.snapshot``
|
|
================ =============================================
|
|
|
|
Per-task ``detail.source`` reports ``"embedded"`` or ``"live"`` so
|
|
bench output distinguishes synthetic from production signal.
|
|
|
|
5R — workspace operators
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Five sub-batteries testing operators applied to a workspace
|
|
(SelfModel + memory_root + audit chain):
|
|
|
|
- **React** — observations integrate into downstream state.
|
|
- **Rearrange** — restructure without semantic shift.
|
|
- **Restore** — retrieve prior facts from history.
|
|
- **Replicate** — π*-determinism across N replicas.
|
|
- **Resonate** — variance-zero across N runs.
|
|
|
|
Cross-modality discipline
|
|
-------------------------
|
|
|
|
Every fixture carries:
|
|
|
|
- ``carrier`` — domain whitelist enforced by
|
|
:data:`bench.batteries.base.PHASE_1_CARRIERS`. Phase 1
|
|
domains: ``text``, ``claim_lattice``, ``memory_snapshot``,
|
|
``selfmodel_snapshot``, ``providence_record``, ``audit_event``,
|
|
``code``, ``arithmetic``, ``logic``, ``time_series``.
|
|
- ``domain`` — sub-domain qualifier (e.g.,
|
|
``rational``, ``propositional``, ``python_ast``).
|
|
- ``pi_star_ref`` — registry key naming the canonicalizer.
|
|
- ``loss_report_refs`` — optional projection-loss links.
|
|
- ``modality_notes`` — scope note.
|
|
|
|
Unsupported carriers fail explicitly with
|
|
``reason="unsupported_carrier"`` — never silently accepted. Hidden-
|
|
channel work is defensive only (detection / flagging, never
|
|
generation).
|
|
|
|
ForkScore consumes battery output
|
|
---------------------------------
|
|
|
|
The v8 ForkScore (see :doc:`v8-fork-score`) reads BatteryResult JSON
|
|
from a parent and child organism, computes a weighted scalar
|
|
verdict with ACCEPT / MARGINAL / REJECT classes:
|
|
|
|
.. code-block:: bash
|
|
|
|
make bench-suite # generates parent.json
|
|
# ... apply changes ...
|
|
make bench-suite # generates child.json
|
|
arborist v8 score --parent parent.json --child child.json
|
|
|
|
Authoring new fixtures
|
|
----------------------
|
|
|
|
See :file:`docs/spec-methodology.md` (per-author checklist) and the
|
|
existing fixture files under :file:`bench/fixtures/`. New
|
|
sub-batteries follow the protocol in :mod:`bench.batteries.base` —
|
|
``Battery.run(fixtures_path) → BatteryResult``, deterministic, no
|
|
LLM-as-judge, carrier metadata mandatory.
|