arborist/docs/_source/v8-fork-score.rst
russell@unturf.com 70ffc01ce4
fan-out: witness audit + 5F extractor + function-sampled demo + docs
Three small streams in one commit:

#000028 follow-up — witness divergence → 5F fixtures
=====================================================

Witness fan-out now writes a `providence_canonical_witness` audit
event when it fires (next to the capital-ledger record landed in
708aa45). Body carries pi_star_ref, question_text, agreement_label,
canonical_answer_text, llm_raw_text, llm_canonical_bytes,
cache_status. Best-effort write — chain failure never fails the
query.

New extractor `bench/scripts/witness_to_5f.py` reads those events
from a qa.db and writes them out as 5F-Falsification fixtures
matching the existing `falsification-live-v1` schema. Filtering
includes only divergence labels (LLM-DIVERGED / KERNEL-LLM-DIVERGED
/ CACHE-DRIFT); skips KERNEL-LLM-AGREE / STRICT-WITNESSED (no
calibration signal) and KERNEL-ONLY (LLM unparseable, not a
supervised-correction sample).

Idempotent: sorted by audit-event seq, so re-running against the
same qa.db produces byte-equal fixture files. The existing
fixture-digest discipline stays valid.

Makefile: `make bench-witness-divergence` (override default
qa.db / output path via WITNESS_QA_DB / WITNESS_OUT env-vars).

Closes the divergence → calibration data loop the witness ticket
imagined: every LLM hallucination on a canonical-shape question
becomes a supervised-correction fixture downstream prompt
improvements can grade against.

#000030 Phase 7 demo — function-sampled@v1 end-to-end
======================================================

`bench/scripts/demo_plot.py` — closes the loop on opencompletion's
activity24-math-plot.yaml. SymPy expression → quantized
integer-vector signature (canonical bytes) → optional matplotlib
PNG. Canonical bytes are the proof; PNG is just a downstream view
of the same evidence.

  $ make demo-plot Q='sin(x)' PNG=/tmp/sin.png

Output JSON contains canonical_bytes_sha256 + canonical_bytes_preview
+ canonical_bytes_total_chars + grid metadata + the optional png_path.
matplotlib is gated — when absent, --png prints a warning to stderr
and skips the render; the canonical bytes still print. Tests skip
the PNG-presence assertion via `pytest.importorskip("matplotlib")`.

Public docs polish (#7)
========================

- docs/_source/bench.rst: updated fixture-count narrative (~660 →
  662 default tasks + ~110 math π* fixtures); `make` quick-reference
  now lists all per-π* 5S targets (tabular, calculus-limit/series,
  linear-algebra, function-sampled) plus bench-real-shard,
  bench-fork-baseline/score, bench-witness-divergence.
- docs/_source/v8-fork-score.rst: CLI section gained --out flag
  documentation + a Make-harness sub-section covering
  bench-fork-baseline / bench-fork-score / FORK_PARENT/CHILD/REPORT
  env-vars.

Tests
=====

- tests/test_witness_to_5f.py — 8 new tests covering the audit-event
  write (3) + extractor logic (5).
- tests/test_demo_plot.py — 6 new tests covering canonical-bytes
  determinism + equivalence-class collapse + matplotlib gating.
Full suite: 1624 passed, 37 skipped (was 1568; +56).
2026-05-09 13:19:30 -04:00

154 lines
5.1 KiB
ReStructuredText
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

v8 ForkScore
============
The Merkle-AGI v8 selection protocol's **scoring half**. Pure
function over a (parent, child) bench-result pair → a single
scalar with ACCEPT / MARGINAL / REJECT verdict. Phase 1a of
ticket ``#000012``.
The complementary canonicalization half (validator state machine,
acceptance protocol, slashing, fork-choice rule) is reserved for
the v8 paper itself; Phase 1a ships only the function ForkScore
without the consensus surface around it.
Formula
-------
.. code-block:: text
ForkScore =
α · Δ5S
+ β · Δ5T
+ γ · Δ5F (incl. efficiency-aware bonus)
+ δ · SelfModelCalibrationGain
+ ε · AuditCompleteness
+ ζ · ValidatorDiversity
- η · RegressionPenalty
- θ · CapitalCostPenalty
- ι · SecurityRiskPenalty (reserved; Phase 1a = 0)
- κ · ComplexityPenalty (reserved; Phase 1a = 0)
- λ · MemoryInvalidationPenalty
Each term consumes the metrics every 5S/5T/5F sub-battery emits in
its :class:`BatteryResult.metrics` dict. The Δ-rate per battery is
the mean of per-sub-battery rate deltas (child — parent).
Δ5F additionally consumes the inf-aware efficiency aggregations
landed under the 2026-05-08 ``fbd99a8`` review:
``adaptation_efficiency_mean_finite``,
``adaptation_efficiency_infinite_count``,
``adaptation_efficiency_neg_infinite_count``,
``feedback_efficiency_mean_finite``,
``feedback_efficiency_infinite_count``.
Verdict thresholds
------------------
========================== ================== ============
Score / flags Verdict CLI exit
========================== ================== ============
``score >= SIGNAL_FLOOR`` **ACCEPT** ``0``
``[0, SIGNAL_FLOOR)`` **MARGINAL** ``0``
``score < 0`` **REJECT** ``1``
hard-regression flag **REJECT** ``1``
``NEG_INF_REGRESSION`` flag **REJECT** ``1``
========================== ================== ============
``SIGNAL_FLOOR`` defaults to ``0.05`` (5pp; matches
:file:`docs/bench-maxing.md`'s noise floor).
Hard-regression flag fires if any single sub-battery rate drops by
``HARD_REGRESSION_FLOOR`` (default 5pp), regardless of net score.
``NEG_INF_REGRESSION`` flag fires when
``*_efficiency_neg_infinite_count`` increases parent → child:
*free regression* is unsafe regardless of other gains.
Default weights
---------------
.. code-block:: python
WeightSet(
alpha=1.0, # Δ5S
beta=1.0, # Δ5T
gamma=1.0, # Δ5F
delta=0.5, # SelfModelCalibrationGain
epsilon=0.3, # AuditCompleteness
zeta=0.0, # ValidatorDiversity (off in single-validator)
eta=2.0, # RegressionPenalty (heavy by design)
theta=0.5, # CapitalCostPenalty
iota=0.0, # SecurityRiskPenalty (reserved)
kappa=0.0, # ComplexityPenalty (reserved)
lambda_=0.5, # MemoryInvalidationPenalty
)
Override via JSON file passed to ``--weights``:
.. code-block:: json
{
"alpha": 1.5,
"eta": 5.0,
"lambda": 1.0
}
The JSON key ``"lambda"`` round-trips into ``WeightSet.lambda_``
because ``lambda`` is a Python reserved word.
CLI
---
.. code-block:: bash
arborist v8 score \
--parent parent-bench.json \
--child child-bench.json \
[--weights weights.json] \
[--capital-delta N] \
[--memory-invalidation-count N] \
[--audit-completeness 0..1] \
[--selfmodel-calibration-gain N] \
[--out path/to/fork_score_report.json]
``parent-bench.json`` and ``child-bench.json`` are the JSON output
of ``bench.batteries.runner --all``.
``--out`` mirrors stdout to a file; CI / mesh peers / downstream
graders ingest the artifact without parsing pipe output.
Output: :class:`arborist.v8.fork_score.ScoredFork` with
``score``, ``verdict``, per-term ``breakdown``, ``flags`` list,
and the active ``weights`` echoed back.
Make harness (Phase 1b)
~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
make bench-fork-baseline # one-shot: pin current bench-suite output
# as the ForkScore parent (writes
# bench/results/baseline-suite.json)
<hack hack hack>
make bench-fork-score # rerun bench, score child vs pinned parent,
# write bench/results/fork_score_report.json.
# Exits 1 on REJECT verdict — CI-gateable.
Override defaults via env-vars: ``FORK_PARENT``, ``FORK_CHILD``,
``FORK_REPORT``.
What's NOT in Phase 1a
----------------------
- Validator state machine (bonding / signing / slashing).
- Acceptance protocol (proposal / quorum / finalization).
- Challenge protocol (audit-replay disagreement).
- Fork-choice rule (which of two competing finalizations wins).
- Mesh wire format extensions for validator gossip.
- Stake mechanics + economic incentives.
- Cross-validator ZK proof exchange.
These are commissioned by the v8 paper itself
(``docs/merkle-agi-v8-consensus.rst``, still open under #000012).
Phase 1a's scoring function is the substrate the paper consumes;
landing it now lets the paper cite measured values instead of
stipulated ones.