arborist/tests/test_prometheus_sigma.py
russell@unturf.com c422216428
#000037 Phase 0: §12 trigger probe + Phase 1 test scaffolding
Phase 0 of ticket #000037 (Prometheus-Σ recursive falsification
controller) is doc-only and gates Phase 1 on §12 measured-pressure
triggers. This commit lands the empirical-evidence harness fox needs
for the go/no-go decision, plus the §16.2 test scaffolding so the
contract is discoverable from the test runner today.

bench/prometheus_sigma_trigger_probe.py
=======================================

Read-only walk of audit_events + capital_ledger across shards.
Reports each §12 trigger:

- Trigger 1 (branch density, ≥4 branches/checkpoint): looks for a
  fork_score branch-set table; reports "no data — single-validator
  ForkScore Phase 1a" when absent. Trigger structurally cannot fire
  until #000012 multi-branch persistence lands.
- Trigger 2 (divergence variance, N≥30 + ratio>0.5 OR abs>0.10):
  aggregates providence_canonical_witness audit-event bodies,
  maps each agreement_label per #000028 §1.2 to a binary
  LLM-divergence score, computes mean/stddev/ratio. Defensive on
  the max(mean,ε) guard from §12 itself.
- Trigger 3 (witness cost share > 0.30): aggregates capital_ledger
  rows; uses `material` (kWh proxy) as the canonical compute axis;
  surfaces both `material` and `financial` so fox can pick a
  different form if needed. Tags the report with the caveat that
  the current ledger reflects ad-hoc activity, not a controlled
  #000026 sweep.
- Trigger 4 (operator mission need): n/a — operator decision.

Pure measurement, no LLM, no schema change, no mutation. Output is
a markdown report at $(PROMETHEUS_PROBE_OUT) (default
bench/results/prometheus-sigma-triggers-<utc-date>.md).

Wired via `make prometheus-trigger-probe`.

bench/results/prometheus-sigma-triggers-2026-05-10.md
=====================================================

First captured baseline. Verdict on current shards:

  Trigger 1: NO  (no fork_score branch-set table; #000012 Phase 1a
                  is single-validator)
  Trigger 2: NO  (16 samples; N_min=30. But mean=0.625, σ=0.5,
                  ratio=0.8 — both ratio AND abs floors would fire
                  if N reaches 30. Signal is there; just needs more
                  samples.)
  Trigger 3: NO  (witness/total material = 0.005, well under 0.30
                  threshold. Caveated: ledger is ad-hoc, not a
                  controlled #000026 sweep.)
  Trigger 4: n/a (operator-stated)

Empirical answer: no §12 measured-pressure trigger has fired yet.
Phase 1 of #000037 remains paper-only unless fox invokes Trigger 4.

tests/test_prometheus_sigma.py
==============================

17 skip-stubs pinning the §16.2 acceptance contract. Each test:

- Collects today via `pytest --collect-only` so the test surface is
  discoverable.
- Skips with reason "Phase 1 not landed; controller module
  arborist/v9/prometheus.py absent" until that import succeeds.
- Carries a one-sentence intent line tying it back to a numbered
  ticket section (§5 / §6 / §7 / §10 / §13 / §14 / §16.1 / §4.4).

When Phase 1 lands, the implementer adds the controller module
under arborist/v9/prometheus.py per §13; CONTROLLER_AVAILABLE
becomes True; each test gets its body filled in. The names, intents,
and skip-reason strings are the durable contract.

Hygiene
=======

- make test → 1623 passed, 45 skipped (was 28; +17 new skips).
- make chain-check-shards → 0 breaks across all 7 shards.
- No code touched outside the new probe + scaffolding files +
  Makefile target wiring. fox's in-flight #000037 ticket
  modifications left untouched.
2026-05-10 07:24:59 -04:00

214 lines
8.4 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

"""Phase 1 test surface for ticket #000037 (Prometheus-Σ controller).
Phase 0 is doc-only — these tests are the **scaffolding** the §16.2
implementation contract names. All of them skip until Phase 1 lands
the controller module (``arborist/v9/prometheus.py`` per §13).
Why land scaffolding while Phase 0 is still doc-only:
- The 17 named tests are the §16.2 acceptance contract; pinning them
here means a future shift can't drift the contract by accident.
- ``pytest --collect-only`` lists them, so the test surface is
discoverable from the test runner today rather than buried in a
ticket.
- When Phase 1 lands, the implementer flips
``CONTROLLER_AVAILABLE = True``, drops the body of each test, and
the contract enforces itself.
Per CLAUDE.md "no half-finished implementations": these stubs are
*explicitly* documented as scaffolding. Each test contains a one-
sentence intent line tied to the controller invariant it pins; that
intent is what the eventual implementation must satisfy.
Phase 1 trigger gating (§12) is a separate concern measured by
``bench/prometheus_sigma_trigger_probe.py`` — these tests will run
regardless of trigger state once the module is in place.
"""
from __future__ import annotations
import pytest
try:
from arborist.v9 import prometheus as _prom # noqa: F401
CONTROLLER_AVAILABLE = True
except ImportError:
CONTROLLER_AVAILABLE = False
skip_until_phase_1 = pytest.mark.skipif(
not CONTROLLER_AVAILABLE,
reason=(
"ticket #000037 Phase 1 not landed; controller module "
"arborist/v9/prometheus.py absent. Skip is the contract — "
"implementer flips this when the module exists."
),
)
# ----------------------------------------------------------- core decision
@skip_until_phase_1
def test_all_vetoed_returns_reject_or_quarantine():
"""§14 row 2: all branches hard-vetoed → emit veto reasons; no
allocation; label ``REJECT`` or ``QUARANTINE`` (depending on the
veto class). Catches the "every branch is poisoned" case."""
pytest.fail("Phase 1 implementation pending; see §13 step 3.")
@skip_until_phase_1
def test_zero_budget_returns_deferred():
"""§14 row 3: ``B = 0`` → no LLM call; queue sleep if useful;
label ``DEFERRED``. Distinguishes "didn't evaluate due to budget"
from "evaluated but uncertain" (``MARGINAL``). Per David review
point 3."""
pytest.fail("Phase 1 implementation pending; see §4.3 + §14.")
@skip_until_phase_1
def test_stable_softmax_no_overflow():
"""§5: softmax must use ``exp(z_i max z) / Σ exp(z_j max z)``.
With large positive utilities (e.g. z = 1000) naïve ``exp(z)``
overflows; the normalized form must not. Per David review point 6."""
pytest.fail("Phase 1 implementation pending; see §5 + §13 step 5.")
@skip_until_phase_1
def test_negative_payoff_gets_zero_allocation():
"""§7 Kelly safety guard: ``b_i ≤ 0`` → zero allocation. A branch
with negative expected payoff must not consume budget. Per David
review point 8."""
pytest.fail("Phase 1 implementation pending; see §7.")
# ----------------------------------------------------------- vetoes
@skip_until_phase_1
def test_unsupported_carrier_quarantines():
"""§6 hard-veto class: claim references a carrier modality that
no live π* library supports → ``QUARANTINE``. Surfaces missing
domain coverage rather than papering over it."""
pytest.fail("Phase 1 implementation pending; see §6.")
@skip_until_phase_1
def test_cache_drift_quarantines():
"""§6 hard-veto class: cache row whose ``pi_star_ref`` no longer
matches a live kernel version → ``QUARANTINE``. Echoes the
CACHE-DRIFT outcome from #000028 §1.2; controller surfaces
rather than silently re-uses."""
pytest.fail("Phase 1 implementation pending; see §6 + #000028 §1.2.")
@skip_until_phase_1
def test_memory_invalidation_above_threshold_escalates():
"""§10 Gödel discipline: a branch whose acceptance would
invalidate too much committed memory → label ``ESCALATE``, not
``REJECT``. The controller explicitly steps aside per the "must
never infer" rule. Per David review point 12."""
pytest.fail("Phase 1 implementation pending; see §10.")
# ----------------------------------------------------------- difficulty / EMA
@skip_until_phase_1
def test_high_entropy_increases_difficulty():
"""§7.1 difficulty update law: high ``H_norm(p)`` → smoothed
increase in ``difficulty_ema``. The EMA smoothing keeps the
update from overshooting on a single noisy sample. Per David
review point 9."""
pytest.fail("Phase 1 implementation pending; see §7.1.")
@skip_until_phase_1
def test_low_entropy_decreases_or_preserves_difficulty():
"""§7.1 difficulty update law: low ``H_norm(p)`` (controller is
confident) → difficulty drops or holds. Symmetric to the
high-entropy test; together they pin the EMA's monotonicity."""
pytest.fail("Phase 1 implementation pending; see §7.1.")
@skip_until_phase_1
def test_divergence_increases_witness_sampling_recommendation():
"""§4.3 output / §17.1: high observed witness divergence → the
controller's recommendation field should bump ``canonical_witness_
sample_rate`` upward (an advisory; runtime decides whether to
apply). Pins the feedback loop into #000028's sample-rate field
we landed in 6d20aeb."""
pytest.fail("Phase 1 implementation pending; see §4.3.")
# ----------------------------------------------------------- discipline
@skip_until_phase_1
def test_no_llm_call_in_controller():
"""§10 Gödel + David review point 12: the controller is a pure
function. No ``ChatClient`` import in the call graph; no network
socket; no env-var that secretly enables one. The witness module
calls the LLM; the controller reads its results. Pins the
"LLM is witness, never authority" doctrine."""
pytest.fail("Phase 1 implementation pending; see §10 + §16.1.")
@skip_until_phase_1
def test_controller_does_not_modify_cache_key_inputs():
"""§4.4 update authority: the controller emits **proposals** for
MemoryRoot / SelfModel updates; it never mutates the cache_key
8-dim input itself. Pins the schema invariant from #000027:
cache_key is computed by the cache_key() function, not by any
advisory layer above it. Per David review point 14."""
pytest.fail("Phase 1 implementation pending; see §4.4 + §13 step 12.")
@skip_until_phase_1
def test_controller_outputs_advisory_event_only():
"""§13 step 10: the controller writes ``controller_decision`` /
``controller_difficulty`` / ``controller_budget_allocation`` as
sibling tags on ``audit_events`` — they do NOT enter
``event_hash`` preimage. Re-running the controller against the
same state cannot break the audit chain. Pins the same sibling-
table invariant the capital_ledger uses."""
pytest.fail("Phase 1 implementation pending; see §13 step 10.")
# ----------------------------------------------------------- §12 trigger guards
@skip_until_phase_1
def test_zero_mean_divergence_does_not_trigger_phase_1():
"""§12 Trigger 2: ``max(mean, ε)`` guard against div-by-zero
when divergence is uniformly low. Probe covers this; the
in-controller guard duplicates it so the controller can be run
on a fresh corpus without crashing on the first call."""
pytest.fail("Phase 1 implementation pending; see §12 Trigger 2.")
@skip_until_phase_1
def test_small_sample_does_not_trigger_phase_1():
"""§12 Trigger 2: ``N_min = 30`` floor below which the variance
trigger does not fire. Pins behavior on early-corpus deployments
where sample-count noise would dominate any signal."""
pytest.fail("Phase 1 implementation pending; see §12 Trigger 2.")
# ----------------------------------------------------------- proposals, not mutations
@skip_until_phase_1
def test_propose_not_mutate_memory_root():
"""§4.4 + §13 step 12: controller emits a MemoryRoot update
*proposal*; the existing memory-root write path validates and
commits. Direct mutation would let the controller poison hard-
hashed state without going through validation."""
pytest.fail("Phase 1 implementation pending; see §4.4.")
@skip_until_phase_1
def test_propose_not_mutate_self_model():
"""§4.4 + §13 step 12: controller emits a SelfModel update
*proposal*; the existing #000014 selfmodel write path validates
and commits. Same boundary as MemoryRoot."""
pytest.fail("Phase 1 implementation pending; see §4.4 + #000014.")