Phase 0 of ticket #000037 (Prometheus-Σ recursive falsification controller) is doc-only and gates Phase 1 on §12 measured-pressure triggers. This commit lands the empirical-evidence harness fox needs for the go/no-go decision, plus the §16.2 test scaffolding so the contract is discoverable from the test runner today. bench/prometheus_sigma_trigger_probe.py ======================================= Read-only walk of audit_events + capital_ledger across shards. Reports each §12 trigger: - Trigger 1 (branch density, ≥4 branches/checkpoint): looks for a fork_score branch-set table; reports "no data — single-validator ForkScore Phase 1a" when absent. Trigger structurally cannot fire until #000012 multi-branch persistence lands. - Trigger 2 (divergence variance, N≥30 + ratio>0.5 OR abs>0.10): aggregates providence_canonical_witness audit-event bodies, maps each agreement_label per #000028 §1.2 to a binary LLM-divergence score, computes mean/stddev/ratio. Defensive on the max(mean,ε) guard from §12 itself. - Trigger 3 (witness cost share > 0.30): aggregates capital_ledger rows; uses `material` (kWh proxy) as the canonical compute axis; surfaces both `material` and `financial` so fox can pick a different form if needed. Tags the report with the caveat that the current ledger reflects ad-hoc activity, not a controlled #000026 sweep. - Trigger 4 (operator mission need): n/a — operator decision. Pure measurement, no LLM, no schema change, no mutation. Output is a markdown report at $(PROMETHEUS_PROBE_OUT) (default bench/results/prometheus-sigma-triggers-<utc-date>.md). Wired via `make prometheus-trigger-probe`. bench/results/prometheus-sigma-triggers-2026-05-10.md ===================================================== First captured baseline. Verdict on current shards: Trigger 1: NO (no fork_score branch-set table; #000012 Phase 1a is single-validator) Trigger 2: NO (16 samples; N_min=30. But mean=0.625, σ=0.5, ratio=0.8 — both ratio AND abs floors would fire if N reaches 30. Signal is there; just needs more samples.) Trigger 3: NO (witness/total material = 0.005, well under 0.30 threshold. Caveated: ledger is ad-hoc, not a controlled #000026 sweep.) Trigger 4: n/a (operator-stated) Empirical answer: no §12 measured-pressure trigger has fired yet. Phase 1 of #000037 remains paper-only unless fox invokes Trigger 4. tests/test_prometheus_sigma.py ============================== 17 skip-stubs pinning the §16.2 acceptance contract. Each test: - Collects today via `pytest --collect-only` so the test surface is discoverable. - Skips with reason "Phase 1 not landed; controller module arborist/v9/prometheus.py absent" until that import succeeds. - Carries a one-sentence intent line tying it back to a numbered ticket section (§5 / §6 / §7 / §10 / §13 / §14 / §16.1 / §4.4). When Phase 1 lands, the implementer adds the controller module under arborist/v9/prometheus.py per §13; CONTROLLER_AVAILABLE becomes True; each test gets its body filled in. The names, intents, and skip-reason strings are the durable contract. Hygiene ======= - make test → 1623 passed, 45 skipped (was 28; +17 new skips). - make chain-check-shards → 0 breaks across all 7 shards. - No code touched outside the new probe + scaffolding files + Makefile target wiring. fox's in-flight #000037 ticket modifications left untouched.
214 lines
8.4 KiB
Python
214 lines
8.4 KiB
Python
"""Phase 1 test surface for ticket #000037 (Prometheus-Σ controller).
|
||
|
||
Phase 0 is doc-only — these tests are the **scaffolding** the §16.2
|
||
implementation contract names. All of them skip until Phase 1 lands
|
||
the controller module (``arborist/v9/prometheus.py`` per §13).
|
||
|
||
Why land scaffolding while Phase 0 is still doc-only:
|
||
|
||
- The 17 named tests are the §16.2 acceptance contract; pinning them
|
||
here means a future shift can't drift the contract by accident.
|
||
- ``pytest --collect-only`` lists them, so the test surface is
|
||
discoverable from the test runner today rather than buried in a
|
||
ticket.
|
||
- When Phase 1 lands, the implementer flips
|
||
``CONTROLLER_AVAILABLE = True``, drops the body of each test, and
|
||
the contract enforces itself.
|
||
|
||
Per CLAUDE.md "no half-finished implementations": these stubs are
|
||
*explicitly* documented as scaffolding. Each test contains a one-
|
||
sentence intent line tied to the controller invariant it pins; that
|
||
intent is what the eventual implementation must satisfy.
|
||
|
||
Phase 1 trigger gating (§12) is a separate concern measured by
|
||
``bench/prometheus_sigma_trigger_probe.py`` — these tests will run
|
||
regardless of trigger state once the module is in place.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import pytest
|
||
|
||
try:
|
||
from arborist.v9 import prometheus as _prom # noqa: F401
|
||
CONTROLLER_AVAILABLE = True
|
||
except ImportError:
|
||
CONTROLLER_AVAILABLE = False
|
||
|
||
skip_until_phase_1 = pytest.mark.skipif(
|
||
not CONTROLLER_AVAILABLE,
|
||
reason=(
|
||
"ticket #000037 Phase 1 not landed; controller module "
|
||
"arborist/v9/prometheus.py absent. Skip is the contract — "
|
||
"implementer flips this when the module exists."
|
||
),
|
||
)
|
||
|
||
|
||
# ----------------------------------------------------------- core decision
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_all_vetoed_returns_reject_or_quarantine():
|
||
"""§14 row 2: all branches hard-vetoed → emit veto reasons; no
|
||
allocation; label ``REJECT`` or ``QUARANTINE`` (depending on the
|
||
veto class). Catches the "every branch is poisoned" case."""
|
||
pytest.fail("Phase 1 implementation pending; see §13 step 3.")
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_zero_budget_returns_deferred():
|
||
"""§14 row 3: ``B = 0`` → no LLM call; queue sleep if useful;
|
||
label ``DEFERRED``. Distinguishes "didn't evaluate due to budget"
|
||
from "evaluated but uncertain" (``MARGINAL``). Per David review
|
||
point 3."""
|
||
pytest.fail("Phase 1 implementation pending; see §4.3 + §14.")
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_stable_softmax_no_overflow():
|
||
"""§5: softmax must use ``exp(z_i − max z) / Σ exp(z_j − max z)``.
|
||
With large positive utilities (e.g. z = 1000) naïve ``exp(z)``
|
||
overflows; the normalized form must not. Per David review point 6."""
|
||
pytest.fail("Phase 1 implementation pending; see §5 + §13 step 5.")
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_negative_payoff_gets_zero_allocation():
|
||
"""§7 Kelly safety guard: ``b_i ≤ 0`` → zero allocation. A branch
|
||
with negative expected payoff must not consume budget. Per David
|
||
review point 8."""
|
||
pytest.fail("Phase 1 implementation pending; see §7.")
|
||
|
||
|
||
# ----------------------------------------------------------- vetoes
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_unsupported_carrier_quarantines():
|
||
"""§6 hard-veto class: claim references a carrier modality that
|
||
no live π* library supports → ``QUARANTINE``. Surfaces missing
|
||
domain coverage rather than papering over it."""
|
||
pytest.fail("Phase 1 implementation pending; see §6.")
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_cache_drift_quarantines():
|
||
"""§6 hard-veto class: cache row whose ``pi_star_ref`` no longer
|
||
matches a live kernel version → ``QUARANTINE``. Echoes the
|
||
CACHE-DRIFT outcome from #000028 §1.2; controller surfaces
|
||
rather than silently re-uses."""
|
||
pytest.fail("Phase 1 implementation pending; see §6 + #000028 §1.2.")
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_memory_invalidation_above_threshold_escalates():
|
||
"""§10 Gödel discipline: a branch whose acceptance would
|
||
invalidate too much committed memory → label ``ESCALATE``, not
|
||
``REJECT``. The controller explicitly steps aside per the "must
|
||
never infer" rule. Per David review point 12."""
|
||
pytest.fail("Phase 1 implementation pending; see §10.")
|
||
|
||
|
||
# ----------------------------------------------------------- difficulty / EMA
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_high_entropy_increases_difficulty():
|
||
"""§7.1 difficulty update law: high ``H_norm(p)`` → smoothed
|
||
increase in ``difficulty_ema``. The EMA smoothing keeps the
|
||
update from overshooting on a single noisy sample. Per David
|
||
review point 9."""
|
||
pytest.fail("Phase 1 implementation pending; see §7.1.")
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_low_entropy_decreases_or_preserves_difficulty():
|
||
"""§7.1 difficulty update law: low ``H_norm(p)`` (controller is
|
||
confident) → difficulty drops or holds. Symmetric to the
|
||
high-entropy test; together they pin the EMA's monotonicity."""
|
||
pytest.fail("Phase 1 implementation pending; see §7.1.")
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_divergence_increases_witness_sampling_recommendation():
|
||
"""§4.3 output / §17.1: high observed witness divergence → the
|
||
controller's recommendation field should bump ``canonical_witness_
|
||
sample_rate`` upward (an advisory; runtime decides whether to
|
||
apply). Pins the feedback loop into #000028's sample-rate field
|
||
we landed in 6d20aeb."""
|
||
pytest.fail("Phase 1 implementation pending; see §4.3.")
|
||
|
||
|
||
# ----------------------------------------------------------- discipline
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_no_llm_call_in_controller():
|
||
"""§10 Gödel + David review point 12: the controller is a pure
|
||
function. No ``ChatClient`` import in the call graph; no network
|
||
socket; no env-var that secretly enables one. The witness module
|
||
calls the LLM; the controller reads its results. Pins the
|
||
"LLM is witness, never authority" doctrine."""
|
||
pytest.fail("Phase 1 implementation pending; see §10 + §16.1.")
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_controller_does_not_modify_cache_key_inputs():
|
||
"""§4.4 update authority: the controller emits **proposals** for
|
||
MemoryRoot / SelfModel updates; it never mutates the cache_key
|
||
8-dim input itself. Pins the schema invariant from #000027:
|
||
cache_key is computed by the cache_key() function, not by any
|
||
advisory layer above it. Per David review point 14."""
|
||
pytest.fail("Phase 1 implementation pending; see §4.4 + §13 step 12.")
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_controller_outputs_advisory_event_only():
|
||
"""§13 step 10: the controller writes ``controller_decision`` /
|
||
``controller_difficulty`` / ``controller_budget_allocation`` as
|
||
sibling tags on ``audit_events`` — they do NOT enter
|
||
``event_hash`` preimage. Re-running the controller against the
|
||
same state cannot break the audit chain. Pins the same sibling-
|
||
table invariant the capital_ledger uses."""
|
||
pytest.fail("Phase 1 implementation pending; see §13 step 10.")
|
||
|
||
|
||
# ----------------------------------------------------------- §12 trigger guards
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_zero_mean_divergence_does_not_trigger_phase_1():
|
||
"""§12 Trigger 2: ``max(mean, ε)`` guard against div-by-zero
|
||
when divergence is uniformly low. Probe covers this; the
|
||
in-controller guard duplicates it so the controller can be run
|
||
on a fresh corpus without crashing on the first call."""
|
||
pytest.fail("Phase 1 implementation pending; see §12 Trigger 2.")
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_small_sample_does_not_trigger_phase_1():
|
||
"""§12 Trigger 2: ``N_min = 30`` floor below which the variance
|
||
trigger does not fire. Pins behavior on early-corpus deployments
|
||
where sample-count noise would dominate any signal."""
|
||
pytest.fail("Phase 1 implementation pending; see §12 Trigger 2.")
|
||
|
||
|
||
# ----------------------------------------------------------- proposals, not mutations
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_propose_not_mutate_memory_root():
|
||
"""§4.4 + §13 step 12: controller emits a MemoryRoot update
|
||
*proposal*; the existing memory-root write path validates and
|
||
commits. Direct mutation would let the controller poison hard-
|
||
hashed state without going through validation."""
|
||
pytest.fail("Phase 1 implementation pending; see §4.4.")
|
||
|
||
|
||
@skip_until_phase_1
|
||
def test_propose_not_mutate_self_model():
|
||
"""§4.4 + §13 step 12: controller emits a SelfModel update
|
||
*proposal*; the existing #000014 selfmodel write path validates
|
||
and commits. Same boundary as MemoryRoot."""
|
||
pytest.fail("Phase 1 implementation pending; see §4.4 + #000014.")
|