#000037 Phase 0: §12 trigger probe + Phase 1 test scaffolding
Phase 0 of ticket #000037 (Prometheus-Σ recursive falsification controller) is doc-only and gates Phase 1 on §12 measured-pressure triggers. This commit lands the empirical-evidence harness fox needs for the go/no-go decision, plus the §16.2 test scaffolding so the contract is discoverable from the test runner today. bench/prometheus_sigma_trigger_probe.py ======================================= Read-only walk of audit_events + capital_ledger across shards. Reports each §12 trigger: - Trigger 1 (branch density, ≥4 branches/checkpoint): looks for a fork_score branch-set table; reports "no data — single-validator ForkScore Phase 1a" when absent. Trigger structurally cannot fire until #000012 multi-branch persistence lands. - Trigger 2 (divergence variance, N≥30 + ratio>0.5 OR abs>0.10): aggregates providence_canonical_witness audit-event bodies, maps each agreement_label per #000028 §1.2 to a binary LLM-divergence score, computes mean/stddev/ratio. Defensive on the max(mean,ε) guard from §12 itself. - Trigger 3 (witness cost share > 0.30): aggregates capital_ledger rows; uses `material` (kWh proxy) as the canonical compute axis; surfaces both `material` and `financial` so fox can pick a different form if needed. Tags the report with the caveat that the current ledger reflects ad-hoc activity, not a controlled #000026 sweep. - Trigger 4 (operator mission need): n/a — operator decision. Pure measurement, no LLM, no schema change, no mutation. Output is a markdown report at $(PROMETHEUS_PROBE_OUT) (default bench/results/prometheus-sigma-triggers-<utc-date>.md). Wired via `make prometheus-trigger-probe`. bench/results/prometheus-sigma-triggers-2026-05-10.md ===================================================== First captured baseline. Verdict on current shards: Trigger 1: NO (no fork_score branch-set table; #000012 Phase 1a is single-validator) Trigger 2: NO (16 samples; N_min=30. But mean=0.625, σ=0.5, ratio=0.8 — both ratio AND abs floors would fire if N reaches 30. Signal is there; just needs more samples.) Trigger 3: NO (witness/total material = 0.005, well under 0.30 threshold. Caveated: ledger is ad-hoc, not a controlled #000026 sweep.) Trigger 4: n/a (operator-stated) Empirical answer: no §12 measured-pressure trigger has fired yet. Phase 1 of #000037 remains paper-only unless fox invokes Trigger 4. tests/test_prometheus_sigma.py ============================== 17 skip-stubs pinning the §16.2 acceptance contract. Each test: - Collects today via `pytest --collect-only` so the test surface is discoverable. - Skips with reason "Phase 1 not landed; controller module arborist/v9/prometheus.py absent" until that import succeeds. - Carries a one-sentence intent line tying it back to a numbered ticket section (§5 / §6 / §7 / §10 / §13 / §14 / §16.1 / §4.4). When Phase 1 lands, the implementer adds the controller module under arborist/v9/prometheus.py per §13; CONTROLLER_AVAILABLE becomes True; each test gets its body filled in. The names, intents, and skip-reason strings are the durable contract. Hygiene ======= - make test → 1623 passed, 45 skipped (was 28; +17 new skips). - make chain-check-shards → 0 breaks across all 7 shards. - No code touched outside the new probe + scaffolding files + Makefile target wiring. fox's in-flight #000037 ticket modifications left untouched.
This commit is contained in:
parent
f6951f4472
commit
c422216428
4 changed files with 768 additions and 0 deletions
11
Makefile
11
Makefile
|
|
@ -33,6 +33,7 @@ SEARCH_Q ?= computer
|
|||
chain-check chain-check-shards \
|
||||
falsify burn burn-kindergarten inspect bootstrap-crawler test-crawler crawl-ingest \
|
||||
recrawl-check bench-qa bench-qa-smoke bench-qa-progressive-and \
|
||||
prometheus-trigger-probe \
|
||||
bootstrap-math clean clean-db clean-data help \
|
||||
textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \
|
||||
crawl-textbooks crawl-textbooks-stats textbook textbook-list
|
||||
|
|
@ -229,6 +230,16 @@ test-live: bootstrap ## live QA quality tests against Hermes (gated; -n auto par
|
|||
ARBORIST_LIVE_TESTS=1 ARBORIST_LIVE_SHARDS_DIR=$(SHARDS_DIR) \
|
||||
.venv/bin/pytest tests/test_qa_quality_live.py -v -n auto
|
||||
|
||||
# Prometheus-Σ §12 trigger probe (ticket #000037 Phase 0). Read-only
|
||||
# walk of audit_events + capital_ledger across shards; reports
|
||||
# whether any of triggers 1-3 have fired empirically. Output is the
|
||||
# evidence fox uses to decide go/no-go on Phase 1. Pure measurement.
|
||||
PROMETHEUS_PROBE_OUT ?= bench/results/prometheus-sigma-triggers-$(shell date -u +%Y-%m-%d).md
|
||||
prometheus-trigger-probe: bootstrap ## #000037 §12 measured-pressure probe → markdown report
|
||||
PYTHONUNBUFFERED=1 $(PY) bench/prometheus_sigma_trigger_probe.py \
|
||||
--shards-dir $(SHARDS_DIR) \
|
||||
--out $(PROMETHEUS_PROBE_OUT)
|
||||
|
||||
# Concept-layer backfill targets. Each runs an extractor across every
|
||||
# wiki shard; per-shard work is independent so we use GNU-parallel-
|
||||
# style concurrency with `xargs -P` to overlap the slow paths
|
||||
|
|
|
|||
484
bench/prometheus_sigma_trigger_probe.py
Normal file
484
bench/prometheus_sigma_trigger_probe.py
Normal file
|
|
@ -0,0 +1,484 @@
|
|||
"""Prometheus-Σ §12 trigger-probe — ticket #000037 Phase 0 evidence-gather.
|
||||
|
||||
Phase 1 of #000037 (the recursive falsification controller) gates on
|
||||
one of four §12 triggers firing. Triggers 1-3 are measurable from
|
||||
already-committed state; trigger 4 is operator decision. This script
|
||||
walks every shard's ``audit_events`` and ``capital_ledger`` tables,
|
||||
computes the §12 quantities, and emits a markdown report.
|
||||
|
||||
Pure measurement. No mutation. No LLM call. No schema change. The
|
||||
report is the deliverable; fox uses it to decide go/no-go on Phase 1.
|
||||
|
||||
The probe is defensive about data-availability: it explicitly reports
|
||||
"no data" when a trigger structurally cannot fire yet (e.g. trigger 1
|
||||
on a single-validator deployment with no fork-score cache table)
|
||||
rather than synthesising a false signal.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import sqlite3
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
# §12 thresholds, transcribed verbatim from the ticket.
|
||||
DIVERGENCE_N_MIN = 30
|
||||
DIVERGENCE_VARIANCE_RATIO_FLOOR = 0.5
|
||||
DIVERGENCE_ABS_STDDEV_FLOOR = 0.10
|
||||
WITNESS_COST_SHARE_FLOOR = 0.30
|
||||
DIVERGENCE_EPSILON = 1e-6 # max(mean, ε) guard against div-by-zero
|
||||
|
||||
|
||||
def _read_witness_events(shards: list[Path]) -> list[dict]:
|
||||
"""Pull every ``providence_canonical_witness`` body across shards."""
|
||||
bodies: list[dict] = []
|
||||
for sp in shards:
|
||||
try:
|
||||
conn = sqlite3.connect(sp)
|
||||
except sqlite3.OperationalError:
|
||||
continue
|
||||
try:
|
||||
cur = conn.execute(
|
||||
"SELECT body, ts FROM audit_events "
|
||||
"WHERE event_type = 'providence_canonical_witness' "
|
||||
"ORDER BY seq"
|
||||
)
|
||||
for row in cur:
|
||||
try:
|
||||
body = json.loads(row[0])
|
||||
except (TypeError, json.JSONDecodeError):
|
||||
continue
|
||||
body["__ts"] = row[1]
|
||||
body["__shard"] = sp.name
|
||||
bodies.append(body)
|
||||
except sqlite3.OperationalError:
|
||||
# Pre-#000028 shard with no audit_events table or schema gap;
|
||||
# not a probe-failure mode.
|
||||
pass
|
||||
finally:
|
||||
conn.close()
|
||||
return bodies
|
||||
|
||||
|
||||
def _read_capital_ledger(shards: list[Path]) -> list[dict]:
|
||||
"""Aggregate every ``capital_ledger`` row across shards."""
|
||||
rows: list[dict] = []
|
||||
for sp in shards:
|
||||
try:
|
||||
conn = sqlite3.connect(sp)
|
||||
conn.row_factory = sqlite3.Row
|
||||
except sqlite3.OperationalError:
|
||||
continue
|
||||
try:
|
||||
cur = conn.execute(
|
||||
"SELECT op_type, living, material, financial, "
|
||||
" intellectual, experiential, social, cultural, "
|
||||
" spiritual, recorded_at "
|
||||
"FROM capital_ledger"
|
||||
)
|
||||
for r in cur:
|
||||
d = dict(r)
|
||||
d["__shard"] = sp.name
|
||||
rows.append(d)
|
||||
except sqlite3.OperationalError:
|
||||
pass
|
||||
finally:
|
||||
conn.close()
|
||||
return rows
|
||||
|
||||
|
||||
def _has_fork_score_table(shards: list[Path]) -> bool:
|
||||
for sp in shards:
|
||||
try:
|
||||
conn = sqlite3.connect(sp)
|
||||
row = conn.execute(
|
||||
"SELECT name FROM sqlite_master "
|
||||
"WHERE type='table' AND name LIKE '%fork_score%' LIMIT 1"
|
||||
).fetchone()
|
||||
conn.close()
|
||||
if row:
|
||||
return True
|
||||
except sqlite3.OperationalError:
|
||||
continue
|
||||
return False
|
||||
|
||||
|
||||
# -------------------------------------------------------------- triggers
|
||||
|
||||
|
||||
def trigger_1_branch_density(shards: list[Path]) -> dict:
|
||||
"""Trigger 1 — branch density: ≥4 candidate branches per checkpoint.
|
||||
|
||||
Requires multi-branch ForkScore (#000012) to be persisting candidate
|
||||
sets. Phase 1a (closed 2026-05-08) is single-validator, so no
|
||||
branch-set table exists; the trigger structurally cannot fire yet.
|
||||
"""
|
||||
has_table = _has_fork_score_table(shards)
|
||||
return {
|
||||
"trigger": 1,
|
||||
"name": "branch density (≥4 branches/checkpoint)",
|
||||
"fires": False,
|
||||
"reason": (
|
||||
"no fork_score branch-set table found across shards; "
|
||||
"ForkScore Phase 1a is single-validator. Multi-branch "
|
||||
"fork-score persistence is the prerequisite — see #000012."
|
||||
)
|
||||
if not has_table
|
||||
else "branch-set table found but density check not yet implemented",
|
||||
"data_available": has_table,
|
||||
}
|
||||
|
||||
|
||||
def _divergence_score(agreement_label: str | None) -> float | None:
|
||||
"""Map a #000028 §1.2 agreement label to a binary LLM-divergence
|
||||
score for trigger 2.
|
||||
|
||||
Returns:
|
||||
- 0.0 when the LLM converged with the kernel
|
||||
- 1.0 when the LLM diverged from the kernel
|
||||
- None when the LLM modality was absent / unparseable, so the
|
||||
event tells us nothing about LLM divergence.
|
||||
|
||||
The N_min=30 sample population is the set of events that produced
|
||||
a 0/1 score. KERNEL-CACHE-AGREE excludes LLM (unparseable) so it
|
||||
contributes no signal to trigger 2; CACHE-DRIFT means K=L (LLM
|
||||
agreed with kernel) so it counts as 0.0.
|
||||
"""
|
||||
if not agreement_label:
|
||||
return None
|
||||
label = agreement_label.upper()
|
||||
LLM_AGREE = {
|
||||
"STRICT-WITNESSED", # K=C=L
|
||||
"KERNEL-LLM-AGREE", # no cache; K=L
|
||||
"CACHE-DRIFT", # K=L; cache stale (counts as agree for LLM)
|
||||
}
|
||||
LLM_DIVERGED = {
|
||||
"LLM-DIVERGED", # K=C; L≠K
|
||||
"KERNEL-LLM-DIVERGED", # no cache; L≠K
|
||||
"LLM-AND-CACHE-DIVERGED", # only K matches itself
|
||||
}
|
||||
LLM_ABSENT = {
|
||||
"KERNEL-CACHE-AGREE", # LLM unparseable; no signal
|
||||
}
|
||||
if label in LLM_AGREE:
|
||||
return 0.0
|
||||
if label in LLM_DIVERGED:
|
||||
return 1.0
|
||||
if label in LLM_ABSENT:
|
||||
return None
|
||||
# Unknown label — substring fallback so renames don't silently
|
||||
# exclude data. Errs on the side of inclusion.
|
||||
if "DIVERGED" in label or "DRIFT_LLM" in label:
|
||||
return 1.0
|
||||
if "AGREE" in label or "WITNESSED" in label:
|
||||
return 0.0
|
||||
return None
|
||||
|
||||
|
||||
def _stddev(samples: list[float]) -> float:
|
||||
n = len(samples)
|
||||
if n < 2:
|
||||
return 0.0
|
||||
mean = sum(samples) / n
|
||||
sq = sum((s - mean) ** 2 for s in samples)
|
||||
return math.sqrt(sq / (n - 1))
|
||||
|
||||
|
||||
def trigger_2_divergence_variance(events: list[dict]) -> dict:
|
||||
"""Trigger 2 — divergence variance: N≥30, ratio>0.5 or abs>0.10.
|
||||
|
||||
Concrete signal from §12:
|
||||
sample_count >= N_min
|
||||
AND (stddev/max(mean, ε) > 0.5 OR absolute stddev > 0.10)
|
||||
"""
|
||||
scores = [
|
||||
s for s in (_divergence_score(e.get("agreement_label")) for e in events)
|
||||
if s is not None
|
||||
]
|
||||
n = len(scores)
|
||||
if n == 0:
|
||||
return {
|
||||
"trigger": 2,
|
||||
"name": "divergence variance",
|
||||
"fires": False,
|
||||
"reason": (
|
||||
"no providence_canonical_witness events with kernel-vs-LLM "
|
||||
"outcomes found; #000028 not exercised on this corpus yet"
|
||||
),
|
||||
"data_available": False,
|
||||
"n": 0,
|
||||
}
|
||||
mean = sum(scores) / n
|
||||
sd = _stddev(scores)
|
||||
ratio = sd / max(mean, DIVERGENCE_EPSILON)
|
||||
abs_fired = sd > DIVERGENCE_ABS_STDDEV_FLOOR
|
||||
ratio_fired = ratio > DIVERGENCE_VARIANCE_RATIO_FLOOR
|
||||
n_ok = n >= DIVERGENCE_N_MIN
|
||||
fires = n_ok and (abs_fired or ratio_fired)
|
||||
if not n_ok:
|
||||
reason = f"only {n} samples; N_min = {DIVERGENCE_N_MIN}"
|
||||
elif fires:
|
||||
which = []
|
||||
if ratio_fired:
|
||||
which.append(f"ratio {ratio:.3f} > {DIVERGENCE_VARIANCE_RATIO_FLOOR}")
|
||||
if abs_fired:
|
||||
which.append(f"abs σ {sd:.3f} > {DIVERGENCE_ABS_STDDEV_FLOOR}")
|
||||
reason = "; ".join(which)
|
||||
else:
|
||||
reason = (
|
||||
f"N={n} ≥ {DIVERGENCE_N_MIN}, but "
|
||||
f"ratio {ratio:.3f} ≤ {DIVERGENCE_VARIANCE_RATIO_FLOOR} "
|
||||
f"and abs σ {sd:.3f} ≤ {DIVERGENCE_ABS_STDDEV_FLOOR}"
|
||||
)
|
||||
return {
|
||||
"trigger": 2,
|
||||
"name": "divergence variance",
|
||||
"fires": fires,
|
||||
"reason": reason,
|
||||
"data_available": True,
|
||||
"n": n,
|
||||
"mean": round(mean, 4),
|
||||
"stddev": round(sd, 4),
|
||||
"ratio": round(ratio, 4),
|
||||
}
|
||||
|
||||
|
||||
def trigger_3_witness_cost_share(rows: list[dict]) -> dict:
|
||||
"""Trigger 3 — witness cost share: witness_compute / total_compute > 0.30.
|
||||
|
||||
"Compute" is read as the ``material`` column (kWh proxy), which is
|
||||
the closest analogue to "compute" in the 8-form ledger. The ticket
|
||||
leaves the form vague; surfacing both ``material`` and ``financial``
|
||||
in the report so fox can choose if the threshold should be against
|
||||
a different form.
|
||||
|
||||
Per §12 the trigger requires #000026 baseline to exist (it does;
|
||||
bench/results/real-shard-baseline.md, 2026-05-08). The ledger
|
||||
here is whatever has been recorded to date — not a controlled
|
||||
workload — so the ratio is informational pending a controlled
|
||||
sweep.
|
||||
"""
|
||||
if not rows:
|
||||
return {
|
||||
"trigger": 3,
|
||||
"name": "witness cost share (material > 0.30)",
|
||||
"fires": False,
|
||||
"reason": "capital_ledger empty across all shards",
|
||||
"data_available": False,
|
||||
"witness_material": 0.0,
|
||||
"total_material": 0.0,
|
||||
}
|
||||
witness_material = sum(
|
||||
r["material"] or 0.0 for r in rows if r["op_type"] == "canonical_witness"
|
||||
)
|
||||
total_material = sum(r["material"] or 0.0 for r in rows)
|
||||
witness_financial = sum(
|
||||
r["financial"] or 0.0 for r in rows if r["op_type"] == "canonical_witness"
|
||||
)
|
||||
total_financial = sum(r["financial"] or 0.0 for r in rows)
|
||||
if total_material == 0.0:
|
||||
return {
|
||||
"trigger": 3,
|
||||
"name": "witness cost share (material > 0.30)",
|
||||
"fires": False,
|
||||
"reason": "total compute denominator is zero (per §12: do not divide)",
|
||||
"data_available": False,
|
||||
"witness_material": witness_material,
|
||||
"total_material": 0.0,
|
||||
}
|
||||
ratio = witness_material / total_material
|
||||
fires = ratio > WITNESS_COST_SHARE_FLOOR
|
||||
return {
|
||||
"trigger": 3,
|
||||
"name": "witness cost share (material > 0.30)",
|
||||
"fires": fires,
|
||||
"reason": (
|
||||
f"witness material / total material = "
|
||||
f"{witness_material:.6f} / {total_material:.6f} = {ratio:.3f}"
|
||||
),
|
||||
"data_available": True,
|
||||
"witness_material": round(witness_material, 6),
|
||||
"total_material": round(total_material, 6),
|
||||
"witness_financial": round(witness_financial, 6),
|
||||
"total_financial": round(total_financial, 6),
|
||||
"ratio": round(ratio, 4),
|
||||
"n_witness_rows": sum(
|
||||
1 for r in rows if r["op_type"] == "canonical_witness"
|
||||
),
|
||||
"n_total_rows": len(rows),
|
||||
"caveat": (
|
||||
"ledger reflects ad-hoc activity, not a controlled #000026 sweep — "
|
||||
"ratio is informational. Re-run after a #000026 baseline sweep "
|
||||
"for a workload-anchored answer."
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def trigger_4_operator_need() -> dict:
|
||||
return {
|
||||
"trigger": 4,
|
||||
"name": "operator mission need",
|
||||
"fires": None,
|
||||
"reason": "operator decision; not measurable from committed state",
|
||||
"data_available": False,
|
||||
}
|
||||
|
||||
|
||||
# -------------------------------------------------------------- report
|
||||
|
||||
|
||||
def render_markdown(
|
||||
shards_dir: Path,
|
||||
shards: list[Path],
|
||||
triggers: list[dict],
|
||||
events: list[dict],
|
||||
rows: list[dict],
|
||||
) -> str:
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
out = []
|
||||
out.append("# Prometheus-Σ §12 trigger probe")
|
||||
out.append("")
|
||||
out.append(f"**Date:** {now}")
|
||||
out.append(f"**Shards directory:** `{shards_dir}`")
|
||||
out.append(f"**Shards walked:** {len(shards)} "
|
||||
f"({', '.join(s.name for s in shards)})")
|
||||
out.append("")
|
||||
out.append(
|
||||
"Phase 1 of ticket #000037 gates on one §12 trigger firing. "
|
||||
"This probe walks `audit_events` + `capital_ledger` and reports "
|
||||
"the empirical state of triggers 1–3. Trigger 4 is operator-stated."
|
||||
)
|
||||
out.append("")
|
||||
out.append("## Verdict")
|
||||
out.append("")
|
||||
out.append("| # | Trigger | Fires? | Reason |")
|
||||
out.append("|---|---------|--------|--------|")
|
||||
for t in triggers:
|
||||
fires = t["fires"]
|
||||
if fires is True:
|
||||
mark = "**YES**"
|
||||
elif fires is False:
|
||||
mark = "no"
|
||||
else:
|
||||
mark = "n/a"
|
||||
out.append(f"| {t['trigger']} | {t['name']} | {mark} | {t['reason']} |")
|
||||
out.append("")
|
||||
any_fires = any(t["fires"] is True for t in triggers)
|
||||
if any_fires:
|
||||
out.append("**Phase 1 trigger has fired.** Prometheus-Σ Phase 1 may proceed.")
|
||||
else:
|
||||
out.append(
|
||||
"**No measurable §12 trigger has fired yet.** Phase 1 of #000037 "
|
||||
"remains paper-only; trigger 4 (operator-stated need) bypasses the "
|
||||
"measured-pressure gates if fox has explicit mission need."
|
||||
)
|
||||
out.append("")
|
||||
out.append("## Trigger 1 — branch density")
|
||||
out.append("")
|
||||
t1 = triggers[0]
|
||||
out.append(f"- Data available: {t1['data_available']}")
|
||||
out.append(f"- {t1['reason']}")
|
||||
out.append("")
|
||||
out.append("## Trigger 2 — divergence variance")
|
||||
out.append("")
|
||||
t2 = triggers[1]
|
||||
if t2.get("data_available"):
|
||||
out.append(f"- Sample count (N): {t2['n']} "
|
||||
f"(N_min = {DIVERGENCE_N_MIN})")
|
||||
out.append(f"- Mean divergence rate: {t2['mean']}")
|
||||
out.append(f"- Stddev: {t2['stddev']}")
|
||||
out.append(f"- Ratio (σ/mean): {t2['ratio']}")
|
||||
out.append(f"- Threshold ratio: > {DIVERGENCE_VARIANCE_RATIO_FLOOR} "
|
||||
f"OR absolute σ > {DIVERGENCE_ABS_STDDEV_FLOOR}")
|
||||
out.append(f"- {t2['reason']}")
|
||||
if events:
|
||||
from collections import Counter
|
||||
agree_counts = Counter(e.get("agreement_label", "?") for e in events)
|
||||
out.append("")
|
||||
out.append("Agreement-label distribution across all shards:")
|
||||
out.append("")
|
||||
out.append("| label | count |")
|
||||
out.append("|-------|-------|")
|
||||
for k, v in sorted(agree_counts.items(), key=lambda kv: (-kv[1], kv[0])):
|
||||
out.append(f"| `{k}` | {v} |")
|
||||
out.append("")
|
||||
out.append("## Trigger 3 — witness cost share")
|
||||
out.append("")
|
||||
t3 = triggers[2]
|
||||
if t3.get("data_available"):
|
||||
out.append(f"- Witness material (kWh proxy): "
|
||||
f"{t3['witness_material']}")
|
||||
out.append(f"- Total material: {t3['total_material']}")
|
||||
out.append(f"- Ratio: {t3['ratio']} "
|
||||
f"(threshold > {WITNESS_COST_SHARE_FLOOR})")
|
||||
out.append(f"- Witness financial: {t3['witness_financial']}")
|
||||
out.append(f"- Total financial: {t3['total_financial']}")
|
||||
out.append(f"- Witness rows: {t3['n_witness_rows']}")
|
||||
out.append(f"- Total ledger rows: {t3['n_total_rows']}")
|
||||
out.append(f"- {t3['reason']}")
|
||||
if t3.get("caveat"):
|
||||
out.append("")
|
||||
out.append(f"**Caveat:** {t3['caveat']}")
|
||||
out.append("")
|
||||
out.append("## Trigger 4 — operator mission need")
|
||||
out.append("")
|
||||
out.append("- Operator-stated; not measurable from committed state.")
|
||||
out.append("- Bypasses §12 measured-pressure gates per ticket §12.")
|
||||
out.append("")
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
# -------------------------------------------------------------- main
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
p.add_argument(
|
||||
"--shards-dir",
|
||||
type=Path,
|
||||
default=Path.home() / ".arborist" / "shards",
|
||||
help="Directory of *.db shard files (default: ~/.arborist/shards)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--out",
|
||||
type=Path,
|
||||
default=None,
|
||||
help="Write markdown report to this path. Default: stdout.",
|
||||
)
|
||||
args = p.parse_args(argv)
|
||||
|
||||
if not args.shards_dir.exists():
|
||||
print(f"shards-dir not found: {args.shards_dir}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
shards = sorted(args.shards_dir.glob("*.db"))
|
||||
if not shards:
|
||||
print(f"no .db files in {args.shards_dir}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
events = _read_witness_events(shards)
|
||||
rows = _read_capital_ledger(shards)
|
||||
|
||||
triggers = [
|
||||
trigger_1_branch_density(shards),
|
||||
trigger_2_divergence_variance(events),
|
||||
trigger_3_witness_cost_share(rows),
|
||||
trigger_4_operator_need(),
|
||||
]
|
||||
|
||||
md = render_markdown(args.shards_dir, shards, triggers, events, rows)
|
||||
if args.out:
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.out.write_text(md, encoding="utf-8")
|
||||
print(f"wrote {args.out}")
|
||||
else:
|
||||
print(md)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
59
bench/results/prometheus-sigma-triggers-2026-05-10.md
Normal file
59
bench/results/prometheus-sigma-triggers-2026-05-10.md
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
# Prometheus-Σ §12 trigger probe
|
||||
|
||||
**Date:** 2026-05-10T11:22:04Z
|
||||
**Shards directory:** `/home/fox/.arborist/shards`
|
||||
**Shards walked:** 7 (000.db, 001.db, 002.db, 003.db, crawl_appliedcombinatorics_org.db, qa.db, snapshots.db)
|
||||
|
||||
Phase 1 of ticket #000037 gates on one §12 trigger firing. This probe walks `audit_events` + `capital_ledger` and reports the empirical state of triggers 1–3. Trigger 4 is operator-stated.
|
||||
|
||||
## Verdict
|
||||
|
||||
| # | Trigger | Fires? | Reason |
|
||||
|---|---------|--------|--------|
|
||||
| 1 | branch density (≥4 branches/checkpoint) | no | no fork_score branch-set table found across shards; ForkScore Phase 1a is single-validator. Multi-branch fork-score persistence is the prerequisite — see #000012. |
|
||||
| 2 | divergence variance | no | only 16 samples; N_min = 30 |
|
||||
| 3 | witness cost share (material > 0.30) | no | witness material / total material = 0.002870 / 0.535350 = 0.005 |
|
||||
| 4 | operator mission need | n/a | operator decision; not measurable from committed state |
|
||||
|
||||
**No measurable §12 trigger has fired yet.** Phase 1 of #000037 remains paper-only; trigger 4 (operator-stated need) bypasses the measured-pressure gates if fox has explicit mission need.
|
||||
|
||||
## Trigger 1 — branch density
|
||||
|
||||
- Data available: False
|
||||
- no fork_score branch-set table found across shards; ForkScore Phase 1a is single-validator. Multi-branch fork-score persistence is the prerequisite — see #000012.
|
||||
|
||||
## Trigger 2 — divergence variance
|
||||
|
||||
- Sample count (N): 16 (N_min = 30)
|
||||
- Mean divergence rate: 0.625
|
||||
- Stddev: 0.5
|
||||
- Ratio (σ/mean): 0.8
|
||||
- Threshold ratio: > 0.5 OR absolute σ > 0.1
|
||||
- only 16 samples; N_min = 30
|
||||
|
||||
Agreement-label distribution across all shards:
|
||||
|
||||
| label | count |
|
||||
|-------|-------|
|
||||
| `KERNEL-LLM-DIVERGED` | 5 |
|
||||
| `LLM-DIVERGED` | 5 |
|
||||
| `KERNEL-LLM-AGREE` | 3 |
|
||||
| `STRICT-WITNESSED` | 3 |
|
||||
|
||||
## Trigger 3 — witness cost share
|
||||
|
||||
- Witness material (kWh proxy): 0.00287
|
||||
- Total material: 0.53535
|
||||
- Ratio: 0.0054 (threshold > 0.3)
|
||||
- Witness financial: 2e-06
|
||||
- Total financial: 2e-06
|
||||
- Witness rows: 16
|
||||
- Total ledger rows: 18
|
||||
- witness material / total material = 0.002870 / 0.535350 = 0.005
|
||||
|
||||
**Caveat:** ledger reflects ad-hoc activity, not a controlled #000026 sweep — ratio is informational. Re-run after a #000026 baseline sweep for a workload-anchored answer.
|
||||
|
||||
## Trigger 4 — operator mission need
|
||||
|
||||
- Operator-stated; not measurable from committed state.
|
||||
- Bypasses §12 measured-pressure gates per ticket §12.
|
||||
214
tests/test_prometheus_sigma.py
Normal file
214
tests/test_prometheus_sigma.py
Normal file
|
|
@ -0,0 +1,214 @@
|
|||
"""Phase 1 test surface for ticket #000037 (Prometheus-Σ controller).
|
||||
|
||||
Phase 0 is doc-only — these tests are the **scaffolding** the §16.2
|
||||
implementation contract names. All of them skip until Phase 1 lands
|
||||
the controller module (``arborist/v9/prometheus.py`` per §13).
|
||||
|
||||
Why land scaffolding while Phase 0 is still doc-only:
|
||||
|
||||
- The 17 named tests are the §16.2 acceptance contract; pinning them
|
||||
here means a future shift can't drift the contract by accident.
|
||||
- ``pytest --collect-only`` lists them, so the test surface is
|
||||
discoverable from the test runner today rather than buried in a
|
||||
ticket.
|
||||
- When Phase 1 lands, the implementer flips
|
||||
``CONTROLLER_AVAILABLE = True``, drops the body of each test, and
|
||||
the contract enforces itself.
|
||||
|
||||
Per CLAUDE.md "no half-finished implementations": these stubs are
|
||||
*explicitly* documented as scaffolding. Each test contains a one-
|
||||
sentence intent line tied to the controller invariant it pins; that
|
||||
intent is what the eventual implementation must satisfy.
|
||||
|
||||
Phase 1 trigger gating (§12) is a separate concern measured by
|
||||
``bench/prometheus_sigma_trigger_probe.py`` — these tests will run
|
||||
regardless of trigger state once the module is in place.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
try:
|
||||
from arborist.v9 import prometheus as _prom # noqa: F401
|
||||
CONTROLLER_AVAILABLE = True
|
||||
except ImportError:
|
||||
CONTROLLER_AVAILABLE = False
|
||||
|
||||
skip_until_phase_1 = pytest.mark.skipif(
|
||||
not CONTROLLER_AVAILABLE,
|
||||
reason=(
|
||||
"ticket #000037 Phase 1 not landed; controller module "
|
||||
"arborist/v9/prometheus.py absent. Skip is the contract — "
|
||||
"implementer flips this when the module exists."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
# ----------------------------------------------------------- core decision
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_all_vetoed_returns_reject_or_quarantine():
|
||||
"""§14 row 2: all branches hard-vetoed → emit veto reasons; no
|
||||
allocation; label ``REJECT`` or ``QUARANTINE`` (depending on the
|
||||
veto class). Catches the "every branch is poisoned" case."""
|
||||
pytest.fail("Phase 1 implementation pending; see §13 step 3.")
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_zero_budget_returns_deferred():
|
||||
"""§14 row 3: ``B = 0`` → no LLM call; queue sleep if useful;
|
||||
label ``DEFERRED``. Distinguishes "didn't evaluate due to budget"
|
||||
from "evaluated but uncertain" (``MARGINAL``). Per David review
|
||||
point 3."""
|
||||
pytest.fail("Phase 1 implementation pending; see §4.3 + §14.")
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_stable_softmax_no_overflow():
|
||||
"""§5: softmax must use ``exp(z_i − max z) / Σ exp(z_j − max z)``.
|
||||
With large positive utilities (e.g. z = 1000) naïve ``exp(z)``
|
||||
overflows; the normalized form must not. Per David review point 6."""
|
||||
pytest.fail("Phase 1 implementation pending; see §5 + §13 step 5.")
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_negative_payoff_gets_zero_allocation():
|
||||
"""§7 Kelly safety guard: ``b_i ≤ 0`` → zero allocation. A branch
|
||||
with negative expected payoff must not consume budget. Per David
|
||||
review point 8."""
|
||||
pytest.fail("Phase 1 implementation pending; see §7.")
|
||||
|
||||
|
||||
# ----------------------------------------------------------- vetoes
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_unsupported_carrier_quarantines():
|
||||
"""§6 hard-veto class: claim references a carrier modality that
|
||||
no live π* library supports → ``QUARANTINE``. Surfaces missing
|
||||
domain coverage rather than papering over it."""
|
||||
pytest.fail("Phase 1 implementation pending; see §6.")
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_cache_drift_quarantines():
|
||||
"""§6 hard-veto class: cache row whose ``pi_star_ref`` no longer
|
||||
matches a live kernel version → ``QUARANTINE``. Echoes the
|
||||
CACHE-DRIFT outcome from #000028 §1.2; controller surfaces
|
||||
rather than silently re-uses."""
|
||||
pytest.fail("Phase 1 implementation pending; see §6 + #000028 §1.2.")
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_memory_invalidation_above_threshold_escalates():
|
||||
"""§10 Gödel discipline: a branch whose acceptance would
|
||||
invalidate too much committed memory → label ``ESCALATE``, not
|
||||
``REJECT``. The controller explicitly steps aside per the "must
|
||||
never infer" rule. Per David review point 12."""
|
||||
pytest.fail("Phase 1 implementation pending; see §10.")
|
||||
|
||||
|
||||
# ----------------------------------------------------------- difficulty / EMA
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_high_entropy_increases_difficulty():
|
||||
"""§7.1 difficulty update law: high ``H_norm(p)`` → smoothed
|
||||
increase in ``difficulty_ema``. The EMA smoothing keeps the
|
||||
update from overshooting on a single noisy sample. Per David
|
||||
review point 9."""
|
||||
pytest.fail("Phase 1 implementation pending; see §7.1.")
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_low_entropy_decreases_or_preserves_difficulty():
|
||||
"""§7.1 difficulty update law: low ``H_norm(p)`` (controller is
|
||||
confident) → difficulty drops or holds. Symmetric to the
|
||||
high-entropy test; together they pin the EMA's monotonicity."""
|
||||
pytest.fail("Phase 1 implementation pending; see §7.1.")
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_divergence_increases_witness_sampling_recommendation():
|
||||
"""§4.3 output / §17.1: high observed witness divergence → the
|
||||
controller's recommendation field should bump ``canonical_witness_
|
||||
sample_rate`` upward (an advisory; runtime decides whether to
|
||||
apply). Pins the feedback loop into #000028's sample-rate field
|
||||
we landed in 6d20aeb."""
|
||||
pytest.fail("Phase 1 implementation pending; see §4.3.")
|
||||
|
||||
|
||||
# ----------------------------------------------------------- discipline
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_no_llm_call_in_controller():
|
||||
"""§10 Gödel + David review point 12: the controller is a pure
|
||||
function. No ``ChatClient`` import in the call graph; no network
|
||||
socket; no env-var that secretly enables one. The witness module
|
||||
calls the LLM; the controller reads its results. Pins the
|
||||
"LLM is witness, never authority" doctrine."""
|
||||
pytest.fail("Phase 1 implementation pending; see §10 + §16.1.")
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_controller_does_not_modify_cache_key_inputs():
|
||||
"""§4.4 update authority: the controller emits **proposals** for
|
||||
MemoryRoot / SelfModel updates; it never mutates the cache_key
|
||||
8-dim input itself. Pins the schema invariant from #000027:
|
||||
cache_key is computed by the cache_key() function, not by any
|
||||
advisory layer above it. Per David review point 14."""
|
||||
pytest.fail("Phase 1 implementation pending; see §4.4 + §13 step 12.")
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_controller_outputs_advisory_event_only():
|
||||
"""§13 step 10: the controller writes ``controller_decision`` /
|
||||
``controller_difficulty`` / ``controller_budget_allocation`` as
|
||||
sibling tags on ``audit_events`` — they do NOT enter
|
||||
``event_hash`` preimage. Re-running the controller against the
|
||||
same state cannot break the audit chain. Pins the same sibling-
|
||||
table invariant the capital_ledger uses."""
|
||||
pytest.fail("Phase 1 implementation pending; see §13 step 10.")
|
||||
|
||||
|
||||
# ----------------------------------------------------------- §12 trigger guards
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_zero_mean_divergence_does_not_trigger_phase_1():
|
||||
"""§12 Trigger 2: ``max(mean, ε)`` guard against div-by-zero
|
||||
when divergence is uniformly low. Probe covers this; the
|
||||
in-controller guard duplicates it so the controller can be run
|
||||
on a fresh corpus without crashing on the first call."""
|
||||
pytest.fail("Phase 1 implementation pending; see §12 Trigger 2.")
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_small_sample_does_not_trigger_phase_1():
|
||||
"""§12 Trigger 2: ``N_min = 30`` floor below which the variance
|
||||
trigger does not fire. Pins behavior on early-corpus deployments
|
||||
where sample-count noise would dominate any signal."""
|
||||
pytest.fail("Phase 1 implementation pending; see §12 Trigger 2.")
|
||||
|
||||
|
||||
# ----------------------------------------------------------- proposals, not mutations
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_propose_not_mutate_memory_root():
|
||||
"""§4.4 + §13 step 12: controller emits a MemoryRoot update
|
||||
*proposal*; the existing memory-root write path validates and
|
||||
commits. Direct mutation would let the controller poison hard-
|
||||
hashed state without going through validation."""
|
||||
pytest.fail("Phase 1 implementation pending; see §4.4.")
|
||||
|
||||
|
||||
@skip_until_phase_1
|
||||
def test_propose_not_mutate_self_model():
|
||||
"""§4.4 + §13 step 12: controller emits a SelfModel update
|
||||
*proposal*; the existing #000014 selfmodel write path validates
|
||||
and commits. Same boundary as MemoryRoot."""
|
||||
pytest.fail("Phase 1 implementation pending; see §4.4 + #000014.")
|
||||
Loading…
Add table
Add a link
Reference in a new issue