diff --git a/docs/tickets/ticket-000037-prometheus-sigma-recursive-falsification-controller.md b/docs/tickets/ticket-000037-prometheus-sigma-recursive-falsification-controller.md new file mode 100644 index 0000000..9522c93 --- /dev/null +++ b/docs/tickets/ticket-000037-prometheus-sigma-recursive-falsification-controller.md @@ -0,0 +1,600 @@ +# Ticket #000037 — Prometheus-Σ recursive falsification controller (bicameral substrate) + +**Status:** open · awaiting go/no-go (doc-only Phase 0) +**Opened:** 2026-05-09 +**Scope:** Spec a named control law that unifies the recursive- +falsification machinery already shipped across #000012 (ForkScore), +#000014 (SelfModel), #000017 (MemoryRoot), #000018 (soft-hash +threat model), #000020 (capital ledger), #000025 (5F battery), +#000027 (canonical-projection persistence), and #000028 (multi- +modality witness) into a Shannon-entropy + Nash-veto + Kelly- +allocated difficulty controller. Frame the deterministic kernel +and the probabilistic LLM as a **bicameral pair** that probes each +other — left brain / right brain, both witness-only authority — +with an unconscious nightly falsification sweep during ingestion +lulls. Phase 0 is doc-only; implementation phases gate on a +measured-pressure trigger, not a calendar date. +**Audience:** fox + future blackops shifts + downstream Merkle-AGI +v9.x authors + anyone proposing a unified mutation-selection layer. +**Hard constraint:** doc-only in Phase 0 — **no** schema bump, +**no** governance_policy_hash bump, **no** LLM-as-judge anywhere +in the controller. Controller is advisory until a measured trigger +(divergence-rate variance, ForkScore candidate density, witness +budget pressure) fires Phase 1. Bicameral pair preserves the +existing rule from #000028: kernel is ground truth on canonical +shapes; LLM is witness, never authority. The controller does not +defeat Gödel — it side-steps the engineering trap by replacing +global completeness with **local, versioned, replayable +admissibility**. + +--- + +## 1. Problem statement + +Arborist now ships every prerequisite for recursive falsification: + +| Piece | Ticket | Role in falsification loop | +|---|---|---| +| 5F Falsification battery | #000025 | failure → fixture → benchmark | +| Multi-witness canonical agreement (kernel · cache · LLM) | #000028 | divergence detection | +| Canonical projection persistence | #000027 | committed objects to falsify | +| ForkScore (accept / marginal / reject) | #000012 | mutation selection verdict | +| MemoryRoot | #000017 | persistent failure recall | +| SelfModel | #000014 | committed self-claims | +| Capital ledger | #000020 | honest fork cost comparison | +| Soft-hash threat model | #000018 | covert-channel discipline | +| Claim-pack source | #000029 | versioned axiom bundles | +| Surface-ingest cited textbooks | #000031 | warrant promotion substrate | + +What's missing is a **named control law** that stitches them into +one loop. Today: + +- ForkScore picks accept/marginal/reject from precomputed battery + deltas, but does not formalize a probability distribution over + candidate branches, does not allocate compute under a budget, + does not scale witness sampling with measured divergence, and + does not name its own difficulty knob. +- 5F Falsification fixtures accumulate from witness divergence, + but no controller closes the loop by widening search width or + raising the acceptance threshold when divergence-rate climbs. +- Multi-witness sweeps run at a fixed sampling rate (#000028's + capital-ledger follow-up). There is no homeostatic feedback: + when the LLM agrees with the kernel for 1 000 consecutive + canonical-shape calls, the controller should narrow witness + sampling and free compute; when it disagrees on 5 of 8 (the + observed rate at #000028 land), it should widen. +- Capital ledger records cost per state-changing op but is not + consulted as a Kelly-style allocation constraint when the system + has multiple candidate mutations and a finite compute budget. + +The shape of the gap, in one diagram: + +```text +[ forkscore ] [ 5F fixtures ] [ witness divergence ] + \ | / + \ | / + \ | / + v v v + ??? no controller ??? + | + v + [ mutation accept / reject ] +``` + +This ticket spec'd the missing controller. + +--- + +## 2. The bicameral hunch (fox, 2026-05-09 thread) + +Verbatim, from the thread that triggered this ticket: + +> recursive falsification is important hence, we can never really +> separate this and the llm … so the hunch was, we need a +> deterministic and a probabilistic llm side by side, like left vs +> right brain, probing each other. that is really how we think. +> +> Gödel is side-stepped, not defeated, by replacing global +> completeness with local, versioned, replayable admissibility. +> +> i really want to reach for a bicameral mind. an unconscious that +> runs the tree falsifying across nodes during periods of sleep or +> new ingestion … but I think that process will slow me down in +> the short term. +> +> hermes can do about 4 concurrent requests — that would be a +> bottleneck. + +Three load-bearing observations from the thread: + +1. **Bicameral pair.** A deterministic π* kernel and a + probabilistic LLM, side by side, each probing the other. This + is the existing #000028 architecture **named** as a cognitive + pattern rather than a witness pipeline. Naming matters for + downstream clients implementing Arborist conventions. +2. **Unconscious falsification.** A nightly / ingestion-lull sweep + that walks committed nodes and re-runs witness comparison. + Distinct from inference-time witness fan-out (#000028) — runs + on **already-cached** answers, and writes new 5F fixtures only + when previously-strict rows now diverge. +3. **Compute is the bottleneck.** Hermes serves ~4 concurrent + requests. This is the binding constraint that forces the + controller to be Kelly-bounded (allocate compute to high- + payoff, high-confidence branches first) rather than uniformly + sampled. + +Phase 0 names these. It does not implement the unconscious sweep +yet — fox's "slow me down in the short term" gates Phase 1. + +--- + +## 3. Architecture — bicameral pair as left/right brain + +```text + ┌──────────────┐ + │ canonical │ deterministic kernel + │ π* witness │ (left brain — exact, slow, + │ (kernel) │ unhallucinable) + └──────┬───────┘ + │ + │ agreement matrix + │ (canonical bytes equality) + │ + ┌──────┴───────┐ + │ LLM witness │ probabilistic LLM + │ (Hermes) │ (right brain — fluent, + │ │ fast, hallucination-prone) + └──────┬───────┘ + │ + v + ┌──────────────┐ + │ controller │ Prometheus-Σ + │ (this ticket)│ homeostasis + └──────────────┘ +``` + +The bicameral pair is **already implemented** by #000028 — the +ticket's contribution is to (a) name it, (b) commit to "kernel is +ground truth on canonical shapes, LLM is witness never authority" +as a doctrine (already enforced by the verifier; this writes it +down), and (c) add the **unconscious sweep** as a Phase 1 +deliverable that walks committed `providence_cache` rows during +ingest lulls and re-runs the witness fan-out on rows older than a +configurable threshold. + +The unconscious sweep is the new mechanical contribution. Today +#000028 fan-out runs at inference time only. The sweep amortizes +falsification across periods when the operator is not actively +querying — the engineering analogue of sleep consolidation. + +### 3.1 Two sweep targets, not one (fox 2026-05-09) + +Original spec covers `providence_cache` re-witnessing. Fox flagged a +second backlog the controller has to drain: **ingested documents +that bypassed meta-cognition at ingest time**. + +Today's ingest pipeline is intentionally fast: fetch → canonicalize +→ chunk → Merkle commit → audit row → done. No witness fan-out, no +canonical-projection cross-check, no warrant resolution. That speed +is the right trade-off for getting bytes into the queryable Merkle + +FTS5 tree, but it leaves a meta-cognition gap: the substrate has +content it never *probed*. + +The unconscious sweep is where that probe runs. Two distinct sweep +targets, both processed during ingest lulls: + +**Target A — `providence_cache` rows older than threshold τ_qa.** +Re-runs the multi-modality witness fan-out (#000028) on cached Q&A +answers. Catches drift: a Hermes update changes the LLM's +divergence pattern; a kernel bump changes canonical bytes; a peer's +mesh sync delivered new evidence that would have changed the +answer. Threshold τ_qa default: 7 days. + +**Target B — `documents` rows ingested without meta-cognition.** +Walks `documents` where `last_swept_at` is NULL or older than τ_doc, +runs whichever cross-checks apply to the document's content: + +- **Canonical-projection probe.** If chunks contain canonical-shape + language (math expressions, propositional logic, time-series JSON + per the existing preflight regexes), run the matching π* and + compare against any stated answer in the document. Divergence → + candidate falsification fixture. +- **Warrant resolver re-run.** Per `arborist/qa/warrant_resolver.py` + (#000031 Phase 2). If the document is a claim-pack record AND + the cited textbook newly appeared in the shard cluster since + the original ingest, write the missing `derivations.proof_blob` + row. Idempotent at the DB layer. +- **Source-URL freshness probe.** Existing + `arborist/sources/crawler/recrawl-check` does HEAD-only + conditional fetches. The sweep folds this in: docs whose + upstream changed get marked stale; docs whose upstream + vanished get marked gone. Provides drift signal without + paying full re-fetch cost. +- **Witness fan-out on document content.** For documents whose + body contains canonical-shape statements (e.g., a Wikipedia + paragraph asserting "0.1 + 0.2 = 0.3"), run the kernel + LLM + + cache witness pair AS IF the document were a Q&A answer + citing itself. Produces high-value falsification fixtures + for documents the corpus *wrote down* but never *probed*. + +**State tracking.** A new column on `documents`: +`last_swept_at INTEGER`. Updated when the sweep runs cross-checks +on a doc. Schema migration is small (one ALTER TABLE), but lives +behind the §12 phase trigger so it doesn't bump the cache_key +invariant prematurely. + +**Why this matters.** Without target B, the substrate's "I have +this content in my Merkle tree" claim is honest at the byte +level but silent on whether the content was ever tested against +the kernel ground truth. Sleep is when you do the tests you +deferred to keep the day moving. The bicameral pair's +unconscious half is exactly where this catch-up happens. + +--- + +## 4. Controller inputs / state / outputs + +### 4.1 Inputs + +```text +battery deltas: 5S, 5T, 5F, 5R (per candidate branch) +witness stats: agreement_rate, divergence_rate, kernel_error +capital ledger: compute_cost, latency_cost, storage_cost, + witness_cost +memory state: stale_claims, invalidation_frontier +selfmodel state: calibration_delta +risk flags: soft_hash_signal, unsupported_carrier, + source_warrant_level +budget: B (concurrent-Hermes ceiling ≈ 4) +``` + +### 4.2 State + +```text +Σ_t = { + Ω_t, # organism state (#000017 + #000014 roots) + H_t, # branch-distribution entropy + ForkScore_t, # latest verdict map (#000012) + divergence_rate_t, # rolling window from #000028 sweep + capital_budget_t, # remaining compute under B + difficulty_t # named knob this ticket introduces +} +``` + +### 4.3 Outputs + +```text +ACCEPT — branch passes all hard vetoes, score positive, + entropy bounded +MARGINAL — useful but uncertain; queue for unconscious sweep +REJECT — negative score OR hard-regression flag +QUARANTINE — integrity issue (carrier, soft-hash, schema) +UNKNOWN — bounded verifier returned no decision +ESCALATE — strictly stronger verifier or human required +``` + +`UNKNOWN` and `ESCALATE` are the labels that side-step Gödel. See §10. + +--- + +## 5. Shannon-entropy fork-selection + +For each candidate branch `b_i ∈ B`: + +```text +U_i = α·5S_i + β·5T_i + γ·5F_i + ρ·5R_i + − λ·C_i − μ·X_i − ν·D_i +``` + +where `C_i` is capital cost, `X_i` is regression / security +penalty, `D_i` is witness divergence. Branch distribution: + +```text +p_i = exp(η · U_i) / Σ_j exp(η · U_j) +H(p) = − Σ_i p_i · log p_i +``` + +The difficulty knob `difficulty_t` directly modulates `η`, search +depth, witness sampling rate, and branch count: + +```text +high H → widen search, raise witness sampling, lower acceptance + confidence, allocate more compute +low H → narrow search, reduce witness sampling, conserve + compute +↑ divergence → raise difficulty, require more verification +↓ divergence → drop difficulty, allow fast path +``` + +This is the homeostatic loop: divergence-rate is the thermostat; +search width and witness sampling are the heat sinks. + +--- + +## 6. Nash-style multi-validator veto + +The bicameral pair generalizes to a multi-validator equilibrium. +Each validator has a different payoff: + +```text +kernel: wants exactness +LLM: wants plausible generation +cache: wants consistency with prior commitments +capital ledger: wants low cost +security check: wants low attack surface +benchmark: wants performance +``` + +A branch is acceptable iff **no critical validator vetoes** and +the equilibrium is stable under the scoring rule. Hard vetoes: + +```text +unsupported carrier → QUARANTINE +verifier failure → REJECT +hard regression → REJECT +cache drift detected → QUARANTINE +soft-hash signal above threshold → QUARANTINE +memory invalidation > κ → ESCALATE +``` + +This is the formal version of "balance computation with +tractability": no objective wins by destroying the others. + +--- + +## 7. Kelly-bounded compute allocation + +With `p_i` from §5, payoff `b_i`, and `q_i = 1 − p_i`: + +```text +f_i = max(0, (p_i · b_i − q_i) / b_i) +allocation_i ∝ f_i · exp(η · U_i) +Σ_i allocation_i ≤ B +``` + +`B` is the binding Hermes-concurrency constraint (~4 today). High- +upside, high-confidence branches get most compute; low-payoff or +noisy branches get little or none; vetoed branches get zero. + +--- + +## 8. Cross-chain matrix solver + +Build `A[i, j]` over (mutation × chain) with chains: + +```text +j=1: 5S delta j=6: memory consistency +j=2: 5T delta j=7: selfmodel calibration +j=3: 5F delta j=8: inverse capital cost +j=4: 5R delta j=9: inverse security risk +j=5: witness agreement j=10: inverse regression penalty +``` + +Solve: + +```text +choose i maximizing A_i · w +subject to: + no hard veto + Σ capital cost_i ≤ B + regression_i ≥ −ε + witness divergence_i ≤ δ + memory invalidation_i ≤ κ +``` + +Recursive: after choosing `i`, update `Ω`, recompute `A`, update +`w` from new falsification pressure, repeat. + +--- + +## 9. Counterfactual temporal search (no "time travel") + +The synthesis that triggered this ticket described "time travel +across latent space." Literal time travel: no. The rigorous form: + +```text +Ω_{t+1} = T(Ω_t, Δ_t) +``` + +Branch search simulates `Ω_t → Ω_{t+1}^{(i)} → Ω_{t+2}^{(i)} → …`, +estimates falsification risk, and chooses the branch with best +expected ForkScore under budget. **Bounded counterfactual search +over a committed cognitive state-space** is the engineering name. + +LLMs propose candidate branches not enumerated in current state. +The controller does not trust them until they pass canonicalization, +verification, witness comparison, and benchmark scoring. Every +proposed branch is a witness, not an authority — same rule as +§3. + +--- + +## 10. Gödel discipline — local, versioned, replayable admissibility + +The controller does **not** defeat Gödel. It side-steps the +engineering trap by refusing single-system closure. Concretely: + +| Technique | Gödel response | +|---|---| +| `UNKNOWN`, `UNSUPPORTED`, `UNDECIDED_UNDER_CURRENT_AXIOMS` labels | the system can return "I don't know" instead of hallucinating proof | +| versioned axiom sets via #000029 claim-packs | every result is "warrant under declared formal conditions," not absolute truth | +| escalation across meta-levels (kernel → stronger kernel → human) | undecidable in `S_0` does not block `S_1` from deciding; audit chain records the escalation | +| separation of empirical / formal / source warrants | a kernel result does not warrant an empirical claim and vice versa — no false closure across types | +| canonical bytes + audit chain + replayable fixtures | every admissibility decision is reproducible; consistency is local-and-versioned, not global | + +This is what fox's thread named: **local, versioned, replayable +admissibility** instead of global completeness. The doctrine is +already implicit across #000005 (label ladder), #000027 (canonical +projections), and the verifier's trichotomy. This ticket commits +it to writing as the controller's design substrate. + +--- + +## 11. Bottleneck: Hermes ~4 concurrent requests + +Per fox's thread, Hermes serves about four concurrent requests at +the single live endpoint (`hermes.ai.unturf.com`). This is the +binding `B` in §7. Implications: + +- Witness fan-out (#000028) can not run uncontrolled; the + controller throttles it via Kelly allocation. +- Unconscious sweep (§3) **must not** compete with inference-time + traffic — runs only when concurrent-Hermes utilization is low, + yields to inference traffic on contention. +- Adding deterministic kernels (the "left brain") buys extra + effective concurrency at zero LLM cost: kernel calls are pure- + Python and parallelize freely. Encourages expanding π* domain + coverage (#000015, #000030, #000032) over LLM-side-only growth. + +The bottleneck is also why the controller is Phase 0 / doc-only +today — Phase 1 work that floods the LLM endpoint under a naive +controller would crowd out fox's actual queries. + +--- + +## 12. Phase gating — measured trigger, not calendar date + +Phase 0 (this ticket) is doc-only. Phase 1 opens **only** when one +of these triggers fires: + +1. ForkScore (#000012 Phase 1a) regularly receives ≥ 4 candidate + branches per checkpoint and the operator wants probability- + weighted compute allocation across them. +2. #000028 witness sweep observes divergence-rate variance large + enough that fixed witness sampling starts wasting budget on + stable rows or under-sampling unstable rows. Concrete signal: + rolling divergence-rate stddev / mean > 0.5 over a 7-day + window. +3. Capital ledger (#000020) shows witness-cost > 30 % of total + compute spend on a real-shard workload (#000026 baseline). +4. Operator-stated mission need for the unconscious sweep — e.g. + nightly re-validation of an ingested textbook (#000031). + +Until then, the controller exists on paper. ForkScore, 5F, and +witness sweep continue to operate as independent modules with no +controller mediating them. Five-step §2 ("delete the part") asks +us to confirm pain before adding mechanism; this gate enforces +that. + +--- + +## 13. Minimal algorithm (Phase 1 sketch) + +```text +Algorithm: Prometheus-Σ Recursive Falsification Controller + +Inputs: + Ω_t, B_t (candidate branches), W_t (witnesses), + M_t (memory + selfmodel), C_t (capital ledger), + P_t (policy constraints) + +Step 1: for each b_i, run available deterministic checks +Step 2: build A[i, j] cross-chain matrix +Step 3: apply hard vetoes (carrier, verifier, regression, cache + drift, security, memory) +Step 4: compute U_i +Step 5: p_i = softmax(η · U_i) +Step 6: H(p) +Step 7: adjust difficulty_t: + ↑ H → wider search + ↑ divergence → stricter witness + ↑ cost → narrower search +Step 8: Kelly-bounded allocation under B +Step 9: pick branch: + ACCEPT if positive, bounded risk, low entropy + MARGINAL if useful but uncertain (queue for sweep) + REJECT if negative or vetoed + QUARANTINE if integrity issue +Step 10: commit decision to audit chain (#000017 + #000014 roots) +Step 11: emit 5F falsification fixtures from divergences +Step 12: update MemoryRoot / SelfModel if warranted + +Output: Ω_{t+1}, decision, audit event, new fixtures, updated + difficulty_t +``` + +Phase 1 implementation lives in `arborist/v9/prometheus.py` (new +namespace; sibling to `arborist/v8/fork_score.py`). No schema +changes — the controller is a pure function over already-committed +state. + +--- + +## 14. Scope boundaries — what this ticket does NOT do + +- **Does not** bump `governance_policy_hash` or any cache_key dim. + The controller reads existing state and emits decisions; its + outputs land as audit-event rows, not as cache_key inputs. +- **Does not** introduce LLM-as-judge anywhere. All judgment is + deterministic — entropy, Kelly fraction, ForkScore verdict, + hard-veto checks. The LLM remains a witness only. +- **Does not** replace ForkScore (#000012). ForkScore is the + single-branch verdict; Prometheus-Σ is the multi-branch + allocator that calls ForkScore on each candidate. +- **Does not** implement the unconscious sweep in Phase 0. Phase 1 + spec'd; Phase 1 implementation gated by §12. +- **Does not** claim completeness, claim Gödel-defeat, or claim + AGI. It implements local-versioned-replayable admissibility, full + stop. See §10. +- **Does not** modify `audit_events.event_hash` preimage — sibling + audit-event tags (`controller_decision`, `controller_difficulty`) + per #000020's advisory-ledger pattern. + +--- + +## 15. Open questions for go/no-go + +1. **`α, β, γ, ρ, λ, μ, ν, η` weights.** The synthesis names them + but does not pick values. Phase 0 should ship a default weight- + set tied to fox's mission priority (intellectual capital ≫ + computational throughput ≫ social capital, per BLACKOPS shard). + Phase 1 may make weights pluggable per operator. +2. **Where do divergence-rate windows live?** New table, or fold + into `audit_events` with a query-time aggregation? Trade-off: + new table = clean separation; aggregation = fewer schema + surfaces. Lean toward aggregation — DRY in context. +3. **Unconscious sweep priority vs inference traffic.** Cooperative + yielding via a concurrent-Hermes utilization probe, or a + strict-priority queue? Probe is simpler; queue gives stronger + guarantees. Probe wins for Phase 1; revisit if it under-yields. +4. **Sympathetic naming.** "Prometheus-Σ" comes from the synthesis. + Internal-arborist convention is `arborist/vN/.py`. Plan: + keep "Prometheus-Σ" as the **doctrine name** (used in tickets, + papers, mesh wire docs), but the module lives at + `arborist/v9/prometheus.py` and the table tags are + `controller_*`. Name does not enter cache_key. + +--- + +## 16. Status + +Phase 0 (this ticket): open · awaiting go/no-go. + +When fox approves: this ticket flips to `in progress · Phase 0 +landed in commit ` with no code change — the doc itself is +the deliverable. + +Phase 1 opens when §12 trigger fires. + +--- + +## 17. References + +- This ticket: design-substrate doc, no code yet. +- #000012 (ForkScore): single-branch verdict layer the controller + consumes. +- #000028 (multi-witness): the bicameral pair, pre-named. +- #000027 (canonical projections): committed objects to falsify. +- #000025 (5F battery): falsification fixture sink. +- #000017 (MemoryRoot), #000014 (SelfModel): persistent state the + controller updates. +- #000020 (capital ledger): the `B` budget input. +- #000018 (soft-hash threat model): the `X_i` security penalty + input. +- #000029 (claim-packs): versioned axiom bundles for §10 + admissibility. +- `docs/v8-fork-score.md`: ForkScore reference. +- BLACKOPS shard "Patch the Planet" / "Balance All Workstations": + the mission anchor for the weight-set in §15 Q1. +- 2026-05-09 fox thread: bicameral hunch, Gödel side-step phrase, + Hermes ~4 concurrent bottleneck.