ticket #000037 §3.1: unconscious sweep also drains ingest backlog

Original ticket spec (drafted earlier today by a parallel-shift
session, untracked in working tree until this commit) covers the
unconscious sweep walking providence_cache rows during ingest
lulls. Fox flagged a second backlog the controller has to drain:
ingested DOCUMENTS that bypassed meta-cognition at ingest time.

Today's ingest pipeline is intentionally fast: fetch →
canonicalize → chunk → Merkle commit → audit row → done. No
witness fan-out, no canonical-projection cross-check, no warrant
resolution. That speed is the right trade-off for getting bytes
into the queryable Merkle + FTS5 tree, but it leaves a
meta-cognition gap: the substrate has content it never *probed*.

New §3.1 spells out two sweep targets, both processed during
ingest lulls:

Target A — providence_cache rows older than τ_qa
  Re-runs multi-modality witness fan-out (#000028) on cached
  Q&A answers. Catches drift from Hermes updates / kernel
  bumps / mesh-delivered new evidence. Default τ_qa: 7 days.

Target B — documents rows ingested without meta-cognition
  Walks `documents` where last_swept_at is NULL or older than
  τ_doc, runs cross-checks per document content:
   - Canonical-projection probe (math/logic statements in body
     vs. matching π* canonical answer)
   - Warrant resolver re-run (newly-ingested textbook can
     resolve previously-unresolved claim-pack records)
   - Source-URL freshness probe (HEAD-only via existing
     recrawl-check; mark stale/gone)
   - Witness fan-out on document content (kernel+LLM+cache as
     if doc were a Q&A answer citing itself)

State tracking: new column documents.last_swept_at, schema
migration deferred to §12 phase trigger so it doesn't bump the
cache_key invariant prematurely.

Why this matters: without target B, the substrate's "I have
this content in my Merkle tree" claim is honest at the byte
level but silent on whether the content was ever tested against
kernel ground truth. Sleep is when you do the tests you
deferred to keep the day moving.
This commit is contained in:
russell@unturf.com 2026-05-09 18:34:09 -04:00
parent 416f956734
commit 178cc422e2
No known key found for this signature in database

View file

@ -0,0 +1,600 @@
# Ticket #000037 — Prometheus-Σ recursive falsification controller (bicameral substrate)
**Status:** open · awaiting go/no-go (doc-only Phase 0)
**Opened:** 2026-05-09
**Scope:** Spec a named control law that unifies the recursive-
falsification machinery already shipped across #000012 (ForkScore),
#000014 (SelfModel), #000017 (MemoryRoot), #000018 (soft-hash
threat model), #000020 (capital ledger), #000025 (5F battery),
#000027 (canonical-projection persistence), and #000028 (multi-
modality witness) into a Shannon-entropy + Nash-veto + Kelly-
allocated difficulty controller. Frame the deterministic kernel
and the probabilistic LLM as a **bicameral pair** that probes each
other — left brain / right brain, both witness-only authority —
with an unconscious nightly falsification sweep during ingestion
lulls. Phase 0 is doc-only; implementation phases gate on a
measured-pressure trigger, not a calendar date.
**Audience:** fox + future blackops shifts + downstream Merkle-AGI
v9.x authors + anyone proposing a unified mutation-selection layer.
**Hard constraint:** doc-only in Phase 0 — **no** schema bump,
**no** governance_policy_hash bump, **no** LLM-as-judge anywhere
in the controller. Controller is advisory until a measured trigger
(divergence-rate variance, ForkScore candidate density, witness
budget pressure) fires Phase 1. Bicameral pair preserves the
existing rule from #000028: kernel is ground truth on canonical
shapes; LLM is witness, never authority. The controller does not
defeat Gödel — it side-steps the engineering trap by replacing
global completeness with **local, versioned, replayable
admissibility**.
---
## 1. Problem statement
Arborist now ships every prerequisite for recursive falsification:
| Piece | Ticket | Role in falsification loop |
|---|---|---|
| 5F Falsification battery | #000025 | failure → fixture → benchmark |
| Multi-witness canonical agreement (kernel · cache · LLM) | #000028 | divergence detection |
| Canonical projection persistence | #000027 | committed objects to falsify |
| ForkScore (accept / marginal / reject) | #000012 | mutation selection verdict |
| MemoryRoot | #000017 | persistent failure recall |
| SelfModel | #000014 | committed self-claims |
| Capital ledger | #000020 | honest fork cost comparison |
| Soft-hash threat model | #000018 | covert-channel discipline |
| Claim-pack source | #000029 | versioned axiom bundles |
| Surface-ingest cited textbooks | #000031 | warrant promotion substrate |
What's missing is a **named control law** that stitches them into
one loop. Today:
- ForkScore picks accept/marginal/reject from precomputed battery
deltas, but does not formalize a probability distribution over
candidate branches, does not allocate compute under a budget,
does not scale witness sampling with measured divergence, and
does not name its own difficulty knob.
- 5F Falsification fixtures accumulate from witness divergence,
but no controller closes the loop by widening search width or
raising the acceptance threshold when divergence-rate climbs.
- Multi-witness sweeps run at a fixed sampling rate (#000028's
capital-ledger follow-up). There is no homeostatic feedback:
when the LLM agrees with the kernel for 1 000 consecutive
canonical-shape calls, the controller should narrow witness
sampling and free compute; when it disagrees on 5 of 8 (the
observed rate at #000028 land), it should widen.
- Capital ledger records cost per state-changing op but is not
consulted as a Kelly-style allocation constraint when the system
has multiple candidate mutations and a finite compute budget.
The shape of the gap, in one diagram:
```text
[ forkscore ] [ 5F fixtures ] [ witness divergence ]
\ | /
\ | /
\ | /
v v v
??? no controller ???
|
v
[ mutation accept / reject ]
```
This ticket spec'd the missing controller.
---
## 2. The bicameral hunch (fox, 2026-05-09 thread)
Verbatim, from the thread that triggered this ticket:
> recursive falsification is important hence, we can never really
> separate this and the llm … so the hunch was, we need a
> deterministic and a probabilistic llm side by side, like left vs
> right brain, probing each other. that is really how we think.
>
> Gödel is side-stepped, not defeated, by replacing global
> completeness with local, versioned, replayable admissibility.
>
> i really want to reach for a bicameral mind. an unconscious that
> runs the tree falsifying across nodes during periods of sleep or
> new ingestion … but I think that process will slow me down in
> the short term.
>
> hermes can do about 4 concurrent requests — that would be a
> bottleneck.
Three load-bearing observations from the thread:
1. **Bicameral pair.** A deterministic π* kernel and a
probabilistic LLM, side by side, each probing the other. This
is the existing #000028 architecture **named** as a cognitive
pattern rather than a witness pipeline. Naming matters for
downstream clients implementing Arborist conventions.
2. **Unconscious falsification.** A nightly / ingestion-lull sweep
that walks committed nodes and re-runs witness comparison.
Distinct from inference-time witness fan-out (#000028) — runs
on **already-cached** answers, and writes new 5F fixtures only
when previously-strict rows now diverge.
3. **Compute is the bottleneck.** Hermes serves ~4 concurrent
requests. This is the binding constraint that forces the
controller to be Kelly-bounded (allocate compute to high-
payoff, high-confidence branches first) rather than uniformly
sampled.
Phase 0 names these. It does not implement the unconscious sweep
yet — fox's "slow me down in the short term" gates Phase 1.
---
## 3. Architecture — bicameral pair as left/right brain
```text
┌──────────────┐
│ canonical │ deterministic kernel
│ π* witness │ (left brain — exact, slow,
│ (kernel) │ unhallucinable)
└──────┬───────┘
│ agreement matrix
│ (canonical bytes equality)
┌──────┴───────┐
│ LLM witness │ probabilistic LLM
│ (Hermes) │ (right brain — fluent,
│ │ fast, hallucination-prone)
└──────┬───────┘
v
┌──────────────┐
│ controller │ Prometheus-Σ
│ (this ticket)│ homeostasis
└──────────────┘
```
The bicameral pair is **already implemented** by #000028 — the
ticket's contribution is to (a) name it, (b) commit to "kernel is
ground truth on canonical shapes, LLM is witness never authority"
as a doctrine (already enforced by the verifier; this writes it
down), and (c) add the **unconscious sweep** as a Phase 1
deliverable that walks committed `providence_cache` rows during
ingest lulls and re-runs the witness fan-out on rows older than a
configurable threshold.
The unconscious sweep is the new mechanical contribution. Today
#000028 fan-out runs at inference time only. The sweep amortizes
falsification across periods when the operator is not actively
querying — the engineering analogue of sleep consolidation.
### 3.1 Two sweep targets, not one (fox 2026-05-09)
Original spec covers `providence_cache` re-witnessing. Fox flagged a
second backlog the controller has to drain: **ingested documents
that bypassed meta-cognition at ingest time**.
Today's ingest pipeline is intentionally fast: fetch → canonicalize
→ chunk → Merkle commit → audit row → done. No witness fan-out, no
canonical-projection cross-check, no warrant resolution. That speed
is the right trade-off for getting bytes into the queryable Merkle +
FTS5 tree, but it leaves a meta-cognition gap: the substrate has
content it never *probed*.
The unconscious sweep is where that probe runs. Two distinct sweep
targets, both processed during ingest lulls:
**Target A — `providence_cache` rows older than threshold τ_qa.**
Re-runs the multi-modality witness fan-out (#000028) on cached Q&A
answers. Catches drift: a Hermes update changes the LLM's
divergence pattern; a kernel bump changes canonical bytes; a peer's
mesh sync delivered new evidence that would have changed the
answer. Threshold τ_qa default: 7 days.
**Target B — `documents` rows ingested without meta-cognition.**
Walks `documents` where `last_swept_at` is NULL or older than τ_doc,
runs whichever cross-checks apply to the document's content:
- **Canonical-projection probe.** If chunks contain canonical-shape
language (math expressions, propositional logic, time-series JSON
per the existing preflight regexes), run the matching π* and
compare against any stated answer in the document. Divergence →
candidate falsification fixture.
- **Warrant resolver re-run.** Per `arborist/qa/warrant_resolver.py`
(#000031 Phase 2). If the document is a claim-pack record AND
the cited textbook newly appeared in the shard cluster since
the original ingest, write the missing `derivations.proof_blob`
row. Idempotent at the DB layer.
- **Source-URL freshness probe.** Existing
`arborist/sources/crawler/recrawl-check` does HEAD-only
conditional fetches. The sweep folds this in: docs whose
upstream changed get marked stale; docs whose upstream
vanished get marked gone. Provides drift signal without
paying full re-fetch cost.
- **Witness fan-out on document content.** For documents whose
body contains canonical-shape statements (e.g., a Wikipedia
paragraph asserting "0.1 + 0.2 = 0.3"), run the kernel + LLM
+ cache witness pair AS IF the document were a Q&A answer
citing itself. Produces high-value falsification fixtures
for documents the corpus *wrote down* but never *probed*.
**State tracking.** A new column on `documents`:
`last_swept_at INTEGER`. Updated when the sweep runs cross-checks
on a doc. Schema migration is small (one ALTER TABLE), but lives
behind the §12 phase trigger so it doesn't bump the cache_key
invariant prematurely.
**Why this matters.** Without target B, the substrate's "I have
this content in my Merkle tree" claim is honest at the byte
level but silent on whether the content was ever tested against
the kernel ground truth. Sleep is when you do the tests you
deferred to keep the day moving. The bicameral pair's
unconscious half is exactly where this catch-up happens.
---
## 4. Controller inputs / state / outputs
### 4.1 Inputs
```text
battery deltas: 5S, 5T, 5F, 5R (per candidate branch)
witness stats: agreement_rate, divergence_rate, kernel_error
capital ledger: compute_cost, latency_cost, storage_cost,
witness_cost
memory state: stale_claims, invalidation_frontier
selfmodel state: calibration_delta
risk flags: soft_hash_signal, unsupported_carrier,
source_warrant_level
budget: B (concurrent-Hermes ceiling ≈ 4)
```
### 4.2 State
```text
Σ_t = {
Ω_t, # organism state (#000017 + #000014 roots)
H_t, # branch-distribution entropy
ForkScore_t, # latest verdict map (#000012)
divergence_rate_t, # rolling window from #000028 sweep
capital_budget_t, # remaining compute under B
difficulty_t # named knob this ticket introduces
}
```
### 4.3 Outputs
```text
ACCEPT — branch passes all hard vetoes, score positive,
entropy bounded
MARGINAL — useful but uncertain; queue for unconscious sweep
REJECT — negative score OR hard-regression flag
QUARANTINE — integrity issue (carrier, soft-hash, schema)
UNKNOWN — bounded verifier returned no decision
ESCALATE — strictly stronger verifier or human required
```
`UNKNOWN` and `ESCALATE` are the labels that side-step Gödel. See §10.
---
## 5. Shannon-entropy fork-selection
For each candidate branch `b_i ∈ B`:
```text
U_i = α·5S_i + β·5T_i + γ·5F_i + ρ·5R_i
λ·C_i μ·X_i ν·D_i
```
where `C_i` is capital cost, `X_i` is regression / security
penalty, `D_i` is witness divergence. Branch distribution:
```text
p_i = exp(η · U_i) / Σ_j exp(η · U_j)
H(p) = Σ_i p_i · log p_i
```
The difficulty knob `difficulty_t` directly modulates `η`, search
depth, witness sampling rate, and branch count:
```text
high H → widen search, raise witness sampling, lower acceptance
confidence, allocate more compute
low H → narrow search, reduce witness sampling, conserve
compute
↑ divergence → raise difficulty, require more verification
↓ divergence → drop difficulty, allow fast path
```
This is the homeostatic loop: divergence-rate is the thermostat;
search width and witness sampling are the heat sinks.
---
## 6. Nash-style multi-validator veto
The bicameral pair generalizes to a multi-validator equilibrium.
Each validator has a different payoff:
```text
kernel: wants exactness
LLM: wants plausible generation
cache: wants consistency with prior commitments
capital ledger: wants low cost
security check: wants low attack surface
benchmark: wants performance
```
A branch is acceptable iff **no critical validator vetoes** and
the equilibrium is stable under the scoring rule. Hard vetoes:
```text
unsupported carrier → QUARANTINE
verifier failure → REJECT
hard regression → REJECT
cache drift detected → QUARANTINE
soft-hash signal above threshold → QUARANTINE
memory invalidation > κ → ESCALATE
```
This is the formal version of "balance computation with
tractability": no objective wins by destroying the others.
---
## 7. Kelly-bounded compute allocation
With `p_i` from §5, payoff `b_i`, and `q_i = 1 p_i`:
```text
f_i = max(0, (p_i · b_i q_i) / b_i)
allocation_i ∝ f_i · exp(η · U_i)
Σ_i allocation_i ≤ B
```
`B` is the binding Hermes-concurrency constraint (~4 today). High-
upside, high-confidence branches get most compute; low-payoff or
noisy branches get little or none; vetoed branches get zero.
---
## 8. Cross-chain matrix solver
Build `A[i, j]` over (mutation × chain) with chains:
```text
j=1: 5S delta j=6: memory consistency
j=2: 5T delta j=7: selfmodel calibration
j=3: 5F delta j=8: inverse capital cost
j=4: 5R delta j=9: inverse security risk
j=5: witness agreement j=10: inverse regression penalty
```
Solve:
```text
choose i maximizing A_i · w
subject to:
no hard veto
Σ capital cost_i ≤ B
regression_i ≥ −ε
witness divergence_i ≤ δ
memory invalidation_i ≤ κ
```
Recursive: after choosing `i`, update `Ω`, recompute `A`, update
`w` from new falsification pressure, repeat.
---
## 9. Counterfactual temporal search (no "time travel")
The synthesis that triggered this ticket described "time travel
across latent space." Literal time travel: no. The rigorous form:
```text
Ω_{t+1} = T(Ω_t, Δ_t)
```
Branch search simulates `Ω_t → Ω_{t+1}^{(i)} → Ω_{t+2}^{(i)} → …`,
estimates falsification risk, and chooses the branch with best
expected ForkScore under budget. **Bounded counterfactual search
over a committed cognitive state-space** is the engineering name.
LLMs propose candidate branches not enumerated in current state.
The controller does not trust them until they pass canonicalization,
verification, witness comparison, and benchmark scoring. Every
proposed branch is a witness, not an authority — same rule as
§3.
---
## 10. Gödel discipline — local, versioned, replayable admissibility
The controller does **not** defeat Gödel. It side-steps the
engineering trap by refusing single-system closure. Concretely:
| Technique | Gödel response |
|---|---|
| `UNKNOWN`, `UNSUPPORTED`, `UNDECIDED_UNDER_CURRENT_AXIOMS` labels | the system can return "I don't know" instead of hallucinating proof |
| versioned axiom sets via #000029 claim-packs | every result is "warrant under declared formal conditions," not absolute truth |
| escalation across meta-levels (kernel → stronger kernel → human) | undecidable in `S_0` does not block `S_1` from deciding; audit chain records the escalation |
| separation of empirical / formal / source warrants | a kernel result does not warrant an empirical claim and vice versa — no false closure across types |
| canonical bytes + audit chain + replayable fixtures | every admissibility decision is reproducible; consistency is local-and-versioned, not global |
This is what fox's thread named: **local, versioned, replayable
admissibility** instead of global completeness. The doctrine is
already implicit across #000005 (label ladder), #000027 (canonical
projections), and the verifier's trichotomy. This ticket commits
it to writing as the controller's design substrate.
---
## 11. Bottleneck: Hermes ~4 concurrent requests
Per fox's thread, Hermes serves about four concurrent requests at
the single live endpoint (`hermes.ai.unturf.com`). This is the
binding `B` in §7. Implications:
- Witness fan-out (#000028) can not run uncontrolled; the
controller throttles it via Kelly allocation.
- Unconscious sweep (§3) **must not** compete with inference-time
traffic — runs only when concurrent-Hermes utilization is low,
yields to inference traffic on contention.
- Adding deterministic kernels (the "left brain") buys extra
effective concurrency at zero LLM cost: kernel calls are pure-
Python and parallelize freely. Encourages expanding π* domain
coverage (#000015, #000030, #000032) over LLM-side-only growth.
The bottleneck is also why the controller is Phase 0 / doc-only
today — Phase 1 work that floods the LLM endpoint under a naive
controller would crowd out fox's actual queries.
---
## 12. Phase gating — measured trigger, not calendar date
Phase 0 (this ticket) is doc-only. Phase 1 opens **only** when one
of these triggers fires:
1. ForkScore (#000012 Phase 1a) regularly receives ≥ 4 candidate
branches per checkpoint and the operator wants probability-
weighted compute allocation across them.
2. #000028 witness sweep observes divergence-rate variance large
enough that fixed witness sampling starts wasting budget on
stable rows or under-sampling unstable rows. Concrete signal:
rolling divergence-rate stddev / mean > 0.5 over a 7-day
window.
3. Capital ledger (#000020) shows witness-cost > 30 % of total
compute spend on a real-shard workload (#000026 baseline).
4. Operator-stated mission need for the unconscious sweep — e.g.
nightly re-validation of an ingested textbook (#000031).
Until then, the controller exists on paper. ForkScore, 5F, and
witness sweep continue to operate as independent modules with no
controller mediating them. Five-step §2 ("delete the part") asks
us to confirm pain before adding mechanism; this gate enforces
that.
---
## 13. Minimal algorithm (Phase 1 sketch)
```text
Algorithm: Prometheus-Σ Recursive Falsification Controller
Inputs:
Ω_t, B_t (candidate branches), W_t (witnesses),
M_t (memory + selfmodel), C_t (capital ledger),
P_t (policy constraints)
Step 1: for each b_i, run available deterministic checks
Step 2: build A[i, j] cross-chain matrix
Step 3: apply hard vetoes (carrier, verifier, regression, cache
drift, security, memory)
Step 4: compute U_i
Step 5: p_i = softmax(η · U_i)
Step 6: H(p)
Step 7: adjust difficulty_t:
↑ H → wider search
↑ divergence → stricter witness
↑ cost → narrower search
Step 8: Kelly-bounded allocation under B
Step 9: pick branch:
ACCEPT if positive, bounded risk, low entropy
MARGINAL if useful but uncertain (queue for sweep)
REJECT if negative or vetoed
QUARANTINE if integrity issue
Step 10: commit decision to audit chain (#000017 + #000014 roots)
Step 11: emit 5F falsification fixtures from divergences
Step 12: update MemoryRoot / SelfModel if warranted
Output: Ω_{t+1}, decision, audit event, new fixtures, updated
difficulty_t
```
Phase 1 implementation lives in `arborist/v9/prometheus.py` (new
namespace; sibling to `arborist/v8/fork_score.py`). No schema
changes — the controller is a pure function over already-committed
state.
---
## 14. Scope boundaries — what this ticket does NOT do
- **Does not** bump `governance_policy_hash` or any cache_key dim.
The controller reads existing state and emits decisions; its
outputs land as audit-event rows, not as cache_key inputs.
- **Does not** introduce LLM-as-judge anywhere. All judgment is
deterministic — entropy, Kelly fraction, ForkScore verdict,
hard-veto checks. The LLM remains a witness only.
- **Does not** replace ForkScore (#000012). ForkScore is the
single-branch verdict; Prometheus-Σ is the multi-branch
allocator that calls ForkScore on each candidate.
- **Does not** implement the unconscious sweep in Phase 0. Phase 1
spec'd; Phase 1 implementation gated by §12.
- **Does not** claim completeness, claim Gödel-defeat, or claim
AGI. It implements local-versioned-replayable admissibility, full
stop. See §10.
- **Does not** modify `audit_events.event_hash` preimage — sibling
audit-event tags (`controller_decision`, `controller_difficulty`)
per #000020's advisory-ledger pattern.
---
## 15. Open questions for go/no-go
1. **`α, β, γ, ρ, λ, μ, ν, η` weights.** The synthesis names them
but does not pick values. Phase 0 should ship a default weight-
set tied to fox's mission priority (intellectual capital ≫
computational throughput ≫ social capital, per BLACKOPS shard).
Phase 1 may make weights pluggable per operator.
2. **Where do divergence-rate windows live?** New table, or fold
into `audit_events` with a query-time aggregation? Trade-off:
new table = clean separation; aggregation = fewer schema
surfaces. Lean toward aggregation — DRY in context.
3. **Unconscious sweep priority vs inference traffic.** Cooperative
yielding via a concurrent-Hermes utilization probe, or a
strict-priority queue? Probe is simpler; queue gives stronger
guarantees. Probe wins for Phase 1; revisit if it under-yields.
4. **Sympathetic naming.** "Prometheus-Σ" comes from the synthesis.
Internal-arborist convention is `arborist/vN/<thing>.py`. Plan:
keep "Prometheus-Σ" as the **doctrine name** (used in tickets,
papers, mesh wire docs), but the module lives at
`arborist/v9/prometheus.py` and the table tags are
`controller_*`. Name does not enter cache_key.
---
## 16. Status
Phase 0 (this ticket): open · awaiting go/no-go.
When fox approves: this ticket flips to `in progress · Phase 0
landed in commit <sha>` with no code change — the doc itself is
the deliverable.
Phase 1 opens when §12 trigger fires.
---
## 17. References
- This ticket: design-substrate doc, no code yet.
- #000012 (ForkScore): single-branch verdict layer the controller
consumes.
- #000028 (multi-witness): the bicameral pair, pre-named.
- #000027 (canonical projections): committed objects to falsify.
- #000025 (5F battery): falsification fixture sink.
- #000017 (MemoryRoot), #000014 (SelfModel): persistent state the
controller updates.
- #000020 (capital ledger): the `B` budget input.
- #000018 (soft-hash threat model): the `X_i` security penalty
input.
- #000029 (claim-packs): versioned axiom bundles for §10
admissibility.
- `docs/v8-fork-score.md`: ForkScore reference.
- BLACKOPS shard "Patch the Planet" / "Balance All Workstations":
the mission anchor for the weight-set in §15 Q1.
- 2026-05-09 fox thread: bicameral hunch, Gödel side-step phrase,
Hermes ~4 concurrent bottleneck.