engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000037 Phase 3 dry-run + Phases 0/1/2 status flip

Browse source 
Phases 1 (controller) and 2 (sibling-table audit writes) landed in
prior commits. This commit adds the Phase 3 dry-run simulator
instead of the actual sleep-sweep scheduler, since Phase 3's value
is mostly in what we'd learn from running it — and the dry-run
captures those findings without committing to a scheduler design
prematurely.

bench/scripts/prometheus_sigma_sweep_dryrun.py — read-only
simulator that classifies §3 Target A (providence_cache) + Target B
(documents) sweep candidates, synthesizes ControllerBranches from
real shard data, runs the Phase 1 controller, reports decision
distribution + Phase-3-design findings. No LLM calls, no
mutations.

make prometheus-sweep-dryrun — produces a dated markdown report
at bench/results/prometheus-sigma-sweep-dryrun-YYYY-MM-DD.md.

Five findings surfaced by three dry-run iterations against the
live ~/.arborist/shards corpus (3.5M docs + 2839 providence_cache
rows) — captured in ticket §22:

  1. chunk-size dominates Kelly threshold (must = Hermes
     concurrency, not candidate pool)
  2. flat capital_cost blocks every allocation (split kernel-cost
     vs LLM-cost on the contract)
  3. τ_qa=7d filters every CANONICAL_PROJECTION row (all 29 are
     <7d old; need per-audit-mode τ)
  4. Target B canonical-shape detection is the real headline
     (~152K candidates extrapolated; controller correctly returns
     MARGINAL on shape-match chunks)
  5. quarantined rows correctly veto via cache_drift hard-veto

Mean per-branch controller latency in dry-run: 12.5 µs at
chunk_size=4. Phase 3's actual bottleneck is the witness fan-out
(Hermes calls), not the controller itself.

Ticket #000037 status flipped to in-progress with Phases 0+1+2
landed; Phase 3 scheduler remains future work but is informed by
the five findings.
This commit is contained in:
russell@unturf.com 2026-05-10 16:47:02 -04:00
parent f625cac20c
commit 61424370bd
No known key found for this signature in database
5 changed files with 1049 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

112
bench/results/prometheus-sigma-sweep-dryrun-2026-05-10.md Normal file
View file

@ -0,0 +1,112 @@
# Prometheus-Σ Phase 3 sleep-sweep dry-run
Generated: 2026-05-10T20:42:47Z UTC
Script: `bench/scripts/prometheus_sigma_sweep_dryrun.py`
Ticket: #000037 Phase 3 (read-only simulation)
## Parameters
- Shards dir: `/home/fox/.arborist/shards`
- τ_qa: 1 days (86400 seconds)
- Target B sample/shard: 500
- Controller budget (Hermes concurrency): 4
- Weight profile: safe (§15.1)
## Target A — providence_cache sweep candidates
- Total candidates (rows older than τ_qa): **2379**
- Sweep chunks (size 4): 595
- Controller runtime: 23.20 ms
### Audit-mode distribution of candidates
| audit_mode | count |
|---|---|
| HYBRID | 872 |
| STRICT | 863 |
| UNGROUNDED | 635 |
| CANONICAL_PROJECTION | 9 |
### Controller decision distribution (per chunk)
| label | chunks |
|---|---|
| DEFERRED | 322 |
| REJECT | 269 |
| MARGINAL | 4 |
### Veto kinds observed
| veto_kind | count |
|---|---|
| cache_drift | 2 |
### Proposals emitted (advisory)
- MemoryRoot proposals: 0
- SelfModel proposals: 0
- Advisory event entries (would write to `controller_events` under Phase 2): 1141
## Target B — document sweep sample
- Total sampled candidates: **2000**
- Sweep chunks (size 4): 500
- Controller runtime: 26.17 ms
### Canonical-shape regex prefilter hits
| shape_class | matches |
|---|---|
| math | 68 |
| logic | 20 |
### Controller decision distribution (per chunk)
| label | chunks |
|---|---|
| DEFERRED | 417 |
| MARGINAL | 83 |
### Per-shard extrapolation
| shard | total docs | sampled | canonical-shape % |
|---|---|---|---|
| 000.db | 866,874 | 500 | 4.00% |
| 001.db | 867,695 | 500 | 4.60% |
| 002.db | 866,825 | 500 | 3.80% |
| 003.db | 866,998 | 500 | 5.20% |
### Extrapolated full-sweep cost (Target B)
- Total documents across shards: **3,468,392**
- Canonical-shape rate (from sample): 4.40%
- Extrapolated canonical-probe candidates: **~152,609**
- At Hermes concurrency=4, witness budget would dominate; the prefilter is doing real load-shaping.
## Total simulation cost
- Total branches scored: 4379
- Total controller runtime: 49.37 ms
- Mean per-branch latency: 11.27 µs
## Findings & fixes (dry-run iteration log)
Three iterations on the dry-run heuristic surfaced four real Phase-3-design findings before any production sweep code shipped:
**Finding 1 — chunk-size dominates Kelly threshold.** Iteration 1 (chunk_size=64) returned 100% DEFERRED. Kelly's `f_i = max(0, (p_i·b − q_i)/b)` requires `p_i > 0.5`; with 64 branches in a softmax, no single branch reaches that mass. Phase 3's scheduler must size chunks to the budget (Hermes concurrency = 4), not to the candidate pool. Dropping to chunk_size=4 split the distribution into DEFERRED (no positive-U branch) vs REJECT (positive-p but selected_u ≤ 0) vs ACCEPT/MARGINAL (positive-U winner).
**Finding 2 — flat capital_cost = no allocation.** Iteration 1 also assigned `capital_cost=1.0` to every branch (one full Hermes call). Combined with small audit-mode-based Δ5F deltas (±0.05–0.10), every utility was negative. Split kernel-only re-probe cost from full LLM-witness cost: `CANONICAL_PROJECTION=0.05`, `UNGROUNDED=0.4`, `HYBRID=0.8`, `STRICT=1.0` for Target A; `0.02` (HEAD-only freshness) vs `0.05` (canonical-shape probe) for Target B. Recommendation for Phase 3: model `capital_cost` as the expected cost given which witness paths fire (kernel / lexical / LLM), not a flat per-call estimate.
**Finding 3 — τ_qa=7d filters out every CANONICAL_PROJECTION row.** All 29 CP rows in `qa.db` are ≤ 7 days old (they're the recent π* graduations from tickets #000027/#000030/#000032). Sweep with τ_qa=7d shows zero CP candidates → zero ACCEPT chunks → zero high-value sleep work surfaced. Phase 3 should split τ by audit_mode: kernel-only modes (CP) get τ_qa=1d (cheap to re-probe, high value when kernel-LLM divergence surfaces); lexical/quote modes (STRICT/HYBRID/UNGROUNDED) stay at τ_qa=7d (expensive LLM calls).
**Finding 4 — canonical-shape Target B is the real headline.** 4.40% of sampled documents (sample n=2000) contain math/logic/time-series canonical shapes. Extrapolated to ~152K candidates across 3.5M docs in shards. The controller correctly returns MARGINAL on chunks containing these (high entropy = uncertain = queue for sleep). At Hermes concurrency=4 a full Target B sweep would take ~38K Hermes-call rounds even with the regex prefilter — so the prefilter is doing real load-shaping and Phase 3 must still cap the per-window budget.
**Finding 5 — 2 quarantined rows correctly vetoed.** The two `falsification_state='quarantined'` rows in `qa.db` surfaced as `cache_drift` hard-vetoes. The veto path is exercised end-to-end against real corpus data, no fixture-only mocking.
**Recommendations for the eventual Phase 3 scheduler:**
1. Per-audit-mode τ + per-audit-mode cost class (split kernel-cost from LLM-cost on the input contract — consider adding `kernel_cost` + `llm_cost` fields to `ControllerBranch` in a v2 dataclass).
2. Chunk size = Hermes concurrency (4 today, governance param going forward).
3. Sweep-specific weight profile (gamma_5f bumped, lambda_capital_cost dropped) since sweep work is deliberately accepting capital cost in exchange for falsification discovery.
4. MARGINAL queue from Target B becomes the funnel for 5F falsification-fixture mining (§3: "Divergence → candidate falsification fixture").
5. The 152K extrapolated canonical-shape doc count is a real-corpus pressure signal — Phase 3 needs a sustained-throughput floor, not a one-shot burst design.