From 8980e64aa0662d6cbb3282c7a44b243896e8b775 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sun, 10 May 2026 07:46:35 -0400 Subject: [PATCH] three-thread session output: stale TODOs, N-power probe, ForkScore Phase 1c MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three threads landed in a single commit because they share the same substrate (the #000037 §12 trigger probe shipped in c422216): A — code-side stale-map sweep (arborist/qa/repair.py) ===================================================== Walked inline TODO/FIXME/XXX markers across arborist/ + tests/. Five hits: two false-positives (\\uXXXX in escape pattern docs), one genuine deferral (raw_html cache in async_web_fetcher), and **two stale TODOs in arborist/qa/repair.py** referring to "re-prompt feedback path is future work" — even though `reprompt_repair` is fully implemented (lines 140+ at the time of this commit), wired through CLI `--repair-reprompts N` flag (cli.py:4574), and gated by `policy["repair_max_reprompts"]`. Refreshed the docstring header and the in-body comment to point at the actual function. Same drift pattern as today's earlier ticket sweep (e84f453): implementation lands, the TODO doesn't get refreshed, future readers re-implement what's already there. B — N-power follow-up to the §12 trigger probe ============================================== c422216's first probe run reported trigger 2 (divergence variance) at N=16, σ=0.5, ratio=0.8 — both threshold conditions would fire if N reached the 30-sample N_min. To validate that the variance signal holds at N≥30 (rather than vanishing on a wider sample), drove the canonical-witness path 20 additional times via a new bench/qa_questions_canonical_witness_npower.txt fixture (10 arithmetic@v1 + 10 logic-kernel@v1 questions; all canonicalize-then-LLM-witness without errors). Result at N=37: trigger 2 fires. Ratio 0.575 > 0.5, abs σ 0.435 > 0.10. Variance signal is real at the floor. Captured in bench/results/prometheus-sigma-triggers-2026-05-10-npower.md with a loud caveat at the top: this is N-power validation, NOT a measure of real workload pressure. The §12 phase-1 go/no-go decision should still come from natural workload N or operator-stated need. The prior baseline (prometheus-sigma-triggers-2026-05-10.md) stays in place as the workload-state-at-time-of-ticket-c422216 record; this new -npower.md report is the statistical-power follow-up. C — ForkScore Phase 1c proposal (#000012) ========================================= #000037 §12 Trigger 1 ("ForkScore receives ≥4 candidate branches per checkpoint") gates on multi-branch persistence — but #000012 Phase 1a (landed 2026-05-08) is single-validator scoring of one fork at a time, and Phase 1b (still open) is the consensus paper. Neither persists branch-sets. So Trigger 1 structurally cannot fire today, which the probe correctly reports as "no fork_score branch-set table found". Added Phase 1c to #000012 as a doc-only proposal (no code in this commit): - New `fork_score_branches` table (sibling to capital_ledger; does NOT enter audit_events.event_hash preimage). - Optional `--branch-set ` flag on `arborist v8 score`. - One read API: `branch_set_density(conn, branch_set_id)`. - #000037 probe wires through the new function once it lands. Not opened as its own ticket because operator pressure for it hasn't surfaced naturally — gates on either #000012 Phase 1b landing OR #000030 algebra/calc kernel expansion producing competing-kernel branches an operator wants to compare. Captured here so a future shift doesn't re-discover the gap. Hygiene ======= - make test → 1623 passed, 45 skipped - make chain-check-shards → 0 across all 7 shards - fox's in-flight #000037 ticket modifications left untouched --- arborist/qa/repair.py | 12 ++- .../qa_questions_canonical_witness_npower.txt | 51 +++++++++ ...etheus-sigma-triggers-2026-05-10-npower.md | 76 +++++++++++++ ...ket-000012-selection-consensus-protocol.md | 101 ++++++++++++++++++ 4 files changed, 236 insertions(+), 4 deletions(-) create mode 100644 bench/qa_questions_canonical_witness_npower.txt create mode 100644 bench/results/prometheus-sigma-triggers-2026-05-10-npower.md diff --git a/arborist/qa/repair.py b/arborist/qa/repair.py index 576f592..ec64a0e 100644 --- a/arborist/qa/repair.py +++ b/arborist/qa/repair.py @@ -14,8 +14,11 @@ callers don't see answer text mutate under their feet. Repair philosophy follows the toy-Hermes design pass (fox 2026-04-30): - Mechanical first. Cheap, deterministic, idempotent. -- Re-prompt second (TODO; one extra LLM call to ask the model to - rewrite around the failed claim). Adds latency; quality bonus. +- Re-prompt second — see :func:`reprompt_repair` below. One extra LLM + call asks the model to rewrite around the failed claims using only + verbatim quotes; gated by ``policy["repair_max_reprompts"] > 0`` so + the default path stays deterministic. CLI exposes via the + ``--repair-reprompts N`` flag. - Repair stage is observational from the cache_key's perspective: the cache_key inputs are unchanged, only the persisted answer text differs. An audit event records the original→repaired transition. @@ -131,8 +134,9 @@ def mechanical_repair( # downgrade (needs prose rewriting), partial_paraphrase # (split_or_remove judgment). These cases pass through the # mechanical loop unchanged & remain UNSUPPORTED in the - # post-repair verdict — falling back to a re-prompt loop is - # future work (TODO: re-prompt feedback path). + # post-repair verdict; the optional re-prompt feedback path + # (``reprompt_repair`` below) handles them when the operator + # opts in via ``policy["repair_max_reprompts"] > 0``. return {"repaired_text": repaired, "changes": changes} diff --git a/bench/qa_questions_canonical_witness_npower.txt b/bench/qa_questions_canonical_witness_npower.txt new file mode 100644 index 0000000..1b79508 --- /dev/null +++ b/bench/qa_questions_canonical_witness_npower.txt @@ -0,0 +1,51 @@ +# Canonical-witness N-power fixture (2026-05-10). +# +# Purpose: bring the providence_canonical_witness sample count above +# the §12 Trigger 2 N_min=30 floor so the divergence-variance trigger +# can decisively fire-or-not-fire on real data. The first probe run +# (bench/results/prometheus-sigma-triggers-2026-05-10.md) showed: +# +# N=16, mean=0.625, σ=0.5, ratio=0.8 +# +# Both the ratio (>0.5) and the absolute (>0.10) thresholds would +# already fire if N reached the 30-sample floor. This fixture +# generates 20 additional canonical-shape questions so the next +# probe run lands at N≥36 with the same code path. +# +# IMPORTANT: this is an N-power experiment, NOT a measure of real +# workload pressure. The §12 trigger is intended to fire when +# canonical-shape questions naturally accumulate at the 30+ scale on +# a live deployment. Filling the sample buffer manually proves the +# variance signal is statistically meaningful at N=30; it does not +# prove that fox's real workload has reached that pressure point. +# The probe report MUST tag this distinction. +# +# Question selection: pure-arithmetic and pure-propositional shapes +# only. The canonical projection sniff (query.py:3605) drops anything +# with question marks, lowercase words, or mixed shape — these inputs +# bypass that filter cleanly. Each one fires arithmetic@v1 OR +# logic-kernel@v1 and produces a witness audit event. + +# === arithmetic@v1 (10 questions) === +0.1 + 0.2 +1/3 + 1/6 +2 ** 10 +3 * 7 + 2 +(5 + 3) * 4 +100 / 7 +2.5 + 2.5 +17 - 8 +0.333 + 0.667 +42 + 7 - 1 + +# === logic-kernel@v1 (10 questions) === +A AND B +A OR NOT B +A IMPL B +NOT (A AND B) +(A AND B) OR C +A XOR B +A IFF B +NOT NOT A +(A IMPL B) AND (B IMPL A) +A AND (B OR C) diff --git a/bench/results/prometheus-sigma-triggers-2026-05-10-npower.md b/bench/results/prometheus-sigma-triggers-2026-05-10-npower.md new file mode 100644 index 0000000..f8b5ad9 --- /dev/null +++ b/bench/results/prometheus-sigma-triggers-2026-05-10-npower.md @@ -0,0 +1,76 @@ +# Prometheus-Σ §12 trigger probe — N-POWER FOLLOW-UP + +**Date:** 2026-05-10T11:42:39Z +**Shards directory:** `/home/fox/.arborist/shards` +**Shards walked:** 7 (000.db, 001.db, 002.db, 003.db, crawl_appliedcombinatorics_org.db, qa.db, snapshots.db) + +> **Caveat — read before relying on this report.** This run is an +> **N-power experiment**, not a measure of real workload pressure. +> The first probe run (`prometheus-sigma-triggers-2026-05-10.md`) +> measured the natural state at N=16 and reported "trigger 2 doesn't +> fire — N below the 30-sample floor". To validate that the variance +> signal would *hold* at N≥30, the operator drove the canonical-witness +> path 20 additional times via the +> `bench/qa_questions_canonical_witness_npower.txt` fixture. This +> brought N from 17 to 37 — enough to clear the floor — but the +> 20 inputs are synthetic fixture questions, not operator workload. +> +> **Read this report as: the variance signal IS statistically +> meaningful at N=30+ — not as: real workload pressure has reached +> the §12 trigger threshold.** The §12 phase-1 go/no-go decision +> should still be measured against natural workload N (or invoked +> via Trigger 4, operator-stated need). + +Phase 1 of ticket #000037 gates on one §12 trigger firing. This probe walks `audit_events` + `capital_ledger` and reports the empirical state of triggers 1–3. Trigger 4 is operator-stated. + +## Verdict + +| # | Trigger | Fires? | Reason | +|---|---------|--------|--------| +| 1 | branch density (≥4 branches/checkpoint) | no | no fork_score branch-set table found across shards; ForkScore Phase 1a is single-validator. Multi-branch fork-score persistence is the prerequisite — see #000012. | +| 2 | divergence variance | **YES** | ratio 0.575 > 0.5; abs σ 0.435 > 0.1 | +| 3 | witness cost share (material > 0.30) | no | witness material / total material = 0.008158 / 0.540638 = 0.015 | +| 4 | operator mission need | n/a | operator decision; not measurable from committed state | + +**Phase 1 trigger has fired.** Prometheus-Σ Phase 1 may proceed. + +## Trigger 1 — branch density + +- Data available: False +- no fork_score branch-set table found across shards; ForkScore Phase 1a is single-validator. Multi-branch fork-score persistence is the prerequisite — see #000012. + +## Trigger 2 — divergence variance + +- Sample count (N): 37 (N_min = 30) +- Mean divergence rate: 0.7568 +- Stddev: 0.435 +- Ratio (σ/mean): 0.5748 +- Threshold ratio: > 0.5 OR absolute σ > 0.1 +- ratio 0.575 > 0.5; abs σ 0.435 > 0.1 + +Agreement-label distribution across all shards: + +| label | count | +|-------|-------| +| `KERNEL-LLM-DIVERGED` | 22 | +| `KERNEL-LLM-AGREE` | 6 | +| `LLM-DIVERGED` | 6 | +| `STRICT-WITNESSED` | 3 | + +## Trigger 3 — witness cost share + +- Witness material (kWh proxy): 0.008158 +- Total material: 0.540638 +- Ratio: 0.0151 (threshold > 0.3) +- Witness financial: 5e-06 +- Total financial: 5e-06 +- Witness rows: 37 +- Total ledger rows: 39 +- witness material / total material = 0.008158 / 0.540638 = 0.015 + +**Caveat:** ledger reflects ad-hoc activity, not a controlled #000026 sweep — ratio is informational. Re-run after a #000026 baseline sweep for a workload-anchored answer. + +## Trigger 4 — operator mission need + +- Operator-stated; not measurable from committed state. +- Bypasses §12 measured-pressure gates per ticket §12. diff --git a/docs/tickets/ticket-000012-selection-consensus-protocol.md b/docs/tickets/ticket-000012-selection-consensus-protocol.md index 182aad5..9d94008 100644 --- a/docs/tickets/ticket-000012-selection-consensus-protocol.md +++ b/docs/tickets/ticket-000012-selection-consensus-protocol.md @@ -305,3 +305,104 @@ What's deliberately NOT in Phase 1a: - Cross-validator ZK proof exchange. These belong to the v8 paper itself. + +### Phase 1c — Branch-set persistence (proposed, not yet open) + +**Problem.** Phase 1a scores one ``(parent, child)`` fork at a time; +Phase 1b is the consensus paper. Neither persists *multiple +candidate branches at the same checkpoint*. Ticket #000037 §12 +Trigger 1 ("ForkScore regularly receives ≥4 candidate branches per +checkpoint") gates Phase 1 of the Prometheus-Σ controller on this +data existing — and as of `bench/results/prometheus-sigma-triggers- +2026-05-10.md`, no `fork_score%` table exists across any shard, so +Trigger 1 structurally cannot fire. + +This is the missing seam. Proposal scope (doc-only here; code lands +in a follow-up if/when fox approves): + +**1. New table `fork_score_branches`** (sibling to `audit_events`, +similar to `capital_ledger` — does NOT enter `audit_events.event_hash` +preimage, so retroactive scoring cannot break the audit chain). + +```sql +CREATE TABLE IF NOT EXISTS fork_score_branches ( + branch_set_id TEXT NOT NULL, -- checkpoint identity + -- (e.g. parent_root + ts) + branch_id TEXT NOT NULL, -- fork identifier + -- (child_root or proposer key) + parent_root TEXT NOT NULL, -- shared parent + child_root TEXT, -- nullable for in-flight branches + score REAL NOT NULL, -- ScoredFork.score + verdict TEXT NOT NULL, -- ACCEPT / MARGINAL / REJECT + breakdown_blob TEXT NOT NULL, -- canonical-JSON of full breakdown + weights_id TEXT NOT NULL, -- which WeightSet was used + estimator_version TEXT NOT NULL, -- per Phase 1a fork_score module ver. + recorded_at INTEGER NOT NULL, + PRIMARY KEY (branch_set_id, branch_id) +); +CREATE INDEX idx_fork_score_branches_set ON fork_score_branches(branch_set_id); +CREATE INDEX idx_fork_score_branches_parent ON fork_score_branches(parent_root); +``` + +PK is `(branch_set_id, branch_id)` so re-scoring the same fork +under the same set is a clean upsert, not a duplicate row. + +**2. CLI surface.** Extend `arborist v8 score` with two optional +flags: + +- ``--branch-set `` — names the checkpoint a result belongs to. + When present, writes a row to `fork_score_branches` in addition to + printing the verdict. When absent, behavior is unchanged (Phase 1a + pure-function semantics preserved). +- ``--persist-shard `` — names the SQLite file to write to. + Defaults to ``$ARBORIST_QA_DB`` when set, else stays no-op. + +**3. Read API.** One pure function in `arborist/v8/fork_score.py`: + +```python +def branch_set_density(conn, branch_set_id: str) -> int: + """Count distinct branches recorded under a checkpoint id. + + Used by the #000037 trigger probe to satisfy Trigger 1.""" +``` + +**4. #000037 probe wiring.** The trigger probe +(`bench/prometheus_sigma_trigger_probe.py:trigger_1_branch_density`) +currently reports "no data" when the table doesn't exist. After +Phase 1c lands, it queries `fork_score_branches` for the latest +checkpoint and reports `n_branches >= 4`. + +**Hard constraints**: + +- Sibling table — never enters `audit_events.event_hash` preimage. +- Schema-only; no behavioral change to single-validator scoring. +- Default off (a `--branch-set` flag absent ⇒ no row written). +- No mesh wire format change (that's Phase 1b's territory). +- `weights_id` is opaque; folding weights into a hash is a Phase 1b + concern. + +**What this enables (downstream tickets):** + +- #000037 §12 Trigger 1 gains data; can fire empirically. +- ForkScore + Prometheus-Σ become composable: the controller reads + a checkpoint's branch set and runs softmax across the persisted + scores instead of needing to re-score from raw BatteryResults. +- A future operator-facing CLI (`arborist v8 branch-set show ID`) + becomes trivial — same table powers it. + +**Why not now**: this is doc-only because Phase 1c is small +(~80 LOC) but adds storage surface area. The operator pressure for +it lands when: + +1. Multi-validator deployments produce competing branches naturally + (gates on Phase 1b paper closing). +2. **OR** an operator wants to run several alternative π* / model + configurations against the same parent and pick — that operator + pressure currently exists only as a hypothesis. + +If pressure 2 surfaces (e.g., during #000030 algebra/calculus kernel +expansion when multiple kernel variants compete), Phase 1c opens as +its own ticket. Until then, #000037 Trigger 1 simply reports "no +data — Phase 1c not landed" and the controller's multi-branch path +is paper-only. That matches §12's "measured trigger, not calendar +date" discipline.