three-thread session output: stale TODOs, N-power probe, ForkScore Phase 1c
Three threads landed in a single commit because they share the same substrate (the #000037 §12 trigger probe shipped inc422216): A — code-side stale-map sweep (arborist/qa/repair.py) ===================================================== Walked inline TODO/FIXME/XXX markers across arborist/ + tests/. Five hits: two false-positives (\\uXXXX in escape pattern docs), one genuine deferral (raw_html cache in async_web_fetcher), and **two stale TODOs in arborist/qa/repair.py** referring to "re-prompt feedback path is future work" — even though `reprompt_repair` is fully implemented (lines 140+ at the time of this commit), wired through CLI `--repair-reprompts N` flag (cli.py:4574), and gated by `policy["repair_max_reprompts"]`. Refreshed the docstring header and the in-body comment to point at the actual function. Same drift pattern as today's earlier ticket sweep (e84f453): implementation lands, the TODO doesn't get refreshed, future readers re-implement what's already there. B — N-power follow-up to the §12 trigger probe ============================================== c422216's first probe run reported trigger 2 (divergence variance) at N=16, σ=0.5, ratio=0.8 — both threshold conditions would fire if N reached the 30-sample N_min. To validate that the variance signal holds at N≥30 (rather than vanishing on a wider sample), drove the canonical-witness path 20 additional times via a new bench/qa_questions_canonical_witness_npower.txt fixture (10 arithmetic@v1 + 10 logic-kernel@v1 questions; all canonicalize-then-LLM-witness without errors). Result at N=37: trigger 2 fires. Ratio 0.575 > 0.5, abs σ 0.435 > 0.10. Variance signal is real at the floor. Captured in bench/results/prometheus-sigma-triggers-2026-05-10-npower.md with a loud caveat at the top: this is N-power validation, NOT a measure of real workload pressure. The §12 phase-1 go/no-go decision should still come from natural workload N or operator-stated need. The prior baseline (prometheus-sigma-triggers-2026-05-10.md) stays in place as the workload-state-at-time-of-ticket-c422216 record; this new -npower.md report is the statistical-power follow-up. C — ForkScore Phase 1c proposal (#000012) ========================================= #000037 §12 Trigger 1 ("ForkScore receives ≥4 candidate branches per checkpoint") gates on multi-branch persistence — but #000012 Phase 1a (landed 2026-05-08) is single-validator scoring of one fork at a time, and Phase 1b (still open) is the consensus paper. Neither persists branch-sets. So Trigger 1 structurally cannot fire today, which the probe correctly reports as "no fork_score branch-set table found". Added Phase 1c to #000012 as a doc-only proposal (no code in this commit): - New `fork_score_branches` table (sibling to capital_ledger; does NOT enter audit_events.event_hash preimage). - Optional `--branch-set <ID>` flag on `arborist v8 score`. - One read API: `branch_set_density(conn, branch_set_id)`. - #000037 probe wires through the new function once it lands. Not opened as its own ticket because operator pressure for it hasn't surfaced naturally — gates on either #000012 Phase 1b landing OR #000030 algebra/calc kernel expansion producing competing-kernel branches an operator wants to compare. Captured here so a future shift doesn't re-discover the gap. Hygiene ======= - make test → 1623 passed, 45 skipped - make chain-check-shards → 0 across all 7 shards - fox's in-flight #000037 ticket modifications left untouched
This commit is contained in:
parent
245fb6ead3
commit
8980e64aa0
4 changed files with 236 additions and 4 deletions
|
|
@ -14,8 +14,11 @@ callers don't see answer text mutate under their feet.
|
|||
Repair philosophy follows the toy-Hermes design pass (fox 2026-04-30):
|
||||
|
||||
- Mechanical first. Cheap, deterministic, idempotent.
|
||||
- Re-prompt second (TODO; one extra LLM call to ask the model to
|
||||
rewrite around the failed claim). Adds latency; quality bonus.
|
||||
- Re-prompt second — see :func:`reprompt_repair` below. One extra LLM
|
||||
call asks the model to rewrite around the failed claims using only
|
||||
verbatim quotes; gated by ``policy["repair_max_reprompts"] > 0`` so
|
||||
the default path stays deterministic. CLI exposes via the
|
||||
``--repair-reprompts N`` flag.
|
||||
- Repair stage is observational from the cache_key's perspective: the
|
||||
cache_key inputs are unchanged, only the persisted answer text differs.
|
||||
An audit event records the original→repaired transition.
|
||||
|
|
@ -131,8 +134,9 @@ def mechanical_repair(
|
|||
# downgrade (needs prose rewriting), partial_paraphrase
|
||||
# (split_or_remove judgment). These cases pass through the
|
||||
# mechanical loop unchanged & remain UNSUPPORTED in the
|
||||
# post-repair verdict — falling back to a re-prompt loop is
|
||||
# future work (TODO: re-prompt feedback path).
|
||||
# post-repair verdict; the optional re-prompt feedback path
|
||||
# (``reprompt_repair`` below) handles them when the operator
|
||||
# opts in via ``policy["repair_max_reprompts"] > 0``.
|
||||
|
||||
return {"repaired_text": repaired, "changes": changes}
|
||||
|
||||
|
|
|
|||
51
bench/qa_questions_canonical_witness_npower.txt
Normal file
51
bench/qa_questions_canonical_witness_npower.txt
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
# Canonical-witness N-power fixture (2026-05-10).
|
||||
#
|
||||
# Purpose: bring the providence_canonical_witness sample count above
|
||||
# the §12 Trigger 2 N_min=30 floor so the divergence-variance trigger
|
||||
# can decisively fire-or-not-fire on real data. The first probe run
|
||||
# (bench/results/prometheus-sigma-triggers-2026-05-10.md) showed:
|
||||
#
|
||||
# N=16, mean=0.625, σ=0.5, ratio=0.8
|
||||
#
|
||||
# Both the ratio (>0.5) and the absolute (>0.10) thresholds would
|
||||
# already fire if N reached the 30-sample floor. This fixture
|
||||
# generates 20 additional canonical-shape questions so the next
|
||||
# probe run lands at N≥36 with the same code path.
|
||||
#
|
||||
# IMPORTANT: this is an N-power experiment, NOT a measure of real
|
||||
# workload pressure. The §12 trigger is intended to fire when
|
||||
# canonical-shape questions naturally accumulate at the 30+ scale on
|
||||
# a live deployment. Filling the sample buffer manually proves the
|
||||
# variance signal is statistically meaningful at N=30; it does not
|
||||
# prove that fox's real workload has reached that pressure point.
|
||||
# The probe report MUST tag this distinction.
|
||||
#
|
||||
# Question selection: pure-arithmetic and pure-propositional shapes
|
||||
# only. The canonical projection sniff (query.py:3605) drops anything
|
||||
# with question marks, lowercase words, or mixed shape — these inputs
|
||||
# bypass that filter cleanly. Each one fires arithmetic@v1 OR
|
||||
# logic-kernel@v1 and produces a witness audit event.
|
||||
|
||||
# === arithmetic@v1 (10 questions) ===
|
||||
0.1 + 0.2
|
||||
1/3 + 1/6
|
||||
2 ** 10
|
||||
3 * 7 + 2
|
||||
(5 + 3) * 4
|
||||
100 / 7
|
||||
2.5 + 2.5
|
||||
17 - 8
|
||||
0.333 + 0.667
|
||||
42 + 7 - 1
|
||||
|
||||
# === logic-kernel@v1 (10 questions) ===
|
||||
A AND B
|
||||
A OR NOT B
|
||||
A IMPL B
|
||||
NOT (A AND B)
|
||||
(A AND B) OR C
|
||||
A XOR B
|
||||
A IFF B
|
||||
NOT NOT A
|
||||
(A IMPL B) AND (B IMPL A)
|
||||
A AND (B OR C)
|
||||
76
bench/results/prometheus-sigma-triggers-2026-05-10-npower.md
Normal file
76
bench/results/prometheus-sigma-triggers-2026-05-10-npower.md
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
# Prometheus-Σ §12 trigger probe — N-POWER FOLLOW-UP
|
||||
|
||||
**Date:** 2026-05-10T11:42:39Z
|
||||
**Shards directory:** `/home/fox/.arborist/shards`
|
||||
**Shards walked:** 7 (000.db, 001.db, 002.db, 003.db, crawl_appliedcombinatorics_org.db, qa.db, snapshots.db)
|
||||
|
||||
> **Caveat — read before relying on this report.** This run is an
|
||||
> **N-power experiment**, not a measure of real workload pressure.
|
||||
> The first probe run (`prometheus-sigma-triggers-2026-05-10.md`)
|
||||
> measured the natural state at N=16 and reported "trigger 2 doesn't
|
||||
> fire — N below the 30-sample floor". To validate that the variance
|
||||
> signal would *hold* at N≥30, the operator drove the canonical-witness
|
||||
> path 20 additional times via the
|
||||
> `bench/qa_questions_canonical_witness_npower.txt` fixture. This
|
||||
> brought N from 17 to 37 — enough to clear the floor — but the
|
||||
> 20 inputs are synthetic fixture questions, not operator workload.
|
||||
>
|
||||
> **Read this report as: the variance signal IS statistically
|
||||
> meaningful at N=30+ — not as: real workload pressure has reached
|
||||
> the §12 trigger threshold.** The §12 phase-1 go/no-go decision
|
||||
> should still be measured against natural workload N (or invoked
|
||||
> via Trigger 4, operator-stated need).
|
||||
|
||||
Phase 1 of ticket #000037 gates on one §12 trigger firing. This probe walks `audit_events` + `capital_ledger` and reports the empirical state of triggers 1–3. Trigger 4 is operator-stated.
|
||||
|
||||
## Verdict
|
||||
|
||||
| # | Trigger | Fires? | Reason |
|
||||
|---|---------|--------|--------|
|
||||
| 1 | branch density (≥4 branches/checkpoint) | no | no fork_score branch-set table found across shards; ForkScore Phase 1a is single-validator. Multi-branch fork-score persistence is the prerequisite — see #000012. |
|
||||
| 2 | divergence variance | **YES** | ratio 0.575 > 0.5; abs σ 0.435 > 0.1 |
|
||||
| 3 | witness cost share (material > 0.30) | no | witness material / total material = 0.008158 / 0.540638 = 0.015 |
|
||||
| 4 | operator mission need | n/a | operator decision; not measurable from committed state |
|
||||
|
||||
**Phase 1 trigger has fired.** Prometheus-Σ Phase 1 may proceed.
|
||||
|
||||
## Trigger 1 — branch density
|
||||
|
||||
- Data available: False
|
||||
- no fork_score branch-set table found across shards; ForkScore Phase 1a is single-validator. Multi-branch fork-score persistence is the prerequisite — see #000012.
|
||||
|
||||
## Trigger 2 — divergence variance
|
||||
|
||||
- Sample count (N): 37 (N_min = 30)
|
||||
- Mean divergence rate: 0.7568
|
||||
- Stddev: 0.435
|
||||
- Ratio (σ/mean): 0.5748
|
||||
- Threshold ratio: > 0.5 OR absolute σ > 0.1
|
||||
- ratio 0.575 > 0.5; abs σ 0.435 > 0.1
|
||||
|
||||
Agreement-label distribution across all shards:
|
||||
|
||||
| label | count |
|
||||
|-------|-------|
|
||||
| `KERNEL-LLM-DIVERGED` | 22 |
|
||||
| `KERNEL-LLM-AGREE` | 6 |
|
||||
| `LLM-DIVERGED` | 6 |
|
||||
| `STRICT-WITNESSED` | 3 |
|
||||
|
||||
## Trigger 3 — witness cost share
|
||||
|
||||
- Witness material (kWh proxy): 0.008158
|
||||
- Total material: 0.540638
|
||||
- Ratio: 0.0151 (threshold > 0.3)
|
||||
- Witness financial: 5e-06
|
||||
- Total financial: 5e-06
|
||||
- Witness rows: 37
|
||||
- Total ledger rows: 39
|
||||
- witness material / total material = 0.008158 / 0.540638 = 0.015
|
||||
|
||||
**Caveat:** ledger reflects ad-hoc activity, not a controlled #000026 sweep — ratio is informational. Re-run after a #000026 baseline sweep for a workload-anchored answer.
|
||||
|
||||
## Trigger 4 — operator mission need
|
||||
|
||||
- Operator-stated; not measurable from committed state.
|
||||
- Bypasses §12 measured-pressure gates per ticket §12.
|
||||
|
|
@ -305,3 +305,104 @@ What's deliberately NOT in Phase 1a:
|
|||
- Cross-validator ZK proof exchange.
|
||||
|
||||
These belong to the v8 paper itself.
|
||||
|
||||
### Phase 1c — Branch-set persistence (proposed, not yet open)
|
||||
|
||||
**Problem.** Phase 1a scores one ``(parent, child)`` fork at a time;
|
||||
Phase 1b is the consensus paper. Neither persists *multiple
|
||||
candidate branches at the same checkpoint*. Ticket #000037 §12
|
||||
Trigger 1 ("ForkScore regularly receives ≥4 candidate branches per
|
||||
checkpoint") gates Phase 1 of the Prometheus-Σ controller on this
|
||||
data existing — and as of `bench/results/prometheus-sigma-triggers-
|
||||
2026-05-10.md`, no `fork_score%` table exists across any shard, so
|
||||
Trigger 1 structurally cannot fire.
|
||||
|
||||
This is the missing seam. Proposal scope (doc-only here; code lands
|
||||
in a follow-up if/when fox approves):
|
||||
|
||||
**1. New table `fork_score_branches`** (sibling to `audit_events`,
|
||||
similar to `capital_ledger` — does NOT enter `audit_events.event_hash`
|
||||
preimage, so retroactive scoring cannot break the audit chain).
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS fork_score_branches (
|
||||
branch_set_id TEXT NOT NULL, -- checkpoint identity
|
||||
-- (e.g. parent_root + ts)
|
||||
branch_id TEXT NOT NULL, -- fork identifier
|
||||
-- (child_root or proposer key)
|
||||
parent_root TEXT NOT NULL, -- shared parent
|
||||
child_root TEXT, -- nullable for in-flight branches
|
||||
score REAL NOT NULL, -- ScoredFork.score
|
||||
verdict TEXT NOT NULL, -- ACCEPT / MARGINAL / REJECT
|
||||
breakdown_blob TEXT NOT NULL, -- canonical-JSON of full breakdown
|
||||
weights_id TEXT NOT NULL, -- which WeightSet was used
|
||||
estimator_version TEXT NOT NULL, -- per Phase 1a fork_score module ver.
|
||||
recorded_at INTEGER NOT NULL,
|
||||
PRIMARY KEY (branch_set_id, branch_id)
|
||||
);
|
||||
CREATE INDEX idx_fork_score_branches_set ON fork_score_branches(branch_set_id);
|
||||
CREATE INDEX idx_fork_score_branches_parent ON fork_score_branches(parent_root);
|
||||
```
|
||||
|
||||
PK is `(branch_set_id, branch_id)` so re-scoring the same fork
|
||||
under the same set is a clean upsert, not a duplicate row.
|
||||
|
||||
**2. CLI surface.** Extend `arborist v8 score` with two optional
|
||||
flags:
|
||||
|
||||
- ``--branch-set <ID>`` — names the checkpoint a result belongs to.
|
||||
When present, writes a row to `fork_score_branches` in addition to
|
||||
printing the verdict. When absent, behavior is unchanged (Phase 1a
|
||||
pure-function semantics preserved).
|
||||
- ``--persist-shard <PATH>`` — names the SQLite file to write to.
|
||||
Defaults to ``$ARBORIST_QA_DB`` when set, else stays no-op.
|
||||
|
||||
**3. Read API.** One pure function in `arborist/v8/fork_score.py`:
|
||||
|
||||
```python
|
||||
def branch_set_density(conn, branch_set_id: str) -> int:
|
||||
"""Count distinct branches recorded under a checkpoint id.
|
||||
|
||||
Used by the #000037 trigger probe to satisfy Trigger 1."""
|
||||
```
|
||||
|
||||
**4. #000037 probe wiring.** The trigger probe
|
||||
(`bench/prometheus_sigma_trigger_probe.py:trigger_1_branch_density`)
|
||||
currently reports "no data" when the table doesn't exist. After
|
||||
Phase 1c lands, it queries `fork_score_branches` for the latest
|
||||
checkpoint and reports `n_branches >= 4`.
|
||||
|
||||
**Hard constraints**:
|
||||
|
||||
- Sibling table — never enters `audit_events.event_hash` preimage.
|
||||
- Schema-only; no behavioral change to single-validator scoring.
|
||||
- Default off (a `--branch-set` flag absent ⇒ no row written).
|
||||
- No mesh wire format change (that's Phase 1b's territory).
|
||||
- `weights_id` is opaque; folding weights into a hash is a Phase 1b
|
||||
concern.
|
||||
|
||||
**What this enables (downstream tickets):**
|
||||
|
||||
- #000037 §12 Trigger 1 gains data; can fire empirically.
|
||||
- ForkScore + Prometheus-Σ become composable: the controller reads
|
||||
a checkpoint's branch set and runs softmax across the persisted
|
||||
scores instead of needing to re-score from raw BatteryResults.
|
||||
- A future operator-facing CLI (`arborist v8 branch-set show ID`)
|
||||
becomes trivial — same table powers it.
|
||||
|
||||
**Why not now**: this is doc-only because Phase 1c is small
|
||||
(~80 LOC) but adds storage surface area. The operator pressure for
|
||||
it lands when:
|
||||
|
||||
1. Multi-validator deployments produce competing branches naturally
|
||||
(gates on Phase 1b paper closing).
|
||||
2. **OR** an operator wants to run several alternative π* / model
|
||||
configurations against the same parent and pick — that operator
|
||||
pressure currently exists only as a hypothesis.
|
||||
|
||||
If pressure 2 surfaces (e.g., during #000030 algebra/calculus kernel
|
||||
expansion when multiple kernel variants compete), Phase 1c opens as
|
||||
its own ticket. Until then, #000037 Trigger 1 simply reports "no
|
||||
data — Phase 1c not landed" and the controller's multi-branch path
|
||||
is paper-only. That matches §12's "measured trigger, not calendar
|
||||
date" discipline.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue