From a53afa61480b511d881e34fe6349a4c233942c82 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Mon, 11 May 2026 08:03:45 -0400 Subject: [PATCH] =?UTF-8?q?#000046=20+=20#000047=20=E2=80=94=20open=20the?= =?UTF-8?q?=20two=20#000025=20=C2=A710.14=20follow-up=20tickets?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 5F threshold calibration (#000025 §10.14) surfaced two loose ends that #000012's v8 acceptance protocol owns; opening them as proper proposals so they don't get lost. #000046 — Harder 5S/5T/5F fixture tier (below-ceiling baselines). Every canonical 5S/5T/5F pack runs at rate 1.0 at HEAD, so fork_score's α·Δ5s+β·Δ5t+γ·Δ5f terms are inert (a child can only move them ≤ 0). The embedded packs can't be made "harder" (their evaluator is the gold); a harder tier is necessarily a live-path tier where a real, imperfect arborist surface produces wrong/partial output. Recommends Option A scoped narrow: a `falsification-hard-v1.jsonl` built around the verify_quotes entity-strategy gap already documented by 5f-fal-live-003, confirm the rate sits below 1.0 and is stable, pin it via `make bench-fork-baseline-hard`, demonstrate a planted surface fix lifting the rate. Status: open · awaiting go/no-go. #000047 — ForkScore _delta_* aggregator (mean vs max vs sum). _delta_* means over a battery's 5 sub-batteries, so a single-sub gain is worth a fifth of face value (the 5× dilution from the §10.14 calibration: +0.06 on one 5F sub → MARGINAL, same +0.06 on all five → ACCEPT). Recommends Option D: parameterize delta_aggregator on WeightSet, default "mean", record it in the breakdown + fork_score_branches; bump ESTIMATOR_VERSION only if the default changes. Defer implementation until #000046 lands a below-ceiling baseline so the aggregator choice is benchable, not guessed. Status: open · awaiting go/no-go; parks behind #000046. TICKETS.md index rows + Next ID bump (000046 → 000048) landed in 8599ce3 (swept in by a concurrent commit; content is correct). ticket-000012 §8 §3/§4 updated to name #000047 / #000046. --- ...ket-000012-selection-consensus-protocol.md | 10 +- .../ticket-000046-harder-5sf-fixture-tier.md | 188 ++++++++++++++++ ...cket-000047-fork-score-delta-aggregator.md | 203 ++++++++++++++++++ 3 files changed, 398 insertions(+), 3 deletions(-) create mode 100644 docs/tickets/ticket-000046-harder-5sf-fixture-tier.md create mode 100644 docs/tickets/ticket-000047-fork-score-delta-aggregator.md diff --git a/docs/tickets/ticket-000012-selection-consensus-protocol.md b/docs/tickets/ticket-000012-selection-consensus-protocol.md index bc0c047..80103df 100644 --- a/docs/tickets/ticket-000012-selection-consensus-protocol.md +++ b/docs/tickets/ticket-000012-selection-consensus-protocol.md @@ -554,7 +554,10 @@ protocol must account for: +`SIGNAL_FLOOR` across every sub of all three batteries scores ≈ 0.15. If v8 wants single-sub improvements to weigh equally, switch `_delta_*` from mean to a max/sum aggregator — a design - call this ticket owns, not a calibration finding. + call this ticket owns, not a calibration finding. Tracked as + **#000047** (recommends parameterizing `delta_aggregator`, default + `mean`; parks until #000046 produces a below-ceiling baseline to + bench the choice against). 4. **Ceiling saturation.** Every 5S/5T/5F pack is at rate 1.0 at HEAD, so `α·Δ5s + β·Δ5t + γ·Δ5f` can only be ≤ 0 — an unchanged @@ -565,5 +568,6 @@ protocol must account for: `feedback_efficiency`) + the non-bench terms (`selfmodel_calibration_gain`, `audit_completeness`, `validator_diversity`, …), not by bench Δ-rate. Adding the - harder tier is tracked as a #000025 follow-up, not a #000012 - blocker. + harder tier is tracked as **#000046** (a #000025 follow-up, + recommends a narrow live-path hard tier starting with 5F + Falsification), not a #000012 blocker. diff --git a/docs/tickets/ticket-000046-harder-5sf-fixture-tier.md b/docs/tickets/ticket-000046-harder-5sf-fixture-tier.md new file mode 100644 index 0000000..928d540 --- /dev/null +++ b/docs/tickets/ticket-000046-harder-5sf-fixture-tier.md @@ -0,0 +1,188 @@ +# Ticket #000046 — Harder 5S/5T/5F fixture tier (below-ceiling baselines) + +**Status:** open · awaiting go/no-go +**Opened:** 2026-05-11 +**Scope:** Give the 5S/5T/5F bench packs *headroom* — a fixture tier +the runtime does **not** pass at rate 1.0 — so the ForkScore Δ-rate +terms (`α·Δ5s + β·Δ5t + γ·Δ5f` in `arborist/substrate/fork_score.py`) +can be strictly positive for a genuine improvement. Doc-only proposal: +options + a narrow recommended start; no fixtures authored in this +ticket. +**Audience:** fox + #000025 follow-up readers + #000012's v8 +acceptance-protocol author + future bench authors. +**Hard constraint:** the existing `*-v1.jsonl` packs keep their pinned +fixture digests — a harder tier is a **new** pack (`*-hard-v1.jsonl` +or a `tier` field), never an edit to a pinned one. No schema / +governance-hash / canonicalization-version change. No LLM-as-judge in +any runner (the discipline that lets the bench be a scoreboard, not a +vibe check). + +--- + +## 1. Problem statement + +The #000025 §10.14 ForkScore threshold calibration +(`bench/scripts/fivef_threshold_calibration.py` → +`bench/results/5f-threshold-calibration-2026-05-11.md`, summarised in +`ticket-000012` §8) surfaced a structural fact: **every canonical 5S, +5T, and 5F pack runs at rate 1.0 at HEAD.** Consequence in +`fork_score`: + +- `_delta_5s / _delta_5t / _delta_5f` are means of per-sub-battery + Δ-rates. A child at ceiling vs a parent at ceiling ⇒ every Δ is 0 + ⇒ the three battery terms contribute exactly 0. An *unchanged* + child therefore scores 0 → MARGINAL (not ACCEPT — which is the + right behaviour, but it means the bench terms are inert). +- A child can only move those terms *negative* (regress a pack) — the + bench Δ-rate has no upside at the current pack difficulty. +- So fork *acceptance* today is carried entirely by the + efficiency-aware bonuses (`adaptation_efficiency` / + `feedback_efficiency` deltas) and the non-bench terms + (`selfmodel_calibration_gain`, `audit_completeness`, + `validator_diversity`, …). The "did the organism's measurable + capabilities improve?" signal is dormant. + +For the bench Δ-rate to carry positive signal, *something the runtime +gets partially wrong* has to be in the corpus. The embedded packs +can't be that something: the embedded evaluators are deterministic +and the gold answers are constructed to match them — a fixture the +embedded evaluator "fails" would be a fixture with a wrong gold, not +a hard problem. So a harder tier is necessarily a **live-path** tier +— inputs fed to a real, imperfect arborist surface (`parse_claims`, +`verify_quotes`, `store_snapshot`, the retrieval pipeline) where the +surface produces wrong / partial / over-confident output and the +fixture's `expected` records the *correct* outcome the surface +misses. Then rate < 1.0, and a future improvement to that surface +moves the rate up — real positive Δ. + +--- + +## 2. Options + +### 2.1 Option A — live-path "hard" tier (recommended, narrow start) + +Add a `*-hard-v1.jsonl` pack to one sub-battery where the live +surface most plausibly fails on difficult input, confirm the rate +sits below 1.0 *and is stable across runs* (the live surfaces are +deterministic — imperfect, not random — so rate must be a fixed +fraction), then a `make bench-fork-baseline-hard` that pins it as a +below-ceiling parent. Iterate to other sub-batteries only if the +loop proves out. + +Two strong candidates for the first hard pack: + +- **5F Falsification (`verify_quotes`)** — there's already a + documented soft-signal gap: `5f-fal-live-003` records that the + entity-strategy verifier soft-matches "Insulin / Penicillin" as one + claim because both name Alexander Fleming. A hard pack of near-miss + quote pairs that *should* land UNGROUNDED but the entity strategy + over-grounds would put the Falsification rate below 1.0 today; a + future tightening of the entity strategy (or the title-relevance + hard check picking up more) raises it. Directly bench-able. +- **5F Formulate (`parse_claims`)** — genuinely messy prose where the + claim-lattice parser produces the wrong claim count / pointer-set / + order. The `match_policy` is already structural (sorted-approx + claim text + exact-set pointer IDs + required order), so a hard + pack just needs inputs the parser mis-segments. + +Cost: ~10-20 hard fixtures for the first pack + a make target + a +test pinning "rate is below 1.0 and equals X". Risk: low — the live +surfaces are deterministic; the only failure mode is "the surface is +actually perfect on the inputs we picked", which just means picking +harder ones. + +### 2.2 Option B — deliberately-degraded parent baseline + +Ship a pinned `bench/results/baseline-degraded-v1.json` with +knocked-down rates (e.g. 0.85 uniformly), so a real child run (at +1.0) shows +0.15 → ACCEPT. Cheap (one JSON file, no fixtures). But +the "improvement" is fictional — the child didn't get better, the +parent was artificially handicapped. This is useful for *exercising +the ForkScore mechanics* (which the §10.14 calibration already does +in-memory via `_degraded_parent`), not for real fork selection. It +does **not** make the bench Δ-rate carry real signal; it just makes +the synthetic demo persist as a file. Probably not worth a ticket on +its own — note it here as the cheap alternative we explicitly chose +*not* to pursue as the primary path. + +### 2.3 Option C — do nothing; revisit when headroom surfaces organically + +Accept that the bench Δ-rate is inert at the current pack difficulty, +keep the §8 note in #000012, and add a hard tier only when one shows +up for free: a new π* domain whose kernel is imperfect, an +LLM-backed sub-battery, or a real regression that drops a pack below +ceiling. Lowest cost; the cost is that the dormant signal stays +dormant indefinitely. + +--- + +## 3. Recommendation + +**Option A, scoped to one sub-battery first — Falsification.** It has +a pre-documented gap (`5f-fal-live-003`), the live surface +(`verify_quotes`) is deterministic, and a hard pack there immediately +gives ForkScore a below-ceiling 5F sub-battery whose rate a future +verifier improvement can lift. If the loop proves out (rate stable +below 1.0, a planted verifier improvement demonstrably raises it, +ForkScore shows a positive `γ·Δ5f` term), extend to Formulate, then +to whichever 5S/5T sub-batteries have an imperfect live surface +(retrieval-backed ones are the obvious next look). Treat Option B as +a documented non-choice and Option C as the fallback if A's first +pack turns out perfect on every input we can author. + +Defer **#000047** (`_delta_*` mean-vs-max-vs-sum aggregation) until at +least one hard pack lands — the aggregator choice is empirically +testable only once there's a below-ceiling baseline to bench against. + +--- + +## 4. Out of scope + +- **Authoring the hard fixtures** — this ticket picks the strategy + and the first sub-battery; the fixture pack lands in the + implementation commit chain this ticket gates. +- **Changing `fork_score`** — the scorer already consumes whatever + rates the packs produce. A hard tier doesn't touch + `fork_score.py`; it changes the *inputs*. +- **Re-pinning the existing `*-v1.jsonl` packs** — hard off-limits; + those digests are referenced across closed tickets. +- **An LLM-backed sub-battery** — a different (larger) way to get + headroom; out of scope here, ticket-able separately if it ever + becomes the path. +- **`_delta_*` aggregator choice** — #000047. + +--- + +## 5. Status + +**Open · awaiting go/no-go.** Doc-only proposal. Opens when fox picks +a strategy (recommendation: Option A / Falsification first). The +implementation commit chain it gates: (1) author the first hard pack, +(2) confirm rate < 1.0 and stable + add the pinning test, (3) `make +bench-fork-baseline-hard`, (4) demonstrate a planted surface +improvement lifting the rate + a positive ForkScore Δ term, (5) +extend to the next sub-battery if the loop holds. Closure: ≥ 1 hard +pack live with a below-ceiling stable rate AND a worked example +showing the ForkScore bench-Δ term going positive on a real +improvement. + +--- + +## 6. References + +- `bench/scripts/fivef_threshold_calibration.py` + + `bench/results/5f-threshold-calibration-2026-05-11.md` — the + calibration that surfaced the ceiling-saturation problem. +- `docs/tickets/ticket-000012-selection-consensus-protocol.md` §8 — + the handoff section recording "ceiling saturation → bench Δ-rate + inert; track a harder tier as a #000025 follow-up". +- `docs/tickets/ticket-000025-5f-battery.md` — parent; this is the + follow-up the §10.14 closure flagged. +- `bench/fixtures/5f/falsification-live-v1.jsonl` (`5f-fal-live-003`) + — the pre-documented entity-strategy soft-signal gap the first + hard pack would build around. +- `arborist/qa/verify.py` (`verify_quotes`, the entity strategy) / + `arborist/qa/parse_claims.py` — the imperfect live surfaces a hard + tier exercises. +- `docs/tickets/ticket-000047-fork-score-delta-aggregator.md` — + sibling; parks until this ticket produces a below-ceiling baseline. diff --git a/docs/tickets/ticket-000047-fork-score-delta-aggregator.md b/docs/tickets/ticket-000047-fork-score-delta-aggregator.md new file mode 100644 index 0000000..f7225ad --- /dev/null +++ b/docs/tickets/ticket-000047-fork-score-delta-aggregator.md @@ -0,0 +1,203 @@ +# Ticket #000047 — ForkScore `_delta_*` aggregator: mean vs max vs sum + +**Status:** open · awaiting go/no-go (parks until #000046 produces a +below-ceiling baseline to bench against) +**Opened:** 2026-05-11 +**Scope:** Decide how `arborist/substrate/fork_score.py:_delta_5s / +_delta_5t / _delta_5f` should aggregate the per-sub-battery Δ-rate +vector into the single `Δ5{s,t,f}` term — keep the current +**mean**, switch to **max-of-positives** / **sum**, or expose it as a +policy parameter. Doc-only proposal: trade-offs + recommendation; no +code change in this ticket. +**Audience:** fox + #000012's v8 acceptance-protocol author + +maintainers of `arborist/substrate/fork_score.py` / +`arborist/substrate/weights.py`. +**Hard constraint:** `ForkScore` constants/aggregators are *estimator +parameters* (pinned by `ESTIMATOR_VERSION`), **not** +`governance_policy_hash` inputs — changing the default bumps +`ESTIMATOR_VERSION` and stales `fork_score_branches` rows written +under the prior estimator, but it does **not** invalidate any cache +record. No schema / canonicalization change. The hard-regression +side (`HARD_REGRESSION_FLOOR` catching a single-sub drop) is +unaffected — this ticket is only about how *positive* per-sub deltas +combine. + +--- + +## 1. Problem statement + +`_delta_5{s,t,f}` currently returns the **arithmetic mean** of the +per-sub-battery Δ-rates (5 subs per battery). The #000025 §10.14 +calibration (`bench/results/5f-threshold-calibration-2026-05-11.md`, +summarised in `ticket-000012` §8) made the consequence concrete — the +**5× averaging dilution**: + +- A child that improves **one** 5F sub-battery by +0.06 scores + `1.0 · (0.06 / 5) = 0.012` on the `γ·Δ5f` term → **MARGINAL**. +- A child that improves **all five** 5F subs by +0.06 scores + `1.0 · 0.06 = 0.06` → **ACCEPT** (boundary). +- Each battery contributes a *separate* term at weight 1.0, so a + uniform +`SIGNAL_FLOOR` across every sub of all three batteries + scores ≈ 0.15. + +So "a fork must improve by `SIGNAL_FLOOR` (0.05)" really means *one +of*: ≈ 0.25 on a single sub-battery, ≈ 0.05 uniform across one +battery's five subs, or ≈ 0.017 uniform across the whole 15-sub +suite. A fork that **specializes** — doubles one capability, leaves +the rest flat — barely moves the score. Is that the intended +sensitivity, or should a real single-sub gain weigh closer to face +value? + +--- + +## 2. Options + +### 2.1 Option A — keep `mean` (status quo) + +`Δ5{s,t,f} = mean(per-sub Δ-rates)`. + +- **Pro:** rewards broad, balanced improvement; no single-sub spike + dominates; a single-sub *regression* is averaged down on the + positive side but caught regardless by `HARD_REGRESSION_FLOOR` on + the flag side — so the asymmetry is deliberate (regressions are + zero-tolerance per-sub; improvements need breadth). +- **Pro:** robust to a noisy sub-battery — one flaky pack can't swing + the whole battery term. +- **Con:** heavily damps specialization gains. A fork that genuinely + advances one capability is told it barely improved. +- **Con:** with packs at ceiling (until #000046), the term is + inert regardless of aggregator — the dilution only bites once + there's headroom, which is *why this ticket parks behind #000046*. + +### 2.2 Option B — `max(0, per-sub Δ)` (best-of-the-improved) + +`Δ5{s,t,f} = max(0, max_i Δ_i)` — the largest single-sub gain, floored +at 0. + +- **Pro:** a single-sub +0.06 gain weighs at face value → ACCEPT. + Rewards specialization; "the fork improved *something*" ⇒ positive + term. +- **Con:** silent on the rest of the vector. A fork that improves one + sub by +0.05 and *regresses* four by −0.01 each (net-negative on + that battery, but each below `HARD_REGRESSION_FLOOR`) scores + positive on the `max` — the regressions are invisible to this term + and slip under the hard-regression flag. Would need the + `regression_penalty` term re-weighted to compensate. +- **Con:** discards how broad an improvement is — a uniform +0.05 and + a single +0.05 score identically. + +### 2.3 Option C — `sum` of per-sub Δs + +`Δ5{s,t,f} = Σ_i Δ_i`. + +- **Pro:** linear, total-improvement semantics; a single +0.06 scores + 0.06 and a uniform +0.06 across five scores 0.30 — breadth *and* + magnitude both count. +- **Con:** over-rewards broad tiny gains: five subs each +0.011 (each + below the 5pp bench noise floor — i.e. statistical noise, not real + signal) sums to 0.055 ≥ `SIGNAL_FLOOR` → ACCEPT on what is plausibly + zero real improvement. The mean is *deliberately* sub-noise-floor + in that case. +- **Con:** the `SIGNAL_FLOOR` / `HARD_REGRESSION_FLOOR` framing in + `docs/bench-maxing.md` is per-rate (5pp on *a* rate); a summed term + no longer maps cleanly onto "5pp" — the floors would need + re-derivation. + +### 2.4 Option D — parameterize it (recommended) + +Add `delta_aggregator ∈ {"mean", "max", "sum"}` to `WeightSet` (or a +sibling policy struct), default `"mean"`. `_delta_*` dispatches on it; +the chosen aggregator is recorded in `ScoredFork.breakdown` (so a +reader can replay the verdict) and in `fork_score_branches`' +`breakdown_blob`. `ESTIMATOR_VERSION` bumps **only if the default +changes** — a per-call override at the non-default value is recorded +but doesn't restate the estimator generation for everyone. + +- **Pro:** #000012's v8 acceptance protocol picks per-deployment + without a code change; the default stays `"mean"` so nothing breaks + silently; the choice is auditable. +- **Con:** another knob (Five-step §2 "delete the part" pushes back — + but this knob has a real, named requester: #000012's protocol + author wanting to tune fork sensitivity, and the alternative is + re-litigating mean-vs-max every time someone wants the other). + +--- + +## 3. Recommendation + +**Option D — parameterize, default `mean`** — but **do not implement +until #000046 lands a below-ceiling baseline.** Rationale: + +1. The dilution only matters once a pack has headroom; until then any + choice is synthetic (the §10.14 calibration already shows the + mechanics in-memory). Implementing now would be guessing. +2. Once #000046's hard pack exists, the aggregator choice becomes a + bench question, not an opinion: pin a parent below ceiling, hand- + craft a "broad small improvement" child and a "single big + improvement" child, run all three aggregators, see which one's + ACCEPT/MARGINAL/REJECT verdicts match the intuition fox wants. The + result decides the default (still recording the others as + selectable). +3. `mean` stays the default regardless — it's the conservative, + noise-robust, asymmetric-with-the-regression-side choice that + matches `docs/bench-maxing.md`'s per-rate floor framing. + +So this ticket parks behind #000046, then opens as a small +implementation (≈ 30 LOC + a `WeightSet` field + tests + an +`ESTIMATOR_VERSION` note if the default moves, which it shouldn't). + +--- + +## 4. Out of scope + +- **Changing the default away from `mean`** — that needs the #000046 + bench data; this ticket recommends *keeping* `mean` as default and + only adding the knob. +- **Re-weighting `regression_penalty`** — Option B would need it; if + fox ever picks `max` as default, that's part of *that* change, not + a separate ticket. +- **Touching `HARD_REGRESSION_FLOOR` / `SIGNAL_FLOOR`** — #000025 + §10.14 already recommended keeping both at 0.05; this ticket + doesn't reopen that. +- **The harder-fixture tier itself** — #000046. +- **`bench/scripts/fivef_threshold_calibration.py`** — it already + exercises the mechanics; if the aggregator becomes a parameter, + the calibration script gains a column showing each aggregator's + verdict (small follow-on, part of the implementation). + +--- + +## 5. Status + +**Open · awaiting go/no-go; parks until #000046 produces a +below-ceiling baseline.** Doc-only spec for the aggregator decision. +Opens when: #000046 lands ≥ 1 hard pack with a stable below-ceiling +rate (so the choice can be benched), OR fox decides the parameter is +worth adding now regardless. Closure: `delta_aggregator` is a +`WeightSet`/policy field with `mean` default, `_delta_*` dispatches on +it, the value is recorded in `breakdown` + `fork_score_branches`, and +the bench data behind the default choice is captured (in a +`bench/results/` note or this ticket). + +--- + +## 6. References + +- `arborist/substrate/fork_score.py` (`_delta_5s` / `_delta_5t` / + `_delta_5f`, `SIGNAL_FLOOR`, `HARD_REGRESSION_FLOOR`, + `ESTIMATOR_VERSION`) — the code this ticket would touch. +- `arborist/substrate/weights.py` (`WeightSet`) — where + `delta_aggregator` would live. +- `bench/results/5f-threshold-calibration-2026-05-11.md` (§3 sanity + checks, §4 "5× averaging dilution") + `ticket-000012` §8 — the + calibration finding that motivates this ticket. +- `docs/bench-maxing.md` — the per-rate 5pp signal-floor framing the + `mean` aggregator preserves and `sum` would break. +- `docs/tickets/ticket-000046-harder-5sf-fixture-tier.md` — the + prerequisite; the aggregator choice is benchable only once a hard + pack exists. +- `docs/tickets/ticket-000012-selection-consensus-protocol.md` (§5 + fork choice, §8 calibration handoff) — the v8 protocol that would + consume `delta_aggregator`. +- `docs/tickets/ticket-000025-5f-battery.md` §10.14 — where this + loose end was flagged.