# Ticket #000046 — Harder 5S/5T/5F fixture tier (below-ceiling baselines)

**Status:** **closed · 2026-05-11.** Phase 1 (Option A) — 5F
Falsification hard pack `bench/fixtures/5f/falsification-hard-v1.jsonl`
(12 near-misses; `make bench-5f-falsification-hard` /
`make bench-fork-baseline-hard`). Phase 2 — 5F Formulate hard pack
`bench/fixtures/5f/formulate-hard-v1.jsonl` (12 mis-segment inputs,
rate 4/12, stable; `make bench-5f-formulate-hard`). Phase 3 (the
closure) — a **real `verify_quotes` tightening**: a paraphrase
numeric-agreement gate (`_numeric_signature` + a check in
`_check_each_with_paraphrase`) that demotes a token-covering span
asserting a digit-number the source lacks (modulo thousands-comma).
It lifted the falsification-hard rate **4/12 → 6/12 = 0.5** and was
bench-gated — `make bench-qa` n=3 × 75 × 3 before/after showed no
STRICT-rate regression on legitimate answers (the only gate-caused
QA shift was correctly demoting a fictional-year claim STRICT →
HYBRID; all other deltas were within the 5-pp noise floor and were
LLM re-answer variance). `fork_score`'s γ·Δ5f went positive on that
real improvement → the loop is closed end-to-end. Headroom remains
(6 over-grounds still in the hard pack — 4 HYBRID_ENTITY + 2
recombined-no-number STRICT_PARAPHRASE — plus the Formulate pack's
8); a bigger order/dependency-aware verifier upgrade would close
those, tracked separately if wanted (see §5). See §5 + §7.
**Opened:** 2026-05-11
**Scope:** Give the 5S/5T/5F bench packs *headroom* — a fixture tier
the runtime does **not** pass at rate 1.0 — so the ForkScore Δ-rate
terms (`α·Δ5s + β·Δ5t + γ·Δ5f` in `arborist/substrate/fork_score.py`)
can be strictly positive for a genuine improvement. Doc-only proposal:
options + a narrow recommended start; no fixtures authored in this
ticket.
**Audience:** fox + #000025 follow-up readers + #000012's v8
acceptance-protocol author + future bench authors.
**Hard constraint:** the existing `*-v1.jsonl` packs keep their pinned
fixture digests — a harder tier is a **new** pack (`*-hard-v1.jsonl`
or a `tier` field), never an edit to a pinned one. No schema /
governance-hash / canonicalization-version change. No LLM-as-judge in
any runner (the discipline that lets the bench be a scoreboard, not a
vibe check).

---

## 1. Problem statement

The #000025 §10.14 ForkScore threshold calibration
(`bench/scripts/fivef_threshold_calibration.py` →
`bench/results/5f-threshold-calibration-2026-05-11.md`, summarised in
`ticket-000012` §8) surfaced a structural fact: **every canonical 5S,
5T, and 5F pack runs at rate 1.0 at HEAD.** Consequence in
`fork_score`:

- `_delta_5s / _delta_5t / _delta_5f` are means of per-sub-battery
  Δ-rates. A child at ceiling vs a parent at ceiling ⇒ every Δ is 0
  ⇒ the three battery terms contribute exactly 0. An *unchanged*
  child therefore scores 0 → MARGINAL (not ACCEPT — which is the
  right behaviour, but it means the bench terms are inert).
- A child can only move those terms *negative* (regress a pack) — the
  bench Δ-rate has no upside at the current pack difficulty.
- So fork *acceptance* today is carried entirely by the
  efficiency-aware bonuses (`adaptation_efficiency` /
  `feedback_efficiency` deltas) and the non-bench terms
  (`selfmodel_calibration_gain`, `audit_completeness`,
  `validator_diversity`, …). The "did the organism's measurable
  capabilities improve?" signal is dormant.

For the bench Δ-rate to carry positive signal, *something the runtime
gets partially wrong* has to be in the corpus. The embedded packs
can't be that something: the embedded evaluators are deterministic
and the gold answers are constructed to match them — a fixture the
embedded evaluator "fails" would be a fixture with a wrong gold, not
a hard problem. So a harder tier is necessarily a **live-path** tier
— inputs fed to a real, imperfect arborist surface (`parse_claims`,
`verify_quotes`, `store_snapshot`, the retrieval pipeline) where the
surface produces wrong / partial / over-confident output and the
fixture's `expected` records the *correct* outcome the surface
misses. Then rate < 1.0, and a future improvement to that surface
moves the rate up — real positive Δ.

---

## 2. Options

### 2.1 Option A — live-path "hard" tier (recommended, narrow start)

Add a `*-hard-v1.jsonl` pack to one sub-battery where the live
surface most plausibly fails on difficult input, confirm the rate
sits below 1.0 *and is stable across runs* (the live surfaces are
deterministic — imperfect, not random — so rate must be a fixed
fraction), then a `make bench-fork-baseline-hard` that pins it as a
below-ceiling parent. Iterate to other sub-batteries only if the
loop proves out.

Two strong candidates for the first hard pack:

- **5F Falsification (`verify_quotes`)** — there's already a
  documented soft-signal gap: `5f-fal-live-003` records that the
  entity-strategy verifier soft-matches "Insulin / Penicillin" as one
  claim because both name Alexander Fleming. A hard pack of near-miss
  quote pairs that *should* land UNGROUNDED but the entity strategy
  over-grounds would put the Falsification rate below 1.0 today; a
  future tightening of the entity strategy (or the title-relevance
  hard check picking up more) raises it. Directly bench-able.
- **5F Formulate (`parse_claims`)** — genuinely messy prose where the
  claim-lattice parser produces the wrong claim count / pointer-set /
  order. The `match_policy` is already structural (sorted-approx
  claim text + exact-set pointer IDs + required order), so a hard
  pack just needs inputs the parser mis-segments.

Cost: ~10-20 hard fixtures for the first pack + a make target + a
test pinning "rate is below 1.0 and equals X". Risk: low — the live
surfaces are deterministic; the only failure mode is "the surface is
actually perfect on the inputs we picked", which just means picking
harder ones.

### 2.2 Option B — deliberately-degraded parent baseline

Ship a pinned `bench/results/baseline-degraded-v1.json` with
knocked-down rates (e.g. 0.85 uniformly), so a real child run (at
1.0) shows +0.15 → ACCEPT. Cheap (one JSON file, no fixtures). But
the "improvement" is fictional — the child didn't get better, the
parent was artificially handicapped. This is useful for *exercising
the ForkScore mechanics* (which the §10.14 calibration already does
in-memory via `_degraded_parent`), not for real fork selection. It
does **not** make the bench Δ-rate carry real signal; it just makes
the synthetic demo persist as a file. Probably not worth a ticket on
its own — note it here as the cheap alternative we explicitly chose
*not* to pursue as the primary path.

### 2.3 Option C — do nothing; revisit when headroom surfaces organically

Accept that the bench Δ-rate is inert at the current pack difficulty,
keep the §8 note in #000012, and add a hard tier only when one shows
up for free: a new π* domain whose kernel is imperfect, an
LLM-backed sub-battery, or a real regression that drops a pack below
ceiling. Lowest cost; the cost is that the dormant signal stays
dormant indefinitely.

---

## 3. Recommendation

**Option A, scoped to one sub-battery first — Falsification.** It has
a pre-documented gap (`5f-fal-live-003`), the live surface
(`verify_quotes`) is deterministic, and a hard pack there immediately
gives ForkScore a below-ceiling 5F sub-battery whose rate a future
verifier improvement can lift. If the loop proves out (rate stable
below 1.0, a planted verifier improvement demonstrably raises it,
ForkScore shows a positive `γ·Δ5f` term), extend to Formulate, then
to whichever 5S/5T sub-batteries have an imperfect live surface
(retrieval-backed ones are the obvious next look). Treat Option B as
a documented non-choice and Option C as the fallback if A's first
pack turns out perfect on every input we can author.

Defer **#000047** (`_delta_*` mean-vs-max-vs-sum aggregation) until at
least one hard pack lands — the aggregator choice is empirically
testable only once there's a below-ceiling baseline to bench against.

---

## 4. Out of scope

- **Authoring the hard fixtures** — this ticket picks the strategy
  and the first sub-battery; the fixture pack lands in the
  implementation commit chain this ticket gates.
- **Changing `fork_score`** — the scorer already consumes whatever
  rates the packs produce. A hard tier doesn't touch
  `fork_score.py`; it changes the *inputs*.
- **Re-pinning the existing `*-v1.jsonl` packs** — hard off-limits;
  those digests are referenced across closed tickets.
- **An LLM-backed sub-battery** — a different (larger) way to get
  headroom; out of scope here, ticket-able separately if it ever
  becomes the path.
- **`_delta_*` aggregator choice** — #000047.

---

## 5. Status

**In progress — Phase 1 + Phase 2 landed 2026-05-11.**

### Phase 1 (Option A) — 5F Falsification hard pack

- **Hard pack:** `bench/fixtures/5f/falsification-hard-v1.jsonl` — 12
  near-misses, each a *false / unsupported* claim whose correct
  verdict is `UNGROUNDED` (recorded in `expected_reason`). 8 of 12
  are over-grounded by `arborist.qa.verify.verify_quotes` at HEAD
  (paraphrase token-coverage → `STRICT_PARAPHRASE`, entity-proximity →
  `HYBRID_ENTITY`, on incidental overlap) so those tasks fail by
  design; 4 the verifier handles correctly. **Rate at HEAD = 4/12 =
  0.333…** — a stable below-ceiling baseline (`verify_quotes` is
  pure-lexical / deterministic). Built around the pre-documented gap
  `5f-fal-live-003`.
- **Make targets:** `make bench-5f-falsification-hard` (runs the pack;
  the runner exits 1 because 8 fixtures fail by design — the targets
  `|| true` past that, the JSON is still written), `make
  bench-fork-baseline-hard` → `bench/results/baseline-falsification-hard.json`
  (pins the below-ceiling rate as a ForkScore parent artifact).
- **Tests:** `tests/test_bench_batteries.py` —
  `test_5f_falsification_hard_pack_below_ceiling` (pins rate 4/12,
  source=live, every fixture asserts `UNGROUNDED`, the 8 fails are
  over-grounds not abstentions) + `test_fork_score_positive_gamma_5f_on_hard_falsification_improvement`
  (the worked example: `fork_score(parent={5f/falsification: 1/3},
  child={5f/falsification: 1.0})` → `γ·Δ5f ≈ +0.133 > 0`, verdict
  ACCEPT, no regression flags — the bench Δ-rate carrying signal it
  can't carry while every canonical pack is at ceiling).
- **Not in** `make bench-5f` / `make bench-5s5t5f` / `make
  bench-fork-baseline` / `runner --all` — the hard pack is a
  separate, deliberately-failing artifact pinned on its own.

### Phase 2 — 5F Formulate hard pack

- **Hard pack:** `bench/fixtures/5f/formulate-hard-v1.jsonl` — 12
  prose inputs that `arborist.qa.parse_claims.parse_pointer_claims`
  *should* segment into a particular claim lattice (recorded in
  `expected_lattice`). The parser is line/bullet-based — one line ⇒
  one claim, `[E#]` tokens attach to it — so 8 of 12 it mis-segments:
  merges and-/semicolon-/dash-joined or `(1)(2)`-enumerated multi-claim
  lines into one claim with all the pointers, or splits a wrapped
  bullet into two. Those 8 fail at HEAD on claim-count mismatch
  (+ pointer-set mismatch); the other 4 are well-formed bullet/
  numbered lists / single claims the parser handles right. **Rate at
  HEAD = 4/12 = 0.333…**, stable (`parse_pointer_claims` is
  deterministic). A claim-lattice parser that does sentence/clause
  segmentation (split on `. `, `;`, subordinating conjunctions, inline
  enumerations) + joins wrapped bullets lifts the rate toward 1.0 →
  positive `γ·Δ5f` for that child fork.
- **Make target:** `make bench-5f-formulate-hard` (runs it; `|| true`
  past the runner's nonzero-on-failures exit). Not in `make bench-5f`
  / `runner --all`.
- **Test:** `tests/test_bench_batteries.py` —
  `test_5f_formulate_hard_pack_below_ceiling` (pins rate 4/12,
  source=live, the 8 fails are claim-count mis-segments).

### Phase 3 (the closure) — `verify_quotes` paraphrase numeric-agreement gate

- **Change:** `arborist/qa/verify.py` —
  `_numeric_signature(text)` (comma-stripped digit-runs: `8,849` and
  `8849` collapse; `300` stays distinct from `300000` ← `300,000`) +
  a gate in `_check_each_with_paraphrase`: a span that token-covers
  the source ≥ `paraphrase_coverage` but asserts a digit-number the
  source lacks is **not** paraphrase-grounded — it goes to
  `unverified`. Catches the near-miss the lexical coverage check is
  blind to ("Water boils at 50 degrees" against a source saying 100
  token-covers 100% because `50`/`100` aren't > 4-char content
  tokens). The gate is **narrow by construction**: only fires on the
  paraphrase fallback (verbatim / span / entity paths untouched), and
  only on a digit-number, modulo thousands-comma. A rounding-
  paraphrase demoting here is the honest verdict — it isn't a
  verbatim grounding.
- **Effect on the hard pack:** lifts `5f-fal-hard-004` (50 vs 100)
  and `-007` (300 vs 300,000) to UNGROUNDED → **falsification-hard
  rate 4/12 → 6/12 = 0.5**. The other 6 over-grounds (4 HYBRID_ENTITY
  + 2 recombined-no-number STRICT_PARAPHRASE — "Mercury is the
  largest" vs "Jupiter is the largest; Mercury is the smallest") are
  unaffected; the Formulate hard pack is unaffected (it exercises
  `parse_pointer_claims`, not `verify_quotes`).
- **Bench gate:** `make bench-qa` (n=3 × 75 questions × 3 modes = 675
  cells) before (`bench/qa_results/2026-05-11T13-42-38Z.md`) and
  after (`...T14-19-51Z.md`). STRICT-rate: quote 0.53 → 0.50, pointer
  0.23 → 0.25, lattice 0.44 → 0.45 — all within the 5-pp noise floor.
  Per-row diff: the *only* clearly gate-attributable QA shift was the
  fictional "our cold fusion breakthrough" year-claim demoting STRICT
  → HYBRID (×3 samples) — a **correct** demotion (the year isn't
  grounded); every other transition was `quote→quote` / `claim_lattice
  →claim_lattice` LLM re-answer variance, which the gate doesn't
  touch. No regression on legitimate answers.
- **Worked example / the loop:** `fork_score` on the real improvement
  — parent `{5f/falsification: 4/12}` → child `{5f/falsification:
  6/12}` → `γ·Δ5f = (1/6)/5 ≈ +0.033 > 0` (positive, just under
  ACCEPT given the ÷5 dilution — a *single* improvement of this size
  is MARGINAL by design; the test pins the full-lift-to-1.0 case at
  ACCEPT). The bench Δ-rate carries signal on the 5F/falsification
  axis now, on a real change.
- **Tests:** `tests/test_verify.py` — `test_numeric_signature_comma_normalized`,
  `test_numeric_signature_subset_matches_comma_variant`,
  `test_paraphrase_numeric_disagreement_rejected`,
  `test_paraphrase_numeric_gate_is_narrow_no_number_unaffected`,
  `test_paraphrase_number_present_in_source_still_verifies`;
  `tests/test_bench_batteries.py` `test_5f_falsification_hard_pack_below_ceiling`
  re-pinned 4/12 → 6/12.

### Headroom (optional follow-up — not a #000046 blocker)

Tracked as **#000048** (opened 2026-05-11). As of 2026-05-11:
#000048 step 2.1 (entity salient-token gate) closed the 4
HYBRID_ENTITY over-grounds → `falsification-hard` 10/12; step 2.4
(`parse_pointer_claims` clause segmentation) closed all 8 Formulate
mis-segments → `formulate-hard` 12/12 (that pack at ceiling). The
last residue: 2 STRICT_PARAPHRASE in `falsification-hard` (hard-003
Mercury, hard-005 Einstein — the false claim recombines source tokens
into a different true statement). #000048 step 2.2 (a single-clause-
containment lexical check) was *attempted and reverted* — it can't
distinguish that recombination from a legitimate cross-sentence
summary, and no threshold separates them (#000048 §5 "What we
learned"); closing those 2 needs an attribution-aware verifier —
its own ticket, **#000049** — so they stand as
documented residue. #000046 is done: the below-ceiling baselines
exist, real surface improvements (numeric gate, entity gate, clause
segmenter) lifted rates, and ForkScore's bench-Δ went positive on
them.

---

## 6. References

- `bench/fixtures/5f/falsification-hard-v1.jsonl` — the Phase-1 hard
  pack (12 near-misses, rate 6/12 after the Phase-3 numeric gate).
  `make bench-5f-falsification-hard` runs it; `make
  bench-fork-baseline-hard` pins it to
  `bench/results/baseline-falsification-hard.json`.
- `bench/fixtures/5f/formulate-hard-v1.jsonl` — the Phase-2 hard pack
  (12 mis-segment inputs, rate 4/12 at HEAD). `make
  bench-5f-formulate-hard` runs it.
- `arborist/qa/verify.py` — `_numeric_signature` + the
  `_check_each_with_paraphrase` numeric-agreement gate (Phase 3).
- `arborist/qa/parse_claims.py` (`parse_pointer_claims`) — the
  line/bullet-based parser the Phase-2 pack exercises.
- `bench/qa_results/2026-05-11T13-42-38Z.md` (before) +
  `bench/qa_results/2026-05-11T14-19-51Z.md` (after) — the QA-bench
  gate for the Phase-3 verifier change.
- `tests/test_verify.py` — the 5 numeric-gate tests (`test_numeric_signature_*`,
  `test_paraphrase_numeric_disagreement_rejected`,
  `test_paraphrase_numeric_gate_is_narrow_no_number_unaffected`,
  `test_paraphrase_number_present_in_source_still_verifies`).
- `tests/test_bench_batteries.py` —
  `test_5f_falsification_hard_pack_below_ceiling` (6/12 after Phase 3),
  `test_5f_formulate_hard_pack_below_ceiling` (4/12) +
  `test_fork_score_positive_gamma_5f_on_hard_falsification_improvement`
  (the worked example).
- `bench/scripts/fivef_threshold_calibration.py` +
  `bench/results/5f-threshold-calibration-2026-05-11.md` — the
  calibration that surfaced the ceiling-saturation problem.
- `docs/tickets/ticket-000012-selection-consensus-protocol.md` §8 —
  the handoff section recording "ceiling saturation → bench Δ-rate
  inert; track a harder tier as a #000025 follow-up".
- `docs/tickets/ticket-000025-5f-battery.md` — parent; this is the
  follow-up the §10.14 closure flagged.
- `bench/fixtures/5f/falsification-live-v1.jsonl` (`5f-fal-live-003`)
  — the pre-documented entity-strategy soft-signal gap the first
  hard pack would build around.
- `arborist/qa/verify.py` (`verify_quotes`, the entity strategy) /
  `arborist/qa/parse_claims.py` — the imperfect live surfaces a hard
  tier exercises.
- `docs/tickets/ticket-000047-fork-score-delta-aggregator.md` —
  sibling; parks until this ticket produces a below-ceiling baseline.
