arborist/docs/tickets/ticket-000046-harder-5sf-fixture-tier.md

Ticket #000046 — Harder 5S/5T/5F fixture tier (below-ceiling baselines)

Status: closed · 2026-05-11. Phase 1 (Option A) — 5F Falsification hard pack bench/fixtures/5f/falsification-hard-v1.jsonl (12 near-misses; make bench-5f-falsification-hard / make bench-fork-baseline-hard). Phase 2 — 5F Formulate hard pack bench/fixtures/5f/formulate-hard-v1.jsonl (12 mis-segment inputs, rate 4/12, stable; make bench-5f-formulate-hard). Phase 3 (the closure) — a real verify_quotes tightening: a paraphrase numeric-agreement gate (_numeric_signature + a check in _check_each_with_paraphrase) that demotes a token-covering span asserting a digit-number the source lacks (modulo thousands-comma). It lifted the falsification-hard rate 4/12 → 6/12 = 0.5 and was bench-gated — make bench-qa n=3 × 75 × 3 before/after showed no STRICT-rate regression on legitimate answers (the only gate-caused QA shift was correctly demoting a fictional-year claim STRICT → HYBRID; all other deltas were within the 5-pp noise floor and were LLM re-answer variance). fork_score's γ·Δ5f went positive on that real improvement → the loop is closed end-to-end. Headroom remains (6 over-grounds still in the hard pack — 4 HYBRID_ENTITY + 2 recombined-no-number STRICT_PARAPHRASE — plus the Formulate pack's 8); a bigger order/dependency-aware verifier upgrade would close those, tracked separately if wanted (see §5). See §5 + §7. Opened: 2026-05-11 Scope: Give the 5S/5T/5F bench packs headroom — a fixture tier the runtime does not pass at rate 1.0 — so the ForkScore Δ-rate terms (α·Δ5s + β·Δ5t + γ·Δ5f in arborist/substrate/fork_score.py) can be strictly positive for a genuine improvement. Doc-only proposal: options + a narrow recommended start; no fixtures authored in this ticket. Audience: fox + #000025 follow-up readers + #000012's v8 acceptance-protocol author + future bench authors. Hard constraint: the existing *-v1.jsonl packs keep their pinned fixture digests — a harder tier is a new pack (*-hard-v1.jsonl or a tier field), never an edit to a pinned one. No schema / governance-hash / canonicalization-version change. No LLM-as-judge in any runner (the discipline that lets the bench be a scoreboard, not a vibe check).

---

1. Problem statement

The #000025 §10.14 ForkScore threshold calibration (bench/scripts/fivef_threshold_calibration.py → bench/results/5f-threshold-calibration-2026-05-11.md, summarised in ticket-000012 §8) surfaced a structural fact: every canonical 5S, 5T, and 5F pack runs at rate 1.0 at HEAD. Consequence in fork_score:

• _delta_5s / _delta_5t / _delta_5f are means of per-sub-battery Δ-rates. A child at ceiling vs a parent at ceiling ⇒ every Δ is 0 ⇒ the three battery terms contribute exactly 0. An unchanged child therefore scores 0 → MARGINAL (not ACCEPT — which is the right behaviour, but it means the bench terms are inert).
• A child can only move those terms negative (regress a pack) — the bench Δ-rate has no upside at the current pack difficulty.
• So fork acceptance today is carried entirely by the efficiency-aware bonuses (adaptation_efficiency / feedback_efficiency deltas) and the non-bench terms (selfmodel_calibration_gain, audit_completeness, validator_diversity, …). The "did the organism's measurable capabilities improve?" signal is dormant.

For the bench Δ-rate to carry positive signal, something the runtime gets partially wrong has to be in the corpus. The embedded packs can't be that something: the embedded evaluators are deterministic and the gold answers are constructed to match them — a fixture the embedded evaluator "fails" would be a fixture with a wrong gold, not a hard problem. So a harder tier is necessarily a live-path tier — inputs fed to a real, imperfect arborist surface (parse_claims, verify_quotes, store_snapshot, the retrieval pipeline) where the surface produces wrong / partial / over-confident output and the fixture's expected records the correct outcome the surface misses. Then rate < 1.0, and a future improvement to that surface moves the rate up — real positive Δ.

---

2. Options

2.1 Option A — live-path "hard" tier (recommended, narrow start)

Add a *-hard-v1.jsonl pack to one sub-battery where the live surface most plausibly fails on difficult input, confirm the rate sits below 1.0 and is stable across runs (the live surfaces are deterministic — imperfect, not random — so rate must be a fixed fraction), then a make bench-fork-baseline-hard that pins it as a below-ceiling parent. Iterate to other sub-batteries only if the loop proves out.

Two strong candidates for the first hard pack:

• 5F Falsification (verify_quotes) — there's already a documented soft-signal gap: 5f-fal-live-003 records that the entity-strategy verifier soft-matches "Insulin / Penicillin" as one claim because both name Alexander Fleming. A hard pack of near-miss quote pairs that should land UNGROUNDED but the entity strategy over-grounds would put the Falsification rate below 1.0 today; a future tightening of the entity strategy (or the title-relevance hard check picking up more) raises it. Directly bench-able.
• 5F Formulate (parse_claims) — genuinely messy prose where the claim-lattice parser produces the wrong claim count / pointer-set / order. The match_policy is already structural (sorted-approx claim text + exact-set pointer IDs + required order), so a hard pack just needs inputs the parser mis-segments.

Cost: ~10-20 hard fixtures for the first pack + a make target + a test pinning "rate is below 1.0 and equals X". Risk: low — the live surfaces are deterministic; the only failure mode is "the surface is actually perfect on the inputs we picked", which just means picking harder ones.

2.2 Option B — deliberately-degraded parent baseline

Ship a pinned bench/results/baseline-degraded-v1.json with knocked-down rates (e.g. 0.85 uniformly), so a real child run (at 1.0) shows +0.15 → ACCEPT. Cheap (one JSON file, no fixtures). But the "improvement" is fictional — the child didn't get better, the parent was artificially handicapped. This is useful for exercising the ForkScore mechanics (which the §10.14 calibration already does in-memory via _degraded_parent), not for real fork selection. It does not make the bench Δ-rate carry real signal; it just makes the synthetic demo persist as a file. Probably not worth a ticket on its own — note it here as the cheap alternative we explicitly chose not to pursue as the primary path.

2.3 Option C — do nothing; revisit when headroom surfaces organically

Accept that the bench Δ-rate is inert at the current pack difficulty, keep the §8 note in #000012, and add a hard tier only when one shows up for free: a new π* domain whose kernel is imperfect, an LLM-backed sub-battery, or a real regression that drops a pack below ceiling. Lowest cost; the cost is that the dormant signal stays dormant indefinitely.

---

3. Recommendation

Option A, scoped to one sub-battery first — Falsification. It has a pre-documented gap (5f-fal-live-003), the live surface (verify_quotes) is deterministic, and a hard pack there immediately gives ForkScore a below-ceiling 5F sub-battery whose rate a future verifier improvement can lift. If the loop proves out (rate stable below 1.0, a planted verifier improvement demonstrably raises it, ForkScore shows a positive γ·Δ5f term), extend to Formulate, then to whichever 5S/5T sub-batteries have an imperfect live surface (retrieval-backed ones are the obvious next look). Treat Option B as a documented non-choice and Option C as the fallback if A's first pack turns out perfect on every input we can author.

Defer #000047 (_delta_* mean-vs-max-vs-sum aggregation) until at least one hard pack lands — the aggregator choice is empirically testable only once there's a below-ceiling baseline to bench against.

---

4. Out of scope

• Authoring the hard fixtures — this ticket picks the strategy and the first sub-battery; the fixture pack lands in the implementation commit chain this ticket gates.
• Changing fork_score — the scorer already consumes whatever rates the packs produce. A hard tier doesn't touch fork_score.py; it changes the inputs.
• Re-pinning the existing *-v1.jsonl packs — hard off-limits; those digests are referenced across closed tickets.
• An LLM-backed sub-battery — a different (larger) way to get headroom; out of scope here, ticket-able separately if it ever becomes the path.
• _delta_* aggregator choice — #000047.

---

5. Status

In progress — Phase 1 + Phase 2 landed 2026-05-11.

Phase 1 (Option A) — 5F Falsification hard pack

• Hard pack: bench/fixtures/5f/falsification-hard-v1.jsonl — 12 near-misses, each a false / unsupported claim whose correct verdict is UNGROUNDED (recorded in expected_reason). 8 of 12 are over-grounded by arborist.qa.verify.verify_quotes at HEAD (paraphrase token-coverage → STRICT_PARAPHRASE, entity-proximity → HYBRID_ENTITY, on incidental overlap) so those tasks fail by design; 4 the verifier handles correctly. Rate at HEAD = 4/12 = 0.333… — a stable below-ceiling baseline (verify_quotes is pure-lexical / deterministic). Built around the pre-documented gap 5f-fal-live-003.
• Make targets: make bench-5f-falsification-hard (runs the pack; the runner exits 1 because 8 fixtures fail by design — the targets || true past that, the JSON is still written), make bench-fork-baseline-hard → bench/results/baseline-falsification-hard.json (pins the below-ceiling rate as a ForkScore parent artifact).
• Tests: tests/test_bench_batteries.py — test_5f_falsification_hard_pack_below_ceiling (pins rate 4/12, source=live, every fixture asserts UNGROUNDED, the 8 fails are over-grounds not abstentions) + test_fork_score_positive_gamma_5f_on_hard_falsification_improvement (the worked example: fork_score(parent={5f/falsification: 1/3}, child={5f/falsification: 1.0}) → γ·Δ5f ≈ +0.133 > 0, verdict ACCEPT, no regression flags — the bench Δ-rate carrying signal it can't carry while every canonical pack is at ceiling).
• Not in make bench-5f / make bench-5s5t5f / make bench-fork-baseline / runner --all — the hard pack is a separate, deliberately-failing artifact pinned on its own.

Phase 2 — 5F Formulate hard pack

• Hard pack: bench/fixtures/5f/formulate-hard-v1.jsonl — 12 prose inputs that arborist.qa.parse_claims.parse_pointer_claims should segment into a particular claim lattice (recorded in expected_lattice). The parser is line/bullet-based — one line ⇒ one claim, [E#] tokens attach to it — so 8 of 12 it mis-segments: merges and-/semicolon-/dash-joined or (1)(2)-enumerated multi-claim lines into one claim with all the pointers, or splits a wrapped bullet into two. Those 8 fail at HEAD on claim-count mismatch (+ pointer-set mismatch); the other 4 are well-formed bullet/ numbered lists / single claims the parser handles right. Rate at HEAD = 4/12 = 0.333…, stable (parse_pointer_claims is deterministic). A claim-lattice parser that does sentence/clause segmentation (split on . , ;, subordinating conjunctions, inline enumerations) + joins wrapped bullets lifts the rate toward 1.0 → positive γ·Δ5f for that child fork.
• Make target: make bench-5f-formulate-hard (runs it; || true past the runner's nonzero-on-failures exit). Not in make bench-5f / runner --all.
• Test: tests/test_bench_batteries.py — test_5f_formulate_hard_pack_below_ceiling (pins rate 4/12, source=live, the 8 fails are claim-count mis-segments).

Phase 3 (the closure) — verify_quotes paraphrase numeric-agreement gate

• Change: arborist/qa/verify.py — _numeric_signature(text) (comma-stripped digit-runs: 8,849 and 8849 collapse; 300 stays distinct from 300000 ← 300,000) + a gate in _check_each_with_paraphrase: a span that token-covers the source ≥ paraphrase_coverage but asserts a digit-number the source lacks is not paraphrase-grounded — it goes to unverified. Catches the near-miss the lexical coverage check is blind to ("Water boils at 50 degrees" against a source saying 100 token-covers 100% because 50/100 aren't > 4-char content tokens). The gate is narrow by construction: only fires on the paraphrase fallback (verbatim / span / entity paths untouched), and only on a digit-number, modulo thousands-comma. A rounding- paraphrase demoting here is the honest verdict — it isn't a verbatim grounding.
• Effect on the hard pack: lifts 5f-fal-hard-004 (50 vs 100) and -007 (300 vs 300,000) to UNGROUNDED → falsification-hard rate 4/12 → 6/12 = 0.5. The other 6 over-grounds (4 HYBRID_ENTITY
  • 2 recombined-no-number STRICT_PARAPHRASE — "Mercury is the largest" vs "Jupiter is the largest; Mercury is the smallest") are unaffected; the Formulate hard pack is unaffected (it exercises parse_pointer_claims, not verify_quotes).
• Bench gate: make bench-qa (n=3 × 75 questions × 3 modes = 675 cells) before (bench/qa_results/2026-05-11T13-42-38Z.md) and after (...T14-19-51Z.md). STRICT-rate: quote 0.53 → 0.50, pointer 0.23 → 0.25, lattice 0.44 → 0.45 — all within the 5-pp noise floor. Per-row diff: the only clearly gate-attributable QA shift was the fictional "our cold fusion breakthrough" year-claim demoting STRICT → HYBRID (×3 samples) — a correct demotion (the year isn't grounded); every other transition was quote→quote / claim_lattice →claim_lattice LLM re-answer variance, which the gate doesn't touch. No regression on legitimate answers.
• Worked example / the loop: fork_score on the real improvement — parent {5f/falsification: 4/12} → child {5f/falsification: 6/12} → γ·Δ5f = (1/6)/5 ≈ +0.033 > 0 (positive, just under ACCEPT given the ÷5 dilution — a single improvement of this size is MARGINAL by design; the test pins the full-lift-to-1.0 case at ACCEPT). The bench Δ-rate carries signal on the 5F/falsification axis now, on a real change.
• Tests: tests/test_verify.py — test_numeric_signature_comma_normalized, test_numeric_signature_subset_matches_comma_variant, test_paraphrase_numeric_disagreement_rejected, test_paraphrase_numeric_gate_is_narrow_no_number_unaffected, test_paraphrase_number_present_in_source_still_verifies; tests/test_bench_batteries.py test_5f_falsification_hard_pack_below_ceiling re-pinned 4/12 → 6/12.

Headroom (optional follow-up — not a #000046 blocker)

Tracked as #000048 (opened 2026-05-11). As of 2026-05-11: #000048 step 2.1 (entity salient-token gate) closed the 4 HYBRID_ENTITY over-grounds → falsification-hard 10/12; step 2.4 (parse_pointer_claims clause segmentation) closed all 8 Formulate mis-segments → formulate-hard 12/12 (that pack at ceiling). The last residue: 2 STRICT_PARAPHRASE in falsification-hard (hard-003 Mercury, hard-005 Einstein — the false claim recombines source tokens into a different true statement). #000048 step 2.2 (a single-clause- containment lexical check) was attempted and reverted — it can't distinguish that recombination from a legitimate cross-sentence summary, and no threshold separates them (#000048 §5 "What we learned"); closing those 2 needs an attribution-aware verifier — its own ticket, #000049 — so they stand as documented residue. #000046 is done: the below-ceiling baselines exist, real surface improvements (numeric gate, entity gate, clause segmenter) lifted rates, and ForkScore's bench-Δ went positive on them.

---

6. References

• bench/fixtures/5f/falsification-hard-v1.jsonl — the Phase-1 hard pack (12 near-misses, rate 6/12 after the Phase-3 numeric gate). make bench-5f-falsification-hard runs it; make bench-fork-baseline-hard pins it to bench/results/baseline-falsification-hard.json.
• bench/fixtures/5f/formulate-hard-v1.jsonl — the Phase-2 hard pack (12 mis-segment inputs, rate 4/12 at HEAD). make bench-5f-formulate-hard runs it.
• arborist/qa/verify.py — _numeric_signature + the _check_each_with_paraphrase numeric-agreement gate (Phase 3).
• arborist/qa/parse_claims.py (parse_pointer_claims) — the line/bullet-based parser the Phase-2 pack exercises.
• bench/qa_results/2026-05-11T13-42-38Z.md (before) + bench/qa_results/2026-05-11T14-19-51Z.md (after) — the QA-bench gate for the Phase-3 verifier change.
• tests/test_verify.py — the 5 numeric-gate tests (test_numeric_signature_*, test_paraphrase_numeric_disagreement_rejected, test_paraphrase_numeric_gate_is_narrow_no_number_unaffected, test_paraphrase_number_present_in_source_still_verifies).
• tests/test_bench_batteries.py — test_5f_falsification_hard_pack_below_ceiling (6/12 after Phase 3), test_5f_formulate_hard_pack_below_ceiling (4/12) + test_fork_score_positive_gamma_5f_on_hard_falsification_improvement (the worked example).
• bench/scripts/fivef_threshold_calibration.py + bench/results/5f-threshold-calibration-2026-05-11.md — the calibration that surfaced the ceiling-saturation problem.
• docs/tickets/ticket-000012-selection-consensus-protocol.md §8 — the handoff section recording "ceiling saturation → bench Δ-rate inert; track a harder tier as a #000025 follow-up".
• docs/tickets/ticket-000025-5f-battery.md — parent; this is the follow-up the §10.14 closure flagged.
• bench/fixtures/5f/falsification-live-v1.jsonl (5f-fal-live-003) — the pre-documented entity-strategy soft-signal gap the first hard pack would build around.
• arborist/qa/verify.py (verify_quotes, the entity strategy) / arborist/qa/parse_claims.py — the imperfect live surfaces a hard tier exercises.
• docs/tickets/ticket-000047-fork-score-delta-aggregator.md — sibling; parks until this ticket produces a below-ceiling baseline.