# Ticket #000060 — H-ABCDEFG same-model substrate-delta harness (+ jaggedness tensor + performance curvature)

**Status:** in progress · v1 deterministic jaggedness instrument
landed 2026-05-21 (`bench/jaggedness.py` + `make bench-jaggedness`);
A-vs-C answer-quality arm already existed under #000057
(`control_ab.py` / `control_sweep.py`) — NOT rebuilt. Curvature +
LLM-arm jaggedness delta still open (see §8).
**Opened:** 2026-05-20
**Scope:** The report's "decisive proof" — a harness that runs the
SAME base model with and without the arborist substrate over
long-horizon / adversarial / non-jagged batteries, and reports the
delta. Produces two new metrics — a jaggedness tensor and discrete
performance curvature — as first-class outputs.
**Audience:** fox + a Dav1d de-novo review of the empirical bet + bench
maintainers (`bench/`, `docs/benchmarks.md`).
**Hard constraint:** deterministic where possible; gate numbers come
from bench-qa pipeline output, never contrived fixtures (CLAUDE.md
bench-maxing rule); n=3, 5pp signal floor. Mined fixtures complement,
never replace, the curated adversarial set.

---

## 1. Origin

Dav1dPrometheus, *Protocol-Layer AGI, Recursive Falsification, and the
Final Bet* (2026-05-20 working report; held outside the repo per fox).
Across **§26, §50, §82-84** the report's single most-repeated claim is
that the architecture is *theoretically strong but not empirically
settled*, and the decisive next step is:

> same model · same compute · same tasks · with and without substrate
> · measure H-ABCDEFG performance.

That is a measurement gap, not a strategy gap — squarely in arborist's
remit and the cheapest high-value thing to build.

## 2. What exists vs the gap

**Exists:** the benchmark surface (5S/5T/5F/5R batteries, deterministic
fixtures, no LLM-as-judge); `bench/mine_questions.py` +
`bench/recall_at_k.py` (deterministic retrieval recall@k); `bench-qa`
QA-quality sweep; the cost/quality matrix work (#000057).

**Gap:** none of these run the **same base model in a substrate-on /
substrate-off A/B** and report the delta. The report's systems:

```
A: base model + standard agent scaffold
B: base model + lab-style memory/tools/evals
C: base model + Arborist / PROMETHEUS-Σ substrate
D: base model + open proof mesh        (out of scope here — #000012/#000016)
```

For arborist the tractable, honest cut is **A vs C** (and optionally B):
toggle the substrate, hold the model + tasks fixed, measure.

## 3. The two new metrics (report §73, §5)

- **Jaggedness tensor (§73).** For a task family with semantically
  nearby variants, `J = variance of performance across variants`,
  normalized by expected difficulty (`J_norm`). A system with high
  benchmark score but high jaggedness fails the Hassabis non-jagged
  bar. Testable prediction the harness checks: *same base model +
  substrate should reduce `J_norm` vs the model alone.* This rides the
  existing `mine_questions` / perturbation tooling — generate nearby
  variants, measure variance.
- **Performance curvature (§5.2).** Discrete second difference
  `κ_t = P_t − 2·P_{t-1} + P_{t-2}` over upgrade cycles, with the
  honest bound correction (§5.3, Erratum 5): performance is bounded in
  `[0,1]`, so **no global-convexity claim** — target positive slope +
  non-destructive curvature in the unsaturated regime, route to new
  constraint surfaces when curvature decays.

## 4. Coupled fold (NOT a new ticket): curvature-aware ForkScore → #000012

The report's **§11 Priority 3 / §7.3** wants ForkScore extended with
ΔHABCDEFG, κ, jaggedness-delta, falsifier-coverage-delta. That is an
**extension of an in-progress ticket, not a new one** — `weights.py`
already reserves the slots (`iota` SecurityRisk, `kappa` Complexity).
Per don't-proliferate, this folds into **#000012** (Selection &
consensus / ForkScore) as a new phase, consuming this harness's metric
outputs. Cross-link both ways; do not spawn a sibling.

Suggested extension (recorded for #000012, hard rejects unchanged —
hard regression / NEG_INF_REGRESSION / rollback-missing / audit-break
all still reject):

```
ExtendedForkScore = ForkScore
  + a·ΔHABCDEFG + b·κ_HABCDEFG + c·ΔFalsifierCoverage
  + d·ΔJaggednessReduction + e·ΔPoisonSurvivalReduction
  − f·LatencyPenalty − g·SecurityRisk
```

## 5. Scope boundaries

- A-vs-C (substrate off/on) is the spine; B (lab-style memory/tools) is
  optional; D (proof mesh) is explicitly OUT (consensus/mesh =
  #000012/#000016).
- Budget discipline: control-arm LLM calls route to Hermes-3-8B +
  Qwen, never Opus, without an explicit fox go (CLAUDE.md budget rule).
  Heavy off-device passes run on the GPU box, never in arborist's
  python+sqlite3 core.
- Deterministic retrieval-recall arms (no LLM) fan out in parallel;
  only same-fixture LLM A/B that toggles state needs serialization.
- ABCDEFG self-validation caveat (report §27.2, Erratum 4 / Anomaly 4):
  include external held-out / mechanism-agnostic variants so the
  harness does not reward the substrate by construction.

## 6. Done when

A `make` target runs the same base model substrate-off vs substrate-on
over the batteries and reports per-battery delta + `J_norm` + curvature,
with gate numbers from bench-qa pipeline output (n≥3); the jaggedness
and curvature outputs are consumable by the #000012 ForkScore
extension. `docs/benchmarks.md` gains the new harness + metric schema.

## 7. Scope refinement — H-ABCDEFG-M / -C / -X split (2026-05-26)

Folded in from Dav1d's de-novo review of the architecture stack
(§1.2, §4.7, §9.1). The harness output is split along three axes so
"the harness ACCEPTed the upgrade" can't mean "we passed the tests we
designed for ourselves":

- **H-ABCDEFG-M (mechanism)** — does the substrate work?  
  Cache-key invalidation correctness, audit-chain integrity, π*
  canonicalization round-trip, falsification-state transitions, ForkScore
  scoring on synthetic per-axis deltas. These are tests of the
  *plumbing* — pass/fail tells you whether the substrate's primitives
  function as specified, not whether they help cognition.

- **H-ABCDEFG-C (capability)** — does the substrate improve task
  performance?  
  The current 5S / 5T / 5F / 5R bench surface over real fixtures.
  This is the "does it actually help?" axis. ACCEPT here on its own
  is the load-bearing improvement claim — but only one of three.

- **H-ABCDEFG-X (external adversarial)** — does it generalize outside
  author-designed fixtures?  
  Held-out / mechanism-agnostic / adversarial variants drawn from
  outside the substrate's own design parameters. Fixtures that the
  substrate's designers did NOT pick. Without this axis, the harness
  is at risk of self-validating benchmark theology (§27.2 in the
  ABCDEFG report; Erratum 4 / Anomaly 4 already noted in §5 of this
  ticket but stated weaker than Dav1d's framing).

ACCEPT requires clearing all three axes. M-only or M+C with X failing
is a *partial* acceptance that should still surface in the harness
output, but cannot trigger ForkScore ACCEPT for substrate upgrades.

The split doesn't change `BatteryResult` row schema (already carries
`carrier` / `domain` / `pi_star_ref`); it changes what the harness
aggregator reports + what ForkScore consumes. Each fixture is tagged
with its axis at fixture-definition time; the harness reports per-axis
roll-ups + a combined verdict.

Implementation order when this ticket gates open:
1. Tag every existing 5S/5T/5F/5R fixture with `{M, C}` (current
   fixtures are mechanism + capability by construction; X is the new
   bucket).
2. Add X axis fixtures: held-out variants, prompt rewording, adversarial
   carrier swaps, π*-collision attempts.
3. Aggregator emits `M_pass_rate`, `C_pass_rate`, `X_pass_rate` +
   combined `ACCEPT_M_C_X` boolean. ForkScore consumes the per-axis
   numbers (already lined up with the reserved weight slots in #000012).
