# Ticket #000057 — Witness-preserving vs ordinary recursive loop: minimal deterministic drift A/B

**Status:** open · awaiting go/no-go · doc-only scaffold (2026-05-19)
**Opened:** 2026-05-19
**Asked by:** fox, relaying a Hamming-framed de-novo review (GPT-5.5
Thinking, 2026-05-19 19:13 Asia/Kuala_Lumpur).
**Scope:** ONE ticket, scaffold only. Define the *smallest*
deterministic experiment that would prove the whitepaper's headline
empirical claim. NOT a build. NOT the 8-task/8-metric flagship.
**Audience:** fox + a Dav1d de-novo review of the experiment design +
the Merkle-Providence-Reverse-RAG whitepaper (this is its missing
empirical headline; the paper currently *asserts* the architecture).

---

## 1. The one claim worth proving (the review's §8.2, compressed)

> In a recursive loop where outputs feed future context, **untracked
> evidence loss → unbounded epistemic drift**. A witness-preserving
> state transition system **bounds** drift by making unsupported
> transitions detectable and reversible.

Everything else in the review (renaming, corpus hierarchy, IQ/talent,
ToE/Riemann/identity/geopolitics) is **deliberately not ticketed** —
positioning/whitepaper-narrative, not arborist engineering;
don't-proliferate + this session is arborist + whitepapers only.

## 2. The discipline this ticket exists to enforce

The blunt review line is *"make the **smallest** irreversible
proof."* The 2026-05-18 session is the cautionary precedent: a grand
target (STRICT-rate) was unmovable for four hypotheses **because the
instrument couldn't resolve it**; the win came only from a
deterministic, noise-free, ground-truth-carrying instrument
(mined recall@k). A "prove recursive trustworthy intelligence"
experiment is the *same trap at larger scale* — it will produce
unfalsifiable mush unless the instrument is built first.

**Therefore, hard scope gates (do not skip in order):**

1. **Instrument before experiment.** Define ONE deterministic drift
   metric — replayable, no-LLM-as-judge, ground-truth-carrying,
   resolvable above a stated noise floor — exactly the
   `mine_questions.py`/`recall_at_k.py` discipline. If the metric
   isn't deterministic and ground-truthed, stop; the experiment is
   not yet attackable.
2. **One task, not eight.** Pick the single task that most directly
   exercises arborist's *distinguishing* mechanism — falsification
   state — under recursion: **recursive stale-source invalidation**
   *or* **recursive contradiction repair**. The other five §8.1
   tasks are explicitly OUT until the one-task A/B clears its floor.
3. **Minimal A/B.** Same task, N recursive iterations, the only
   difference being witness-binding/falsification-state ON vs OFF.
   Metric = a deterministic count of unsupported claims that survive
   K iterations (and whether they are reversible). Before vs after,
   like every fold in #000001-family work.
4. **Non-claims, pinned.** NOT an AGI claim. The provable statement
   is "a *necessary substrate* for recursive trustworthy
   retrieval-grounded answering, demonstrated on one task." The
   review agrees (§9 AGI). No STRICT/global-QA/multilingual claim.

## 3. What this is NOT (recorded so it isn't re-litigated)

- Not the multi-task flagship matrix (§8.1's 6 tasks × 8 metrics) —
  that is the *grandness* the review itself warns against; it earns
  consideration only after gate 3 clears with measured signal.
- Not a rename (`Witness-Preserving Recursive AI` etc.) — narrative,
  not engineering; whitepaper editorial, separate from this ticket.
- Not corpus-triage / physics / Riemann / identity — out of arborist
  scope entirely.
- Not autonomous-executable: this is a scope/strategy decision (I
  propose, fox decides) — unlike the folds, which were measured,
  bounded, and self-evidently shippable.

## 4. Why a ticket at all (don't-proliferate justification)

Crosses the split bar: a self-contained, Dav1d-reviewable
architectural-inflection design decision (the whitepaper's headline
experiment), distinct audience, not a sub-tweak of an existing
ticket. Exactly **one** ticket — the rest of the review is
explicitly refused above.

## 4b. Control-arm + external judge (fox ruling 2026-05-19)

The sharpest point in the review (§4.1) and fox's correction:
**every measurement so far is *within* Arborist; there has been no
control arm.** A treatment-vs-control experiment needs (a) the
control = **Hermes-solo** (no Arborist retrieval / claim-lattice /
provenance — pure parametric answer), and (b) a judge **outside
Arborist** (scoring with Arborist's own verifier = treatment grading
itself = invalid).

**CORRECTION (fox 2026-05-19, supersedes the earlier draft):** the
"no LLM-as-judge" rule is a **proof-path** rule (an LLM judge
*inside* Arborist's verifier is the fabrication/circularity risk
Arborist exists to kill; #000049's 0.24–0.48 reliability is about a
model *in the proof path*, cheap/pinned/shadow-constrained). The
control experiment's judge is **external science** — outside *both*
arms (Arborist and Hermes-solo), scoring outputs post-hoc. It does
NOT touch either system's internals, so it cannot "break the
experiment." Importing the proof-path constraint here was dogma.

**Resolution — a strong SOTA model as the external judge.** A judge
**stronger than both systems under test** (frontier-class grading
Hermes-8B-class output) is standard, sound ML-eval. This also
removes the atomic-fact-only limit: it judges paraphrase **and**
non-atomic grounding ("describe DNA structure") — the #000049
exclusion was right for the proof path, wrong for external science.

Science hygiene (kept — methodology, not dogma):
- **blinded / order-randomised** — judge unaware which answer is
  Arborist vs Hermes-solo (kills pro-treatment bias);
- **reference-grounded against FIXED INDEPENDENT GOLD, not either
  arm's retrieval** (fox Q 2026-05-19). The judge's source = the
  mined question's ground-truth target article + gold fact, fixed
  at fixture-construction time, **identical for both arms**. NOT
  "the 8 retrieved / cited docs": (a) undefined for Hermes-solo —
  no retrieval, so per-arm-docs breaks A/B symmetry; (b) judging
  Arborist vs its *own* retrieval re-imports circularity; (c)
  "supported by cited docs" = citation self-consistency (Arborist's
  own verifier already computes that), not truth. Judge sees
  `(question, answer, fixed gold)` → correct-grounded / wrong /
  fabricated / honestly-abstained. The ground-truth-carrying mined
  fixture supplies exactly this fixed gold (same substrate that
  broke the fold-search treadmill);
- **the experiment is itself audited** — log judge model+version,
  prompt, per-item verdict; the science measuring Arborist's
  auditability must itself be replayable;
- **deterministic gold-key demoted to a cheap cross-check** on the
  atomic-fact subset: judge↔key disagreement = a judge-quality
  flag (belt-and-suspenders, not the primary instrument).
Both arms scored by the identical blinded reference-grounded judge.

**Metric — fabrication vs honest abstention, not raw accuracy.**
A common-knowledge fixture would falsely show "Arborist adds
nothing" (parametric memory already knows "capital of France"). The
claim Arborist actually makes is: on corpus-specific / stale /
false-premise / fabrication-bait questions, Hermes-solo
*confidently fabricates* while Arborist *fails closed (UNGROUNDED)*.
The headline delta = Hermes-solo confidently-wrong rate vs Arborist
honest-abstention rate, deterministically computable from the gold
key (gold absent AND a competing wrong fact asserted = fabrication).

**Scope (corrected):** the atomic-fact-only limit is **lifted** — a
strong external judge handles non-atomic/essay grounding too. The
v1 fixture is still chosen for *claim-isolating power* (next §),
not because the judge can't do prose. Documented extensions, NOT
v1, each measuring a *different* question (must never be conflated
with the control delta): (i) a middle arm (Hermes + naive RAG, no
Arborist verification) isolates verification-layer vs retrieval
delta; (ii) a **faithfulness ablation** — judge Arborist's answer
against *Arborist's own assembled context* (did the model stay true
to what it was given). (ii) is Arborist-arm-only and measures
faithfulness, not correctness-vs-truth — explicitly labelled so it
is never read as the headline.

## 4c. Judge instrument — BUILT + VERIFIED (2026-05-19)

fox ruled the judge = Opus via `claude -p`. `bench/judge.py` built
to the §4b hygiene: hermetic (`env -u CLAUDECODE claude -p`, fresh
process, context = only (Q, answer, gold)), blinded-by-caller,
reference-grounded, **structured via a `FINAL_VERDICT=<TOKEN>`
sentinel parsed last-match** (the first cut parsed first-match over
the model's chain-of-thought → a 0/3 self-test; the judge reasoned
correctly, the *parser* was the defect — instrument-before-
experiment gate working). Hardened + re-verified: `make
judge-self-test` = **4/4** on known-verdict triples
(CORRECT_GROUNDED / WRONG / FABRICATED / ABSTAINED) via real
`claude -p`. This make target IS the precondition gate; no control
run may trust the judge until it passes.

Threat to validity (recorded, not hidden): same model *family*
judging (this repo's agent is also Opus); blinding + no-stake +
reference-grounding mitigate, do not eliminate — a different-family
SOTA cross-check is the only full removal; stated as a known limit.

**Experiment harness — BUILT + smoke-verified end-to-end
(2026-05-19).** `bench/control_ab.py` + `make control-ab` (gated on
`judge-self-test` as a *make dependency* — the instrument gate
cannot be skipped). Hermes-solo (`OpenAICompatibleClient`, question
only, neutral abstention-permitting system) vs Arborist (`query()`),
same model; gold = target-article text by `target_root`; Arborist
`[E…]` scaffolding stripped so format can't betray the arm
(blinding); Arborist `UNGROUNDED` credited as honest abstention;
hermetic Opus judge; deterministic aggregate; self-auditing JSONL.
**N=2 smoke: clean, 0 JUDGE_ERROR**, and already produced a case
*against* the treatment (solo correctly ABSTAINED; Arborist
HYBRID-WRONG) — the instrument can falsify the Arborist value claim,
which is precisely what makes it worth running. n=2 proves nothing
(report states this); the verdict needs a real N.

**Open (fox spend call):** the real run — `make control-ab
N=<…> FIXTURE=<…>`. v1 default N=12 (smallest proof, bounded:
2N Hermes + 2N `claude -p`). Fixture population still a fox call
(mined = partially claim-isolating; a deliberately-adversarial
fabrication-bait set would sharpen it). No strong claim until a
real-N run; the instrument is done, the science is now executable.

## 5. Decision status & the smallest-proof decomposition

fox's control-arm framing (§4b) reorders the experiment honestly:
the **single-shot treatment-vs-control** is *more foundational and
smaller* than the recursive-drift loop. So:

- **v1 (the smallest irreversible proof):** single-shot **Hermes-solo
  vs Arborist** on a fabrication-bait atomic-fact fixture, judged by
  the external deterministic gold-fact key (§4b); headline = the
  fabrication-vs-honest-abstention delta. This *is* the "force the
  architecture to beat the baseline" result; no recursion needed yet.
- **v2 (the §1 claim):** the recursive-drift loop (witness ON/OFF, N
  iterations) — only after v1 establishes a measurable single-shot
  delta. Building v2 before v1 measures is the precedent error.

**Resolved by fox (2026-05-19):** control = Hermes-solo; judge = a
**strong SOTA model, external to the experiment**, blinded +
reference-grounded (§4b); atomic-only limit lifted; gold-key kept
only as a cheap cross-check.
**Still open (genuine fox calls, not autonomous):** (a) the
fabrication-bait fixture population — corpus-specific / stale /
false-premise / known-fabrication-bait, *not* common-knowledge where
parametric memory wins anyway (the resolving-fixture lesson:
prevalence ≠ what isolates the claim); (b) **which SOTA judge model**
— and the spend/egress note: an external frontier judge is
experiment-time API cost + Operation-Voyeur egress (low-sensitivity
2003-Wikipedia QA, but a fox call, and the judge transcript must be
logged for replay); (c) spend now vs parked. No design past §2
gate-1 / §4b until (a)–(c) ruled.
