diff --git a/docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md b/docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md index 8cdc88e..c35f25c 100644 --- a/docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md +++ b/docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md @@ -110,10 +110,19 @@ exclusion was right for the proof path, wrong for external science. Science hygiene (kept — methodology, not dogma): - **blinded / order-randomised** — judge unaware which answer is Arborist vs Hermes-solo (kills pro-treatment bias); -- **reference-grounded** — judge scores "supported by *this source - text*" (source supplied), NOT vs its own parametric knowledge - (else judge cutoff/hallucination contaminates; essential for the - fabrication metric); +- **reference-grounded against FIXED INDEPENDENT GOLD, not either + arm's retrieval** (fox Q 2026-05-19). The judge's source = the + mined question's ground-truth target article + gold fact, fixed + at fixture-construction time, **identical for both arms**. NOT + "the 8 retrieved / cited docs": (a) undefined for Hermes-solo — + no retrieval, so per-arm-docs breaks A/B symmetry; (b) judging + Arborist vs its *own* retrieval re-imports circularity; (c) + "supported by cited docs" = citation self-consistency (Arborist's + own verifier already computes that), not truth. Judge sees + `(question, answer, fixed gold)` → correct-grounded / wrong / + fabricated / honestly-abstained. The ground-truth-carrying mined + fixture supplies exactly this fixed gold (same substrate that + broke the fold-search treadmill); - **the experiment is itself audited** — log judge model+version, prompt, per-item verdict; the science measuring Arborist's auditability must itself be replayable; @@ -135,10 +144,15 @@ key (gold absent AND a competing wrong fact asserted = fabrication). **Scope (corrected):** the atomic-fact-only limit is **lifted** — a strong external judge handles non-atomic/essay grounding too. The v1 fixture is still chosen for *claim-isolating power* (next §), -not because the judge can't do prose. Optional documented -extension: a middle arm (Hermes + naive RAG, no -Arborist verification) isolates the verification-layer delta vs the -retrieval delta — not v1. +not because the judge can't do prose. Documented extensions, NOT +v1, each measuring a *different* question (must never be conflated +with the control delta): (i) a middle arm (Hermes + naive RAG, no +Arborist verification) isolates verification-layer vs retrieval +delta; (ii) a **faithfulness ablation** — judge Arborist's answer +against *Arborist's own assembled context* (did the model stay true +to what it was given). (ii) is Arborist-arm-only and measures +faithfulness, not correctness-vs-truth — explicitly labelled so it +is never read as the headline. ## 5. Decision status & the smallest-proof decomposition