From fa81b97c5c86c5e2d48a5f96fc71b38caf958e29 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Tue, 19 May 2026 08:36:30 -0400 Subject: [PATCH] docs(#000057): judge reference = fixed independent gold, not either arm's retrieval (fox Q) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fox: should the judge get the same docs, or the 8 retrieved/cited? Neither. Per-arm retrieved docs (a) are undefined for Hermes-solo (no retrieval → breaks A/B symmetry), (b) re-import circularity (judging the treatment vs its own retrieval), (c) measure citation self-consistency (Arborist's verifier already does that), not truth. Judge's reference = the mined question's fixed ground-truth target/gold, identical for both arms, independent of retrieval — the ground-truth-carrying fixture supplies exactly this. Recorded a distinct, non-headline faithfulness ablation (judge Arborist vs its own context) so it is never conflated with the control delta. Index row unchanged (skim surface; ticket body is the design log). --- ...7-witness-preserving-recursive-drift-ab.md | 30 ++++++++++++++----- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md b/docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md index 8cdc88e..c35f25c 100644 --- a/docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md +++ b/docs/tickets/ticket-000057-witness-preserving-recursive-drift-ab.md @@ -110,10 +110,19 @@ exclusion was right for the proof path, wrong for external science. Science hygiene (kept — methodology, not dogma): - **blinded / order-randomised** — judge unaware which answer is Arborist vs Hermes-solo (kills pro-treatment bias); -- **reference-grounded** — judge scores "supported by *this source - text*" (source supplied), NOT vs its own parametric knowledge - (else judge cutoff/hallucination contaminates; essential for the - fabrication metric); +- **reference-grounded against FIXED INDEPENDENT GOLD, not either + arm's retrieval** (fox Q 2026-05-19). The judge's source = the + mined question's ground-truth target article + gold fact, fixed + at fixture-construction time, **identical for both arms**. NOT + "the 8 retrieved / cited docs": (a) undefined for Hermes-solo — + no retrieval, so per-arm-docs breaks A/B symmetry; (b) judging + Arborist vs its *own* retrieval re-imports circularity; (c) + "supported by cited docs" = citation self-consistency (Arborist's + own verifier already computes that), not truth. Judge sees + `(question, answer, fixed gold)` → correct-grounded / wrong / + fabricated / honestly-abstained. The ground-truth-carrying mined + fixture supplies exactly this fixed gold (same substrate that + broke the fold-search treadmill); - **the experiment is itself audited** — log judge model+version, prompt, per-item verdict; the science measuring Arborist's auditability must itself be replayable; @@ -135,10 +144,15 @@ key (gold absent AND a competing wrong fact asserted = fabrication). **Scope (corrected):** the atomic-fact-only limit is **lifted** — a strong external judge handles non-atomic/essay grounding too. The v1 fixture is still chosen for *claim-isolating power* (next §), -not because the judge can't do prose. Optional documented -extension: a middle arm (Hermes + naive RAG, no -Arborist verification) isolates the verification-layer delta vs the -retrieval delta — not v1. +not because the judge can't do prose. Documented extensions, NOT +v1, each measuring a *different* question (must never be conflated +with the control delta): (i) a middle arm (Hermes + naive RAG, no +Arborist verification) isolates verification-layer vs retrieval +delta; (ii) a **faithfulness ablation** — judge Arborist's answer +against *Arborist's own assembled context* (did the model stay true +to what it was given). (ii) is Arborist-arm-only and measures +faithfulness, not correctness-vs-truth — explicitly labelled so it +is never read as the headline. ## 5. Decision status & the smallest-proof decomposition