docs(#000057): judge reference = fixed independent gold, not either arm's retrieval (fox Q)

fox: should the judge get the same docs, or the 8 retrieved/cited?
Neither. Per-arm retrieved docs (a) are undefined for Hermes-solo
(no retrieval → breaks A/B symmetry), (b) re-import circularity
(judging the treatment vs its own retrieval), (c) measure citation
self-consistency (Arborist's verifier already does that), not
truth. Judge's reference = the mined question's fixed ground-truth
target/gold, identical for both arms, independent of retrieval —
the ground-truth-carrying fixture supplies exactly this. Recorded a
distinct, non-headline faithfulness ablation (judge Arborist vs its
own context) so it is never conflated with the control delta.
Index row unchanged (skim surface; ticket body is the design log).
This commit is contained in:
russell@unturf.com 2026-05-19 08:36:30 -04:00
parent 1fdae8830a
commit fa81b97c5c
No known key found for this signature in database

View file

@ -110,10 +110,19 @@ exclusion was right for the proof path, wrong for external science.
Science hygiene (kept — methodology, not dogma):
- **blinded / order-randomised** — judge unaware which answer is
Arborist vs Hermes-solo (kills pro-treatment bias);
- **reference-grounded** — judge scores "supported by *this source
text*" (source supplied), NOT vs its own parametric knowledge
(else judge cutoff/hallucination contaminates; essential for the
fabrication metric);
- **reference-grounded against FIXED INDEPENDENT GOLD, not either
arm's retrieval** (fox Q 2026-05-19). The judge's source = the
mined question's ground-truth target article + gold fact, fixed
at fixture-construction time, **identical for both arms**. NOT
"the 8 retrieved / cited docs": (a) undefined for Hermes-solo —
no retrieval, so per-arm-docs breaks A/B symmetry; (b) judging
Arborist vs its *own* retrieval re-imports circularity; (c)
"supported by cited docs" = citation self-consistency (Arborist's
own verifier already computes that), not truth. Judge sees
`(question, answer, fixed gold)` → correct-grounded / wrong /
fabricated / honestly-abstained. The ground-truth-carrying mined
fixture supplies exactly this fixed gold (same substrate that
broke the fold-search treadmill);
- **the experiment is itself audited** — log judge model+version,
prompt, per-item verdict; the science measuring Arborist's
auditability must itself be replayable;
@ -135,10 +144,15 @@ key (gold absent AND a competing wrong fact asserted = fabrication).
**Scope (corrected):** the atomic-fact-only limit is **lifted** — a
strong external judge handles non-atomic/essay grounding too. The
v1 fixture is still chosen for *claim-isolating power* (next §),
not because the judge can't do prose. Optional documented
extension: a middle arm (Hermes + naive RAG, no
Arborist verification) isolates the verification-layer delta vs the
retrieval delta — not v1.
not because the judge can't do prose. Documented extensions, NOT
v1, each measuring a *different* question (must never be conflated
with the control delta): (i) a middle arm (Hermes + naive RAG, no
Arborist verification) isolates verification-layer vs retrieval
delta; (ii) a **faithfulness ablation** — judge Arborist's answer
against *Arborist's own assembled context* (did the model stay true
to what it was given). (ii) is Arborist-arm-only and measures
faithfulness, not correctness-vs-truth — explicitly labelled so it
is never read as the headline.
## 5. Decision status & the smallest-proof decomposition