docs(#000057): judge reference = fixed independent gold, not either arm's retrieval (fox Q)
fox: should the judge get the same docs, or the 8 retrieved/cited? Neither. Per-arm retrieved docs (a) are undefined for Hermes-solo (no retrieval → breaks A/B symmetry), (b) re-import circularity (judging the treatment vs its own retrieval), (c) measure citation self-consistency (Arborist's verifier already does that), not truth. Judge's reference = the mined question's fixed ground-truth target/gold, identical for both arms, independent of retrieval — the ground-truth-carrying fixture supplies exactly this. Recorded a distinct, non-headline faithfulness ablation (judge Arborist vs its own context) so it is never conflated with the control delta. Index row unchanged (skim surface; ticket body is the design log).
This commit is contained in:
parent
1fdae8830a
commit
fa81b97c5c
1 changed files with 22 additions and 8 deletions
|
|
@ -110,10 +110,19 @@ exclusion was right for the proof path, wrong for external science.
|
|||
Science hygiene (kept — methodology, not dogma):
|
||||
- **blinded / order-randomised** — judge unaware which answer is
|
||||
Arborist vs Hermes-solo (kills pro-treatment bias);
|
||||
- **reference-grounded** — judge scores "supported by *this source
|
||||
text*" (source supplied), NOT vs its own parametric knowledge
|
||||
(else judge cutoff/hallucination contaminates; essential for the
|
||||
fabrication metric);
|
||||
- **reference-grounded against FIXED INDEPENDENT GOLD, not either
|
||||
arm's retrieval** (fox Q 2026-05-19). The judge's source = the
|
||||
mined question's ground-truth target article + gold fact, fixed
|
||||
at fixture-construction time, **identical for both arms**. NOT
|
||||
"the 8 retrieved / cited docs": (a) undefined for Hermes-solo —
|
||||
no retrieval, so per-arm-docs breaks A/B symmetry; (b) judging
|
||||
Arborist vs its *own* retrieval re-imports circularity; (c)
|
||||
"supported by cited docs" = citation self-consistency (Arborist's
|
||||
own verifier already computes that), not truth. Judge sees
|
||||
`(question, answer, fixed gold)` → correct-grounded / wrong /
|
||||
fabricated / honestly-abstained. The ground-truth-carrying mined
|
||||
fixture supplies exactly this fixed gold (same substrate that
|
||||
broke the fold-search treadmill);
|
||||
- **the experiment is itself audited** — log judge model+version,
|
||||
prompt, per-item verdict; the science measuring Arborist's
|
||||
auditability must itself be replayable;
|
||||
|
|
@ -135,10 +144,15 @@ key (gold absent AND a competing wrong fact asserted = fabrication).
|
|||
**Scope (corrected):** the atomic-fact-only limit is **lifted** — a
|
||||
strong external judge handles non-atomic/essay grounding too. The
|
||||
v1 fixture is still chosen for *claim-isolating power* (next §),
|
||||
not because the judge can't do prose. Optional documented
|
||||
extension: a middle arm (Hermes + naive RAG, no
|
||||
Arborist verification) isolates the verification-layer delta vs the
|
||||
retrieval delta — not v1.
|
||||
not because the judge can't do prose. Documented extensions, NOT
|
||||
v1, each measuring a *different* question (must never be conflated
|
||||
with the control delta): (i) a middle arm (Hermes + naive RAG, no
|
||||
Arborist verification) isolates verification-layer vs retrieval
|
||||
delta; (ii) a **faithfulness ablation** — judge Arborist's answer
|
||||
against *Arborist's own assembled context* (did the model stay true
|
||||
to what it was given). (ii) is Arborist-arm-only and measures
|
||||
faithfulness, not correctness-vs-truth — explicitly labelled so it
|
||||
is never read as the headline.
|
||||
|
||||
## 5. Decision status & the smallest-proof decomposition
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue