docs(#000057): Addendum 8 — control sweep retrieval × model × framing × reasoning
Durable record of the #000057 sweep in the bench journal. Captures: - The question: is Hermes-8B's confident present-day-officeholder fabrication an 8B weakness, a framing artefact, or does retrieval fix it? Crosses {hermes, qwen-nothink, qwen-think} × {plain, source_relative, as_of_corpus} × {solo, arborist} on a 386-item office-holder fixture with corpus-vintage gold. - The judge methodology: Opus headless judge burned quota (79.5% JUDGE_ERROR), replaced with the deterministic code judge (bench/judge_code.py), calibrated against Opus's gradeable records (CG agreement 13->47%, WRONG 56->89%, ABSTAINED 80->95%). - Consolidated CG% scorecard, all arms on the identical final judge. - Three findings: 1. Retrieval dominates — arb/qwen-nothink/plain 82% vs 7% solo; no solo config approaches the retrieval arms. 2. Reasoning does NOT improve raw correctness — qwen-think/as_of 44% vs nothink 50%. 3. Reasoning's real cost is broken honest-abstention — qwen-nothink/source_relative abstains 97% (clean); qwen-think only 61%, reasoning itself into wrong parametric answers. - Production recommendation: arborist + qwen-nothink, plain framing, reasoning OFF (82% CG, ~0% abstain, 11% wrong-assert). - Held cell noted: arborist+qwen-think running at write time, result to be appended. Bench %s are point-in-time measurements (not repo-derived counts), so no AUTOCOUNT tags — consistent with addenda 1-7. test_doc_counts 3/3.
This commit is contained in:
parent
5863559445
commit
42f614a501
1 changed files with 102 additions and 0 deletions
|
|
@ -529,3 +529,105 @@ Bench artifacts: `bench/qa_results/2026-05-11T17-12-41Z.{jsonl,md}`
|
|||
(before — HEAD's parse_claims.py) · `bench/qa_results/2026-05-11T20-26-37Z.{jsonl,md}`
|
||||
(after). Full per-ticket detail: `docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md`
|
||||
§5 step 2.4.
|
||||
|
||||
### Addendum 8 — #000057 control sweep: retrieval × model × framing × reasoning (2026-05-19/20)
|
||||
|
||||
**The question.** Hermes-3-8B confidently states the *present-day*
|
||||
office-holder ("the president of France is Emmanuel Macron") against a
|
||||
~2010-vintage corpus (Sarkozy). Is that an 8B-model weakness, an unfair
|
||||
question framing, or does retrieval fix it regardless? The sweep
|
||||
crosses **{model} × {framing} × {retrieval on/off}** over a 386-item
|
||||
office-holder fixture (`bench/qa_questions_stale_map.json`), each item
|
||||
with a fixed corpus-vintage gold article.
|
||||
|
||||
- **models:** `hermes` (Hermes-3-8B), `qwen-nothink` (Qwen3.6-27B,
|
||||
reasoning off), `qwen-think` (same weights, reasoning on). Qwen runs
|
||||
on llama.cpp; Hermes on vLLM.
|
||||
- **framings:** `plain` ("who is the president of France?"),
|
||||
`source_relative` ("According to the reference knowledge base, …"),
|
||||
`as_of_corpus` ("As of 2010, who was …").
|
||||
- **arms:** `solo` (model alone — measures parametric prior) vs
|
||||
`arborist` (model + retrieval over the 2010 corpus, `answer_mode=
|
||||
claim_lattice`).
|
||||
|
||||
**Judge methodology — the code judge.** The original sweep used the
|
||||
Opus headless judge (`bench/judge.py`); it burned our Anthropic quota
|
||||
and **79.5 % of its verdicts came back JUDGE_ERROR** (rate-limited).
|
||||
Replaced with a deterministic code judge (`bench/judge_code.py`):
|
||||
abstention regex → short-answer entity-grounding fast path → NLI
|
||||
contradiction (θ=0.85) → lexical verifier (quote/span/entity/
|
||||
paraphrase) → WRONG-vs-FABRICATED tie-break on subject-in-gold. **No
|
||||
LLM, no quota, fully replayable.** Calibrated against the records Opus
|
||||
*did* grade: agreement rose CG 13 %→47 %, WRONG 56 %→89 %, ABSTAINED
|
||||
80 %→95 % across four targeted fixes (raise NLI contradiction
|
||||
threshold off the manifest's 0.5; expand abstention patterns for the
|
||||
"I do not have … information/access" family; demote FABRICATED→WRONG
|
||||
when gold mentions the question subject; short-answer entity-grounding
|
||||
for terse-name answers the verifier's prose-shape extractor whiffs on).
|
||||
A claim-lattice JSON-unwrap lets the same judge grade the Arborist
|
||||
arm's `{"claims":[…]}` envelopes as plain prose. Verdicts are a
|
||||
deterministic *proxy* for grounding (lexical + NLI), not Opus-grade
|
||||
reading — JUDGE_ERROR residue (HYBRID-via-entity with no NLI
|
||||
corroboration) is the natural input to a later LLM-batch pass.
|
||||
|
||||
**Consolidated scorecard — CG% (correct-grounded rate), all arms on
|
||||
the identical calibrated judge.** `arborist/hermes` is n=40 (phase 1's
|
||||
smaller arborist arm); every other cell is n=386.
|
||||
|
||||
| arm / model | plain | source_relative | as_of_corpus |
|
||||
|-------------|-------|-----------------|--------------|
|
||||
| solo / hermes | 9 % | 5 % | 18 % |
|
||||
| solo / qwen-nothink | 7 % | 0 % | **50 %** |
|
||||
| solo / qwen-think | 6 % | 5 % | 44 % |
|
||||
| arborist / hermes (n=40) | 60 % | 62 % | 25 % |
|
||||
| **arborist / qwen-nothink** | **82 %** | **65 %** | 50 % |
|
||||
|
||||
Grounding-fidelity detail (abstain% / wrong-assert% = W+F over n) for
|
||||
the cells the findings turn on:
|
||||
|
||||
| cell | CG% | abstain% | wrong-assert% |
|
||||
|------|-----|----------|---------------|
|
||||
| solo qwen-nothink / source_relative | 0 % | **97 %** | 1 % |
|
||||
| solo qwen-think / source_relative | 5 % | 61 % | 32 % |
|
||||
| solo qwen-nothink / plain | 7 % | 9 % | 79 % |
|
||||
| solo qwen-think / plain | 6 % | 43 % | 48 % |
|
||||
| arborist qwen-nothink / plain | 82 % | 0 % | 11 % |
|
||||
|
||||
**Findings.**
|
||||
|
||||
1. **Retrieval dominates every other lever.** arborist/qwen-nothink/
|
||||
plain = 82 % CG vs 7 % solo; arborist/hermes/plain = 60 % vs 9 %
|
||||
solo. No solo configuration — no model size, no framing — approaches
|
||||
the retrieval arms. The production answer is retrieval.
|
||||
|
||||
2. **Reasoning does not improve raw correctness.** solo qwen-think vs
|
||||
qwen-nothink: as_of_corpus **44 % vs 50 %** (nothink wins), plain
|
||||
6 %/7 %, source_relative 5 %/0 %. Chain-of-thought trades correctness
|
||||
for caution; it does not recall corpus-vintage facts better.
|
||||
|
||||
3. **Reasoning's real cost is broken honest-abstention.** The cleanest
|
||||
grounding signal in the sweep is solo qwen-nothink/source_relative:
|
||||
**97 % abstain** — told to use a reference it lacks, it declines.
|
||||
Reasoning *breaks* this: qwen-think/source_relative abstains only
|
||||
61 % and reasons itself into wrong parametric answers (wrong-assert
|
||||
1 %→32 %). On plain framing reasoning helps caution (wrong-assert
|
||||
79 %→48 %); on the source-grounding framing it hurts.
|
||||
|
||||
**Production recommendation:** arborist + qwen-nothink, plain framing —
|
||||
82 % CG, ~0 % abstain, 11 % wrong-assert. Reasoning OFF: it degrades
|
||||
the honest-abstention property and buys nothing once retrieval supplies
|
||||
the source.
|
||||
|
||||
**Held / in flight:** arborist + qwen-think (the last cell) is running
|
||||
at write time — tests whether reasoning hurts the retrieval arm the way
|
||||
it hurt solo/source_relative. Will append the result.
|
||||
|
||||
**Artifacts.** Solo + arborist-hermes: `control_sweep_2026-05-19T21-52-56Z*`
|
||||
(phase 1). qwen-think solo: `…23-22-21Z*` (phase 2). arborist+qwen-nothink:
|
||||
`…2026-05-20T00-07-38Z*` (phase 3). All re-graded under the final judge
|
||||
(`*_code_judge_final.{md,jsonl}`). Judge + sweep tooling:
|
||||
`bench/judge_code.py`, `bench/score_with_code_judge.py`,
|
||||
`bench/analyze_judge_disagreement.py`, `bench/control_sweep.py`
|
||||
(`--judge {code,opus}`, `--skip-solo`). Engine-agnostic JSON-schema
|
||||
enforcement for the Arborist arm:
|
||||
`arborist.qa.verify.claim_lattice_structured_output_extras`.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue