diff --git a/docs/qa-modes-bench.md b/docs/qa-modes-bench.md index e62a317..84ecb97 100644 --- a/docs/qa-modes-bench.md +++ b/docs/qa-modes-bench.md @@ -529,3 +529,105 @@ Bench artifacts: `bench/qa_results/2026-05-11T17-12-41Z.{jsonl,md}` (before — HEAD's parse_claims.py) · `bench/qa_results/2026-05-11T20-26-37Z.{jsonl,md}` (after). Full per-ticket detail: `docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md` §5 step 2.4. + +### Addendum 8 — #000057 control sweep: retrieval × model × framing × reasoning (2026-05-19/20) + +**The question.** Hermes-3-8B confidently states the *present-day* +office-holder ("the president of France is Emmanuel Macron") against a +~2010-vintage corpus (Sarkozy). Is that an 8B-model weakness, an unfair +question framing, or does retrieval fix it regardless? The sweep +crosses **{model} × {framing} × {retrieval on/off}** over a 386-item +office-holder fixture (`bench/qa_questions_stale_map.json`), each item +with a fixed corpus-vintage gold article. + +- **models:** `hermes` (Hermes-3-8B), `qwen-nothink` (Qwen3.6-27B, + reasoning off), `qwen-think` (same weights, reasoning on). Qwen runs + on llama.cpp; Hermes on vLLM. +- **framings:** `plain` ("who is the president of France?"), + `source_relative` ("According to the reference knowledge base, …"), + `as_of_corpus` ("As of 2010, who was …"). +- **arms:** `solo` (model alone — measures parametric prior) vs + `arborist` (model + retrieval over the 2010 corpus, `answer_mode= + claim_lattice`). + +**Judge methodology — the code judge.** The original sweep used the +Opus headless judge (`bench/judge.py`); it burned our Anthropic quota +and **79.5 % of its verdicts came back JUDGE_ERROR** (rate-limited). +Replaced with a deterministic code judge (`bench/judge_code.py`): +abstention regex → short-answer entity-grounding fast path → NLI +contradiction (θ=0.85) → lexical verifier (quote/span/entity/ +paraphrase) → WRONG-vs-FABRICATED tie-break on subject-in-gold. **No +LLM, no quota, fully replayable.** Calibrated against the records Opus +*did* grade: agreement rose CG 13 %→47 %, WRONG 56 %→89 %, ABSTAINED +80 %→95 % across four targeted fixes (raise NLI contradiction +threshold off the manifest's 0.5; expand abstention patterns for the +"I do not have … information/access" family; demote FABRICATED→WRONG +when gold mentions the question subject; short-answer entity-grounding +for terse-name answers the verifier's prose-shape extractor whiffs on). +A claim-lattice JSON-unwrap lets the same judge grade the Arborist +arm's `{"claims":[…]}` envelopes as plain prose. Verdicts are a +deterministic *proxy* for grounding (lexical + NLI), not Opus-grade +reading — JUDGE_ERROR residue (HYBRID-via-entity with no NLI +corroboration) is the natural input to a later LLM-batch pass. + +**Consolidated scorecard — CG% (correct-grounded rate), all arms on +the identical calibrated judge.** `arborist/hermes` is n=40 (phase 1's +smaller arborist arm); every other cell is n=386. + +| arm / model | plain | source_relative | as_of_corpus | +|-------------|-------|-----------------|--------------| +| solo / hermes | 9 % | 5 % | 18 % | +| solo / qwen-nothink | 7 % | 0 % | **50 %** | +| solo / qwen-think | 6 % | 5 % | 44 % | +| arborist / hermes (n=40) | 60 % | 62 % | 25 % | +| **arborist / qwen-nothink** | **82 %** | **65 %** | 50 % | + +Grounding-fidelity detail (abstain% / wrong-assert% = W+F over n) for +the cells the findings turn on: + +| cell | CG% | abstain% | wrong-assert% | +|------|-----|----------|---------------| +| solo qwen-nothink / source_relative | 0 % | **97 %** | 1 % | +| solo qwen-think / source_relative | 5 % | 61 % | 32 % | +| solo qwen-nothink / plain | 7 % | 9 % | 79 % | +| solo qwen-think / plain | 6 % | 43 % | 48 % | +| arborist qwen-nothink / plain | 82 % | 0 % | 11 % | + +**Findings.** + +1. **Retrieval dominates every other lever.** arborist/qwen-nothink/ + plain = 82 % CG vs 7 % solo; arborist/hermes/plain = 60 % vs 9 % + solo. No solo configuration — no model size, no framing — approaches + the retrieval arms. The production answer is retrieval. + +2. **Reasoning does not improve raw correctness.** solo qwen-think vs + qwen-nothink: as_of_corpus **44 % vs 50 %** (nothink wins), plain + 6 %/7 %, source_relative 5 %/0 %. Chain-of-thought trades correctness + for caution; it does not recall corpus-vintage facts better. + +3. **Reasoning's real cost is broken honest-abstention.** The cleanest + grounding signal in the sweep is solo qwen-nothink/source_relative: + **97 % abstain** — told to use a reference it lacks, it declines. + Reasoning *breaks* this: qwen-think/source_relative abstains only + 61 % and reasons itself into wrong parametric answers (wrong-assert + 1 %→32 %). On plain framing reasoning helps caution (wrong-assert + 79 %→48 %); on the source-grounding framing it hurts. + +**Production recommendation:** arborist + qwen-nothink, plain framing — +82 % CG, ~0 % abstain, 11 % wrong-assert. Reasoning OFF: it degrades +the honest-abstention property and buys nothing once retrieval supplies +the source. + +**Held / in flight:** arborist + qwen-think (the last cell) is running +at write time — tests whether reasoning hurts the retrieval arm the way +it hurt solo/source_relative. Will append the result. + +**Artifacts.** Solo + arborist-hermes: `control_sweep_2026-05-19T21-52-56Z*` +(phase 1). qwen-think solo: `…23-22-21Z*` (phase 2). arborist+qwen-nothink: +`…2026-05-20T00-07-38Z*` (phase 3). All re-graded under the final judge +(`*_code_judge_final.{md,jsonl}`). Judge + sweep tooling: +`bench/judge_code.py`, `bench/score_with_code_judge.py`, +`bench/analyze_judge_disagreement.py`, `bench/control_sweep.py` +(`--judge {code,opus}`, `--skip-solo`). Engine-agnostic JSON-schema +enforcement for the Arborist arm: +`arborist.qa.verify.claim_lattice_structured_output_extras`.