From 092cac4e80fdf96b20f91418b864dd4aabf5c667 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Thu, 30 Apr 2026 16:32:41 -0400 Subject: [PATCH] docs(qa): final stop-sequence bench row, day-journey net summary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-stop-sequence bench (2026-04-30T20-15-11Z): quote 29S 21H 16U strict-rate 44% grounded 50 6.0s pointer 14S 44H 8U strict-rate 21% grounded 58 4.8s JSON 37S 23H 6U strict-rate 56% grounded 60 4.5s JSON mode delta vs prior post-pointer-ID bench: STRICT: 31 → 37 (+6) UNGROUNDED: 10 → 6 (-4) strict-rate: 47% → 56% (+9pp) apollo runaway: 3/3 → 2/3 (one sample recovered) latency: 5.5s → 4.5s (runaway cases were dragging avg) Day's net for JSON mode (morning baseline → late-evening): STRICT: 26 → 37 (+11) errors: 19 → 0 (-19) grounded: 38 → 60 (+22) strict-rate: 39% → 56% (+17pp) Each step was a named-failure → fix → re-bench cycle: retry → trim-and-verify → pointer-IDs → stop-sequence. --- docs/qa-modes-bench-2026-04-30.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/qa-modes-bench-2026-04-30.md b/docs/qa-modes-bench-2026-04-30.md index 8e41304..341f284 100644 --- a/docs/qa-modes-bench-2026-04-30.md +++ b/docs/qa-modes-bench-2026-04-30.md @@ -72,7 +72,9 @@ Fix landed in commit `f23d3a3`: pass `stop=["\n\n"]` to vLLM in JSON mode. Well- | baseline (no improvements) | morning n=3 | 31 | 14 | 26 | **19** | 38 | | post-retry + trim-and-verify | midday n=3 | 31 | 16 | **33** | 0 | 54 | | post-pointer-ID switch | evening n=3 | 27 | 15 | 31 | 0 | 56 | -| post-stop-sequence | (in progress) | — | — | — | — | — | +| post-stop-sequence | late-evening n=3 | 29 | 14 | **37** | 0 | **60** | + +JSON mode net: 26 → 37 STRICT (+11), 19 → 0 errors, 38 → 60 grounded (+22) over the day. Strict-rate climbed 39% → 56% (+17pp). Each of the four improvements addressed a named failure mode surfaced by the prior bench. Each step in the journey was a fix to a specific failure mode named by the prior step's bench. The methodology delivered: name the failure → fix in code → re-bench → confirm or surface the next failure.