engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(qa): final stop-sequence bench row, day-journey net summary

Browse source 
Post-stop-sequence bench (2026-04-30T20-15-11Z):
  quote   29S 21H 16U   strict-rate 44%   grounded 50   6.0s
  pointer 14S 44H  8U   strict-rate 21%   grounded 58   4.8s
  JSON    37S 23H  6U   strict-rate 56%   grounded 60   4.5s

JSON mode delta vs prior post-pointer-ID bench:
  STRICT:     31 → 37  (+6)
  UNGROUNDED: 10 →  6  (-4)
  strict-rate: 47% → 56% (+9pp)
  apollo runaway: 3/3 → 2/3 (one sample recovered)
  latency:    5.5s → 4.5s (runaway cases were dragging avg)

Day's net for JSON mode (morning baseline → late-evening):
  STRICT:     26 → 37  (+11)
  errors:     19 →  0  (-19)
  grounded:   38 → 60  (+22)
  strict-rate: 39% → 56%  (+17pp)

Each step was a named-failure → fix → re-bench cycle:
retry → trim-and-verify → pointer-IDs → stop-sequence.
This commit is contained in:
russell@unturf.com 2026-04-30 16:32:41 -04:00
parent 4aaa253dfd
commit 092cac4e80
No known key found for this signature in database
1 changed files with 3 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

4
docs/qa-modes-bench-2026-04-30.md
View file

@ -72,7 +72,9 @@ Fix landed in commit `f23d3a3`: pass `stop=["\n\n"]` to vLLM in JSON mode. Well-
| baseline (no improvements) | morning n=3 | 31 | 14 | 26 | **19** | 38 |
| post-retry + trim-and-verify | midday n=3 | 31 | 16 | **33** | 0 | 54 |
| post-pointer-ID switch | evening n=3 | 27 | 15 | 31 | 0 | 56 |
| post-stop-sequence | (in progress) | — | — | — | — | — |
| post-stop-sequence | late-evening n=3 | 29 | 14 | **37** | 0 | **60** |
JSON mode net: 26 → 37 STRICT (+11), 19 → 0 errors, 38 → 60 grounded (+22) over the day. Strict-rate climbed 39% → 56% (+17pp). Each of the four improvements addressed a named failure mode surfaced by the prior bench.
Each step in the journey was a fix to a specific failure mode named by the prior step's bench. The methodology delivered: name the failure → fix in code → re-bench → confirm or surface the next failure.