arborist/docs/bench-maxing.md

Bench-maxing — measure deltas, not opinions

The QA-quality bench (make bench-qa) and live functional fixtures (make test-live) are not decoration. They are the discipline that keeps prompt + verifier work honest. Patterns established 2026-04-30 during the JSON-mode hardening journey:

• Bench before AND after every change. Single-sample bench has ±20pp noise; n=3 narrows the band but signal under 5pp is still noise. The only way to know a change moved the needle is to run the same harness on identical inputs immediately before and immediately after. Don't skip this — opinions about whether a prompt change "should help" are routinely wrong.

• Slight prompt adjustments are fine — local-minima are not. Prompts can be nudged with a worked example or a single-sentence rule. They should NOT be padded with many rules that incentivize empty / over-cautious output ("if uncertain, omit" can collapse the model into refusing valid answers). When in doubt, change one thing per bench cycle.

• Avoid negation in every prompt. Small instruction-tuned models (Hermes-3-8B observed) struggle with "do NOT X" / "never Y" / "without Z" — the negation often pattern-matches away or inverts under attention. Rewrite every rule to its positive form. Some swaps that work:

  • "Do not invent IDs" → "Use only the IDs that appear in the EVIDENCE blocks above"
  • "Never begin a line with E1:" → "Each line begins with the claim text, then a bracket tag"
  • "No commentary, no preamble" → "Your output begins with { and ends with }"
  • "If no evidence supports a claim, omit the claim" → "Emit only claims that an EVIDENCE block directly supports"
  • "No double-quote characters in text" → "Write text as plain prose; punctuation appears in the source span the runtime renders for you"

  Audit all the *_system_prompt and *_grounding_reminder policy fields for residual negation when a model behaves erratically — it is the cheapest fix in the prompt-iteration toolkit.

• Bench is the scoreboard; live fixtures are the gates. Bench measures aggregate (STRICT/HYBRID/UNGROUNDED counts across N×Q samples). Live fixtures (tests/test_qa_quality_live.py) gate on specific known-good answers. When a change improves things the bench number climbs AND every fixture passes; when something regresses the bench falls AND named fixtures fail by name, pointing at where the regression landed.

• Self-heal beats retry. When the model emits malformed output (truncated JSON, trailing comma, partial key), repair the artifact rather than re-running the LLM call. Self-healing preserves whatever partial content the model produced and lands in the unverified bucket honestly. Retrying spends another inference round and may produce identical garbage. See _repair_truncated_json in arborist/qa/verify.py for the pattern: walk once tracking string state + bracket stack, close / drop / balance at end-of-input. Conservative repairs only — never insert content, never fabricate keys.

• Name the failure → fix in code → re-bench → confirm. Each bench-driven commit should reference the specific failure mode it addresses (e.g. "Apollo runaway: 3/3 → 2/3" or "JSON hallucinated near-miss content-IDs on cross-doc relationships"). When the next bench shows the named failure didn't budge, you fixed the wrong thing. Name it again, try again.

• Honest verdicts beat optimistic ones. A change that drops STRICT count by 5 but moves those 5 to HYBRID with real grounding is a WIN — false-positive STRICTs are corruption. The bench grade improves when the verifier reports closer to ground truth, not closer to 100%. Trust HYBRID with smell-sidecar warnings over STRICT-with-bogus-citation every time.

• Per-question fixtures > marker-string assertions. When designing a live fixture, prefer entity-presence checks ("Burns in the answer", "October or 1991") over byte-identical matching. Hermes is non-deterministic; the right entity in the answer is the gate, not the exact phrasing. When a fixture starts failing intermittently, treat it as quality drift on that question shape and tighten retrieval / verifier upstream rather than loosening the test.

• Old maps vs runtime maps — move authority toward runtime artifacts. Every base model carries old maps: trained patterns from the time the weights froze. It "knows" how API X used to work, what JSON shape Y used to take, which ritual citation phrase was popular. Those priors collide with the present-day runtime — the actual retrieved chunks, the current evidence map, the live policy hash, the ID space the runtime built this minute. When the model and the runtime disagree, the runtime wins. Engineering moves authority OUT of the model's old map and INTO runtime artifacts:

  • Pointer IDs (E1, E2, …) the runtime mints, so the model can't fabricate a content-addressed hash that "looks right."
  • Source spans the runtime interpolates by (chunk_root, offset_start, offset_end), so the model never owns the quote text.
  • Evidence maps the retrieval pipeline assembles per-query, so the model can't self-supply context from training memory.
  • Policy hashes that fold prompt + verifier + retrieval choices into the cache_key, so old governance can't silently reuse new-rule records.
  • Hard checks run by the verifier, never the model self-grading.

  When a defect smells like "model brought a stale map," the fix is to make the runtime more authoritative, not to negotiate with the model's prior. Hermes' content-addressed-evidence-id hallucination (commit bb8450d, 2026-04-30) is the canonical case: the fix was to swap the prompt-facing surface to runtime-minted pointer IDs the model can't fabricate, not to add an "are you sure?" step.

• Doc captures the journey, not just the end state. docs/qa-modes-bench-2026-04-30.md shows the day's progression (baseline → retry → trim-and-verify → pointer-IDs → stop-sequence → self-heal). docs/qa-modes-bench.md continues the journey through Rule 8 promotion, anchor-class warrant generalization, per-mode context budgets, and the bench-harness upgrades named below. Each row is a named failure + fix + bench delta. Future-you (or a new agent) can read the docs and see WHY each commit landed, not just WHAT.

Bench harness — the speed playbook

Patterns added 2026-05-02. The bench is now the inner-loop tool; treat it like a CI gate, not a once-a-week ritual.

• Sample-level shuffled scheduling, not cell-grouped. Cell-grouped scheduling (run all n samples of one question/mode back-to-back before the next cell) confounds two things: (1) it correlates the n samples — vLLM's continuous batcher fills with same-cache_key requests, KV-cache locality and queue load become shared signal across "samples 1-3"; (2) it starves the batcher of diverse work, so per-call latency stretches and aggregate throughput drops. Sample-level scheduling — every (q, mode, sample_idx) is its own task, shuffled with a deterministic seed — gives true i.i.d. variance for n>=3 and delivers the batcher uncorrelated requests. Per-cell Lock dict serializes the rare adjacent-same-cell case so burn-then-write on the shared cache_key never races. Measured on 2026-05-02: +58% throughput at the same concurrency.

• Persistent HTTP client. The OpenAI-compatible client used to build a fresh httpx.Client per chat_completion call, paying a TLS handshake on every request. Move it to __init__ so HTTP/1.1 keep-alive holds the TCP+TLS connection across calls from the same client instance. httpx's pool is thread-safe; one client serves all worker threads. Saved 1-2 min on a 426-call bench, more on longer sweeps.

• Concurrency tuning, empirically. vLLM has a sweet spot — more concurrent requests fill the batch better, but only up to the point where per-call latency growth outpaces parallelism gain. On hermes.ai.unturf.com the sweep result (15-task smoke fixture, 2026-05-02):

  c=3   102s   8.8 tasks/min  ← peak
  c=4   106s   8.5 tasks/min  ← within 4%; chosen for runs
  c=5   119s   7.6 tasks/min  ← 12% slower
  c=6   185s   4.9 tasks/min  ← 45% slower (saturation)
  

  Run a small smoke at multiple concurrency levels before committing to a long full-bench. The peak shifts when vLLM is upgraded, when the corpus changes the average context size, or when other tenants load the endpoint differently.

• Per-mode max_context_chars from bench feedback. The bench's "recommended context budget" table (per-mode strict-rate by context-size bucket) is the substrate's voice telling the policy where each mode peaks. As of the 2026-05-02 bench: quote 8-16KB, pointer 16-32KB, JSON 32-64KB. Those defaults live in DEFAULT_QUERY_POLICY["max_context_chars_by_mode"] and fold into governance_policy_hash so any retuning cleanly partitions the cache namespace. Tighten the cap when bench shows degradation past a smaller bucket; loosen it when JSON starts paging in 64-128KB. The harness reports the table on every run — let it drive the policy, not the other way around.

• --seed for reproducibility. Same --seed reproduces the shuffle. Useful when a regression bench needs to be re-run against the exact same task order; useful when the --resume path needs the post-resume order to align with the pre-kill order. Default --seed 0.

• --resume <jsonl-path> — stop/start-able. A 60-90 minute full bench is annoying to re-run from zero when something interrupts (Ctrl-C, kernel panic, network outage). The bench reads the existing JSONL, skips already-completed (q, mode, sample_idx) triples, appends fresh rows to the same file, and re-renders the markdown summary from the union. Same --seed required so the remaining tasks come out in the original order. Stops can now be cheap.

• Smoke fixture for inner-loop. bench/qa_questions_smoke.txt carries 5 questions covering all anchor classes — date, place, relation, entity-list, why-cause — each chosen because pointer mode currently fails 100% on them and JSON aces 100%. Run via make bench-qa-smoke (~1-3 min wall-clock). The smoke is the inner-loop tool for prompt iteration; the full 71-question bench stays the scoreboard. The bench-as-build-gate doctrine: smoke before full.

• pytest -n auto for the test suite. pytest-xdist lands the test suite in 10s instead of 36s on this hardware (3.6× speedup). Wired via make test. Cycle-time enabler for prompt-iteration loops where the test suite gates the bench.