arborist/docs/bench-maxing.md

Bench-maxing — measure deltas, not opinions

The QA-quality bench (make bench-qa) and live functional fixtures (make test-live) are not decoration. They are the discipline that keeps prompt + verifier work honest. Patterns established 2026-04-30 during the JSON-mode hardening journey:

• Bench before AND after every change. Single-sample bench has ±20pp noise; n=3 narrows the band but signal under 5pp is still noise. The only way to know a change moved the needle is to run the same harness on identical inputs immediately before and immediately after. Don't skip this — opinions about whether a prompt change "should help" are routinely wrong.

• Slight prompt adjustments are fine — local-minima are not. Prompts can be nudged with a worked example or a single-sentence rule. They should NOT be padded with many rules that incentivize empty / over-cautious output ("if uncertain, omit" can collapse the model into refusing valid answers). When in doubt, change one thing per bench cycle.

• Avoid negation in every prompt. Small instruction-tuned models (Hermes-3-8B observed) struggle with "do NOT X" / "never Y" / "without Z" — the negation often pattern-matches away or inverts under attention. Rewrite every rule to its positive form. Some swaps that work:

  • "Do not invent IDs" → "Use only the IDs that appear in the EVIDENCE blocks above"
  • "Never begin a line with E1:" → "Each line begins with the claim text, then a bracket tag"
  • "No commentary, no preamble" → "Your output begins with { and ends with }"
  • "If no evidence supports a claim, omit the claim" → "Emit only claims that an EVIDENCE block directly supports"
  • "No double-quote characters in text" → "Write text as plain prose; punctuation appears in the source span the runtime renders for you"

  Audit all the *_system_prompt and *_grounding_reminder policy fields for residual negation when a model behaves erratically — it is the cheapest fix in the prompt-iteration toolkit.

• Bench is the scoreboard; live fixtures are the gates. Bench measures aggregate (STRICT/HYBRID/UNGROUNDED counts across N×Q samples). Live fixtures (tests/test_qa_quality_live.py) gate on specific known-good answers. When a change improves things the bench number climbs AND every fixture passes; when something regresses the bench falls AND named fixtures fail by name, pointing at where the regression landed.

• Self-heal beats retry. When the model emits malformed output (truncated JSON, trailing comma, partial key), repair the artifact rather than re-running the LLM call. Self-healing preserves whatever partial content the model produced and lands in the unverified bucket honestly. Retrying spends another inference round and may produce identical garbage. See _repair_truncated_json in aborist/qa/verify.py for the pattern: walk once tracking string state + bracket stack, close / drop / balance at end-of-input. Conservative repairs only — never insert content, never fabricate keys.

• Name the failure → fix in code → re-bench → confirm. Each bench-driven commit should reference the specific failure mode it addresses (e.g. "Apollo runaway: 3/3 → 2/3" or "JSON hallucinated near-miss content-IDs on cross-doc relationships"). When the next bench shows the named failure didn't budge, you fixed the wrong thing. Name it again, try again.

• Honest verdicts beat optimistic ones. A change that drops STRICT count by 5 but moves those 5 to HYBRID with real grounding is a WIN — false-positive STRICTs are corruption. The bench grade improves when the verifier reports closer to ground truth, not closer to 100%. Trust HYBRID with smell-sidecar warnings over STRICT-with-bogus-citation every time.

• Per-question fixtures > marker-string assertions. When designing a live fixture, prefer entity-presence checks ("Burns in the answer", "October or 1991") over byte-identical matching. Hermes is non-deterministic; the right entity in the answer is the gate, not the exact phrasing. When a fixture starts failing intermittently, treat it as quality drift on that question shape and tighten retrieval / verifier upstream rather than loosening the test.

• Old maps vs runtime maps — move authority toward runtime artifacts. Every base model carries old maps: trained patterns from the time the weights froze. It "knows" how API X used to work, what JSON shape Y used to take, which ritual citation phrase was popular. Those priors collide with the present-day runtime — the actual retrieved chunks, the current evidence map, the live policy hash, the ID space the runtime built this minute. When the model and the runtime disagree, the runtime wins. Engineering moves authority OUT of the model's old map and INTO runtime artifacts:

  • Pointer IDs (E1, E2, …) the runtime mints, so the model can't fabricate a content-addressed hash that "looks right."
  • Source spans the runtime interpolates by (chunk_root, offset_start, offset_end), so the model never owns the quote text.
  • Evidence maps the retrieval pipeline assembles per-query, so the model can't self-supply context from training memory.
  • Policy hashes that fold prompt + verifier + retrieval choices into the cache_key, so old governance can't silently reuse new-rule records.
  • Hard checks run by the verifier, never the model self-grading.

  When a defect smells like "model brought a stale map," the fix is to make the runtime more authoritative, not to negotiate with the model's prior. Hermes' content-addressed-evidence-id hallucination (commit bb8450d, 2026-04-30) is the canonical case: the fix was to swap the prompt-facing surface to runtime-minted pointer IDs the model can't fabricate, not to add an "are you sure?" step.

• Doc captures the journey, not just the end state. docs/qa-modes-bench-2026-04-30.md shows the day's progression (baseline → retry → trim-and-verify → pointer-IDs → stop-sequence → self-heal). Each row is a named failure + fix + bench delta. Future-you (or a new agent) can read the doc and see WHY each commit landed, not just WHAT.