# Bench-maxing — measure deltas, not opinions

The QA-quality bench (`make bench-qa`) and live functional fixtures
(`make test-live`) are not decoration. They are the discipline that
keeps prompt + verifier work honest. Patterns established 2026-04-30
during the JSON-mode hardening journey:

- **Bench before AND after every change.** Single-sample bench has
  ±20pp noise; n=3 narrows the band but signal under 5pp is still
  noise. The only way to know a change moved the needle is to run
  the same harness on identical inputs immediately before and
  immediately after. Don't skip this — opinions about whether a
  prompt change "should help" are routinely wrong.

- **Slight prompt adjustments are fine — local-minima are not.**
  Prompts can be nudged with a worked example or a single-sentence
  rule. They should NOT be padded with many rules that incentivize
  empty / over-cautious output ("if uncertain, omit" can collapse
  the model into refusing valid answers). When in doubt, change
  one thing per bench cycle.

- **Avoid negation in every prompt.** Small instruction-tuned models
  (Hermes-3-8B observed) struggle with "do NOT X" / "never Y" /
  "without Z" — the negation often pattern-matches away or inverts
  under attention. Rewrite every rule to its positive form. Some
  swaps that work:
  - "Do not invent IDs" → "Use only the IDs that appear in the EVIDENCE blocks above"
  - "Never begin a line with E1:" → "Each line begins with the claim text, then a bracket tag"
  - "No commentary, no preamble" → "Your output begins with `{` and ends with `}`"
  - "If no evidence supports a claim, omit the claim" → "Emit only claims that an EVIDENCE block directly supports"
  - "No double-quote characters in text" → "Write text as plain prose; punctuation appears in the source span the runtime renders for you"

  Audit all the `*_system_prompt` and `*_grounding_reminder` policy
  fields for residual negation when a model behaves erratically — it
  is the cheapest fix in the prompt-iteration toolkit.

- **Bench is the scoreboard; live fixtures are the gates.** Bench
  measures aggregate (STRICT/HYBRID/UNGROUNDED counts across N×Q
  samples). Live fixtures (`tests/test_qa_quality_live.py`) gate on
  specific known-good answers. When a change improves things the
  bench number climbs AND every fixture passes; when something
  regresses the bench falls AND named fixtures fail by name,
  pointing at where the regression landed.

- **Self-heal beats retry.** When the model emits malformed output
  (truncated JSON, trailing comma, partial key), repair the artifact
  rather than re-running the LLM call. Self-healing preserves
  whatever partial content the model produced and lands in the
  unverified bucket honestly. Retrying spends another inference
  round and may produce identical garbage. See
  `_repair_truncated_json` in `aborist/qa/verify.py` for the
  pattern: walk once tracking string state + bracket stack, close /
  drop / balance at end-of-input. Conservative repairs only — never
  insert content, never fabricate keys.

- **Name the failure → fix in code → re-bench → confirm.** Each
  bench-driven commit should reference the specific failure mode it
  addresses (e.g. "Apollo runaway: 3/3 → 2/3" or "JSON hallucinated
  near-miss content-IDs on cross-doc relationships"). When the next
  bench shows the named failure didn't budge, you fixed the wrong
  thing. Name it again, try again.

- **Honest verdicts beat optimistic ones.** A change that drops
  STRICT count by 5 but moves those 5 to HYBRID with real grounding
  is a WIN — false-positive STRICTs are corruption. The bench grade
  improves when the verifier reports closer to ground truth, not
  closer to 100%. Trust HYBRID with smell-sidecar warnings over
  STRICT-with-bogus-citation every time.

- **Per-question fixtures > marker-string assertions.** When
  designing a live fixture, prefer entity-presence checks ("Burns
  in the answer", "October or 1991") over byte-identical matching.
  Hermes is non-deterministic; the right entity in the answer is
  the gate, not the exact phrasing. When a fixture starts failing
  intermittently, treat it as quality drift on that question shape
  and tighten retrieval / verifier upstream rather than loosening
  the test.

- **Old maps vs runtime maps — move authority toward runtime artifacts.**
  Every base model carries old maps: trained patterns from the time
  the weights froze. It "knows" how API X used to work, what JSON
  shape Y used to take, which ritual citation phrase was popular.
  Those priors collide with the present-day runtime — the actual
  retrieved chunks, the current evidence map, the live policy hash,
  the ID space the runtime built this minute. When the model and
  the runtime disagree, the runtime wins. Engineering moves
  authority OUT of the model's old map and INTO runtime artifacts:
  - Pointer IDs (E1, E2, …) the runtime mints, so the model can't
    fabricate a content-addressed hash that "looks right."
  - Source spans the runtime interpolates by
    `(chunk_root, offset_start, offset_end)`, so the model never
    owns the quote text.
  - Evidence maps the retrieval pipeline assembles per-query, so
    the model can't self-supply context from training memory.
  - Policy hashes that fold prompt + verifier + retrieval choices
    into the cache_key, so old governance can't silently reuse
    new-rule records.
  - Hard checks run by the verifier, never the model self-grading.

  When a defect smells like "model brought a stale map," the fix is
  to make the runtime more authoritative, not to negotiate with the
  model's prior. Hermes' content-addressed-evidence-id hallucination
  (commit `bb8450d`, 2026-04-30) is the canonical case: the fix was
  to swap the prompt-facing surface to runtime-minted pointer IDs
  the model can't fabricate, not to add an "are you sure?" step.

- **Doc captures the journey, not just the end state.**
  `docs/qa-modes-bench-2026-04-30.md` shows the day's progression
  (baseline → retry → trim-and-verify → pointer-IDs → stop-sequence
  → self-heal). Each row is a named failure + fix + bench delta.
  Future-you (or a new agent) can read the doc and see WHY each
  commit landed, not just WHAT.
