arborist/docs/bench-maxing.md

# Bench-maxing — measure deltas, not opinions

The QA-quality bench (`make bench-qa`) and live functional fixtures
(`make test-live`) are not decoration. They are the discipline that
keeps prompt + verifier work honest. Patterns established 2026-04-30
during the JSON-mode hardening journey:

- **Bench before AND after every change.** Single-sample bench has
  ±20pp noise; n=3 narrows the band but signal under 5pp is still
  noise. The only way to know a change moved the needle is to run
  the same harness on identical inputs immediately before and
  immediately after. Don't skip this — opinions about whether a
  prompt change "should help" are routinely wrong.

- **Slight prompt adjustments are fine — local-minima are not.**
  Prompts can be nudged with a worked example or a single-sentence
  rule. They should NOT be padded with many rules that incentivize
  empty / over-cautious output ("if uncertain, omit" can collapse
  the model into refusing valid answers). When in doubt, change
  one thing per bench cycle.

- **Avoid negation in every prompt.** Small instruction-tuned models
  (Hermes-3-8B observed) struggle with "do NOT X" / "never Y" /
  "without Z" — the negation often pattern-matches away or inverts
  under attention. Rewrite every rule to its positive form. Some
  swaps that work:
  - "Do not invent IDs" → "Use only the IDs that appear in the EVIDENCE blocks above"
  - "Never begin a line with E1:" → "Each line begins with the claim text, then a bracket tag"
  - "No commentary, no preamble" → "Your output begins with `{` and ends with `}`"
  - "If no evidence supports a claim, omit the claim" → "Emit only claims that an EVIDENCE block directly supports"
  - "No double-quote characters in text" → "Write text as plain prose; punctuation appears in the source span the runtime renders for you"

  Audit all the `*_system_prompt` and `*_grounding_reminder` policy
  fields for residual negation when a model behaves erratically — it
  is the cheapest fix in the prompt-iteration toolkit.

- **Bench is the scoreboard; live fixtures are the gates.** Bench
  measures aggregate (STRICT/HYBRID/UNGROUNDED counts across N×Q
  samples). Live fixtures (`tests/test_qa_quality_live.py`) gate on
  specific known-good answers. When a change improves things the
  bench number climbs AND every fixture passes; when something
  regresses the bench falls AND named fixtures fail by name,
  pointing at where the regression landed.

- **Self-heal beats retry.** When the model emits malformed output
  (truncated JSON, trailing comma, partial key), repair the artifact
  rather than re-running the LLM call. Self-healing preserves
  whatever partial content the model produced and lands in the
  unverified bucket honestly. Retrying spends another inference
  round and may produce identical garbage. See
  `_repair_truncated_json` in `arborist/qa/verify.py` for the
  pattern: walk once tracking string state + bracket stack, close /
  drop / balance at end-of-input. Conservative repairs only — never
  insert content, never fabricate keys.

- **Name the failure → fix in code → re-bench → confirm.** Each
  bench-driven commit should reference the specific failure mode it
  addresses (e.g. "Apollo runaway: 3/3 → 2/3" or "JSON hallucinated
  near-miss content-IDs on cross-doc relationships"). When the next
  bench shows the named failure didn't budge, you fixed the wrong
  thing. Name it again, try again.

- **Honest verdicts beat optimistic ones.** A change that drops
  STRICT count by 5 but moves those 5 to HYBRID with real grounding
  is a WIN — false-positive STRICTs are corruption. The bench grade
  improves when the verifier reports closer to ground truth, not
  closer to 100%. Trust HYBRID with smell-sidecar warnings over
  STRICT-with-bogus-citation every time.

- **Per-question fixtures > marker-string assertions.** When
  designing a live fixture, prefer entity-presence checks ("Burns
  in the answer", "October or 1991") over byte-identical matching.
  Hermes is non-deterministic; the right entity in the answer is
  the gate, not the exact phrasing. When a fixture starts failing
  intermittently, treat it as quality drift on that question shape
  and tighten retrieval / verifier upstream rather than loosening
  the test.

- **Old maps vs runtime maps — move authority toward runtime artifacts.**
  Every base model carries old maps: trained patterns from the time
  the weights froze. It "knows" how API X used to work, what JSON
  shape Y used to take, which ritual citation phrase was popular.
  Those priors collide with the present-day runtime — the actual
  retrieved chunks, the current evidence map, the live policy hash,
  the ID space the runtime built this minute. When the model and
  the runtime disagree, the runtime wins. Engineering moves
  authority OUT of the model's old map and INTO runtime artifacts:
  - Pointer IDs (E1, E2, …) the runtime mints, so the model can't
    fabricate a content-addressed hash that "looks right."
  - Source spans the runtime interpolates by
    `(chunk_root, offset_start, offset_end)`, so the model never
    owns the quote text.
  - Evidence maps the retrieval pipeline assembles per-query, so
    the model can't self-supply context from training memory.
  - Policy hashes that fold prompt + verifier + retrieval choices
    into the cache_key, so old governance can't silently reuse
    new-rule records.
  - Hard checks run by the verifier, never the model self-grading.

  When a defect smells like "model brought a stale map," the fix is
  to make the runtime more authoritative, not to negotiate with the
  model's prior. Hermes' content-addressed-evidence-id hallucination
  (commit `bb8450d`, 2026-04-30) is the canonical case: the fix was
  to swap the prompt-facing surface to runtime-minted pointer IDs
  the model can't fabricate, not to add an "are you sure?" step.

- **Doc captures the journey, not just the end state.**
  `docs/qa-modes-bench-2026-04-30.md` shows the day's progression
  (baseline → retry → trim-and-verify → pointer-IDs → stop-sequence
  → self-heal). `docs/qa-modes-bench.md` continues the
  journey through Rule 8 promotion, anchor-class warrant
  generalization, per-mode context budgets, and the bench-harness
  upgrades named below. Each row is a named failure + fix + bench
  delta. Future-you (or a new agent) can read the docs and see WHY
  each commit landed, not just WHAT.

## Bench harness — the speed playbook

Patterns added 2026-05-02. The bench is now the inner-loop tool;
treat it like a CI gate, not a once-a-week ritual.

- **Sample-level shuffled scheduling, not cell-grouped.**
  Cell-grouped scheduling (run all n samples of one question/mode
  back-to-back before the next cell) confounds two things:
  (1) it correlates the n samples — vLLM's continuous batcher fills
  with same-cache_key requests, KV-cache locality and queue load
  become shared signal across "samples 1-3"; (2) it starves the
  batcher of diverse work, so per-call latency stretches and
  aggregate throughput drops. Sample-level scheduling — every
  `(q, mode, sample_idx)` is its own task, shuffled with a
  deterministic seed — gives true i.i.d. variance for n>=3 and
  delivers the batcher uncorrelated requests. Per-cell `Lock` dict
  serializes the rare adjacent-same-cell case so burn-then-write
  on the shared `cache_key` never races. Measured on 2026-05-02:
  +58% throughput at the same concurrency.

- **Persistent HTTP client.** The OpenAI-compatible client used to
  build a fresh `httpx.Client` per `chat_completion` call, paying
  a TLS handshake on every request. Move it to `__init__` so
  HTTP/1.1 keep-alive holds the TCP+TLS connection across calls
  from the same client instance. httpx's pool is thread-safe; one
  client serves all worker threads. Saved 1-2 min on a 426-call
  bench, more on longer sweeps.

- **Concurrency tuning, empirically.** vLLM has a sweet spot —
  more concurrent requests fill the batch better, but only up to
  the point where per-call latency growth outpaces parallelism
  gain. On `hermes.ai.unturf.com` the sweep result (15-task smoke
  fixture, 2026-05-02):

      c=3   102s   8.8 tasks/min  ← peak
      c=4   106s   8.5 tasks/min  ← within 4%; chosen for runs
      c=5   119s   7.6 tasks/min  ← 12% slower
      c=6   185s   4.9 tasks/min  ← 45% slower (saturation)

  Run a small smoke at multiple concurrency levels before
  committing to a long full-bench. The peak shifts when vLLM is
  upgraded, when the corpus changes the average context size, or
  when other tenants load the endpoint differently.

- **Per-mode `max_context_chars` from bench feedback.** The
  bench's "recommended context budget" table (per-mode strict-rate
  by context-size bucket) is the substrate's voice telling the
  policy where each mode peaks. As of the 2026-05-02 bench:
  quote 8-16KB, pointer 16-32KB, JSON 32-64KB. Those defaults
  live in `DEFAULT_QUERY_POLICY["max_context_chars_by_mode"]` and
  fold into `governance_policy_hash` so any retuning cleanly
  partitions the cache namespace. Tighten the cap when bench
  shows degradation past a smaller bucket; loosen it when JSON
  starts paging in 64-128KB. The harness reports the table on
  every run — let it drive the policy, not the other way around.

- **`--seed` for reproducibility.** Same `--seed` reproduces the
  shuffle. Useful when a regression bench needs to be re-run
  against the exact same task order; useful when the `--resume`
  path needs the post-resume order to align with the pre-kill
  order. Default `--seed 0`.

- **`--resume <jsonl-path>` — stop/start-able.** A 60-90 minute
  full bench is annoying to re-run from zero when something
  interrupts (Ctrl-C, kernel panic, network outage). The bench
  reads the existing JSONL, skips already-completed
  `(q, mode, sample_idx)` triples, appends fresh rows to the same
  file, and re-renders the markdown summary from the union. Same
  `--seed` required so the remaining tasks come out in the
  original order. Stops can now be cheap.

- **Smoke fixture for inner-loop.** `bench/qa_questions_smoke.txt`
  carries 5 questions covering all anchor classes — date, place,
  relation, entity-list, why-cause — each chosen because pointer
  mode currently fails 100% on them and JSON aces 100%. Run via
  `make bench-qa-smoke` (~1-3 min wall-clock). The smoke is the
  inner-loop tool for prompt iteration; the full 71-question
  bench stays the scoreboard. The bench-as-build-gate doctrine:
  smoke before full.

- **`pytest -n auto` for the test suite.** `pytest-xdist` lands
  the test suite in 10s instead of 36s on this hardware (3.6×
  speedup). Wired via `make test`. Cycle-time enabler for
  prompt-iteration loops where the test suite gates the bench.