arborist/docs/benchmarks.md

# Benchmarks — orientation, harnesses, fixtures, discipline

Arborist ships two bench harnesses, four question fixtures, and a
signal-floor discipline pinned across the substrate. This doc is
the canonical entry point. Three companion docs cover specifics:

| File                                   | Purpose                                    |
|----------------------------------------|--------------------------------------------|
| `docs/bench-maxing.md`                 | Discipline rules (5pp signal floor, etc.)  |
| `docs/qa-modes-bench.md`               | Historical journal — running addendum log  |
| `docs/bench-emergent-design.md`        | Random-word stress-test design rationale   |
| `docs/qa-modes-bench-2026-04-30.md`    | Frozen point-in-time bench (Apr-30 baseline) |

Read those when you need the why or the long-form data. This file
covers the what, where, and how.

## 1. Two harnesses, two purposes

### 1.1 `bench/qa_sweep.py` — curated QA-quality bench

Runs a fixture file × answer modes × n samples. Persists a JSONL
of per-row results plus a markdown summary. Used for:

- **Regression checks** before merging substrate changes.
- **A/B cycles** on policy knobs (cap on/off, reminder on/off,
  preflight enabled/disabled).
- **Per-mode strict-rate tracking** over time.

Run via `make bench-qa` or `bench/qa_sweep.py` directly. Bench
rows carry ~25 fields per cell, including the four-rung label
(`audit_mode`), the verifier-method, the violation kinds, the
preflight QuestionState projection (`preflight_logical_statuses`,
`preflight_question_shape`, etc.), the quantifier classifier
output (`quantifier_intensity`, `scope_bound_hint`,
`claim_cap_applied`), the preflight stage hash 12-char prefix
(#000009 §7.2), and a per-stage timing breakdown
(`preflight_ms`, `soft_preflight_ms`, `search_ms`, `context_ms`,
`llm_ms`, `persist_ms`, `total_ms`).

### 1.2 `scripts/bench_emergent.py` — random-word stress test

Picks three random words from `/usr/share/dict/words`, asks
Hermes (at temp 0.8) to weave them into a creative question
paragraph, then sends that paragraph to arborist. Logs every cycle
to `bench/emergent_log.jsonl` as one JSONL line. Designed for
**blue-moon cadence**, not every-commit benching — the
combinatoric word space surfaces failure shapes the curated bench
doesn't reach.

The teacher review step is intentionally **not automated.** Fox
brings interesting log entries to a teacher model (Claude Opus
4.7 in the current setup) and asks for guidance: did the answer
match the question? Novelty class
(known_truth_grounding / emergent_synthesis / novel_claim /
no_signal)? Which hyperparam to tune next?

Run via `make bench-emergent` (default `EMERGENT_N=10`) or
`make bench-emergent EMERGENT_N=100` for a longer cycle. See
`docs/bench-emergent-design.md` for the design rationale.

## 2. Four question fixtures

```
bench/qa_questions.txt                                — 75 questions
bench/qa_questions_smoke.txt                          —  smoke test
bench/qa_questions_quantifier_subset.txt              —  9 questions
bench/qa_questions_quantifier_baseline.txt            —  1 question
bench/qa_questions_metacog_subset.txt                 — 28 questions
```

| Fixture                              | Use case                                                                                    | Cell size at n=3 × 3 modes |
|--------------------------------------|---------------------------------------------------------------------------------------------|----------------------------|
| `qa_questions.txt`                   | Full regression bench. Run before merging substrate changes. Wide coverage + corner cases. | 675 runs, ~3-3.5h          |
| `qa_questions_smoke.txt`             | Smoke test on 1-2 questions. Use to confirm a CLI flag wires through correctly.            | 3-6 runs, ~30s             |
| `qa_questions_quantifier_subset.txt` | 9 broad-quantifier questions (7 unbounded + 2 bounded). Use for A/B cycles on #000008 knobs. | 81 runs, ~7-15min          |
| `qa_questions_quantifier_baseline.txt` | Single broad question for narrow n=3 baseline measurement.                              | 9 runs, ~2-4min            |
| `qa_questions_metacog_subset.txt`    | 28 questions targeting each #000010 detector (temporal, contradiction, false-premise, out-of-corpus) plus mixed multi-trigger + well-formed controls. | 252 runs, ~25-35min |

When choosing a fixture: ask whether you're measuring the
**substrate as a whole** (full bench), a **specific feature**
(broad-quantifier subset, metacog subset), or just **wiring
correctness** (smoke).

## 3. The signal floor — n=3 × 9 = 27 sample minimum, 5pp delta floor

Per `docs/bench-maxing.md`:

- **5pp floor** for STRICT-rate deltas at n=3 × any-fixture sample
  size. Differences below this are Hermes nondeterminism, not
  signal. Quote-mode often jitters ±10pp on 27-sample cells.
- **Mean-ratio + UNGROUNDED-rate** are tighter signals at the same
  sample size — they aggregate per-row values rather than
  counting discrete verdicts.
- **n=5 verification** when an A/B cell sits within the noise
  band (e.g. 0.30 vs 0.33 STRICT-rate is 3pp — within noise; n=5
  on the closer cell tightens the variance).
- **vLLM concurrency saturates at c=3-4.** Going higher slows
  per-call latency without parallelizing harder. Default `c=4`.

Don't tune to a single bench cycle. Always have a baseline +
an A/B cell + (sometimes) a verification cell.

## 4. Make targets cheat sheet

```
# Curated QA bench (default: bench/qa_questions.txt × 3 modes × n=3)
make bench-qa

# Same but with limit / different file
.venv/bin/python bench/qa_sweep.py \
    --questions bench/qa_questions_quantifier_subset.txt \
    --shards-dir ~/.arborist/shards --n 3 --concurrency 4 \
    --out-dir bench/qa_results

# A/B cycle: flip a policy knob for one bench cell
.venv/bin/python bench/qa_sweep.py \
    --questions bench/qa_questions_quantifier_subset.txt \
    --shards-dir ~/.arborist/shards --n 3 --concurrency 4 \
    --policy quantifier_guard_apply_caps=true \
    --policy quantifier_reminder_enabled=true

# Resume an interrupted bench (same --seed, append to existing JSONL)
.venv/bin/python bench/qa_sweep.py \
    --resume bench/qa_results/2026-05-03T19-30-13Z.jsonl \
    --seed 0

# Smoke test — quick sanity of the wiring
.venv/bin/python bench/qa_sweep.py \
    --questions bench/qa_questions_smoke.txt --n 1 --concurrency 1

# Random-word stress test (blue-moon cadence)
make bench-emergent              # n=10 default
make bench-emergent EMERGENT_N=100
```

## 5. The bench-row schema

Every row in `bench/qa_results/<stamp>.jsonl` carries (current
shape — fields are additive across versions):

**Identity & verdict:**
- `question`, `answer_mode`, `audit_mode`, `verifier_method`
- `n_quotes`, `n_verified`, `ratio`
- `cache_key` (12-char prefix)
- `preflight_hash` (12-char prefix; see #000009 §7.2)
- `model_profile_id`

**Diagnostics:**
- `format_collapsed` (pointer-mode only signal; #000008)
- `violation_kinds` (sorted list of unique kinds)
- `lazy_anchor_ratio`, `pointer_id_distribution`
- `answer_brackets`, `answer_pointer_count`,
  `answer_chars_with_brackets`, `raw_meaningful_line_count`
- `deflection_kind`, `subject_anchor`, `subject_in_answer`
- `metaphor_deflection_kind`, `metaphor_cue_count`,
  `metaphor_overlap_count`

**Preflight projection (#000010):**
- `preflight_logical_statuses` (list)
- `preflight_question_shape`, `preflight_result`
- `preflight_temporal_sensitivity`
- `preflight_has_false_premise`, `preflight_has_contradiction`
- `preflight_corpus_requirement`

**Quantifier classifier (#000008):**
- `quantifier_intensity`, `quantifier_matched_token`
- `scope_bound_hint`, `quantifier_explicit_count`
- `claim_cap_applied`

**Capacity:**
- `prompt_chars_total`, `prompt_chars_evidence`,
  `prompt_chars_system`, `prompt_chars_question`
- `answer_chars`

**Directive compliance** (per-row pass/fail for the seven-point
program directives observable from a single bench row):
- `directive_compliance` dict keyed on D2/D3/D4/D6/D7

**Time:**
- `elapsed_s`, `iso_ts`, `error`

The markdown summary alongside the JSONL renders aggregate views
(per-mode strict-rate, format-collapse + violation kinds table,
strict-rate-by-prompt-size buckets, directive coverage).

## 6. The journal — where headlines live

`docs/qa-modes-bench.md` is the **rolling research log** for the
QA bench. Each substantial bench cycle adds a new addendum (or
refines an existing one). The 2026-05-02 baseline is the frozen
authoritative state of the substrate before #000008/9/10/11
landed; subsequent addenda compare against it.

Current addenda:
- **2026-05-02 baseline** — pre-preflight authoritative state.
- **Addendum 1** (broad-quantifier A/B from #000008 §12.6).
- **Addendum 2** (preflight on vs off, 9-question broad subset).
- **Addendum 3** (full 75-question regression check post-flip).
- **Addendum 4** (metacog-trigger detector validation, 6 → 28
  question fixture).

When a new bench cycle produces a substantive finding, add an
addendum or refine an existing one. Bench artifacts (JSONL +
markdown) live in `bench/qa_results/` and are gitignored — only
headlines + cross-references go in the journal.

Per-ticket bench data lives in the relevant ticket file's `§12`
or `§13` (e.g. `docs/tickets/ticket-000008-...md` §12 carries the
four-cell A/B; #000010 §13 has three sub-validations; #000006 has
the rolling-amend pattern across multiple cycle counts).

## 7. The bench-emergent log

`bench/emergent_log.jsonl` is a long-running append-only log of
random-word stress cycles. Each entry has:

```json
{
  "ts": <unix>, "iso_ts": "2026-05-04T...",
  "words": ["dismally", "heats", "Jamaicans"],
  "question": "<model-generated question paragraph>",
  "answer": "<arborist's answer>",
  "audit_mode": "STRICT|HYBRID|UNGROUNDED",
  "verifier_method": "claim_lattice",
  "n_quotes": 3, "n_verified": 0,
  "violation_kinds": ["CITATION_MISMATCH", "TOO_MANY_EVIDENCE_IDS"],
  "metaphor_deflection_kind": "no_signal",
  "metaphor_cue_count": 2, "metaphor_overlap_count": 2,
  "sources": [{"title": "...", "uri": "...", "used": false}, ...],
  "answer_seconds": <float>, "total_seconds": <float>,
  "teacher": <reserved for human review>
}
```

Findings live in `docs/tickets/ticket-000006-bench-emergent-findings.md`
as rolling amends — each new cycle batch produces a new amend with
the verdict shift, violation profile, and tuning candidates.

## 8. How to run a focused A/B

Pattern from #000008 §12 (broad-quantifier four-cell A/B):

1. **Pick a fixture** matching the feature under test
   (broad-subset for quantifier work; metacog-subset for
   metacognition work; smoke for wiring sanity).
2. **Run the baseline** — same fixture, default policy.
3. **Run the cell** — same fixture, one policy knob flipped via
   `--policy KEY=VALUE`. Repeat for each independent knob.
4. **Compare** — STRICT-rate, mean-ratio, UNGROUNDED-rate,
   FORMAT_COLLAPSED rate, dominant violation kinds. Apply the
   5pp floor to STRICT-rate; mean-ratio and UNGROUNDED tend to
   be tighter.
5. **Verify with n=5** if a cell sits within the noise band.
6. **Document** — add an addendum to `docs/qa-modes-bench.md` or
   a new `§12.x` / `§13.x` to the relevant ticket file.

The four-cell A/B in #000008 (preflight off / reminder only / cap
only / cap+reminder) is the canonical pattern for measuring
multi-knob features.

## 9. How to interpret results

**STRICT-rate** = the fraction of cells where the verifier could
prove every claim grounded against cited evidence. Most direct
quality metric, but noisy at small sample sizes.

**HYBRID-rate** = mixed verdicts (some claims grounded, some
not). Often more informative than STRICT-rate when a feature's
job is to rescue UNGROUNDED → HYBRID rather than HYBRID → STRICT.

**UNGROUNDED-rate** = honest "the verifier could not ground."
Lower is generally better, but UNGROUNDED on questions the corpus
genuinely can't answer is the *correct* outcome — the substrate
preferring UNGROUNDED to fabricated STRICT is the architectural
win named in `docs/seven-point-program.md` D7.

**Mean ratio** = mean of `n_verified / n_quotes` across rows.
Aggregates per-row values rather than counting verdicts; tighter
at small sample sizes than STRICT-rate.

**FORMAT_COLLAPSED rate** = pointer-mode-specific gauge of
whether the model is following the `[E\d+]` citation protocol.
Zero is the goal; non-zero means the model emitted free prose
without pointer tags.

**Violation kind distribution** = per-mode tallies of which hard
checks fired most often. Gives the operator a per-failure-mode
view of the substrate's weaknesses.

**Directive compliance** = per-row pass/fail for the
seven-point-program directives (D2/D3/D4/D6/D7). Substrate-level
hygiene check; should be ~100% on lattice modes.

**Audit-line tails** (rendered by
`arborist/cli.py:_render_warrant_tail`):
- Hard tails: `· warrant missing`, `· title mismatch`,
  `· format collapsed`, `· broad cap N`, `· broad rejected`,
  `· broad unbounded`, `· broad runaway`, `· false premise`,
  `· contradictory`, `· stale risk`, `· out of corpus`,
  `· frame ambiguous`.
- Soft tails (advisory): `· soft: <label>` (#000011 sidecar
  output; SOFT_DISABLED / SOFT_PARSE_FAIL / SOFT_WELL_FORMED
  suppress).

A row with multiple tails composes them: `EVIDENCE-WARRANTED ·
via claim_lattice · false premise · soft: time sensitive` is a
verdict the substrate reached but with two distinct preflight
warnings worth the operator's attention.

## 10. Operator commands cheat sheet

```bash
# Inspect preflight stage payload from any cache row
arborist providence --show-preflight CACHE_KEY_PREFIX

# Single query with all preflight knobs flipped
arborist query "winners of all major sports?" \
    --apply-quantifier-caps --reject-broad --soft-preflight

# Dry-run — assemble context but skip LLM
arborist query "..." --dry-run

# Force fresh inference (delete prior cache row first)
arborist query "..." --burn

# Show raw JSON record instead of human render
arborist query "..." --json
```

The `Makefile` exposes most of these via shorthand:

```
make query Q="..." BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 \
    BURN=1 JSON=1 ANSWER_MODE=claim_lattice
```

See the `Makefile` directly for the full set.

## 11. Deterministic retrieval instruments (no-LLM)

The bench-maxing rule (CLAUDE.md): when a lever's failure class is
below the n=3 / 5pp noise floor, fix the instrument — mine
ground-truth-carrying questions and grade by deterministic retrieval,
not by the LLM verifier. These instruments take `query --dry-run`
output only: no LLM, no verifier, no judge, no noise floor, scalable
to the corpus. They measure *retrieval surfacing* —
necessary-not-sufficient for STRICT, and they **complement, never
replace**, the curated adversarial set (the verifier-honesty gate).

- **`bench/mine_questions.py`** — mines questions whose target article
  is known by construction (surface-variant of a corpus title).
  Classes: `numeral` (Roman↔ordinal), `accent`, `hyphen`,
  `honorific` (Mt↔Mount), `amp` (&↔and), `brit` (US↔UK spelling),
  `stale` (temporal fabrication-bait, control_ab only).
- **`bench/recall_at_k.py`** — recall@1/@3/@k of the known target on
  a mined fixture. Returns rank, so recall at every k is free from one
  retrieval (a too-lenient k hides a rank-only lift — report @1/@3/@k).
- **`bench/jaggedness.py`** (`make bench-jaggedness`, #000060) — does
  retrieval surface the SAME target for a question's *canonical*
  phrasing (title verbatim) AND its *surface-perturbed* phrasing? A
  non-jagged retriever agrees; a jagged one finds "Henry VIII" but
  misses "Henry the eighth". Reuses `recall_at_k.probe` +
  `mine_questions._surface_variant`.
  - `J_norm` (binary) = fraction of titles where canonical & perturbed
    DISAGREE on surfacing the target @k (XOR). Lower = more non-jagged.
  - graded mean `|Δrank|` over titles where both surfaced — rank
    instability the binary metric can't see.
  - Distinguishes a true recall-miss (both phrasings miss = not
    jagged) from jaggedness (one surfaces, one doesn't). Feeds
    #000012's ForkScore `ΔJaggednessReduction`: a retrieval fold that
    lowers `J_norm` without lowering recall is a non-jagged win.
  - The A-vs-C *answer-quality* jaggedness delta (does the substrate
    make answers more consistent across variants than the bare model?)
    is the LLM+judge version — rides `bench/control_sweep.py` over
    variant groups, gated v2.