arborist/docs/benchmarks.md

Benchmarks — orientation, harnesses, fixtures, discipline

Arborist ships two bench harnesses, four question fixtures, and a signal-floor discipline pinned across the substrate. This doc is the canonical entry point. Three companion docs cover specifics:

File	Purpose
docs/bench-maxing.md	Discipline rules (5pp signal floor, etc.)
docs/qa-modes-bench.md	Historical journal — running addendum log
docs/bench-emergent-design.md	Random-word stress-test design rationale
docs/qa-modes-bench-2026-04-30.md	Frozen point-in-time bench (Apr-30 baseline)

Read those when you need the why or the long-form data. This file covers the what, where, and how.

1. Two harnesses, two purposes

1.1 bench/qa_sweep.py — curated QA-quality bench

Runs a fixture file × answer modes × n samples. Persists a JSONL of per-row results plus a markdown summary. Used for:

• Regression checks before merging substrate changes.
• A/B cycles on policy knobs (cap on/off, reminder on/off, preflight enabled/disabled).
• Per-mode strict-rate tracking over time.

Run via make bench-qa or bench/qa_sweep.py directly. Bench rows carry ~25 fields per cell, including the four-rung label (audit_mode), the verifier-method, the violation kinds, the preflight QuestionState projection (preflight_logical_statuses, preflight_question_shape, etc.), the quantifier classifier output (quantifier_intensity, scope_bound_hint, claim_cap_applied), the preflight stage hash 12-char prefix (#000009 §7.2), and a per-stage timing breakdown (preflight_ms, soft_preflight_ms, search_ms, context_ms, llm_ms, persist_ms, total_ms).

1.2 scripts/bench_emergent.py — random-word stress test

Picks three random words from /usr/share/dict/words, asks Hermes (at temp 0.8) to weave them into a creative question paragraph, then sends that paragraph to arborist. Logs every cycle to bench/emergent_log.jsonl as one JSONL line. Designed for blue-moon cadence, not every-commit benching — the combinatoric word space surfaces failure shapes the curated bench doesn't reach.

The teacher review step is intentionally not automated. Fox brings interesting log entries to a teacher model (Claude Opus 4.7 in the current setup) and asks for guidance: did the answer match the question? Novelty class (known_truth_grounding / emergent_synthesis / novel_claim / no_signal)? Which hyperparam to tune next?

Run via make bench-emergent (default EMERGENT_N=10) or make bench-emergent EMERGENT_N=100 for a longer cycle. See docs/bench-emergent-design.md for the design rationale.

2. Four question fixtures

bench/qa_questions.txt                                — 75 questions
bench/qa_questions_smoke.txt                          —  smoke test
bench/qa_questions_quantifier_subset.txt              —  9 questions
bench/qa_questions_quantifier_baseline.txt            —  1 question
bench/qa_questions_metacog_subset.txt                 — 28 questions

Fixture	Use case	Cell size at n=3 × 3 modes
qa_questions.txt	Full regression bench. Run before merging substrate changes. Wide coverage + corner cases.	675 runs, ~3-3.5h
qa_questions_smoke.txt	Smoke test on 1-2 questions. Use to confirm a CLI flag wires through correctly.	3-6 runs, ~30s
qa_questions_quantifier_subset.txt	9 broad-quantifier questions (7 unbounded + 2 bounded). Use for A/B cycles on #000008 knobs.	81 runs, ~7-15min
qa_questions_quantifier_baseline.txt	Single broad question for narrow n=3 baseline measurement.	9 runs, ~2-4min
qa_questions_metacog_subset.txt	28 questions targeting each #000010 detector (temporal, contradiction, false-premise, out-of-corpus) plus mixed multi-trigger + well-formed controls.	252 runs, ~25-35min

When choosing a fixture: ask whether you're measuring the substrate as a whole (full bench), a specific feature (broad-quantifier subset, metacog subset), or just wiring correctness (smoke).

3. The signal floor — n=3 × 9 = 27 sample minimum, 5pp delta floor

Per docs/bench-maxing.md:

• 5pp floor for STRICT-rate deltas at n=3 × any-fixture sample size. Differences below this are Hermes nondeterminism, not signal. Quote-mode often jitters ±10pp on 27-sample cells.
• Mean-ratio + UNGROUNDED-rate are tighter signals at the same sample size — they aggregate per-row values rather than counting discrete verdicts.
• n=5 verification when an A/B cell sits within the noise band (e.g. 0.30 vs 0.33 STRICT-rate is 3pp — within noise; n=5 on the closer cell tightens the variance).
• vLLM concurrency saturates at c=3-4. Going higher slows per-call latency without parallelizing harder. Default c=4.

Don't tune to a single bench cycle. Always have a baseline + an A/B cell + (sometimes) a verification cell.

4. Make targets cheat sheet

# Curated QA bench (default: bench/qa_questions.txt × 3 modes × n=3)
make bench-qa

# Same but with limit / different file
.venv/bin/python bench/qa_sweep.py \
    --questions bench/qa_questions_quantifier_subset.txt \
    --shards-dir ~/.arborist/shards --n 3 --concurrency 4 \
    --out-dir bench/qa_results

# A/B cycle: flip a policy knob for one bench cell
.venv/bin/python bench/qa_sweep.py \
    --questions bench/qa_questions_quantifier_subset.txt \
    --shards-dir ~/.arborist/shards --n 3 --concurrency 4 \
    --policy quantifier_guard_apply_caps=true \
    --policy quantifier_reminder_enabled=true

# Resume an interrupted bench (same --seed, append to existing JSONL)
.venv/bin/python bench/qa_sweep.py \
    --resume bench/qa_results/2026-05-03T19-30-13Z.jsonl \
    --seed 0

# Smoke test — quick sanity of the wiring
.venv/bin/python bench/qa_sweep.py \
    --questions bench/qa_questions_smoke.txt --n 1 --concurrency 1

# Random-word stress test (blue-moon cadence)
make bench-emergent              # n=10 default
make bench-emergent EMERGENT_N=100

5. The bench-row schema

Every row in bench/qa_results/<stamp>.jsonl carries (current shape — fields are additive across versions):

Identity & verdict:

• question, answer_mode, audit_mode, verifier_method
• n_quotes, n_verified, ratio
• cache_key (12-char prefix)
• preflight_hash (12-char prefix; see #000009 §7.2)
• model_profile_id

Diagnostics:

• format_collapsed (pointer-mode only signal; #000008)
• violation_kinds (sorted list of unique kinds)
• lazy_anchor_ratio, pointer_id_distribution
• answer_brackets, answer_pointer_count, answer_chars_with_brackets, raw_meaningful_line_count
• deflection_kind, subject_anchor, subject_in_answer
• metaphor_deflection_kind, metaphor_cue_count, metaphor_overlap_count

Preflight projection (#000010):

• preflight_logical_statuses (list)
• preflight_question_shape, preflight_result
• preflight_temporal_sensitivity
• preflight_has_false_premise, preflight_has_contradiction
• preflight_corpus_requirement

Quantifier classifier (#000008):

• quantifier_intensity, quantifier_matched_token
• scope_bound_hint, quantifier_explicit_count
• claim_cap_applied

Capacity:

• prompt_chars_total, prompt_chars_evidence, prompt_chars_system, prompt_chars_question
• answer_chars

Directive compliance (per-row pass/fail for the seven-point program directives observable from a single bench row):

• directive_compliance dict keyed on D2/D3/D4/D6/D7

Time:

• elapsed_s, iso_ts, error

The markdown summary alongside the JSONL renders aggregate views (per-mode strict-rate, format-collapse + violation kinds table, strict-rate-by-prompt-size buckets, directive coverage).

6. The journal — where headlines live

docs/qa-modes-bench.md is the rolling research log for the QA bench. Each substantial bench cycle adds a new addendum (or refines an existing one). The 2026-05-02 baseline is the frozen authoritative state of the substrate before #000008/9/10/11 landed; subsequent addenda compare against it.

Current addenda:

• 2026-05-02 baseline — pre-preflight authoritative state.
• Addendum 1 (broad-quantifier A/B from #000008 §12.6).
• Addendum 2 (preflight on vs off, 9-question broad subset).
• Addendum 3 (full 75-question regression check post-flip).
• Addendum 4 (metacog-trigger detector validation, 6 → 28 question fixture).

When a new bench cycle produces a substantive finding, add an addendum or refine an existing one. Bench artifacts (JSONL + markdown) live in bench/qa_results/ and are gitignored — only headlines + cross-references go in the journal.

Per-ticket bench data lives in the relevant ticket file's §12 or §13 (e.g. docs/tickets/ticket-000008-...md §12 carries the four-cell A/B; #000010 §13 has three sub-validations; #000006 has the rolling-amend pattern across multiple cycle counts).

7. The bench-emergent log

bench/emergent_log.jsonl is a long-running append-only log of random-word stress cycles. Each entry has:

{
  "ts": <unix>, "iso_ts": "2026-05-04T...",
  "words": ["dismally", "heats", "Jamaicans"],
  "question": "<model-generated question paragraph>",
  "answer": "<arborist's answer>",
  "audit_mode": "STRICT|HYBRID|UNGROUNDED",
  "verifier_method": "claim_lattice",
  "n_quotes": 3, "n_verified": 0,
  "violation_kinds": ["CITATION_MISMATCH", "TOO_MANY_EVIDENCE_IDS"],
  "metaphor_deflection_kind": "no_signal",
  "metaphor_cue_count": 2, "metaphor_overlap_count": 2,
  "sources": [{"title": "...", "uri": "...", "used": false}, ...],
  "answer_seconds": <float>, "total_seconds": <float>,
  "teacher": <reserved for human review>
}

Findings live in docs/tickets/ticket-000006-bench-emergent-findings.md as rolling amends — each new cycle batch produces a new amend with the verdict shift, violation profile, and tuning candidates.

8. How to run a focused A/B

Pattern from #000008 §12 (broad-quantifier four-cell A/B):

1. Pick a fixture matching the feature under test (broad-subset for quantifier work; metacog-subset for metacognition work; smoke for wiring sanity).
2. Run the baseline — same fixture, default policy.
3. Run the cell — same fixture, one policy knob flipped via --policy KEY=VALUE. Repeat for each independent knob.
4. Compare — STRICT-rate, mean-ratio, UNGROUNDED-rate, FORMAT_COLLAPSED rate, dominant violation kinds. Apply the 5pp floor to STRICT-rate; mean-ratio and UNGROUNDED tend to be tighter.
5. Verify with n=5 if a cell sits within the noise band.
6. Document — add an addendum to docs/qa-modes-bench.md or a new §12.x / §13.x to the relevant ticket file.

The four-cell A/B in #000008 (preflight off / reminder only / cap only / cap+reminder) is the canonical pattern for measuring multi-knob features.

9. How to interpret results

STRICT-rate = the fraction of cells where the verifier could prove every claim grounded against cited evidence. Most direct quality metric, but noisy at small sample sizes.

HYBRID-rate = mixed verdicts (some claims grounded, some not). Often more informative than STRICT-rate when a feature's job is to rescue UNGROUNDED → HYBRID rather than HYBRID → STRICT.

UNGROUNDED-rate = honest "the verifier could not ground." Lower is generally better, but UNGROUNDED on questions the corpus genuinely can't answer is the correct outcome — the substrate preferring UNGROUNDED to fabricated STRICT is the architectural win named in docs/seven-point-program.md D7.

Mean ratio = mean of n_verified / n_quotes across rows. Aggregates per-row values rather than counting verdicts; tighter at small sample sizes than STRICT-rate.

FORMAT_COLLAPSED rate = pointer-mode-specific gauge of whether the model is following the [E\d+] citation protocol. Zero is the goal; non-zero means the model emitted free prose without pointer tags.

Violation kind distribution = per-mode tallies of which hard checks fired most often. Gives the operator a per-failure-mode view of the substrate's weaknesses.

Directive compliance = per-row pass/fail for the seven-point-program directives (D2/D3/D4/D6/D7). Substrate-level hygiene check; should be ~100% on lattice modes.

Audit-line tails (rendered by arborist/cli.py:_render_warrant_tail):

• Hard tails: · warrant missing, · title mismatch, · format collapsed, · broad cap N, · broad rejected, · broad unbounded, · broad runaway, · false premise, · contradictory, · stale risk, · out of corpus, · frame ambiguous.
• Soft tails (advisory): · soft: <label> (#000011 sidecar output; SOFT_DISABLED / SOFT_PARSE_FAIL / SOFT_WELL_FORMED suppress).

A row with multiple tails composes them: EVIDENCE-WARRANTED · via claim_lattice · false premise · soft: time sensitive is a verdict the substrate reached but with two distinct preflight warnings worth the operator's attention.

10. Operator commands cheat sheet

# Inspect preflight stage payload from any cache row
arborist providence --show-preflight CACHE_KEY_PREFIX

# Single query with all preflight knobs flipped
arborist query "winners of all major sports?" \
    --apply-quantifier-caps --reject-broad --soft-preflight

# Dry-run — assemble context but skip LLM
arborist query "..." --dry-run

# Force fresh inference (delete prior cache row first)
arborist query "..." --burn

# Show raw JSON record instead of human render
arborist query "..." --json

The Makefile exposes most of these via shorthand:

make query Q="..." BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 \
    BURN=1 JSON=1 ANSWER_MODE=claim_lattice

See the Makefile directly for the full set.

11. Deterministic retrieval instruments (no-LLM)

The bench-maxing rule (CLAUDE.md): when a lever's failure class is below the n=3 / 5pp noise floor, fix the instrument — mine ground-truth-carrying questions and grade by deterministic retrieval, not by the LLM verifier. These instruments take query --dry-run output only: no LLM, no verifier, no judge, no noise floor, scalable to the corpus. They measure retrieval surfacing — necessary-not-sufficient for STRICT, and they complement, never replace, the curated adversarial set (the verifier-honesty gate).

• bench/mine_questions.py — mines questions whose target article is known by construction (surface-variant of a corpus title). Classes: numeral (Roman↔ordinal), accent, hyphen, honorific (Mt↔Mount), amp (&↔and), brit (US↔UK spelling), stale (temporal fabrication-bait, control_ab only).
• bench/recall_at_k.py — recall@1/@3/@k of the known target on a mined fixture. Returns rank, so recall at every k is free from one retrieval (a too-lenient k hides a rank-only lift — report @1/@3/@k).
• bench/jaggedness.py (make bench-jaggedness, #000060) — does retrieval surface the SAME target for a question's canonical phrasing (title verbatim) AND its surface-perturbed phrasing? A non-jagged retriever agrees; a jagged one finds "Henry VIII" but misses "Henry the eighth". Reuses recall_at_k.probe + mine_questions._surface_variant.
  • J_norm (binary) = fraction of titles where canonical & perturbed DISAGREE on surfacing the target @k (XOR). Lower = more non-jagged.
  • graded mean |Δrank| over titles where both surfaced — rank instability the binary metric can't see.
  • Distinguishes a true recall-miss (both phrasings miss = not jagged) from jaggedness (one surfaces, one doesn't). Feeds #000012's ForkScore ΔJaggednessReduction: a retrieval fold that lowers J_norm without lowering recall is a non-jagged win.
  • The A-vs-C answer-quality jaggedness delta (does the substrate make answers more consistent across variants than the bare model?) is the LLM+judge version — rides bench/control_sweep.py over variant groups, gated v2.