From 17d637e99f98a1d0c772334d311ad5b0e0fe88b7 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Mon, 4 May 2026 07:05:46 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20add=20docs/benchmarks.md=20=E2=80=94=20?= =?UTF-8?q?orientation=20doc=20for=20bench=20harnesses?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Single canonical entry point that ties together the four existing bench-related docs (qa-modes-bench.md / bench-maxing.md / bench-emergent-design.md / qa-modes-bench-2026-04-30.md) plus the make targets, fixtures, and bench-row schema. Sections: 1. Two harnesses, two purposes bench/qa_sweep.py — curated regression bench scripts/bench_emergent.py — random-word stress test 2. Four question fixtures (75 / 28 / 9 / 1 / smoke) with cell sizes + use cases per fixture 3. Signal floor (5pp / n=3 × 9 = 27 / vLLM c=3-4 saturation) 4. Make targets cheat sheet (bench-qa, bench-emergent, --policy KEY=VALUE A/B pattern, --resume) 5. Bench-row schema — every field a row carries (identity / verdict / diagnostics / preflight projection / quantifier classifier / capacity / directive compliance / time) 6. Where headlines live (qa-modes-bench.md addenda, per-ticket §12/§13 bench sections) 7. bench-emergent log shape + #000006 rolling-amend pattern 8. How to run a focused A/B (the four-cell pattern from #000008) 9. How to interpret results (STRICT-rate, mean-ratio, UNGROUNDED-rate, FORMAT_COLLAPSED rate, violation kind distribution, audit-line tails) 10. Operator commands cheat sheet (--show-preflight, --apply-quantifier-caps, --reject-broad, --soft-preflight, Makefile shortcuts) CLAUDE.md docs index updated to point to benchmarks.md as the "read first" entry for bench work, and to add bench-emergent-design.md to the index (was missing). --- CLAUDE.md | 11 +- docs/benchmarks.md | 337 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 344 insertions(+), 4 deletions(-) create mode 100644 docs/benchmarks.md diff --git a/CLAUDE.md b/CLAUDE.md index 4396a22..0f2ff5b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -350,12 +350,15 @@ Architecture / ongoing work: - `docs/cti-architecture.md` — CTI Clause Tree Intelligence. - `docs/mesh.md`, `docs/mesh-deploy.md` — mesh wire + deploy. -- `docs/qa-modes-bench.md` — latest bench (post-Rule-8 / - post-warrant-generalization / post-frame-detector). Quote 0.50, - pointer 0.23, JSON 0.44. Concurrency landed alongside. +- `docs/benchmarks.md` — orientation: harnesses, fixtures, + signal floor, make targets, bench-row schema, addenda index. + Read first when running a bench. +- `docs/qa-modes-bench.md` — bench journal (rolling addenda). + Headlines + cross-references to per-ticket bench data. - `docs/qa-modes-bench-2026-04-30.md` — JSON-mode hardening journey (prior bench reference). -- `docs/bench-maxing.md` — bench discipline. +- `docs/bench-maxing.md` — bench discipline (5pp signal floor etc). +- `docs/bench-emergent-design.md` — random-word stress-test design. - `docs/verifier-semantic-gap-design.md` — soft-signal NLI proposal. - `docs/self-reference-design.md`, `docs/self-reference-design.md` — recursive distillation. diff --git a/docs/benchmarks.md b/docs/benchmarks.md new file mode 100644 index 0000000..7986dfd --- /dev/null +++ b/docs/benchmarks.md @@ -0,0 +1,337 @@ +# Benchmarks — orientation, harnesses, fixtures, discipline + +Aborist ships two bench harnesses, four question fixtures, and a +signal-floor discipline pinned across the substrate. This doc is +the canonical entry point. Three companion docs cover specifics: + +| File | Purpose | +|----------------------------------------|--------------------------------------------| +| `docs/bench-maxing.md` | Discipline rules (5pp signal floor, etc.) | +| `docs/qa-modes-bench.md` | Historical journal — running addendum log | +| `docs/bench-emergent-design.md` | Random-word stress-test design rationale | +| `docs/qa-modes-bench-2026-04-30.md` | Frozen point-in-time bench (Apr-30 baseline) | + +Read those when you need the why or the long-form data. This file +covers the what, where, and how. + +## 1. Two harnesses, two purposes + +### 1.1 `bench/qa_sweep.py` — curated QA-quality bench + +Runs a fixture file × answer modes × n samples. Persists a JSONL +of per-row results plus a markdown summary. Used for: + +- **Regression checks** before merging substrate changes. +- **A/B cycles** on policy knobs (cap on/off, reminder on/off, + preflight enabled/disabled). +- **Per-mode strict-rate tracking** over time. + +Run via `make bench-qa` or `bench/qa_sweep.py` directly. Bench +rows carry ~25 fields per cell, including the four-rung label +(`audit_mode`), the verifier-method, the violation kinds, the +preflight QuestionState projection (`preflight_logical_statuses`, +`preflight_question_shape`, etc.), the quantifier classifier +output (`quantifier_intensity`, `scope_bound_hint`, +`claim_cap_applied`), the preflight stage hash 12-char prefix +(#000009 §7.2), and a per-stage timing breakdown +(`preflight_ms`, `soft_preflight_ms`, `search_ms`, `context_ms`, +`llm_ms`, `persist_ms`, `total_ms`). + +### 1.2 `scripts/bench_emergent.py` — random-word stress test + +Picks three random words from `/usr/share/dict/words`, asks +Hermes (at temp 0.8) to weave them into a creative question +paragraph, then sends that paragraph to aborist. Logs every cycle +to `bench/emergent_log.jsonl` as one JSONL line. Designed for +**blue-moon cadence**, not every-commit benching — the +combinatoric word space surfaces failure shapes the curated bench +doesn't reach. + +The teacher review step is intentionally **not automated.** Fox +brings interesting log entries to a teacher model (Claude Opus +4.7 in the current setup) and asks for guidance: did the answer +match the question? Novelty class +(known_truth_grounding / emergent_synthesis / novel_claim / +no_signal)? Which hyperparam to tune next? + +Run via `make bench-emergent` (default `EMERGENT_N=10`) or +`make bench-emergent EMERGENT_N=100` for a longer cycle. See +`docs/bench-emergent-design.md` for the design rationale. + +## 2. Four question fixtures + +``` +bench/qa_questions.txt — 75 questions +bench/qa_questions_smoke.txt — smoke test +bench/qa_questions_quantifier_subset.txt — 9 questions +bench/qa_questions_quantifier_baseline.txt — 1 question +bench/qa_questions_metacog_subset.txt — 28 questions +``` + +| Fixture | Use case | Cell size at n=3 × 3 modes | +|--------------------------------------|---------------------------------------------------------------------------------------------|----------------------------| +| `qa_questions.txt` | Full regression bench. Run before merging substrate changes. Wide coverage + corner cases. | 675 runs, ~3-3.5h | +| `qa_questions_smoke.txt` | Smoke test on 1-2 questions. Use to confirm a CLI flag wires through correctly. | 3-6 runs, ~30s | +| `qa_questions_quantifier_subset.txt` | 9 broad-quantifier questions (7 unbounded + 2 bounded). Use for A/B cycles on #000008 knobs. | 81 runs, ~7-15min | +| `qa_questions_quantifier_baseline.txt` | Single broad question for narrow n=3 baseline measurement. | 9 runs, ~2-4min | +| `qa_questions_metacog_subset.txt` | 28 questions targeting each #000010 detector (temporal, contradiction, false-premise, out-of-corpus) plus mixed multi-trigger + well-formed controls. | 252 runs, ~25-35min | + +When choosing a fixture: ask whether you're measuring the +**substrate as a whole** (full bench), a **specific feature** +(broad-quantifier subset, metacog subset), or just **wiring +correctness** (smoke). + +## 3. The signal floor — n=3 × 9 = 27 sample minimum, 5pp delta floor + +Per `docs/bench-maxing.md`: + +- **5pp floor** for STRICT-rate deltas at n=3 × any-fixture sample + size. Differences below this are Hermes nondeterminism, not + signal. Quote-mode often jitters ±10pp on 27-sample cells. +- **Mean-ratio + UNGROUNDED-rate** are tighter signals at the same + sample size — they aggregate per-row values rather than + counting discrete verdicts. +- **n=5 verification** when an A/B cell sits within the noise + band (e.g. 0.30 vs 0.33 STRICT-rate is 3pp — within noise; n=5 + on the closer cell tightens the variance). +- **vLLM concurrency saturates at c=3-4.** Going higher slows + per-call latency without parallelizing harder. Default `c=4`. + +Don't tune to a single bench cycle. Always have a baseline + +an A/B cell + (sometimes) a verification cell. + +## 4. Make targets cheat sheet + +``` +# Curated QA bench (default: bench/qa_questions.txt × 3 modes × n=3) +make bench-qa + +# Same but with limit / different file +.venv/bin/python bench/qa_sweep.py \ + --questions bench/qa_questions_quantifier_subset.txt \ + --shards-dir ~/.aborist/shards --n 3 --concurrency 4 \ + --out-dir bench/qa_results + +# A/B cycle: flip a policy knob for one bench cell +.venv/bin/python bench/qa_sweep.py \ + --questions bench/qa_questions_quantifier_subset.txt \ + --shards-dir ~/.aborist/shards --n 3 --concurrency 4 \ + --policy quantifier_guard_apply_caps=true \ + --policy quantifier_reminder_enabled=true + +# Resume an interrupted bench (same --seed, append to existing JSONL) +.venv/bin/python bench/qa_sweep.py \ + --resume bench/qa_results/2026-05-03T19-30-13Z.jsonl \ + --seed 0 + +# Smoke test — quick sanity of the wiring +.venv/bin/python bench/qa_sweep.py \ + --questions bench/qa_questions_smoke.txt --n 1 --concurrency 1 + +# Random-word stress test (blue-moon cadence) +make bench-emergent # n=10 default +make bench-emergent EMERGENT_N=100 +``` + +## 5. The bench-row schema + +Every row in `bench/qa_results/.jsonl` carries (current +shape — fields are additive across versions): + +**Identity & verdict:** +- `question`, `answer_mode`, `audit_mode`, `verifier_method` +- `n_quotes`, `n_verified`, `ratio` +- `cache_key` (12-char prefix) +- `preflight_hash` (12-char prefix; see #000009 §7.2) +- `model_profile_id` + +**Diagnostics:** +- `format_collapsed` (pointer-mode only signal; #000008) +- `violation_kinds` (sorted list of unique kinds) +- `lazy_anchor_ratio`, `pointer_id_distribution` +- `answer_brackets`, `answer_pointer_count`, + `answer_chars_with_brackets`, `raw_meaningful_line_count` +- `deflection_kind`, `subject_anchor`, `subject_in_answer` +- `metaphor_deflection_kind`, `metaphor_cue_count`, + `metaphor_overlap_count` + +**Preflight projection (#000010):** +- `preflight_logical_statuses` (list) +- `preflight_question_shape`, `preflight_result` +- `preflight_temporal_sensitivity` +- `preflight_has_false_premise`, `preflight_has_contradiction` +- `preflight_corpus_requirement` + +**Quantifier classifier (#000008):** +- `quantifier_intensity`, `quantifier_matched_token` +- `scope_bound_hint`, `quantifier_explicit_count` +- `claim_cap_applied` + +**Capacity:** +- `prompt_chars_total`, `prompt_chars_evidence`, + `prompt_chars_system`, `prompt_chars_question` +- `answer_chars` + +**Directive compliance** (per-row pass/fail for the seven-point +program directives observable from a single bench row): +- `directive_compliance` dict keyed on D2/D3/D4/D6/D7 + +**Time:** +- `elapsed_s`, `iso_ts`, `error` + +The markdown summary alongside the JSONL renders aggregate views +(per-mode strict-rate, format-collapse + violation kinds table, +strict-rate-by-prompt-size buckets, directive coverage). + +## 6. The journal — where headlines live + +`docs/qa-modes-bench.md` is the **rolling research log** for the +QA bench. Each substantial bench cycle adds a new addendum (or +refines an existing one). The 2026-05-02 baseline is the frozen +authoritative state of the substrate before #000008/9/10/11 +landed; subsequent addenda compare against it. + +Current addenda: +- **2026-05-02 baseline** — pre-preflight authoritative state. +- **Addendum 1** (broad-quantifier A/B from #000008 §12.6). +- **Addendum 2** (preflight on vs off, 9-question broad subset). +- **Addendum 3** (full 75-question regression check post-flip). +- **Addendum 4** (metacog-trigger detector validation, 6 → 28 + question fixture). + +When a new bench cycle produces a substantive finding, add an +addendum or refine an existing one. Bench artifacts (JSONL + +markdown) live in `bench/qa_results/` and are gitignored — only +headlines + cross-references go in the journal. + +Per-ticket bench data lives in the relevant ticket file's `§12` +or `§13` (e.g. `docs/tickets/ticket-000008-...md` §12 carries the +four-cell A/B; #000010 §13 has three sub-validations; #000006 has +the rolling-amend pattern across multiple cycle counts). + +## 7. The bench-emergent log + +`bench/emergent_log.jsonl` is a long-running append-only log of +random-word stress cycles. Each entry has: + +```json +{ + "ts": , "iso_ts": "2026-05-04T...", + "words": ["dismally", "heats", "Jamaicans"], + "question": "", + "answer": "", + "audit_mode": "STRICT|HYBRID|UNGROUNDED", + "verifier_method": "claim_lattice", + "n_quotes": 3, "n_verified": 0, + "violation_kinds": ["CITATION_MISMATCH", "TOO_MANY_EVIDENCE_IDS"], + "metaphor_deflection_kind": "no_signal", + "metaphor_cue_count": 2, "metaphor_overlap_count": 2, + "sources": [{"title": "...", "uri": "...", "used": false}, ...], + "answer_seconds": , "total_seconds": , + "teacher": +} +``` + +Findings live in `docs/tickets/ticket-000006-bench-emergent-findings.md` +as rolling amends — each new cycle batch produces a new amend with +the verdict shift, violation profile, and tuning candidates. + +## 8. How to run a focused A/B + +Pattern from #000008 §12 (broad-quantifier four-cell A/B): + +1. **Pick a fixture** matching the feature under test + (broad-subset for quantifier work; metacog-subset for + metacognition work; smoke for wiring sanity). +2. **Run the baseline** — same fixture, default policy. +3. **Run the cell** — same fixture, one policy knob flipped via + `--policy KEY=VALUE`. Repeat for each independent knob. +4. **Compare** — STRICT-rate, mean-ratio, UNGROUNDED-rate, + FORMAT_COLLAPSED rate, dominant violation kinds. Apply the + 5pp floor to STRICT-rate; mean-ratio and UNGROUNDED tend to + be tighter. +5. **Verify with n=5** if a cell sits within the noise band. +6. **Document** — add an addendum to `docs/qa-modes-bench.md` or + a new `§12.x` / `§13.x` to the relevant ticket file. + +The four-cell A/B in #000008 (preflight off / reminder only / cap +only / cap+reminder) is the canonical pattern for measuring +multi-knob features. + +## 9. How to interpret results + +**STRICT-rate** = the fraction of cells where the verifier could +prove every claim grounded against cited evidence. Most direct +quality metric, but noisy at small sample sizes. + +**HYBRID-rate** = mixed verdicts (some claims grounded, some +not). Often more informative than STRICT-rate when a feature's +job is to rescue UNGROUNDED → HYBRID rather than HYBRID → STRICT. + +**UNGROUNDED-rate** = honest "the verifier could not ground." +Lower is generally better, but UNGROUNDED on questions the corpus +genuinely can't answer is the *correct* outcome — the substrate +preferring UNGROUNDED to fabricated STRICT is the architectural +win named in `docs/seven-point-program.md` D7. + +**Mean ratio** = mean of `n_verified / n_quotes` across rows. +Aggregates per-row values rather than counting verdicts; tighter +at small sample sizes than STRICT-rate. + +**FORMAT_COLLAPSED rate** = pointer-mode-specific gauge of +whether the model is following the `[E\d+]` citation protocol. +Zero is the goal; non-zero means the model emitted free prose +without pointer tags. + +**Violation kind distribution** = per-mode tallies of which hard +checks fired most often. Gives the operator a per-failure-mode +view of the substrate's weaknesses. + +**Directive compliance** = per-row pass/fail for the +seven-point-program directives (D2/D3/D4/D6/D7). Substrate-level +hygiene check; should be ~100% on lattice modes. + +**Audit-line tails** (rendered by +`aborist/cli.py:_render_warrant_tail`): +- Hard tails: `· warrant missing`, `· title mismatch`, + `· format collapsed`, `· broad cap N`, `· broad rejected`, + `· broad unbounded`, `· broad runaway`, `· false premise`, + `· contradictory`, `· stale risk`, `· out of corpus`, + `· frame ambiguous`. +- Soft tails (advisory): `· soft: