docs: add docs/benchmarks.md — orientation doc for bench harnesses
Single canonical entry point that ties together the four existing
bench-related docs (qa-modes-bench.md / bench-maxing.md /
bench-emergent-design.md / qa-modes-bench-2026-04-30.md) plus the
make targets, fixtures, and bench-row schema.
Sections:
1. Two harnesses, two purposes
bench/qa_sweep.py — curated regression bench
scripts/bench_emergent.py — random-word stress test
2. Four question fixtures (75 / 28 / 9 / 1 / smoke) with cell
sizes + use cases per fixture
3. Signal floor (5pp / n=3 × 9 = 27 / vLLM c=3-4 saturation)
4. Make targets cheat sheet (bench-qa, bench-emergent,
--policy KEY=VALUE A/B pattern, --resume)
5. Bench-row schema — every field a row carries
(identity / verdict / diagnostics / preflight projection /
quantifier classifier / capacity / directive compliance / time)
6. Where headlines live (qa-modes-bench.md addenda, per-ticket
§12/§13 bench sections)
7. bench-emergent log shape + #000006 rolling-amend pattern
8. How to run a focused A/B (the four-cell pattern from #000008)
9. How to interpret results (STRICT-rate, mean-ratio,
UNGROUNDED-rate, FORMAT_COLLAPSED rate, violation kind
distribution, audit-line tails)
10. Operator commands cheat sheet (--show-preflight,
--apply-quantifier-caps, --reject-broad, --soft-preflight,
Makefile shortcuts)
CLAUDE.md docs index updated to point to benchmarks.md as the
"read first" entry for bench work, and to add bench-emergent-design.md
to the index (was missing).
This commit is contained in:
parent
453e340e08
commit
17d637e99f
2 changed files with 344 additions and 4 deletions
11
CLAUDE.md
11
CLAUDE.md
|
|
@ -350,12 +350,15 @@ Architecture / ongoing work:
|
|||
|
||||
- `docs/cti-architecture.md` — CTI Clause Tree Intelligence.
|
||||
- `docs/mesh.md`, `docs/mesh-deploy.md` — mesh wire + deploy.
|
||||
- `docs/qa-modes-bench.md` — latest bench (post-Rule-8 /
|
||||
post-warrant-generalization / post-frame-detector). Quote 0.50,
|
||||
pointer 0.23, JSON 0.44. Concurrency landed alongside.
|
||||
- `docs/benchmarks.md` — orientation: harnesses, fixtures,
|
||||
signal floor, make targets, bench-row schema, addenda index.
|
||||
Read first when running a bench.
|
||||
- `docs/qa-modes-bench.md` — bench journal (rolling addenda).
|
||||
Headlines + cross-references to per-ticket bench data.
|
||||
- `docs/qa-modes-bench-2026-04-30.md` — JSON-mode hardening journey
|
||||
(prior bench reference).
|
||||
- `docs/bench-maxing.md` — bench discipline.
|
||||
- `docs/bench-maxing.md` — bench discipline (5pp signal floor etc).
|
||||
- `docs/bench-emergent-design.md` — random-word stress-test design.
|
||||
- `docs/verifier-semantic-gap-design.md` — soft-signal NLI proposal.
|
||||
- `docs/self-reference-design.md`,
|
||||
`docs/self-reference-design.md` — recursive distillation.
|
||||
|
|
|
|||
337
docs/benchmarks.md
Normal file
337
docs/benchmarks.md
Normal file
|
|
@ -0,0 +1,337 @@
|
|||
# Benchmarks — orientation, harnesses, fixtures, discipline
|
||||
|
||||
Aborist ships two bench harnesses, four question fixtures, and a
|
||||
signal-floor discipline pinned across the substrate. This doc is
|
||||
the canonical entry point. Three companion docs cover specifics:
|
||||
|
||||
| File | Purpose |
|
||||
|----------------------------------------|--------------------------------------------|
|
||||
| `docs/bench-maxing.md` | Discipline rules (5pp signal floor, etc.) |
|
||||
| `docs/qa-modes-bench.md` | Historical journal — running addendum log |
|
||||
| `docs/bench-emergent-design.md` | Random-word stress-test design rationale |
|
||||
| `docs/qa-modes-bench-2026-04-30.md` | Frozen point-in-time bench (Apr-30 baseline) |
|
||||
|
||||
Read those when you need the why or the long-form data. This file
|
||||
covers the what, where, and how.
|
||||
|
||||
## 1. Two harnesses, two purposes
|
||||
|
||||
### 1.1 `bench/qa_sweep.py` — curated QA-quality bench
|
||||
|
||||
Runs a fixture file × answer modes × n samples. Persists a JSONL
|
||||
of per-row results plus a markdown summary. Used for:
|
||||
|
||||
- **Regression checks** before merging substrate changes.
|
||||
- **A/B cycles** on policy knobs (cap on/off, reminder on/off,
|
||||
preflight enabled/disabled).
|
||||
- **Per-mode strict-rate tracking** over time.
|
||||
|
||||
Run via `make bench-qa` or `bench/qa_sweep.py` directly. Bench
|
||||
rows carry ~25 fields per cell, including the four-rung label
|
||||
(`audit_mode`), the verifier-method, the violation kinds, the
|
||||
preflight QuestionState projection (`preflight_logical_statuses`,
|
||||
`preflight_question_shape`, etc.), the quantifier classifier
|
||||
output (`quantifier_intensity`, `scope_bound_hint`,
|
||||
`claim_cap_applied`), the preflight stage hash 12-char prefix
|
||||
(#000009 §7.2), and a per-stage timing breakdown
|
||||
(`preflight_ms`, `soft_preflight_ms`, `search_ms`, `context_ms`,
|
||||
`llm_ms`, `persist_ms`, `total_ms`).
|
||||
|
||||
### 1.2 `scripts/bench_emergent.py` — random-word stress test
|
||||
|
||||
Picks three random words from `/usr/share/dict/words`, asks
|
||||
Hermes (at temp 0.8) to weave them into a creative question
|
||||
paragraph, then sends that paragraph to aborist. Logs every cycle
|
||||
to `bench/emergent_log.jsonl` as one JSONL line. Designed for
|
||||
**blue-moon cadence**, not every-commit benching — the
|
||||
combinatoric word space surfaces failure shapes the curated bench
|
||||
doesn't reach.
|
||||
|
||||
The teacher review step is intentionally **not automated.** Fox
|
||||
brings interesting log entries to a teacher model (Claude Opus
|
||||
4.7 in the current setup) and asks for guidance: did the answer
|
||||
match the question? Novelty class
|
||||
(known_truth_grounding / emergent_synthesis / novel_claim /
|
||||
no_signal)? Which hyperparam to tune next?
|
||||
|
||||
Run via `make bench-emergent` (default `EMERGENT_N=10`) or
|
||||
`make bench-emergent EMERGENT_N=100` for a longer cycle. See
|
||||
`docs/bench-emergent-design.md` for the design rationale.
|
||||
|
||||
## 2. Four question fixtures
|
||||
|
||||
```
|
||||
bench/qa_questions.txt — 75 questions
|
||||
bench/qa_questions_smoke.txt — smoke test
|
||||
bench/qa_questions_quantifier_subset.txt — 9 questions
|
||||
bench/qa_questions_quantifier_baseline.txt — 1 question
|
||||
bench/qa_questions_metacog_subset.txt — 28 questions
|
||||
```
|
||||
|
||||
| Fixture | Use case | Cell size at n=3 × 3 modes |
|
||||
|--------------------------------------|---------------------------------------------------------------------------------------------|----------------------------|
|
||||
| `qa_questions.txt` | Full regression bench. Run before merging substrate changes. Wide coverage + corner cases. | 675 runs, ~3-3.5h |
|
||||
| `qa_questions_smoke.txt` | Smoke test on 1-2 questions. Use to confirm a CLI flag wires through correctly. | 3-6 runs, ~30s |
|
||||
| `qa_questions_quantifier_subset.txt` | 9 broad-quantifier questions (7 unbounded + 2 bounded). Use for A/B cycles on #000008 knobs. | 81 runs, ~7-15min |
|
||||
| `qa_questions_quantifier_baseline.txt` | Single broad question for narrow n=3 baseline measurement. | 9 runs, ~2-4min |
|
||||
| `qa_questions_metacog_subset.txt` | 28 questions targeting each #000010 detector (temporal, contradiction, false-premise, out-of-corpus) plus mixed multi-trigger + well-formed controls. | 252 runs, ~25-35min |
|
||||
|
||||
When choosing a fixture: ask whether you're measuring the
|
||||
**substrate as a whole** (full bench), a **specific feature**
|
||||
(broad-quantifier subset, metacog subset), or just **wiring
|
||||
correctness** (smoke).
|
||||
|
||||
## 3. The signal floor — n=3 × 9 = 27 sample minimum, 5pp delta floor
|
||||
|
||||
Per `docs/bench-maxing.md`:
|
||||
|
||||
- **5pp floor** for STRICT-rate deltas at n=3 × any-fixture sample
|
||||
size. Differences below this are Hermes nondeterminism, not
|
||||
signal. Quote-mode often jitters ±10pp on 27-sample cells.
|
||||
- **Mean-ratio + UNGROUNDED-rate** are tighter signals at the same
|
||||
sample size — they aggregate per-row values rather than
|
||||
counting discrete verdicts.
|
||||
- **n=5 verification** when an A/B cell sits within the noise
|
||||
band (e.g. 0.30 vs 0.33 STRICT-rate is 3pp — within noise; n=5
|
||||
on the closer cell tightens the variance).
|
||||
- **vLLM concurrency saturates at c=3-4.** Going higher slows
|
||||
per-call latency without parallelizing harder. Default `c=4`.
|
||||
|
||||
Don't tune to a single bench cycle. Always have a baseline +
|
||||
an A/B cell + (sometimes) a verification cell.
|
||||
|
||||
## 4. Make targets cheat sheet
|
||||
|
||||
```
|
||||
# Curated QA bench (default: bench/qa_questions.txt × 3 modes × n=3)
|
||||
make bench-qa
|
||||
|
||||
# Same but with limit / different file
|
||||
.venv/bin/python bench/qa_sweep.py \
|
||||
--questions bench/qa_questions_quantifier_subset.txt \
|
||||
--shards-dir ~/.aborist/shards --n 3 --concurrency 4 \
|
||||
--out-dir bench/qa_results
|
||||
|
||||
# A/B cycle: flip a policy knob for one bench cell
|
||||
.venv/bin/python bench/qa_sweep.py \
|
||||
--questions bench/qa_questions_quantifier_subset.txt \
|
||||
--shards-dir ~/.aborist/shards --n 3 --concurrency 4 \
|
||||
--policy quantifier_guard_apply_caps=true \
|
||||
--policy quantifier_reminder_enabled=true
|
||||
|
||||
# Resume an interrupted bench (same --seed, append to existing JSONL)
|
||||
.venv/bin/python bench/qa_sweep.py \
|
||||
--resume bench/qa_results/2026-05-03T19-30-13Z.jsonl \
|
||||
--seed 0
|
||||
|
||||
# Smoke test — quick sanity of the wiring
|
||||
.venv/bin/python bench/qa_sweep.py \
|
||||
--questions bench/qa_questions_smoke.txt --n 1 --concurrency 1
|
||||
|
||||
# Random-word stress test (blue-moon cadence)
|
||||
make bench-emergent # n=10 default
|
||||
make bench-emergent EMERGENT_N=100
|
||||
```
|
||||
|
||||
## 5. The bench-row schema
|
||||
|
||||
Every row in `bench/qa_results/<stamp>.jsonl` carries (current
|
||||
shape — fields are additive across versions):
|
||||
|
||||
**Identity & verdict:**
|
||||
- `question`, `answer_mode`, `audit_mode`, `verifier_method`
|
||||
- `n_quotes`, `n_verified`, `ratio`
|
||||
- `cache_key` (12-char prefix)
|
||||
- `preflight_hash` (12-char prefix; see #000009 §7.2)
|
||||
- `model_profile_id`
|
||||
|
||||
**Diagnostics:**
|
||||
- `format_collapsed` (pointer-mode only signal; #000008)
|
||||
- `violation_kinds` (sorted list of unique kinds)
|
||||
- `lazy_anchor_ratio`, `pointer_id_distribution`
|
||||
- `answer_brackets`, `answer_pointer_count`,
|
||||
`answer_chars_with_brackets`, `raw_meaningful_line_count`
|
||||
- `deflection_kind`, `subject_anchor`, `subject_in_answer`
|
||||
- `metaphor_deflection_kind`, `metaphor_cue_count`,
|
||||
`metaphor_overlap_count`
|
||||
|
||||
**Preflight projection (#000010):**
|
||||
- `preflight_logical_statuses` (list)
|
||||
- `preflight_question_shape`, `preflight_result`
|
||||
- `preflight_temporal_sensitivity`
|
||||
- `preflight_has_false_premise`, `preflight_has_contradiction`
|
||||
- `preflight_corpus_requirement`
|
||||
|
||||
**Quantifier classifier (#000008):**
|
||||
- `quantifier_intensity`, `quantifier_matched_token`
|
||||
- `scope_bound_hint`, `quantifier_explicit_count`
|
||||
- `claim_cap_applied`
|
||||
|
||||
**Capacity:**
|
||||
- `prompt_chars_total`, `prompt_chars_evidence`,
|
||||
`prompt_chars_system`, `prompt_chars_question`
|
||||
- `answer_chars`
|
||||
|
||||
**Directive compliance** (per-row pass/fail for the seven-point
|
||||
program directives observable from a single bench row):
|
||||
- `directive_compliance` dict keyed on D2/D3/D4/D6/D7
|
||||
|
||||
**Time:**
|
||||
- `elapsed_s`, `iso_ts`, `error`
|
||||
|
||||
The markdown summary alongside the JSONL renders aggregate views
|
||||
(per-mode strict-rate, format-collapse + violation kinds table,
|
||||
strict-rate-by-prompt-size buckets, directive coverage).
|
||||
|
||||
## 6. The journal — where headlines live
|
||||
|
||||
`docs/qa-modes-bench.md` is the **rolling research log** for the
|
||||
QA bench. Each substantial bench cycle adds a new addendum (or
|
||||
refines an existing one). The 2026-05-02 baseline is the frozen
|
||||
authoritative state of the substrate before #000008/9/10/11
|
||||
landed; subsequent addenda compare against it.
|
||||
|
||||
Current addenda:
|
||||
- **2026-05-02 baseline** — pre-preflight authoritative state.
|
||||
- **Addendum 1** (broad-quantifier A/B from #000008 §12.6).
|
||||
- **Addendum 2** (preflight on vs off, 9-question broad subset).
|
||||
- **Addendum 3** (full 75-question regression check post-flip).
|
||||
- **Addendum 4** (metacog-trigger detector validation, 6 → 28
|
||||
question fixture).
|
||||
|
||||
When a new bench cycle produces a substantive finding, add an
|
||||
addendum or refine an existing one. Bench artifacts (JSONL +
|
||||
markdown) live in `bench/qa_results/` and are gitignored — only
|
||||
headlines + cross-references go in the journal.
|
||||
|
||||
Per-ticket bench data lives in the relevant ticket file's `§12`
|
||||
or `§13` (e.g. `docs/tickets/ticket-000008-...md` §12 carries the
|
||||
four-cell A/B; #000010 §13 has three sub-validations; #000006 has
|
||||
the rolling-amend pattern across multiple cycle counts).
|
||||
|
||||
## 7. The bench-emergent log
|
||||
|
||||
`bench/emergent_log.jsonl` is a long-running append-only log of
|
||||
random-word stress cycles. Each entry has:
|
||||
|
||||
```json
|
||||
{
|
||||
"ts": <unix>, "iso_ts": "2026-05-04T...",
|
||||
"words": ["dismally", "heats", "Jamaicans"],
|
||||
"question": "<model-generated question paragraph>",
|
||||
"answer": "<aborist's answer>",
|
||||
"audit_mode": "STRICT|HYBRID|UNGROUNDED",
|
||||
"verifier_method": "claim_lattice",
|
||||
"n_quotes": 3, "n_verified": 0,
|
||||
"violation_kinds": ["CITATION_MISMATCH", "TOO_MANY_EVIDENCE_IDS"],
|
||||
"metaphor_deflection_kind": "no_signal",
|
||||
"metaphor_cue_count": 2, "metaphor_overlap_count": 2,
|
||||
"sources": [{"title": "...", "uri": "...", "used": false}, ...],
|
||||
"answer_seconds": <float>, "total_seconds": <float>,
|
||||
"teacher": <reserved for human review>
|
||||
}
|
||||
```
|
||||
|
||||
Findings live in `docs/tickets/ticket-000006-bench-emergent-findings.md`
|
||||
as rolling amends — each new cycle batch produces a new amend with
|
||||
the verdict shift, violation profile, and tuning candidates.
|
||||
|
||||
## 8. How to run a focused A/B
|
||||
|
||||
Pattern from #000008 §12 (broad-quantifier four-cell A/B):
|
||||
|
||||
1. **Pick a fixture** matching the feature under test
|
||||
(broad-subset for quantifier work; metacog-subset for
|
||||
metacognition work; smoke for wiring sanity).
|
||||
2. **Run the baseline** — same fixture, default policy.
|
||||
3. **Run the cell** — same fixture, one policy knob flipped via
|
||||
`--policy KEY=VALUE`. Repeat for each independent knob.
|
||||
4. **Compare** — STRICT-rate, mean-ratio, UNGROUNDED-rate,
|
||||
FORMAT_COLLAPSED rate, dominant violation kinds. Apply the
|
||||
5pp floor to STRICT-rate; mean-ratio and UNGROUNDED tend to
|
||||
be tighter.
|
||||
5. **Verify with n=5** if a cell sits within the noise band.
|
||||
6. **Document** — add an addendum to `docs/qa-modes-bench.md` or
|
||||
a new `§12.x` / `§13.x` to the relevant ticket file.
|
||||
|
||||
The four-cell A/B in #000008 (preflight off / reminder only / cap
|
||||
only / cap+reminder) is the canonical pattern for measuring
|
||||
multi-knob features.
|
||||
|
||||
## 9. How to interpret results
|
||||
|
||||
**STRICT-rate** = the fraction of cells where the verifier could
|
||||
prove every claim grounded against cited evidence. Most direct
|
||||
quality metric, but noisy at small sample sizes.
|
||||
|
||||
**HYBRID-rate** = mixed verdicts (some claims grounded, some
|
||||
not). Often more informative than STRICT-rate when a feature's
|
||||
job is to rescue UNGROUNDED → HYBRID rather than HYBRID → STRICT.
|
||||
|
||||
**UNGROUNDED-rate** = honest "the verifier could not ground."
|
||||
Lower is generally better, but UNGROUNDED on questions the corpus
|
||||
genuinely can't answer is the *correct* outcome — the substrate
|
||||
preferring UNGROUNDED to fabricated STRICT is the architectural
|
||||
win named in `docs/seven-point-program.md` D7.
|
||||
|
||||
**Mean ratio** = mean of `n_verified / n_quotes` across rows.
|
||||
Aggregates per-row values rather than counting verdicts; tighter
|
||||
at small sample sizes than STRICT-rate.
|
||||
|
||||
**FORMAT_COLLAPSED rate** = pointer-mode-specific gauge of
|
||||
whether the model is following the `[E\d+]` citation protocol.
|
||||
Zero is the goal; non-zero means the model emitted free prose
|
||||
without pointer tags.
|
||||
|
||||
**Violation kind distribution** = per-mode tallies of which hard
|
||||
checks fired most often. Gives the operator a per-failure-mode
|
||||
view of the substrate's weaknesses.
|
||||
|
||||
**Directive compliance** = per-row pass/fail for the
|
||||
seven-point-program directives (D2/D3/D4/D6/D7). Substrate-level
|
||||
hygiene check; should be ~100% on lattice modes.
|
||||
|
||||
**Audit-line tails** (rendered by
|
||||
`aborist/cli.py:_render_warrant_tail`):
|
||||
- Hard tails: `· warrant missing`, `· title mismatch`,
|
||||
`· format collapsed`, `· broad cap N`, `· broad rejected`,
|
||||
`· broad unbounded`, `· broad runaway`, `· false premise`,
|
||||
`· contradictory`, `· stale risk`, `· out of corpus`,
|
||||
`· frame ambiguous`.
|
||||
- Soft tails (advisory): `· soft: <label>` (#000011 sidecar
|
||||
output; SOFT_DISABLED / SOFT_PARSE_FAIL / SOFT_WELL_FORMED
|
||||
suppress).
|
||||
|
||||
A row with multiple tails composes them: `EVIDENCE-WARRANTED ·
|
||||
via claim_lattice · false premise · soft: time sensitive` is a
|
||||
verdict the substrate reached but with two distinct preflight
|
||||
warnings worth the operator's attention.
|
||||
|
||||
## 10. Operator commands cheat sheet
|
||||
|
||||
```bash
|
||||
# Inspect preflight stage payload from any cache row
|
||||
aborist providence --show-preflight CACHE_KEY_PREFIX
|
||||
|
||||
# Single query with all preflight knobs flipped
|
||||
aborist query "winners of all major sports?" \
|
||||
--apply-quantifier-caps --reject-broad --soft-preflight
|
||||
|
||||
# Dry-run — assemble context but skip LLM
|
||||
aborist query "..." --dry-run
|
||||
|
||||
# Force fresh inference (delete prior cache row first)
|
||||
aborist query "..." --burn
|
||||
|
||||
# Show raw JSON record instead of human render
|
||||
aborist query "..." --json
|
||||
```
|
||||
|
||||
The `Makefile` exposes most of these via shorthand:
|
||||
|
||||
```
|
||||
make query Q="..." BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 \
|
||||
BURN=1 JSON=1 ANSWER_MODE=claim_lattice
|
||||
```
|
||||
|
||||
See the `Makefile` directly for the full set.
|
||||
Loading…
Add table
Add a link
Reference in a new issue