bench: 2026-05-02 journal + CLAUDE.md headline refresh
Captures the post-Rule-8 / post-warrant-generalization / post-frame-detector bench taken with --concurrency 4 (~51 min wall-clock on 426 runs). Headlines (vs 2026-04-30 post-retry rerun): quote 0.47 → 0.50 (+3pp) claim_lattice_pointer 0.24 → 0.23 (−1pp) claim_lattice (JSON) 0.50 → 0.44 (−6pp) Pointer mode is essentially flat. The earlier 'pointer regressed 13pp' alarm was wrong — that compared to the pre-retry 2026-04-30 numbers (0.36) instead of the post-retry rerun (0.24) which is the correct reference. Real news: JSON's 6pp drop is the honesty cost of Rule 8 title-relevance promotion + warrant-class generalization. Lattice modes now hit 99% directive coverage (D2/D3/D4/D6/D7); the 1% gap is 2 UTF-16 surrogate errors per mode on one question. Pointer-mode failure pattern shifted to lazy-anchor: 49/142 rows have lazy_anchor_ratio >= 0.75. JSON's structured-per-claim linkage absorbs Rule 8's pressure better than pointer's prose-with-tags. 19/71 questions show JSON > pointer by 50pp+. Recommended context budget surfaced for the first time: - claim_lattice (JSON) peaks at 32-64KB (0.51) - pointer + quote peak at 16-32KB CLAUDE.md headline numbers refreshed; docs index points to the new journal alongside the prior 2026-04-30 reference.
This commit is contained in:
parent
0870af60af
commit
5bc42d277e
2 changed files with 143 additions and 3 deletions
15
CLAUDE.md
15
CLAUDE.md
|
|
@ -143,11 +143,16 @@ revert without reading why. When in doubt, walk the
|
|||
distillation). Soft never enters proof path.
|
||||
- **Three answer modes**: `policy["answer_mode"] ∈ {"quote",
|
||||
"claim_lattice_pointer", "claim_lattice"}`, default `"quote"`.
|
||||
Bench 2026-04-30 on Hermes-3-8B: quote 0.47 strict-rate, pointer
|
||||
0.36, JSON 0.49 — JSON wins. Both lattice modes share
|
||||
Bench 2026-05-02 on Hermes-3-8B (post-Rule-8, post-warrant-
|
||||
generalization, n=2 × 71 questions): quote 0.50 strict-rate,
|
||||
pointer 0.23, JSON 0.44 — quote leads on raw lexical signal;
|
||||
JSON leads among lattice modes & peaks at 32-64KB context (vs
|
||||
pointer's 16-32KB). 99% directive coverage (D2/D3/D4/D6/D7) on
|
||||
lattice modes. Both lattice modes share
|
||||
`verifier_method="claim_lattice"`; `answer_mode` on the run-DAG
|
||||
+ `json_fixups` disambiguate. Each mode folds into
|
||||
`governance_policy_hash`. See `aborist/qa/verify.py`,
|
||||
`docs/qa-modes-bench-2026-05-02.md` (latest), prior journal
|
||||
`docs/qa-modes-bench-2026-04-30.md`.
|
||||
- **Four-rung ladder (lattice-mode display layer)**: POINTER-LINKED
|
||||
→ ANCHOR-WARRANTED → EVIDENCE-WARRANTED → (ENTAILMENT-VERIFIED
|
||||
|
|
@ -320,7 +325,11 @@ Architecture / ongoing work:
|
|||
|
||||
- `docs/cti-architecture.md` — CTI Clause Tree Intelligence.
|
||||
- `docs/mesh.md`, `docs/mesh-deploy.md` — mesh wire + deploy.
|
||||
- `docs/qa-modes-bench-2026-04-30.md` — JSON-mode hardening journey.
|
||||
- `docs/qa-modes-bench-2026-05-02.md` — latest bench (post-Rule-8 /
|
||||
post-warrant-generalization / post-frame-detector). Quote 0.50,
|
||||
pointer 0.23, JSON 0.44. Concurrency landed alongside.
|
||||
- `docs/qa-modes-bench-2026-04-30.md` — JSON-mode hardening journey
|
||||
(prior bench reference).
|
||||
- `docs/reference-frame-failure-class.md` — Orwell case + phrase-route fix.
|
||||
- `docs/bench-maxing.md` — bench discipline.
|
||||
- `docs/test-coverage-audit-2026-05-01.md` — test-suite coverage audit.
|
||||
|
|
|
|||
131
docs/qa-modes-bench-2026-05-02.md
Normal file
131
docs/qa-modes-bench-2026-05-02.md
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
# QA-modes bench — 2026-05-02 (post-Rule-8, post-frame, post-warrant-generalization)
|
||||
|
||||
**Date:** 2026-05-02
|
||||
**Bench:** `bench/qa_sweep.py`, 71 questions × 3 modes × 2 samples = 426 LLM calls (213 cells)
|
||||
**Endpoint:** `https://hermes.ai.unturf.com/v1` (Hermes-3-Llama-3.1-8B-FP8-Dynamic, vLLM, 82K ctx)
|
||||
**Corpus:** Wikipedia 2003-05-16 cur snapshot, sharded under `~/.aborist/shards`
|
||||
**Wall-clock:** ~51 min with `--concurrency 4`
|
||||
**Verifier hardening since 2026-04-30:** Rule 8 title-relevance promoted to hard check, anchor-class warrant generalized to entity-list / count / why-cause shapes (Ticket #000003), retrieval-plan hash bound into the run-DAG (Ticket #000001), reference-frame polarity contract (Ticket #000002), four-rung ladder display (Ticket #000005), bench harness extended with directive coverage + log-scale buckets to 1M (Ticket #000004).
|
||||
|
||||
## Aggregate
|
||||
|
||||
| mode | runs | STRICT | HYBRID | UNGROUNDED | err | strict-rate | mean ratio | mean latency |
|
||||
|------|------|--------|--------|------------|-----|-------------|------------|--------------|
|
||||
| `quote` | 142 | 71 | 45 | 26 | 0 | **0.50** | 0.705 | 29.1s |
|
||||
| `claim_lattice_pointer` | 142 | 32 | 83 | 25 | 2 | **0.23** | 0.564 | 26.7s |
|
||||
| `claim_lattice` (JSON) | 142 | 62 | 51 | 27 | 2 | **0.44** | 0.698 | 28.2s |
|
||||
|
||||
**Δ vs 2026-04-30** (post-retry rerun reference):
|
||||
|
||||
| mode | 2026-04-30 | 2026-05-02 | Δ |
|
||||
|------|-----------|-----------|----|
|
||||
| `quote` | 0.47 | **0.50** | +3pp |
|
||||
| `claim_lattice_pointer` | 0.24 | **0.23** | −1pp |
|
||||
| `claim_lattice` (JSON) | 0.50 | **0.44** | −6pp |
|
||||
|
||||
(Earlier message used the pre-retry 2026-04-30 numbers (0.36 / 0.49) — the post-retry rerun in the same journal is the correct reference; pointer mode is essentially flat, JSON mode took a 6pp honesty cost.)
|
||||
|
||||
## Directive coverage (seven-point program)
|
||||
|
||||
| mode | D2 pointer | D3 cti-ready | D4 ev-map bound | D6 warrant | D7 honest label |
|
||||
|------|-----------|--------------|-----------------|-----------|----------------|
|
||||
| `quote` | 0/142 (0%) | 0/142 (0%) | 142/142 (100%) | 0/142 (0%) | 142/142 (100%) |
|
||||
| `claim_lattice_pointer` | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) |
|
||||
| `claim_lattice` | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) |
|
||||
|
||||
Lattice modes hit **99% on every observable directive**; the 1% gap is the 2 errors per mode (UTF-16 surrogate from the Roman Empire question — model output had lone surrogates that broke UTF-8 encoding). Quote mode shows 0% on D2/D3/D6 by construction (those are claim-lattice-only directives) and 100% on D4/D7.
|
||||
|
||||
## Strict-rate by prompt size
|
||||
|
||||
Buckets capacity in `messages_total` chars. Tells you whether the model strict-rate degrades with input size.
|
||||
|
||||
| mode | bucket | runs | STRICT | strict-rate | mean evidence chars | mean answer chars |
|
||||
|------|--------|------|--------|-------------|---------------------|-------------------|
|
||||
| `quote` | 8-16KB | 4 | 2 | 0.50 | 8460 | 267 |
|
||||
| `quote` | 16-32KB | 26 | 14 | 0.54 | 26946 | 1095 |
|
||||
| `quote` | 32-64KB | 112 | 55 | 0.49 | 37038 | 783 |
|
||||
| `claim_lattice_pointer` | 8-16KB | 2 | 0 | 0.00 | 13066 | 2335 |
|
||||
| `claim_lattice_pointer` | 16-32KB | 62 | 18 | 0.29 | 22846 | 1522 |
|
||||
| `claim_lattice_pointer` | 32-64KB | 76 | 14 | 0.18 | 36205 | 931 |
|
||||
| `claim_lattice` | 8-16KB | 2 | 0 | 0.00 | 13066 | 140 |
|
||||
| `claim_lattice` | 16-32KB | 66 | 25 | 0.38 | 23274 | 1044 |
|
||||
| `claim_lattice` | 32-64KB | 72 | 37 | 0.51 | 36555 | 991 |
|
||||
|
||||
## Recommended context budget
|
||||
|
||||
Per-mode peak strict-rate bucket (5-run minimum to reduce noise):
|
||||
|
||||
| mode | peak bucket | strict-rate | n |
|
||||
|------|-------------|-------------|---|
|
||||
| `quote` | 16-32KB | 0.54 | 26 |
|
||||
| `claim_lattice_pointer` | 16-32KB | 0.29 | 62 |
|
||||
| `claim_lattice` | **32-64KB** | **0.51** | **72** |
|
||||
|
||||
Notable: `claim_lattice` (JSON) **peaks at LARGER context** than the other modes. The structured per-claim evidence linkage that JSON enforces appears to *benefit* from more evidence per claim. Pointer & quote degrade past 16-32KB.
|
||||
|
||||
## Pointer-mode signal — where the gap lives
|
||||
|
||||
Pointer mode's strict-rate is unchanged from 2026-04-30 in absolute terms (0.24 → 0.23) but the failure mode has shifted. The **lazy-anchor ratio histogram** is the load-bearing signal:
|
||||
|
||||
| `lazy_anchor_ratio` bucket | pointer-mode rows |
|
||||
|----------------------------|-------------------|
|
||||
| `0.00` | 25 |
|
||||
| `<0.25` | 9 |
|
||||
| `<0.5` | 11 |
|
||||
| `<0.75` | 46 |
|
||||
| `>=0.75` | **49** |
|
||||
|
||||
49 of 142 pointer rows (35%) anchored ≥75% of claims to a topic-level chunk rather than the specific claim-supporting chunk. **This is the failure pattern Rule 8 was promoted to catch** — when the lazy-anchor target's source title shares zero stemmed content tokens with the claim, the run demotes STRICT → HYBRID via `TITLE_MISMATCH`.
|
||||
|
||||
19 of 71 questions show JSON > pointer by 50pp+:
|
||||
|
||||
```
|
||||
pointer=HH json=SS did napoleon really die on saint helena?
|
||||
pointer=HH json=SS name simpsons family members including pets?
|
||||
pointer=HH json=SS what are the planets of our solar system?
|
||||
pointer=HH json=SS what country is the city of prague in?
|
||||
pointer=HH json=SS what is the boltzmann constant?
|
||||
pointer=UU json=SS what is the difference between http and ftp?
|
||||
pointer=HH json=SS when did the soviet union dissolve?
|
||||
pointer=HH json=SS where does the nile river begin?
|
||||
pointer=HH json=SS where is mount kilimanjaro located?
|
||||
pointer=HH json=SS who is supermans girlfriend?
|
||||
pointer=HH json=SS who painted the mona lisa?
|
||||
pointer=HH json=SS who said may the force be with you?
|
||||
pointer=HH json=SS who were the original seven mercury astronauts?
|
||||
pointer=HH json=SS why did the dinosaurs go extinct?
|
||||
```
|
||||
|
||||
(plus 5 more.) These are factual single-fact questions where JSON's structured per-claim evidence linkage forces the model to pick the specific chunk; pointer's prose-with-tags lets the model lazy-anchor to the topic article and Rule 8 catches it.
|
||||
|
||||
`subject_in_answer` is high across all rungs (STRICT 100%, HYBRID 92%, UNGROUNDED 84%) — the subject is in the answer; the answer just doesn't anchor cleanly to a single chunk. This is **structural-honesty signal**, not deflection.
|
||||
|
||||
## Verdict
|
||||
|
||||
**Pointer-mode strict-rate is essentially flat from the prior bench.** The honesty cost of Rule 8 hits both lattice modes; JSON's structured-per-claim linkage absorbs it better than pointer's prose-with-tags.
|
||||
|
||||
**JSON's 6pp drop is the real news.** Likely causes:
|
||||
1. Rule 8 demoting more cases (same as pointer, less severe).
|
||||
2. Frame detector / polarity preamble changing prompt budget on reference-frame questions.
|
||||
3. Anchor-class warrant generalization (entity-list / count / why-cause) catching cases the prior bench missed entirely.
|
||||
|
||||
The directive-coverage table (99% across every observable D2-D7 on lattice modes) is the architectural win. The strict-rate cost is the price of that calibration.
|
||||
|
||||
## Per-mode recommendation
|
||||
|
||||
- **`claim_lattice` (JSON)** stays the recommended default for cache-grade provenance. Best strict-rate among lattice modes (0.44), JSON-mode wins on structured per-claim discipline, peaks at larger context (32-64KB) so it scales with evidence growth.
|
||||
- **`quote` mode** is best for raw lexical-grounding signal (0.50 strict-rate) but doesn't surface anchor-class warrants — use when the four-rung ladder isn't needed.
|
||||
- **`claim_lattice_pointer`** stays useful for low-context-budget scenarios (16-32KB peak) and for prose-distribution models that struggle with JSON grammar; the lazy-anchor honesty cost is the trade.
|
||||
|
||||
## Concurrency landed alongside this bench
|
||||
|
||||
The bench harness gained a `--concurrency N` flag (commit `0870af6`) for parallel (question, mode) cells. First attempt at concurrency=4 surfaced a thread-safety defect in `aborist/compress.py`: the module-level `ZstdCompressor` / `ZstdDecompressor` singletons were not thread-safe — concurrent decompress calls corrupted the libzstd context and raised `ZstdError: Data corruption detected` on ~19% of retrievals. Fixed via `threading.local()` caches; same commit.
|
||||
|
||||
Wall-clock for this 426-run sweep dropped from ~107 min projected sequential to ~51 min actual at concurrency=4. Speedup is below theoretical 4× because vLLM's continuous batcher queues at 4 concurrent requests, stretching individual call latency. Concurrency=2 may give better per-call latency at lower throughput; concurrency=8 may give higher throughput at the cost of per-call latency. Not yet tuned.
|
||||
|
||||
## Outputs
|
||||
|
||||
- `bench/qa_results/2026-05-02T11-31-55Z.jsonl` — 426 rows, durable artifact
|
||||
- `bench/qa_results/2026-05-02T11-31-55Z.md` — auto-generated summary
|
||||
|
||||
(Both gitignored under `bench/qa_results/`. Headlines & analysis live in this journal.)
|
||||
Loading…
Add table
Add a link
Reference in a new issue