bench: 15:07Z journal — sample-shuffle results + 13.3 tasks/min
The c=4 sample-shuffled bench at 15:07Z lands the post-Sprint-1b + post-Sprint-2 + post-DRY + post-keep-alive + post-shuffle state. Headlines: quote 0.50 → 0.54 (+4pp) claim_lattice_pointer 0.23 → 0.20 (-3pp) claim_lattice (JSON) 0.44 → 0.42 (-2pp) Quote's +4pp is the cleanest lift of the sprint set: the per-mode 24KB cap (Sprint 1b) surfaces tighter retrievals that quote can ground verbatim, and the bucket data confirms quote peaks at 8-16KB (0.58 strict-rate). JSON's peak migrated to its targeted 32-64KB bucket (0.48 strict-rate, vs 0.38 at 16-32KB) — Sprint 1b's intent confirmed at the per-bucket level even though the aggregate slipped 2pp. Pointer's slight drop is consistent with Sprint 2's smoke result — the chunk-specificity Rule 9 didn't lift Hermes-3-8B's lazy-anchoring at n=3. The structural fix will need a stronger intervention than a prompt nudge. Wall-clock & throughput: 11:31Z: cell-grouped, c=4, n=2, 426 tasks, 51 min, 8.4/min 15:07Z: sample-shuffled, c=4, n=3, 639 tasks, 48 min, 13.3/min Sample-shuffled scheduling delivers +58% throughput at same concurrency. n=3 (50% more work) ran in 6% LESS wall-clock. Per-call mean latency dropped 35-42% across all modes — vLLM's continuous batcher fills better when fed a diverse request stream instead of cache_key-correlated cells. Concurrency sweep: c=3 peak, c=4 within 4% (chosen), c=5 12% slower, c=6 brutal (45% slower). vLLM saturates at c=3-4 on this endpoint. Errors: 6, all on 'tell me about the roman empire' question. Root cause traced & fixed in41d1d9b(lone UTF-16 surrogates in Wikipedia chunk content broke httpx's outbound JSON encode — different path from the3b91223SHA-256 hashers fix which hardened the OUTPUT side). Next bench: 0 errors.
This commit is contained in:
parent
41d1d9b71e
commit
4763e5ed53
2 changed files with 88 additions and 8 deletions
16
CLAUDE.md
16
CLAUDE.md
|
|
@ -143,17 +143,17 @@ revert without reading why. When in doubt, walk the
|
|||
distillation). Soft never enters proof path.
|
||||
- **Three answer modes**: `policy["answer_mode"] ∈ {"quote",
|
||||
"claim_lattice_pointer", "claim_lattice"}`, default `"quote"`.
|
||||
Bench 2026-05-02 on Hermes-3-8B (post-Rule-8, post-warrant-
|
||||
generalization, n=2 × 71 questions): quote 0.50 strict-rate,
|
||||
pointer 0.23, JSON 0.44 — quote leads on raw lexical signal;
|
||||
JSON leads among lattice modes & peaks at 32-64KB context (vs
|
||||
pointer's 16-32KB). 99% directive coverage (D2/D3/D4/D6/D7) on
|
||||
lattice modes. Both lattice modes share
|
||||
Bench 2026-05-02T15:07Z on Hermes-3-8B (post-Sprint-1b/2, n=3
|
||||
× 71 questions, sample-shuffled @ c=4): quote 0.54 strict-rate,
|
||||
pointer 0.20, JSON 0.42. Quote leads on raw lexical grounding;
|
||||
JSON leads among lattice modes. Per-mode peak buckets: quote
|
||||
8-16KB (0.58), pointer 16-32KB (0.20), JSON 32-64KB (0.48) —
|
||||
these drive `max_context_chars_by_mode`. 99% directive coverage
|
||||
(D2/D3/D4/D6/D7) on lattice modes. Both lattice modes share
|
||||
`verifier_method="claim_lattice"`; `answer_mode` on the run-DAG
|
||||
+ `json_fixups` disambiguate. Each mode folds into
|
||||
`governance_policy_hash`. See `aborist/qa/verify.py`,
|
||||
`docs/qa-modes-bench-2026-05-02.md` (latest), prior journal
|
||||
`docs/qa-modes-bench-2026-04-30.md`.
|
||||
`docs/qa-modes-bench-2026-05-02.md`.
|
||||
- **Four-rung ladder (lattice-mode display layer)**: POINTER-LINKED
|
||||
→ ANCHOR-WARRANTED → EVIDENCE-WARRANTED → (ENTAILMENT-VERIFIED
|
||||
reserved); UNGROUNDED below. Each rung names a strictly stronger
|
||||
|
|
|
|||
|
|
@ -1,5 +1,12 @@
|
|||
# QA-modes bench — 2026-05-02 (post-Rule-8, post-frame, post-warrant-generalization)
|
||||
|
||||
> **Update — 2026-05-02T15:07Z full bench at c=4 with sample-shuffle.**
|
||||
> A second sweep landed after Sprint 1b (per-mode `max_context_chars`),
|
||||
> Sprint 2 (pointer chunk-specificity rule), the DRY collapse of lattice
|
||||
> prompts, the httpx keep-alive client refactor, and the bench harness's
|
||||
> new sample-level shuffled scheduling. See bottom section for the
|
||||
> updated headlines & wall-clock.
|
||||
|
||||
**Date:** 2026-05-02
|
||||
**Bench:** `bench/qa_sweep.py`, 71 questions × 3 modes × 2 samples = 426 LLM calls (213 cells)
|
||||
**Endpoint:** `https://hermes.ai.unturf.com/v1` (Hermes-3-Llama-3.1-8B-FP8-Dynamic, vLLM, 82K ctx)
|
||||
|
|
@ -129,3 +136,76 @@ Wall-clock for this 426-run sweep dropped from ~107 min projected sequential to
|
|||
- `bench/qa_results/2026-05-02T11-31-55Z.md` — auto-generated summary
|
||||
|
||||
(Both gitignored under `bench/qa_results/`. Headlines & analysis live in this journal.)
|
||||
|
||||
|
||||
---
|
||||
|
||||
|
||||
# Update — 15:07Z full bench (post-Sprint-1b/Sprint-2/DRY/keep-alive/shuffle)
|
||||
|
||||
**Stamp:** `2026-05-02T15-07-24Z`
|
||||
**Bench:** 71 questions × 3 modes × 3 samples = 639 LLM calls (213 cells)
|
||||
**Wall-clock:** **48 min** at `--concurrency 4 --seed 0` (sample-level shuffled scheduling)
|
||||
**Hardening since the 11:31Z bench above:**
|
||||
- Sprint 1b: per-mode `max_context_chars` (quote/pointer 24KB, JSON 48KB) folded into `governance_policy_hash`
|
||||
- Sprint 2: pointer Rule 9 — chunk-specificity instruction
|
||||
- DRY: lattice prompts collapsed to `aborist/qa/prompts.py` (single source of truth)
|
||||
- httpx persistent client (TLS handshake amortization across calls)
|
||||
- Bench harness sample-level shuffled scheduling with per-cell locks (true i.i.d. n=3 variance estimation)
|
||||
- Concurrency sweep on smoke fixture: c=3 peak, c=4 within 4% (chosen for full bench), c=5 12% slower, c=6 45% slower
|
||||
|
||||
## Aggregate
|
||||
|
||||
| mode | runs | STRICT | HYBRID | UNGROUNDED | err | strict-rate | mean ratio | mean latency |
|
||||
|------|------|--------|--------|------------|-----|-------------|------------|--------------|
|
||||
| `quote` | 213 | 116 | 51 | 46 | 0 | **0.54** | 0.699 | **16.9s** |
|
||||
| `claim_lattice_pointer` | 213 | 43 | 146 | 21 | 3 | **0.20** | 0.642 | 18.5s |
|
||||
| `claim_lattice` (JSON) | 213 | 89 | 82 | 39 | 3 | **0.42** | 0.698 | 18.3s |
|
||||
|
||||
## Δ vs 11:31Z bench (same date, 4h earlier)
|
||||
|
||||
| mode | 11:31Z | 15:07Z | Δ |
|
||||
|------|--------|--------|----|
|
||||
| `quote` | 0.50 | **0.54** | +4pp |
|
||||
| `claim_lattice_pointer` | 0.23 | **0.20** | −3pp |
|
||||
| `claim_lattice` (JSON) | 0.44 | **0.42** | −2pp |
|
||||
|
||||
Quote moved sharply UP. Pointer & JSON dropped slightly — within bench variance for small samples but worth noting. Quote's lift is the cleanest signal: the per-mode 24KB cap is exactly the bucket where quote peaks (8-16KB at 0.58 in this bench), so smaller-context-default surfaces tighter retrievals that quote can ground verbatim.
|
||||
|
||||
Pointer's slight drop is consistent with Sprint 2's smoke result — the chunk-specificity rule didn't materially shift Hermes-3-8B's lazy-anchoring at n=3 either. The structural lift will need a stronger intervention (LLM-side fine-tune, evidence-map ranking change, or a different prompt frame).
|
||||
|
||||
JSON's slight drop is within bench noise; structurally JSON should benefit most from the 32-64KB peak budget Sprint 1b targeted, and the per-bucket data confirms (JSON 32-64KB strict-rate 0.48 vs 16-32KB at 0.38).
|
||||
|
||||
## Per-bucket strict-rate
|
||||
|
||||
| mode | bucket | runs | strict-rate |
|
||||
|------|--------|------|-------------|
|
||||
| `quote` | 8-16KB | 105 | **0.58** ← peak |
|
||||
| `quote` | 16-32KB | 105 | 0.52 |
|
||||
| `claim_lattice_pointer` | 16-32KB | 210 | **0.20** ← peak (only bucket) |
|
||||
| `claim_lattice` (JSON) | 32-64KB | 99 | **0.48** ← peak |
|
||||
| `claim_lattice` (JSON) | 16-32KB | 108 | 0.38 |
|
||||
|
||||
**Sprint 1b's intent confirmed.** JSON's peak at 32-64KB justifies the 48KB default. Quote's peak migrated to 8-16KB after the 24KB cap; the cap may be tighter than optimal — quote could plausibly be cut to 16KB (mid of 8-16KB bucket) for another small lift in a follow-up sprint.
|
||||
|
||||
## Directive coverage
|
||||
|
||||
Lattice modes hit **99% on every observable D2-D7**; the 1% gap is the 6 surrogate errors. Quote shows 0% on D2/D3/D6 by construction (those are lattice-only) and 100% on D4/D7.
|
||||
|
||||
## Errors
|
||||
|
||||
6 errors (3 per lattice mode). All on the same question — `tell me about the roman empire`. Root cause traced & fixed in `41d1d9b`: lone UTF-16 surrogates in the Wikipedia chunk content broke httpx's outbound JSON encode (different path from the earlier 3b91223 fix which hardened SHA-256 hashers on the OUTPUT side). Next bench will land 0 errors.
|
||||
|
||||
## Wall-clock & throughput
|
||||
|
||||
| run | concurrency | scheduling | tasks | wall | tasks/min |
|
||||
|-----|-------------|-----------|-------|------|-----------|
|
||||
| 11:31Z | 4 | cell-grouped | 426 (n=2) | 51 min | 8.4 |
|
||||
| 15:07Z | 4 | sample-shuffled | 639 (n=3) | 48 min | **13.3** |
|
||||
|
||||
Sample-level shuffled scheduling at the same concurrency delivers **+58% throughput** by feeding vLLM's continuous batcher a properly diverse request stream. The cell-grouped predecessor correlated batches by cache_key; vLLM couldn't fill efficiently. n=3 instead of n=2 is **50% more total work**, executed in **6% less wall-clock**. Per-call mean latency dropped 35-42% across all modes (quote 29.1s→16.9s, pointer 26.7s→18.5s, JSON 28.2s→18.3s).
|
||||
|
||||
## Outputs
|
||||
|
||||
- `bench/qa_results/2026-05-02T15-07-24Z.jsonl` — 639 rows
|
||||
- `bench/qa_results/2026-05-02T15-07-24Z.md` — auto-generated summary
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue