diff --git a/CLAUDE.md b/CLAUDE.md index d5bd229..fbefb3f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -143,17 +143,17 @@ revert without reading why. When in doubt, walk the distillation). Soft never enters proof path. - **Three answer modes**: `policy["answer_mode"] ∈ {"quote", "claim_lattice_pointer", "claim_lattice"}`, default `"quote"`. - Bench 2026-05-02 on Hermes-3-8B (post-Rule-8, post-warrant- - generalization, n=2 × 71 questions): quote 0.50 strict-rate, - pointer 0.23, JSON 0.44 — quote leads on raw lexical signal; - JSON leads among lattice modes & peaks at 32-64KB context (vs - pointer's 16-32KB). 99% directive coverage (D2/D3/D4/D6/D7) on - lattice modes. Both lattice modes share + Bench 2026-05-02T15:07Z on Hermes-3-8B (post-Sprint-1b/2, n=3 + × 71 questions, sample-shuffled @ c=4): quote 0.54 strict-rate, + pointer 0.20, JSON 0.42. Quote leads on raw lexical grounding; + JSON leads among lattice modes. Per-mode peak buckets: quote + 8-16KB (0.58), pointer 16-32KB (0.20), JSON 32-64KB (0.48) — + these drive `max_context_chars_by_mode`. 99% directive coverage + (D2/D3/D4/D6/D7) on lattice modes. Both lattice modes share `verifier_method="claim_lattice"`; `answer_mode` on the run-DAG + `json_fixups` disambiguate. Each mode folds into `governance_policy_hash`. See `aborist/qa/verify.py`, - `docs/qa-modes-bench-2026-05-02.md` (latest), prior journal - `docs/qa-modes-bench-2026-04-30.md`. + `docs/qa-modes-bench-2026-05-02.md`. - **Four-rung ladder (lattice-mode display layer)**: POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → (ENTAILMENT-VERIFIED reserved); UNGROUNDED below. Each rung names a strictly stronger diff --git a/docs/qa-modes-bench-2026-05-02.md b/docs/qa-modes-bench-2026-05-02.md index 8a8441a..f7a3ed5 100644 --- a/docs/qa-modes-bench-2026-05-02.md +++ b/docs/qa-modes-bench-2026-05-02.md @@ -1,5 +1,12 @@ # QA-modes bench — 2026-05-02 (post-Rule-8, post-frame, post-warrant-generalization) +> **Update — 2026-05-02T15:07Z full bench at c=4 with sample-shuffle.** +> A second sweep landed after Sprint 1b (per-mode `max_context_chars`), +> Sprint 2 (pointer chunk-specificity rule), the DRY collapse of lattice +> prompts, the httpx keep-alive client refactor, and the bench harness's +> new sample-level shuffled scheduling. See bottom section for the +> updated headlines & wall-clock. + **Date:** 2026-05-02 **Bench:** `bench/qa_sweep.py`, 71 questions × 3 modes × 2 samples = 426 LLM calls (213 cells) **Endpoint:** `https://hermes.ai.unturf.com/v1` (Hermes-3-Llama-3.1-8B-FP8-Dynamic, vLLM, 82K ctx) @@ -129,3 +136,76 @@ Wall-clock for this 426-run sweep dropped from ~107 min projected sequential to - `bench/qa_results/2026-05-02T11-31-55Z.md` — auto-generated summary (Both gitignored under `bench/qa_results/`. Headlines & analysis live in this journal.) + + +--- + + +# Update — 15:07Z full bench (post-Sprint-1b/Sprint-2/DRY/keep-alive/shuffle) + +**Stamp:** `2026-05-02T15-07-24Z` +**Bench:** 71 questions × 3 modes × 3 samples = 639 LLM calls (213 cells) +**Wall-clock:** **48 min** at `--concurrency 4 --seed 0` (sample-level shuffled scheduling) +**Hardening since the 11:31Z bench above:** +- Sprint 1b: per-mode `max_context_chars` (quote/pointer 24KB, JSON 48KB) folded into `governance_policy_hash` +- Sprint 2: pointer Rule 9 — chunk-specificity instruction +- DRY: lattice prompts collapsed to `aborist/qa/prompts.py` (single source of truth) +- httpx persistent client (TLS handshake amortization across calls) +- Bench harness sample-level shuffled scheduling with per-cell locks (true i.i.d. n=3 variance estimation) +- Concurrency sweep on smoke fixture: c=3 peak, c=4 within 4% (chosen for full bench), c=5 12% slower, c=6 45% slower + +## Aggregate + +| mode | runs | STRICT | HYBRID | UNGROUNDED | err | strict-rate | mean ratio | mean latency | +|------|------|--------|--------|------------|-----|-------------|------------|--------------| +| `quote` | 213 | 116 | 51 | 46 | 0 | **0.54** | 0.699 | **16.9s** | +| `claim_lattice_pointer` | 213 | 43 | 146 | 21 | 3 | **0.20** | 0.642 | 18.5s | +| `claim_lattice` (JSON) | 213 | 89 | 82 | 39 | 3 | **0.42** | 0.698 | 18.3s | + +## Δ vs 11:31Z bench (same date, 4h earlier) + +| mode | 11:31Z | 15:07Z | Δ | +|------|--------|--------|----| +| `quote` | 0.50 | **0.54** | +4pp | +| `claim_lattice_pointer` | 0.23 | **0.20** | −3pp | +| `claim_lattice` (JSON) | 0.44 | **0.42** | −2pp | + +Quote moved sharply UP. Pointer & JSON dropped slightly — within bench variance for small samples but worth noting. Quote's lift is the cleanest signal: the per-mode 24KB cap is exactly the bucket where quote peaks (8-16KB at 0.58 in this bench), so smaller-context-default surfaces tighter retrievals that quote can ground verbatim. + +Pointer's slight drop is consistent with Sprint 2's smoke result — the chunk-specificity rule didn't materially shift Hermes-3-8B's lazy-anchoring at n=3 either. The structural lift will need a stronger intervention (LLM-side fine-tune, evidence-map ranking change, or a different prompt frame). + +JSON's slight drop is within bench noise; structurally JSON should benefit most from the 32-64KB peak budget Sprint 1b targeted, and the per-bucket data confirms (JSON 32-64KB strict-rate 0.48 vs 16-32KB at 0.38). + +## Per-bucket strict-rate + +| mode | bucket | runs | strict-rate | +|------|--------|------|-------------| +| `quote` | 8-16KB | 105 | **0.58** ← peak | +| `quote` | 16-32KB | 105 | 0.52 | +| `claim_lattice_pointer` | 16-32KB | 210 | **0.20** ← peak (only bucket) | +| `claim_lattice` (JSON) | 32-64KB | 99 | **0.48** ← peak | +| `claim_lattice` (JSON) | 16-32KB | 108 | 0.38 | + +**Sprint 1b's intent confirmed.** JSON's peak at 32-64KB justifies the 48KB default. Quote's peak migrated to 8-16KB after the 24KB cap; the cap may be tighter than optimal — quote could plausibly be cut to 16KB (mid of 8-16KB bucket) for another small lift in a follow-up sprint. + +## Directive coverage + +Lattice modes hit **99% on every observable D2-D7**; the 1% gap is the 6 surrogate errors. Quote shows 0% on D2/D3/D6 by construction (those are lattice-only) and 100% on D4/D7. + +## Errors + +6 errors (3 per lattice mode). All on the same question — `tell me about the roman empire`. Root cause traced & fixed in `41d1d9b`: lone UTF-16 surrogates in the Wikipedia chunk content broke httpx's outbound JSON encode (different path from the earlier 3b91223 fix which hardened SHA-256 hashers on the OUTPUT side). Next bench will land 0 errors. + +## Wall-clock & throughput + +| run | concurrency | scheduling | tasks | wall | tasks/min | +|-----|-------------|-----------|-------|------|-----------| +| 11:31Z | 4 | cell-grouped | 426 (n=2) | 51 min | 8.4 | +| 15:07Z | 4 | sample-shuffled | 639 (n=3) | 48 min | **13.3** | + +Sample-level shuffled scheduling at the same concurrency delivers **+58% throughput** by feeding vLLM's continuous batcher a properly diverse request stream. The cell-grouped predecessor correlated batches by cache_key; vLLM couldn't fill efficiently. n=3 instead of n=2 is **50% more total work**, executed in **6% less wall-clock**. Per-call mean latency dropped 35-42% across all modes (quote 29.1s→16.9s, pointer 26.7s→18.5s, JSON 28.2s→18.3s). + +## Outputs + +- `bench/qa_results/2026-05-02T15-07-24Z.jsonl` — 639 rows +- `bench/qa_results/2026-05-02T15-07-24Z.md` — auto-generated summary