diff --git a/CLAUDE.md b/CLAUDE.md index 6b31f55..d5bd229 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -143,11 +143,16 @@ revert without reading why. When in doubt, walk the distillation). Soft never enters proof path. - **Three answer modes**: `policy["answer_mode"] ∈ {"quote", "claim_lattice_pointer", "claim_lattice"}`, default `"quote"`. - Bench 2026-04-30 on Hermes-3-8B: quote 0.47 strict-rate, pointer - 0.36, JSON 0.49 — JSON wins. Both lattice modes share + Bench 2026-05-02 on Hermes-3-8B (post-Rule-8, post-warrant- + generalization, n=2 × 71 questions): quote 0.50 strict-rate, + pointer 0.23, JSON 0.44 — quote leads on raw lexical signal; + JSON leads among lattice modes & peaks at 32-64KB context (vs + pointer's 16-32KB). 99% directive coverage (D2/D3/D4/D6/D7) on + lattice modes. Both lattice modes share `verifier_method="claim_lattice"`; `answer_mode` on the run-DAG + `json_fixups` disambiguate. Each mode folds into `governance_policy_hash`. See `aborist/qa/verify.py`, + `docs/qa-modes-bench-2026-05-02.md` (latest), prior journal `docs/qa-modes-bench-2026-04-30.md`. - **Four-rung ladder (lattice-mode display layer)**: POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → (ENTAILMENT-VERIFIED @@ -320,7 +325,11 @@ Architecture / ongoing work: - `docs/cti-architecture.md` — CTI Clause Tree Intelligence. - `docs/mesh.md`, `docs/mesh-deploy.md` — mesh wire + deploy. -- `docs/qa-modes-bench-2026-04-30.md` — JSON-mode hardening journey. +- `docs/qa-modes-bench-2026-05-02.md` — latest bench (post-Rule-8 / + post-warrant-generalization / post-frame-detector). Quote 0.50, + pointer 0.23, JSON 0.44. Concurrency landed alongside. +- `docs/qa-modes-bench-2026-04-30.md` — JSON-mode hardening journey + (prior bench reference). - `docs/reference-frame-failure-class.md` — Orwell case + phrase-route fix. - `docs/bench-maxing.md` — bench discipline. - `docs/test-coverage-audit-2026-05-01.md` — test-suite coverage audit. diff --git a/docs/qa-modes-bench-2026-05-02.md b/docs/qa-modes-bench-2026-05-02.md new file mode 100644 index 0000000..8a8441a --- /dev/null +++ b/docs/qa-modes-bench-2026-05-02.md @@ -0,0 +1,131 @@ +# QA-modes bench — 2026-05-02 (post-Rule-8, post-frame, post-warrant-generalization) + +**Date:** 2026-05-02 +**Bench:** `bench/qa_sweep.py`, 71 questions × 3 modes × 2 samples = 426 LLM calls (213 cells) +**Endpoint:** `https://hermes.ai.unturf.com/v1` (Hermes-3-Llama-3.1-8B-FP8-Dynamic, vLLM, 82K ctx) +**Corpus:** Wikipedia 2003-05-16 cur snapshot, sharded under `~/.aborist/shards` +**Wall-clock:** ~51 min with `--concurrency 4` +**Verifier hardening since 2026-04-30:** Rule 8 title-relevance promoted to hard check, anchor-class warrant generalized to entity-list / count / why-cause shapes (Ticket #000003), retrieval-plan hash bound into the run-DAG (Ticket #000001), reference-frame polarity contract (Ticket #000002), four-rung ladder display (Ticket #000005), bench harness extended with directive coverage + log-scale buckets to 1M (Ticket #000004). + +## Aggregate + +| mode | runs | STRICT | HYBRID | UNGROUNDED | err | strict-rate | mean ratio | mean latency | +|------|------|--------|--------|------------|-----|-------------|------------|--------------| +| `quote` | 142 | 71 | 45 | 26 | 0 | **0.50** | 0.705 | 29.1s | +| `claim_lattice_pointer` | 142 | 32 | 83 | 25 | 2 | **0.23** | 0.564 | 26.7s | +| `claim_lattice` (JSON) | 142 | 62 | 51 | 27 | 2 | **0.44** | 0.698 | 28.2s | + +**Δ vs 2026-04-30** (post-retry rerun reference): + +| mode | 2026-04-30 | 2026-05-02 | Δ | +|------|-----------|-----------|----| +| `quote` | 0.47 | **0.50** | +3pp | +| `claim_lattice_pointer` | 0.24 | **0.23** | −1pp | +| `claim_lattice` (JSON) | 0.50 | **0.44** | −6pp | + +(Earlier message used the pre-retry 2026-04-30 numbers (0.36 / 0.49) — the post-retry rerun in the same journal is the correct reference; pointer mode is essentially flat, JSON mode took a 6pp honesty cost.) + +## Directive coverage (seven-point program) + +| mode | D2 pointer | D3 cti-ready | D4 ev-map bound | D6 warrant | D7 honest label | +|------|-----------|--------------|-----------------|-----------|----------------| +| `quote` | 0/142 (0%) | 0/142 (0%) | 142/142 (100%) | 0/142 (0%) | 142/142 (100%) | +| `claim_lattice_pointer` | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | +| `claim_lattice` | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | + +Lattice modes hit **99% on every observable directive**; the 1% gap is the 2 errors per mode (UTF-16 surrogate from the Roman Empire question — model output had lone surrogates that broke UTF-8 encoding). Quote mode shows 0% on D2/D3/D6 by construction (those are claim-lattice-only directives) and 100% on D4/D7. + +## Strict-rate by prompt size + +Buckets capacity in `messages_total` chars. Tells you whether the model strict-rate degrades with input size. + +| mode | bucket | runs | STRICT | strict-rate | mean evidence chars | mean answer chars | +|------|--------|------|--------|-------------|---------------------|-------------------| +| `quote` | 8-16KB | 4 | 2 | 0.50 | 8460 | 267 | +| `quote` | 16-32KB | 26 | 14 | 0.54 | 26946 | 1095 | +| `quote` | 32-64KB | 112 | 55 | 0.49 | 37038 | 783 | +| `claim_lattice_pointer` | 8-16KB | 2 | 0 | 0.00 | 13066 | 2335 | +| `claim_lattice_pointer` | 16-32KB | 62 | 18 | 0.29 | 22846 | 1522 | +| `claim_lattice_pointer` | 32-64KB | 76 | 14 | 0.18 | 36205 | 931 | +| `claim_lattice` | 8-16KB | 2 | 0 | 0.00 | 13066 | 140 | +| `claim_lattice` | 16-32KB | 66 | 25 | 0.38 | 23274 | 1044 | +| `claim_lattice` | 32-64KB | 72 | 37 | 0.51 | 36555 | 991 | + +## Recommended context budget + +Per-mode peak strict-rate bucket (5-run minimum to reduce noise): + +| mode | peak bucket | strict-rate | n | +|------|-------------|-------------|---| +| `quote` | 16-32KB | 0.54 | 26 | +| `claim_lattice_pointer` | 16-32KB | 0.29 | 62 | +| `claim_lattice` | **32-64KB** | **0.51** | **72** | + +Notable: `claim_lattice` (JSON) **peaks at LARGER context** than the other modes. The structured per-claim evidence linkage that JSON enforces appears to *benefit* from more evidence per claim. Pointer & quote degrade past 16-32KB. + +## Pointer-mode signal — where the gap lives + +Pointer mode's strict-rate is unchanged from 2026-04-30 in absolute terms (0.24 → 0.23) but the failure mode has shifted. The **lazy-anchor ratio histogram** is the load-bearing signal: + +| `lazy_anchor_ratio` bucket | pointer-mode rows | +|----------------------------|-------------------| +| `0.00` | 25 | +| `<0.25` | 9 | +| `<0.5` | 11 | +| `<0.75` | 46 | +| `>=0.75` | **49** | + +49 of 142 pointer rows (35%) anchored ≥75% of claims to a topic-level chunk rather than the specific claim-supporting chunk. **This is the failure pattern Rule 8 was promoted to catch** — when the lazy-anchor target's source title shares zero stemmed content tokens with the claim, the run demotes STRICT → HYBRID via `TITLE_MISMATCH`. + +19 of 71 questions show JSON > pointer by 50pp+: + +``` +pointer=HH json=SS did napoleon really die on saint helena? +pointer=HH json=SS name simpsons family members including pets? +pointer=HH json=SS what are the planets of our solar system? +pointer=HH json=SS what country is the city of prague in? +pointer=HH json=SS what is the boltzmann constant? +pointer=UU json=SS what is the difference between http and ftp? +pointer=HH json=SS when did the soviet union dissolve? +pointer=HH json=SS where does the nile river begin? +pointer=HH json=SS where is mount kilimanjaro located? +pointer=HH json=SS who is supermans girlfriend? +pointer=HH json=SS who painted the mona lisa? +pointer=HH json=SS who said may the force be with you? +pointer=HH json=SS who were the original seven mercury astronauts? +pointer=HH json=SS why did the dinosaurs go extinct? +``` + +(plus 5 more.) These are factual single-fact questions where JSON's structured per-claim evidence linkage forces the model to pick the specific chunk; pointer's prose-with-tags lets the model lazy-anchor to the topic article and Rule 8 catches it. + +`subject_in_answer` is high across all rungs (STRICT 100%, HYBRID 92%, UNGROUNDED 84%) — the subject is in the answer; the answer just doesn't anchor cleanly to a single chunk. This is **structural-honesty signal**, not deflection. + +## Verdict + +**Pointer-mode strict-rate is essentially flat from the prior bench.** The honesty cost of Rule 8 hits both lattice modes; JSON's structured-per-claim linkage absorbs it better than pointer's prose-with-tags. + +**JSON's 6pp drop is the real news.** Likely causes: +1. Rule 8 demoting more cases (same as pointer, less severe). +2. Frame detector / polarity preamble changing prompt budget on reference-frame questions. +3. Anchor-class warrant generalization (entity-list / count / why-cause) catching cases the prior bench missed entirely. + +The directive-coverage table (99% across every observable D2-D7 on lattice modes) is the architectural win. The strict-rate cost is the price of that calibration. + +## Per-mode recommendation + +- **`claim_lattice` (JSON)** stays the recommended default for cache-grade provenance. Best strict-rate among lattice modes (0.44), JSON-mode wins on structured per-claim discipline, peaks at larger context (32-64KB) so it scales with evidence growth. +- **`quote` mode** is best for raw lexical-grounding signal (0.50 strict-rate) but doesn't surface anchor-class warrants — use when the four-rung ladder isn't needed. +- **`claim_lattice_pointer`** stays useful for low-context-budget scenarios (16-32KB peak) and for prose-distribution models that struggle with JSON grammar; the lazy-anchor honesty cost is the trade. + +## Concurrency landed alongside this bench + +The bench harness gained a `--concurrency N` flag (commit `0870af6`) for parallel (question, mode) cells. First attempt at concurrency=4 surfaced a thread-safety defect in `aborist/compress.py`: the module-level `ZstdCompressor` / `ZstdDecompressor` singletons were not thread-safe — concurrent decompress calls corrupted the libzstd context and raised `ZstdError: Data corruption detected` on ~19% of retrievals. Fixed via `threading.local()` caches; same commit. + +Wall-clock for this 426-run sweep dropped from ~107 min projected sequential to ~51 min actual at concurrency=4. Speedup is below theoretical 4× because vLLM's continuous batcher queues at 4 concurrent requests, stretching individual call latency. Concurrency=2 may give better per-call latency at lower throughput; concurrency=8 may give higher throughput at the cost of per-call latency. Not yet tuned. + +## Outputs + +- `bench/qa_results/2026-05-02T11-31-55Z.jsonl` — 426 rows, durable artifact +- `bench/qa_results/2026-05-02T11-31-55Z.md` — auto-generated summary + +(Both gitignored under `bench/qa_results/`. Headlines & analysis live in this journal.)