# QA-modes bench — 2026-05-02

**Date:** 2026-05-02
**Endpoint:** `https://hermes.ai.unturf.com/v1` (Hermes-3-Llama-3.1-8B-FP8-Dynamic, vLLM, 82K ctx)
**Corpus:** Wikipedia 2003-05-16 cur snapshot, sharded under `~/.arborist/shards`

Two sweeps landed today, each 71 questions × 3 modes:

| stamp | samples | concurrency | scheduling | runs | wall-clock |
|-------|---------|-------------|------------|------|------------|
| `11:31Z` | n=2 | c=4 | cell-grouped | 426 | 51 min |
| `15:07Z` | n=3 | c=4 | sample-shuffled, `--seed 0` | 639 | **48 min** |

The 15:07Z sweep is the **authoritative state of the substrate** at end-of-day; the 11:31Z sweep is the intermediate witnessed before the second wave of hardening landed. Headlines below show both.

## Hardening since the prior bench (2026-04-30 post-retry)

Substrate-side (between `2026-04-30T18:46Z` and `2026-05-02T11:31Z`):

- Rule 8 — title-relevance promoted to hard verifier check (`TITLE_MISMATCH` violation demotes `STRICT → HYBRID` when no cited evidence's source title shares a stemmed token with the claim).
- Anchor-class warrant generalized to entity-list / count / why-cause shapes (Ticket #000003).
- Retrieval-plan hash bound into the run-DAG retrieval stage (Ticket #000001).
- Reference-frame polarity contract (Ticket #000002).
- Four-rung ladder display (Ticket #000005) — `POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → (ENTAILMENT-VERIFIED reserved)` mapped from the v9.8 audit_mode trichotomy at render time.
- Bench harness extended with directive coverage + log-scale buckets to 1M (Ticket #000004).

Substrate-side (between `2026-05-02T11:31Z` and `2026-05-02T15:07Z`):

- **Sprint 1b** — per-mode `max_context_chars`. Bench's recommended-context-budget table flows back into `DEFAULT_QUERY_POLICY["max_context_chars_by_mode"]`: quote 24 KB, pointer 24 KB, JSON 48 KB. Folds into `governance_policy_hash`.
- **Sprint 2** — pointer Rule 9: chunk-specificity instruction added to the lattice-pointer system prompt.
- **DRY collapse** — the four lattice prompts (system + grounding × pointer + JSON) lifted to `arborist/qa/prompts.py` as a single source of truth, imported by both `runner.DEFAULT_POLICY` and `query.DEFAULT_QUERY_POLICY`.
- **httpx persistent client** — the chat-completion path used to construct a fresh `httpx.Client` per call, paying a TLS handshake every request. Move to `__init__`; HTTP/1.1 keep-alive across calls. Save 1-2 min on a 426-call bench.
- **Sample-level shuffled bench scheduling** — every `(question, mode, sample_idx)` is a task, shuffled with `--seed`, dispatched concurrently. Per-cell `Lock` dict serializes burn-then-write on the shared cache_key. True i.i.d. n=3 variance; vLLM batcher fed a diverse request stream.
- **Bench `--resume`** — read existing JSONL, skip done tasks, append fresh rows. Stop/start-able.
- **Concurrency sweep on the smoke fixture.** vLLM peaks at c=3-4; saturates badly past c=4 (see Wall-clock & throughput below).
- **UTF-16 surrogate fix v2** (`41d1d9b`) — corpus chunks with lone surrogates broke httpx's outbound JSON encode. Scrub at the client boundary by WTF-8 → UTF-8-with-replace roundtrip. (The earlier v1 fix `3b91223` hardened the OUTPUT side — SHA-256 hashers — but missed the INPUT side. Both paths now safe.)

## Aggregate (15:07Z, authoritative)

| mode | runs | STRICT | HYBRID | UNGROUNDED | err | strict-rate | mean ratio | mean latency |
|------|------|--------|--------|------------|-----|-------------|------------|--------------|
| `quote` | 213 | 116 | 51 | 46 | 0 | **0.54** | 0.699 | **16.9 s** |
| `claim_lattice_pointer` | 213 | 43 | 146 | 21 | 3 | **0.20** | 0.642 | 18.5 s |
| `claim_lattice` (JSON) | 213 | 89 | 82 | 39 | 3 | **0.42** | 0.698 | 18.3 s |

## Δ across the day

| mode | 2026-04-30 (post-retry) | 2026-05-02T11:31Z | 2026-05-02T15:07Z | net Δ |
|------|------------------------|-------------------|-------------------|-------|
| `quote` | 0.47 | 0.50 (+3pp) | **0.54** (+4pp) | **+7pp** |
| `claim_lattice_pointer` | 0.24 | 0.23 (−1pp) | 0.20 (−3pp) | −4pp |
| `claim_lattice` (JSON) | 0.50 | 0.44 (−6pp) | 0.42 (−2pp) | −8pp |

**Quote climbed the most across the day**, ending at 0.54. The Sprint 1b 24 KB cap surfaces tighter retrievals that quote can ground verbatim (the bucket data confirms — quote's peak migrated to 8-16 KB at 0.58 strict-rate).

**Pointer & JSON each took an honesty cost.** Rule 8 promotion + warrant-class generalization demote more cases that *would* have classified STRICT under the looser pre-2026-05-02 verifier. The architectural win — 99% directive coverage on lattice modes — is the price for those drops; false-positive STRICT was corruption, and we converted it to honest HYBRID.

## Per-bucket strict-rate (15:07Z)

| mode | bucket | runs | strict-rate | note |
|------|--------|------|-------------|------|
| `quote` | 8-16 KB | 105 | **0.58** | peak |
| `quote` | 16-32 KB | 105 | 0.52 | |
| `claim_lattice_pointer` | 16-32 KB | 210 | **0.20** | peak (only bucket) |
| `claim_lattice` (JSON) | 32-64 KB | 99 | **0.48** | peak |
| `claim_lattice` (JSON) | 16-32 KB | 108 | 0.38 | |

**Sprint 1b's intent confirmed.** JSON's peak at 32-64 KB justifies the 48 KB default. Quote's peak migrated to 8-16 KB after the 24 KB cap; the cap may be tighter than optimal — quote could plausibly be cut to 16 KB (mid of 8-16 KB bucket) for another small lift in a follow-up sprint.

`claim_lattice` (JSON) **peaks at LARGER context** than the other modes. The structured per-claim evidence linkage that JSON enforces benefits from more evidence per claim. Pointer & quote degrade past 16-32 KB.

## Recommended context budget (per-mode)

5-run minimum to reduce noise; from the 15:07Z bucket data:

| mode | peak bucket | strict-rate | n |
|------|-------------|-------------|---|
| `quote` | 8-16 KB | 0.58 | 105 |
| `claim_lattice_pointer` | 16-32 KB | 0.20 | 210 |
| `claim_lattice` (JSON) | **32-64 KB** | **0.48** | 99 |

These flow into `DEFAULT_QUERY_POLICY["max_context_chars_by_mode"]` (currently 24 / 24 / 48 KB — mid of each peak bucket). When the bench shifts those peaks, retune the policy and let `governance_policy_hash` partition the cache.

## Directive coverage (seven-point program)

| mode | D2 pointer | D3 cti-ready | D4 ev-map bound | D6 warrant | D7 honest label |
|------|-----------|--------------|-----------------|-----------|----------------|
| `quote` | 0/213 (0%) | 0/213 (0%) | 213/213 (100%) | 0/213 (0%) | 213/213 (100%) |
| `claim_lattice_pointer` | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) |
| `claim_lattice` | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) |

Lattice modes hit **99% on every observable directive**; the 1% gap is the 6 surrogate errors (already fixed in `41d1d9b`, next bench will hit 100%). Quote shows 0% on D2 / D3 / D6 by construction (those are lattice-only directives) and 100% on D4 / D7.

## Pointer-mode signal — where the gap lives

Pointer mode's strict-rate did not lift on Sprint 2's chunk-specificity prompt nudge. The load-bearing failure pattern is **lazy-anchoring**: the model cites a topic-overview chunk for every claim instead of the specific chunk that supports each individual claim. Lazy-anchor ratio histogram (11:31Z bench, 142 pointer rows):

| `lazy_anchor_ratio` bucket | rows |
|----------------------------|------|
| `0.00` | 25 |
| `<0.25` | 9 |
| `<0.5` | 11 |
| `<0.75` | 46 |
| `>=0.75` | **49** (35%) |

35% of pointer rows anchor ≥75% of claims to a topic chunk. **This is the failure pattern Rule 8 was promoted to catch** — when the lazy-anchor target's source title shares zero stemmed content tokens with the claim, the run demotes `STRICT → HYBRID` via `TITLE_MISMATCH`. The honesty surfaces; the structural lift requires a stronger intervention (evidence-map ranking change, model fine-tune, or a different prompt frame).

19 of 71 questions show JSON ≥ 50pp above pointer, including:

```
pointer=HH  json=SS   did napoleon really die on saint helena?
pointer=HH  json=SS   name simpsons family members including pets?
pointer=HH  json=SS   what are the planets of our solar system?
pointer=HH  json=SS   what country is the city of prague in?
pointer=HH  json=SS   what is the boltzmann constant?
pointer=UU  json=SS   what is the difference between http and ftp?
pointer=HH  json=SS   when did the soviet union dissolve?
pointer=HH  json=SS   where does the nile river begin?
pointer=HH  json=SS   where is mount kilimanjaro located?
pointer=HH  json=SS   who is supermans girlfriend?
pointer=HH  json=SS   who painted the mona lisa?
pointer=HH  json=SS   who said may the force be with you?
pointer=HH  json=SS   who were the original seven mercury astronauts?
pointer=HH  json=SS   why did the dinosaurs go extinct?
```

(plus 5 more.) These are factual single-fact questions where JSON's structured per-claim evidence linkage forces the model to pick the specific chunk; pointer's prose-with-tags lets the model lazy-anchor to the topic article and Rule 8 catches it.

`subject_in_answer` is high across all rungs (STRICT 100%, HYBRID 92%, UNGROUNDED 84%) — the subject is in the answer; the answer just doesn't anchor cleanly to a single chunk. This is **structural-honesty signal**, not deflection.

## Wall-clock & throughput evolution

| stamp | concurrency | scheduling | n | tasks | wall-clock | tasks/min | mean latency (mode) |
|-------|-------------|-----------|---|-------|------------|-----------|---------------------|
| 11:31Z | 4 | cell-grouped | 2 | 426 | 51 min | 8.4 | 26-29 s |
| 15:07Z | 4 | sample-shuffled | 3 | 639 | **48 min** | **13.3** | **17-19 s** |

Sample-level shuffled scheduling at the same concurrency delivers **+58% throughput**. n=3 (50% more total work) ran in 6% LESS wall-clock than n=2 cell-grouped. Per-call mean latency dropped 35-42% across all modes (quote 29.1 s → 16.9 s, pointer 26.7 s → 18.5 s, JSON 28.2 s → 18.3 s) because vLLM's continuous batcher fills better when fed a diverse, uncorrelated request stream.

### Concurrency sweep (smoke fixture, 15-task)

| c | wall-clock | tasks/min | normalized |
|---|------------|-----------|------------|
| 3 | 102 s | 8.8 | **1.00** (peak) |
| 4 | 106 s | 8.5 | 0.96 (chosen — within 4% of peak, more forgiving on single-call hiccups) |
| 5 | 119 s | 7.6 | 0.86 |
| 6 | 185 s | 4.9 | 0.55 (vLLM batching ceiling, brutal) |

vLLM saturates at c=3-4 on this endpoint. More concurrent requests fill the batch better, but only up to the point where per-call latency growth outpaces parallelism gain. Re-sweep when the endpoint is upgraded, when corpus shape changes the average context size, or when other tenants change the queue.

## Errors

Across both benches: **2 errors at 11:31Z, 6 errors at 15:07Z** — all on the same question (`tell me about the roman empire`). Lone UTF-16 surrogates in Wikipedia chunk content, two distinct paths:

- v1 (`3b91223`) — hardened SHA-256 hashers on the OUTPUT side (`arborist/qa/dag.py`, `keys.py`, `evidence.py`, `store.py` audit chain) with `errors='surrogatepass'` so the run-DAG roots survive surrogate-bearing model output.
- v2 (`41d1d9b`) — scrubs message content INSIDE `OpenAICompatibleClient.chat_completion` before httpx's outbound JSON encode. The corpus chunk text was the path; httpx's `.encode('utf-8')` on the request body raised before the call left the client.

Verified: `tell me about the roman empire` under `claim_lattice` now classifies HYBRID 5/7 instead of erroring. Next bench will land 0 errors.

## Verdict & per-mode recommendation

**Quote leads on raw lexical grounding (0.54 strict-rate).** Best when the four-rung ladder isn't needed and the answer is short-and-cite-able. Doesn't surface anchor-class warrants.

**JSON leads among lattice modes (0.42).** Recommended default for cache-grade provenance. Structured per-claim evidence linkage; peaks at larger context (32-64 KB) so it scales with evidence growth.

**Pointer (0.20) stays useful for low-context-budget scenarios** (16-32 KB peak) and prose-distribution models that struggle with JSON grammar; the lazy-anchor honesty cost is the trade.

**The directive-coverage table (99% across every observable D2-D7 on lattice modes) is the architectural win.** False-positive STRICT was corruption; converting it to honest HYBRID was the price of calibration. Strict-rate slipped on lattice modes; the substrate is more honest.

## Outputs

- `bench/qa_results/2026-05-02T11-31-55Z.{jsonl,md}` — 426 rows
- `bench/qa_results/2026-05-02T15-07-24Z.{jsonl,md}` — 639 rows

(Both gitignored under `bench/qa_results/`. Headlines & analysis live in this journal.)

---

## Addendum — 2026-05-03 broad-quantifier A/B (Ticket #000008)

This journal froze the 2026-05-02 substrate baseline. On 2026-05-03
Ticket #000008 (broad-quantifier preflight guard) landed Phases 0–4
and ran a four-cell A/B on a 9-question broad subset
(`bench/qa_questions_quantifier_subset.txt`, 81 runs per cell).

Findings relevant to this journal's per-mode recommendations:

- **JSON-mode strict-rate moves under cap-only**: 0.19 baseline →
  0.33 with `quantifier_guard_apply_caps=true` (+14pp on the broad
  subset). The recommendation here that "JSON leads among lattice
  modes" still holds, but the *headroom above 0.42 globally* is
  partly dependent on broad-vs-narrow question mix.
- **Pointer-mode stays at 0/27 STRICT** on broad questions across
  all four A/B cells. The 0.20 strict-rate above is *across the
  full 71-question set*; the broad subset alone is structurally
  unfavorable to pointer mode regardless of cap or reminder.
- **Reminder eliminates FORMAT_COLLAPSED**: 2/27 → 0/27 on pointer
  mode under `quantifier_reminder_enabled=true`.
- **Cap and reminder help DIFFERENT failure modes**: reminder
  rescues UNGROUNDED → HYBRID (restates citation rule); cap
  rescues HYBRID → STRICT (forces fewer-but-better claims).
  Compound effect on pointer mean-ratio is best at 0.684 (vs
  0.473 baseline).

See `docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md`
§12 for the four-cell data and the §10.8 decision-tree verdict.
Bench artifacts: `bench/qa_results/2026-05-03T12-{29-53,38-53,47-23,54-11}Z.{jsonl,md}`.

### Addendum 2 — preflight ON vs OFF validation (2026-05-03T23-06-21Z)

After ticket #000010 flipped `quantifier_reminder_enabled=True` and
shipped `metacognition_enabled=True` defaults, ran a preflight-OFF
cell to validate the flip didn't regress baseline behavior. Same
9-question broad subset (`bench/qa_questions_quantifier_subset.txt`),
n=3, policy override
`--policy metacognition_enabled=false --policy quantifier_reminder_enabled=false`.

Compared against §12.6 reminder-only baseline (cleanest single-knob
on-cell). On this 9-question subset, none of the metacognition
detectors fire (no temporal / contradiction / false-premise / out-
of-corpus shapes), so the comparison effectively isolates the
reminder contribution.

| Metric                       | OFF       | ON        | Δ        |
|------------------------------|-----------|-----------|----------|
| `quote` strict-rate          | 0.63      | 0.52      | −11pp (noise band on 27 samples) |
| `pointer` strict-rate        | 0.00      | 0.00      | 0        |
| `JSON` strict-rate           | 0.26      | 0.22      | −4pp     |
| **`pointer` mean ratio**     | 0.483     | **0.643** | **+16pp** |
| **`JSON` mean ratio**        | 0.570     | **0.735** | **+17pp** |
| **`JSON` UNGROUNDED rate**   | 7/27 (26%) | **1/27 (4%)** | **−22pp** |
| **`pointer` FORMAT_COLLAPSED** | 2/27   | **0/27**  | **−100%** |
| `pointer` NO_EVIDENCE_POINTER | 8/27    | 6/27      | −7pp     |

**Interpretation:**

- **Mean-ratio improvement is robust** (+16-17pp on lattice modes).
  Grounded rows are MORE thoroughly grounded under preflight ON.
- **JSON UNGROUNDED collapse is dramatic** (−22pp). The "didn't
  ground" pool reclassifies into "partially grounded" — operator-
  visible win.
- **FORMAT_COLLAPSED elimination on pointer mode** (2 → 0). The
  reminder restates the [E\d+] citation rule and Hermes follows
  it.
- **STRICT-rate moves are within noise.** Quote-mode dipped 11pp,
  but quote is mode-gated off the guard so this is pure Hermes
  nondeterminism — the §10.8 5pp floor exists exactly to filter
  this. JSON SR moved −4pp (within floor).

**Verdict:** the #000010 default-on flip is doing what was
claimed. Mean-ratio + UNGROUNDED + FORMAT_COLLAPSED metrics all
improve by ≥5pp on lattice modes; STRICT-rate is within noise.
Defaults stay on.

Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`.

### Addendum 3 — full-bench regression check (2026-05-03T23-30-12Z)

Validates that the #000010 default flip (reminder default-on for
lattice modes) doesn't regress narrow-question performance. The
prior validations in Addendum 1 + Addendum 2 used the 9-question
broad subset only; this run sweeps the full 75-question
`bench/qa_questions.txt` (~10% broad, ~89% narrow factoid /
descriptive / list shapes).

Same harness, n=3 × 3 modes × 75 questions = 225 runs per mode.
Compared against the frozen `15:07Z` baseline (the authoritative
pre-#000008/9/10 state of the substrate, 71 questions × n=3 = 213
runs per mode):

| Mode                    | Pre-flip (15:07Z) | **Post-flip (23:30Z)** | Δ SR | Δ mean ratio |
|-------------------------|-------------------|------------------------|------|--------------|
| `quote`                 | 116S/51H/46U · SR 0.54 · ratio 0.699 | 116S/60H/49U · **SR 0.52** · **ratio 0.692** | −2pp | −1pp |
| `claim_lattice_pointer` | 43S/146H/21U · SR 0.20 · ratio 0.642 | 48S/145H/32U · **SR 0.21** · **ratio 0.663** | +1pp | +2pp |
| `claim_lattice` (JSON)  | 89S/82H/39U · SR 0.42 · ratio 0.698 | 99S/80H/46U · **SR 0.44** · **ratio 0.730** | +2pp | +3pp |

All STRICT-rate deltas within the 5pp signal floor. Same question
set ±2 (3 added: `winners of all major sports?`, `name all members
of the beatles`, `list all planets in the solar system`).

**Substrate-level findings:**

- **Pointer-mode `FORMAT_COLLAPSED`: 0/225** across the full sweep.
  The default-on reminder eliminates the collapse mode globally,
  not just on broad questions where it was originally measured.
- **Pointer-mode `NO_EVIDENCE_POINTER`: 29/225** (13%) — lower than
  the 33% we saw on broad-only (9/27) when reminder was off.
  Reminder discipline propagates to non-broad questions even though
  the reminder text only fires on broad shapes (the broader signal
  is that the reminder text reinforces the citation rule across
  the model's session attention).
- **Quote-mode is essentially unchanged** (-2pp SR, -1pp ratio).
  Quote opts out of the guard (`quantifier_guard_modes` default
  excludes quote), so this is pure Hermes nondeterminism on a
  225-sample cell.
- **JSON-mode mean ratio +3pp** — small but positive, consistent
  with the broad-subset finding that reminder boosts per-claim
  grounding density.

**Latency note.** Wall-clock per call jumped from ~17-19s (pre-flip
baseline) to ~33-35s on this run. This is **endpoint contention**
on Hermes-3 at the time of measurement, not a feature regression
(the new substrate adds zero LLM calls — preflight is pure regex).
The throughput numbers in §6 of the journal pre-date this evening's
network conditions; future bench cycles should normalize.

**Verdict — no regression.** The #000010 default flip ships clean
across the full corpus, not just the broad-quantifier subset where
it was originally validated. Defaults stay on. The substrate is
strictly more honest under preflight (FORMAT_COLLAPSED → 0,
NO_EVIDENCE_POINTER ↓) without sacrificing throughput on
non-broad questions.

Bench artifact: `bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}`.

### Addendum 4 — metacog-trigger detector validation (2026-05-04T02-18-42Z)

The validations above (Addendum 1-3) covered the broad-quantifier
subset and the full corpus, neither of which contains questions
that fire the #000010 metacog detectors (temporal sensitivity,
contradiction, false-premise-lite, out-of-corpus). This addendum
closes that empirical gap.

Fixture: `bench/qa_questions_metacog_subset.txt` — 6 questions,
one per detector kind plus a control:

```
who is the current CEO of OpenAI?               # stale_risk
which character is alive and dead?              # contradictory
when did George Washington stop being president  # false_premise +
  of France?                                    #   stale_risk (multi)
what does my uploaded contract say about        # out_of_corpus
  clause 9?
when did the current CEO of Twitter stop        # stale_risk +
  being CEO of Tesla?                           #   false_premise
who painted the mona lisa?                      # control (well_formed)
```

Run: n=3 × 6 questions × 3 modes = 54 rows.

**Per-question verdict matrix:**

| question (detector)       | quote   | pointer  | JSON    |
|---------------------------|---------|----------|---------|
| Current CEO (stale_risk)  | 0/0/3 U | 0/0/3 U  | 0/0/3 U |
| Alive+dead (contradictory)| 0/0/3 U | 3/0/0 S  | 3/0/0 S |
| Wash. stop France (FP)    | 0/0/3 U | 0/3/0 H  | 3/0/0 S |
| Uploaded contract (OOC)   | 0/3/0 H | 0/1/2 U  | 3/0/0 S |
| Twitter→Tesla (multi)     | 0/0/3 U | 3/0/0 S  | 0/0/3 U |
| Mona Lisa (control)       | 3/0/0 S | 1/2/0 S  | 3/0/0 S |

(S/H/U = STRICT/HYBRID/UNGROUNDED; n=3 each cell.)

**Findings:**

1. **Detector accuracy is 6/6.** All trigger questions fire the
   expected `logical_statuses` value during preflight (verified
   programmatically before the bench: `stale_risk`,
   `contradictory_question`, `false_premise_suspected`,
   `out_of_corpus_risk` — matched 1:1 with fixture intent). The
   detectors are doing what their unit tests claim.

2. **Quote mode is the most honest fallback.** 4 of 5 trigger
   questions land all-UNGROUNDED on quote mode. The paraphrase
   verifier won't substring-match across the corpus when the
   question's premise has no anchor. Quote mode's mode-gated-off
   guard works in our favor here.

3. **Lattice modes accidentally ground 2 trigger questions to
   STRICT.** Schrödinger's cat (alive+dead) JSON STRICT is
   *defensible* — the corpus contains quantum-mechanics articles
   that legitimately discuss the state. But:

   - JSON STRICT on **"when did George Washington stop being
     president of France?"** is **NOT defensible**. False premise;
     the model invented an answer that lexically grounded against
     some chunk. The metacog detector correctly flagged
     `false_premise_suspected`; the audit-line tail surfaced
     `· false premise`; but the verdict still lands STRICT.
   - JSON STRICT on **"what does my uploaded contract say about
     clause 9?"** is the same shape: out-of-corpus reference,
     model fabricates a grounding.

4. **Audit-line tails are doing operator-warning duty correctly.**
   The bench rows persist `preflight_logical_statuses` and the
   render layer tails (`· stale risk`, `· false premise`, etc.)
   even on STRICT verdicts — operator sees the warning. But the
   substrate doesn't refuse execution by default for these shapes.

**Implication for #000011 (SOFT_PREFLIGHT_HINT).** This bench
validates the design rationale for the soft-sidecar ticket: the
deterministic metacog detectors flag these shapes correctly, but
the corpus accidentally grounds 2/5 of them to STRICT. A model-
assisted soft preflight could add independent semantic skepticism
("does George Washington being president of France match
historical reality?") that the lexical detectors can't supply.
Soft sidecar output would surface as `· soft: false_premise_*`
on the audit-line, distinct from the hard `· false premise` tail,
giving operators a stronger warning when both signals fire.

**Implication for default policy.** Keep
`metacognition_block_on_contradiction=False` as the default.
Schrödinger's cat (alive+dead) would have been rejected
unnecessarily under a hard-block, and that's a real-world
question with a legitimate answer. The label-only default is
correct; operators wanting strictness opt in via
`--block-on-contradiction`.

Bench artifact: `bench/qa_results/2026-05-04T02-18-42Z.{jsonl,md}`.


### Addendum 5 — #000046 Phase 3 verifier numeric-gate regression check (2026-05-11)

`arborist/qa/verify.py` gained a paraphrase numeric-agreement gate
(`_numeric_signature` + a check in `_check_each_with_paraphrase`):
a span that token-covers the source ≥ `paraphrase_coverage` but
asserts a digit-number the source lacks (modulo thousands-comma) is
no longer paraphrase-grounded — it goes to `unverified`. Closes
#000046 by lifting the `falsification-hard-v1.jsonl` rate 4/12 → 6/12
(`5f-fal-hard-004` 50-vs-100 and `-007` 300-vs-300,000 → UNGROUNDED).

Before/after `make bench-qa` (n=3 × 75 questions × 3 modes = 675
cells):

| mode | STRICT-rate before → after | Δ |
|------|----------------------------|---|
| quote | 0.53 → 0.50 | −3pp |
| claim_lattice_pointer | 0.23 → 0.25 | +2pp |
| claim_lattice | 0.44 → 0.45 | +1pp |

All within the 5-pp noise floor. Per-row diff (675 common cells, 74
changed audit_mode): the *only* clearly gate-attributable QA shift
was the fictional "our cold fusion breakthrough" year-claim demoting
STRICT → HYBRID across all 3 samples — a **correct** demotion (the
year isn't grounded). Every other transition was `quote→quote` /
`claim_lattice→claim_lattice` LLM re-answer variance — the answer
text changed on the re-run, not the verifier (the gate touches only
the paraphrase fallback, never the verbatim/span/entity/claim-lattice
paths). No regression on legitimate answers; the gate ships.

Bench artifacts: `bench/qa_results/2026-05-11T13-42-38Z.{jsonl,md}`
(before) · `bench/qa_results/2026-05-11T14-19-51Z.{jsonl,md}` (after).
Full per-ticket detail: `docs/tickets/ticket-000046-harder-5sf-fixture-tier.md`
§5 Phase 3.


### Addendum 6 — #000048 step 2.1 verifier entity-gate regression check (2026-05-11)

`arborist/qa/verify.py` gained an entity salient-token-disagreement
gate (`_entity_salient_disagrees` + `_is_single_sentence`): in
`verify_quotes`' entity branch (proximity policy), the *weakest*
grounding — `not cluster AND len(verified) <= 1 AND single sentence` —
is declined (→ UNGROUNDED) when the answer asserts a > 4-char
capitalized content token (stopword-filtered) or a digit-number the
source lacks. Catches "Insulin was discovered by Alexander Fleming"
against "Penicillin was discovered by Alexander Fleming" (shared
"Alexander Fleming" matched; swapped subject "Insulin" trips the
gate). Lifts the `falsification-hard-v1.jsonl` rate 6/12 → 10/12 (the
4 HYBRID_ENTITY over-grounds: Insulin / Berlin / 1889 / Pacific).

Before/after `make bench-qa` (n=3 × 75 questions × 3 modes = 675
cells):

| mode | STRICT-rate before → after | Δ |
|------|----------------------------|---|
| quote | 0.50 → 0.54 | +4pp |
| claim_lattice_pointer | 0.25 → 0.22 | −3pp |
| claim_lattice | 0.45 → 0.43 | −2pp |

All within the 5-pp noise floor. Per-row diff (675 common cells, 30
quote-mode rows changed audit_mode): **0 quote-mode rows demoted to
UNGROUNDED from the entity path** — the gate didn't fire on a single
legitimate QA answer in the whole bench. Every quote-mode transition
was LLM re-answer variance (`verifier quote→quote` with the verdict
flipping = a different answer text); the pointer/lattice deltas are
noise too (the gate is in `verify_quotes` / quote mode, not the
claim-lattice verifier). The gate is provably narrow on real traffic;
it ships.

Bench artifacts: `bench/qa_results/2026-05-11T14-19-51Z.{jsonl,md}`
(before — HEAD's verify.py) · `bench/qa_results/2026-05-11T17-12-41Z.{jsonl,md}`
(after). Full per-ticket detail: `docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md`
§5 step 2.1.


### Addendum 7 — #000048 step 2.4 parse_pointer_claims clause-segmentation regression check (2026-05-11)

`arborist/qa/parse_claims.py` gained a clause segmenter: a line that
crams several well-pointered claims onto one row is split into one
claim per clause (split on `;`, sentence boundaries, spaced dashes,
` and `/` or `/` because `/` although `/` since `/` while `, inline
`(N)` enumeration markers, commas — with `(?![^\[]*\])` so a comma
inside a `[E1, E2]` bracket never splits it). `_segment_line` keeps
the split *only if every resulting segment is well-pointered* (a legit
single claim like "The cat is black and white [E1]." or "The cast: A,
B, C [E1]." is never broken — splitting would create pointer-less
fragments → guard rejects; a leading colon-terminated header — "Two
facts:", "Key points:" — is dropped). Plus a wrapped-bullet join: a
continuation line (leading whitespace then a lowercase letter, no
bullet) folds into the previous claim. Closes the 8 mis-segments in
`formulate-hard-v1.jsonl` → rate 4/12 → 12/12 (that pack now at
ceiling).

Before/after `make bench-qa` (n=3 × 75 questions × 3 modes = 675
cells; `parse_pointer_claims` feeds the 450 `claim_lattice_pointer` +
`claim_lattice` cells):

| mode | STRICT-rate before → after | Δ |
|------|----------------------------|---|
| quote | 0.54 → 0.55 | +1pp |
| claim_lattice_pointer | 0.22 → 0.22 | 0pp |
| claim_lattice | 0.43 → 0.45 | +2pp |

All within the 5-pp noise floor. Per-row diff (675 common cells): the
segmenter changed the parsed-claim count *on the same answer text*
for **7 of the 450 lattice cells** (0 in `claim_lattice`, 7 in
`claim_lattice_pointer`); of those, 2 caused an `audit_mode` change —
both **correct**: (a) a wrap-join recovered an answer's intended
structure (4 claims, 2 of which were pointer-less wrap-fragments →
HYBRID) into 2 well-pointered claims → STRICT; (b) a crammed-one-line
blob (1 monolithic claim, all pointers → STRICT) split into 8 claims,
some of which don't individually verify → HYBRID — the honest verdict
(false-positive STRICT was the corruption). Every other lattice/quote
delta is LLM re-answer variance (`answer_chars` changed, often
drastically). No regression — the segmenter's only visible effects on
real traffic are honest improvements.

Bench artifacts: `bench/qa_results/2026-05-11T17-12-41Z.{jsonl,md}`
(before — HEAD's parse_claims.py) · `bench/qa_results/2026-05-11T20-26-37Z.{jsonl,md}`
(after). Full per-ticket detail: `docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md`
§5 step 2.4.

### Addendum 8 — #000057 control sweep: retrieval × model × framing × reasoning (2026-05-19/20)

**The question.** Hermes-3-8B confidently states the *present-day*
office-holder ("the president of France is Emmanuel Macron") against a
~2010-vintage corpus (Sarkozy). Is that an 8B-model weakness, an unfair
question framing, or does retrieval fix it regardless? The sweep
crosses **{model} × {framing} × {retrieval on/off}** over a 386-item
office-holder fixture (`bench/qa_questions_stale_map.json`), each item
with a fixed corpus-vintage gold article.

- **models:** `hermes` (Hermes-3-8B), `qwen-nothink` (Qwen3.6-27B,
  reasoning off), `qwen-think` (same weights, reasoning on). Qwen runs
  on llama.cpp; Hermes on vLLM.
- **framings:** `plain` ("who is the president of France?"),
  `source_relative` ("According to the reference knowledge base, …"),
  `as_of_corpus` ("As of 2010, who was …").
- **arms:** `solo` (model alone — measures parametric prior) vs
  `arborist` (model + retrieval over the 2010 corpus, `answer_mode=
  claim_lattice`).

**Judge methodology — the code judge.** The original sweep used the
Opus headless judge (`bench/judge.py`); it burned our Anthropic quota
and **79.5 % of its verdicts came back JUDGE_ERROR** (rate-limited).
Replaced with a deterministic code judge (`bench/judge_code.py`):
abstention regex → short-answer entity-grounding fast path → NLI
contradiction (θ=0.85) → lexical verifier (quote/span/entity/
paraphrase) → WRONG-vs-FABRICATED tie-break on subject-in-gold. **No
LLM, no quota, fully replayable.** Calibrated against the records Opus
*did* grade: agreement rose CG 13 %→47 %, WRONG 56 %→89 %, ABSTAINED
80 %→95 % across four targeted fixes (raise NLI contradiction
threshold off the manifest's 0.5; expand abstention patterns for the
"I do not have … information/access" family; demote FABRICATED→WRONG
when gold mentions the question subject; short-answer entity-grounding
for terse-name answers the verifier's prose-shape extractor whiffs on).
A claim-lattice JSON-unwrap lets the same judge grade the Arborist
arm's `{"claims":[…]}` envelopes as plain prose. Verdicts are a
deterministic *proxy* for grounding (lexical + NLI), not Opus-grade
reading — JUDGE_ERROR residue (HYBRID-via-entity with no NLI
corroboration) is the natural input to a later LLM-batch pass.

**Consolidated scorecard — CG% (correct-grounded rate), all arms on
the identical calibrated judge.** `arborist/hermes` is n=40 (phase 1's
smaller arborist arm); every other cell is n=386.

| arm / model | plain | source_relative | as_of_corpus |
|-------------|-------|-----------------|--------------|
| solo / hermes | 9 % | 5 % | 18 % |
| solo / qwen-nothink | 7 % | 0 % | **50 %** |
| solo / qwen-think | 6 % | 5 % | 44 % |
| arborist / hermes (n=40) | 60 % | 62 % | 25 % |
| **arborist / qwen-nothink** | **82 %** | **65 %** | 50 % |

Grounding-fidelity detail (abstain% / wrong-assert% = W+F over n) for
the cells the findings turn on:

| cell | CG% | abstain% | wrong-assert% |
|------|-----|----------|---------------|
| solo qwen-nothink / source_relative | 0 % | **97 %** | 1 % |
| solo qwen-think / source_relative | 5 % | 61 % | 32 % |
| solo qwen-nothink / plain | 7 % | 9 % | 79 % |
| solo qwen-think / plain | 6 % | 43 % | 48 % |
| arborist qwen-nothink / plain | 82 % | 0 % | 11 % |

**Findings.**

1. **Retrieval dominates every other lever.** arborist/qwen-nothink/
   plain = 82 % CG vs 7 % solo; arborist/hermes/plain = 60 % vs 9 %
   solo. No solo configuration — no model size, no framing — approaches
   the retrieval arms. The production answer is retrieval.

2. **Reasoning does not improve raw correctness.** solo qwen-think vs
   qwen-nothink: as_of_corpus **44 % vs 50 %** (nothink wins), plain
   6 %/7 %, source_relative 5 %/0 %. Chain-of-thought trades correctness
   for caution; it does not recall corpus-vintage facts better.

3. **Reasoning's real cost is broken honest-abstention.** The cleanest
   grounding signal in the sweep is solo qwen-nothink/source_relative:
   **97 % abstain** — told to use a reference it lacks, it declines.
   Reasoning *breaks* this: qwen-think/source_relative abstains only
   61 % and reasons itself into wrong parametric answers (wrong-assert
   1 %→32 %). On plain framing reasoning helps caution (wrong-assert
   79 %→48 %); on the source-grounding framing it hurts.

**Production recommendation:** arborist + qwen-nothink, plain framing —
82 % CG, ~0 % abstain, 11 % wrong-assert. Reasoning OFF: it degrades
the honest-abstention property and buys nothing once retrieval supplies
the source.

**Held / in flight:** arborist + qwen-think (the last cell) is running
at write time — tests whether reasoning hurts the retrieval arm the way
it hurt solo/source_relative. Will append the result.

**Artifacts.** Solo + arborist-hermes: `control_sweep_2026-05-19T21-52-56Z*`
(phase 1). qwen-think solo: `…23-22-21Z*` (phase 2). arborist+qwen-nothink:
`…2026-05-20T00-07-38Z*` (phase 3). All re-graded under the final judge
(`*_code_judge_final.{md,jsonl}`). Judge + sweep tooling:
`bench/judge_code.py`, `bench/score_with_code_judge.py`,
`bench/analyze_judge_disagreement.py`, `bench/control_sweep.py`
(`--judge {code,opus}`, `--skip-solo`). Engine-agnostic JSON-schema
enforcement for the Arborist arm:
`arborist.qa.verify.claim_lattice_structured_output_extras`.