# User-payload layout — where the question sits relative to evidence A prompt-structure knob that decides where the question text appears in the final user-turn message relative to the evidence block. Pure lever on the LLM's attention budget — no change to the verifier, the audit chain, or the corpus. Policy field: `user_payload_layout ∈ {"tail", "bookend", "per_chunk"}`, default `"tail"`. CLI: `--user-payload-layout` on `arborist query` and `arborist ask`. Make: `LAYOUT=...` on `make query`. Folds into `governance_policy_hash` so each layout cache-partitions cleanly. ## Verdict ``` GO: static layouts shipped as opt-in policy knobs. NO-GO: promote bookend/per_chunk as default. The n=3×75q curated bench (2026-05-27) found bookend statistically indistinguish- able from tail and per_chunk regressing −9.78pp on aggregate STRICT-rate against Hermes-3-8B. See §"Bench results". DONE: companion missed-answer falsification guard shipped as Ticket #000068 (Phase 1 sidecar + Phase 2 bench evidence + Phase 3 opt-in demote flag). Default OFF; Phase 4 default flip blocks on wider bench + human spot-check. See §"Verifier- blind missed-answer class" and docs/tickets/ticket-000068-*.md. ``` Layout is a real but narrow lever — useful for specific failure shapes (small-model evidence-negation under long context), not a universal default. Keep `tail` default; route `bookend`/`per_chunk` via the broad-quantifier classifier on query types that match the target failure mode. The verifier-side companion guard (#000068) catches the residual failure class layout doesn't fully close. ## Why this exists — the Ballestrini case Same query, same corpus, same retrieval, same evidence, same model (`hermes-3-8B`, `https://hermes.ai.unturf.com/v1`), same 22.8 KB prompt. The default `tail` layout produced: > Veronica Ballestrini is a country music singer and songwriter who > has released several songs. **However, the specific songs by her > are not mentioned in the provided evidence blocks.** Evidence block `E2` literally contained the song names: "Amazing", "Out There Somewhere", "Fascinated", and references to music videos for "What's Up With That" and "Don't Say". The model received an evidence chunk packed with answers and synthesized a negation. The verifier marked the run `EVIDENCE-WARRANTED` because the negation literally matched no claim that needed grounding. The guard fires on false positives — claims unsupported by evidence — not on false negatives — answers the model declined to give. From the verifier's view the run was clean; from a user's view it was broken. Root cause is the classic lost-in-the-middle / lost-at-the-front attention failure mode on small (≤8B) models. The final user message under `tail` was shaped: ``` EVIDENCE: === E1 (Veronica Ballestrini | primary_answer_source) === === E2 (Veronica Ballestrini | primary_answer_source) === === E3 ... E8 === --- QUESTION: songs by veronica ballestrini ``` The question is 28 characters at the tail of a 21,176-char evidence wall. Hermes-3-8B's attention budget loses track of which question it was asked while reading through eight evidence chunks. By the time generation begins, the model's prior is "summarize the evidence I just read" — and the evidence as a whole is mostly biographical prose, so the summary leans biographical and skirts the specific-song question. ## Three layouts ### `tail` (default — preserves prior cache) ``` EVIDENCE: --- QUESTION: ``` Original shape. Adequate for large models with long-context attention (Qwen-27B, Claude, GPT-4, etc.). Brittle on small models with long evidence blocks. ### `bookend` (top + bottom) ``` QUESTION: EVIDENCE: --- QUESTION: ``` Question is repeated before AND after the evidence. The leading copy seeds the attention heads on the actual ask; the trailing copy re-anchors right before generation. This is the standard lost-in-the-middle mitigation (Liu et al. 2023, "Lost in the Middle: How Language Models Use Long Contexts"). Effect on Hermes-3-8B for the Ballestrini case: model goes from *"specific songs are not mentioned"* to extracting "Fascinated", "Don't Say", album "What I'm All About", producer Cliff Downs. Trade-off observed in this run: the model became over-eager and conflated Veronica Ballestrini ↔ The Veronicas (Australian pop rock duo whose articles were also in the top-8 retrieval), pulling "This Love", "Revolution" into the answer as if they were Ballestrini songs. ### `per_chunk` (bookend + per-block reminder) ``` QUESTION: EVIDENCE: [for: ] === E1 (...) === [for: ] === E2 (...) === [for: ] === E3 ... E8 === --- QUESTION: ``` Bookend, plus a one-line `[for: ]` reminder injected before each evidence block (cheap — `<28` chars × `N` blocks on a 22 KB prompt). The question reactivates per-chunk; each chunk's attention window has the question text immediately preceding it. Effect on Hermes-3-8B for the Ballestrini case: model correctly extracts "Amazing" as Ballestrini's single, then **disambiguates** The Veronicas as a separate Australian pop rock duo with their own songs ("Popular", "This Love", "Revolution", "Leave Me Alone"), and additionally surfaces Elvis Costello's "Veronica" as a third distinct entity. 6/8 evidence sources used (vs 1/8 for tail, 3/8 for bookend). No false attribution. The `per_chunk` reminder is the question text verbatim, not a paraphrase — keeping it byte-identical so the model's attention heads activate on the same tokens repeatedly. Reminders are not fired for the leading block in lattice mode (the leading bookend copy already covers it) — they appear before every internal block boundary (`\n\n=== `). ## Where layout matters — model-size lever Measured 2026-05-26 on the same Ballestrini query, same retrieval, three layouts × two models (n=1 burn each — directional, not defensible): | Model | tail | bookend | per_chunk | |--------------|---------------------------------------|----------------------------------------------------|------------------------------------------------| | Hermes-3-8B | broken — negates evidence | works, but conflates Ballestrini ↔ The Veronicas | works, correctly disambiguates three entities | | Qwen-27B | works | works (~identical output to tail) | works (~identical output to tail) | Qwen-27B is unaffected by layout on this query — its attention budget is large enough that the question stays salient through 21 KB of evidence regardless of where it appears. Hermes-3-8B is broken under `tail` and recovered by either `bookend` or `per_chunk` on this specific case. Layout matters most for **small models on long-context list/ extraction queries**. For larger models, layout may be behaviorally near-neutral in observed output — but it is **not cache-neutral or cost-neutral**: every layout change shifts `governance_policy_hash`, changes `cache_key`, alters token count, and can affect latency & answer wording even when correctness is unchanged. ## Bench results — 2026-05-27 Curated QA set, 76 questions (75 at sweep start + 1 Ballestrini regression added mid-session — won't affect already-run sweep results), n=3 samples per cell, claim_lattice mode, Hermes-3-8B via `https://hermes.ai.unturf.com/v1`, three layouts sequentially with concurrency=4. 225 runs per layout, 675 total LLM calls. Output under `bench/qa_results/layout-{tail,bookend,per_chunk}/`. | Layout | STRICT | HYBRID | UNGROUNDED | strict-rate | mean ratio | mean latency | Δ vs tail | |-------------|--------|--------|------------|-------------|------------|--------------|------------------------| | tail | 94 | 83 | 48 | 0.418 | 0.696 | 12.8s | — (control) | | bookend | 95 | 83 | 47 | 0.422 | 0.726 | 11.9s | +0.44pp (noise) | | per_chunk | 72 | 112 | 41 | 0.320 | 0.734 | 12.6s | **−9.78pp** (real regression) | Per the bench-maxing rule (5pp signal floor at n=3): - **bookend ≈ tail.** 1 STRICT delta on 225 runs is noise. `mean ratio` (n_verified / n_quotes) improved 0.696 → 0.726, but the strict-rate verdict is the gate. Bookend is a safe no-op aggregate — no regression, no win. - **per_chunk regresses significantly.** −22 STRICT, +29 HYBRID. Per-chunk reminder over-anchors the model into citing more evidence per claim: - `TOO_MANY_EVIDENCE_IDS` violations: tail 20 → bookend 28 → **per_chunk 54** - mean answer chars in the 32-64 KB prompt bucket: tail 720 → bookend 728 → **per_chunk 1,660** (2.3× longer answers) - `WARRANT_MISSING` stable across all three (~28-30), so the issue is **claim-volume inflation**, not deflection or warrant failure. - **Per-prompt-size bucket** (claim_lattice, n_runs in brackets): | bucket | tail [n] | bookend [n] | per_chunk [n] | |----------|----------|-------------|---------------| | 16-32 KB | 0.40 [141] | 0.41 [141] | 0.34 [138] | | 32-64 KB | 0.42 [81] | 0.46 [81] | 0.26 [84] | Bookend nudges +4pp on the largest-prompt bucket (marginal, ~1.5σ). Per_chunk collapses by −16pp on the same bucket. Layout's aggregate effect is opposite to what the n=1 Ballestrini anecdote suggested. ### Honest read The Ballestrini case (small-model evidence-negation under long-context tail) is a **real failure mode but rare in the curated set**. Most questions don't trigger that specific attention failure, so layout fixes don't move the aggregate needle. `per_chunk` fixes the rare case at the cost of ~10pp aggregate STRICT — a bad trade as a default. `bookend` is a wash. **Decision**: keep `tail` as the default. Ship the policy field as an operator opt-in. The Ballestrini case becomes a regression fixture in `bench/qa_questions.txt`. Two follow-up options: 1. **Adaptive routing**: gate `per_chunk` behind the broad-quantifier classifier (`arborist/qa/quantifier.py`) so only list/extraction queries see the per-chunk reminder, narrow factoids stay on `tail`. The classifier already exists and folds into `governance_policy_hash`. 2. **Filtered re-bench**: run the same n=3 sweep on the "entity list" + "broad descriptive" + "bounded universals" subsections of `qa_questions.txt` only. Confirms whether `per_chunk` is a net positive *when the query shape actually matches* the failure class it targets, before wiring (1). (1) without (2) risks installing a class-gated layout that helps the gated class but hasn't been measured net-positive even within the gated class. (2) is the prudent gate before (1). ## Caveats - **`per_chunk` broadens recall on small models — and that broadening is the regression.** The per-chunk reminder activates attention on every evidence chunk, including chunks that are semantically unrelated. Good for "the answer is in chunk 7 of 8" cases; harmful when chunks 4–6 are off-topic. The Rule 8 title-mismatch verifier check catches the worst of these (drops STRICT → HYBRID), but the answer text still grows longer and more digressive. The 2026-05-27 bench confirms this is a net negative on the curated set. - **Quote-mode (`answer_mode=quote`) has no per-chunk boundary.** The body is the flat `Sources: ...` context string. `per_chunk` falls back to `bookend` in quote mode (same `format_user_payload` helper, `per_chunk_marker=None` for the runner's single-document path or `per_chunk_marker="=== Source: "` for query's multi-source context — see `arborist/qa/prompts.py`). - **Cache scope.** Flipping the layout changes `governance_policy_hash` → changes `cache_key` → all prior cached answers are bypassed on lookup. The corpus (`documents`, `chunks`, FTS5 index, audit chain) is content-addressed and unaffected. Re-ingestion is a no-op (same `document_root` → idempotent upsert). - **Verifier untouched.** `verifier_policy_hash` does not depend on `user_payload_layout`. The same hard checks (quote / span / entity / paraphrase verification, Rule 8 title-relevance, Rule 9 subject-tokens-absent, claim-count ceiling) run regardless of layout. Layout is a model-input knob, not a verification knob. ## Verifier-blind missed-answer class The Ballestrini failure under `tail` is not a hallucination. It is a **false negative**: ``` Evidence contains the answer. Model says the evidence does not contain the answer. Verifier sees no unsupported positive claim → marks run clean. User receives a false negative under EVIDENCE-WARRANTED. ``` This is a verifier-blind class. The existing layered verifier (quote / span / entity / paraphrase + Rule 8 / Rule 9 / claim ceiling) guards against *unsupported positive claims* — it has no hook for *unsupported absences*. Layout fixes the attention placement that produced this specific instance, but layout alone cannot close the class — a sufficiently large prompt or an adversarial phrasing can resurface the failure under any layout. ### Companion missed-answer guard — Ticket #000068 (shipped 2026-05-27) Shipped as a deterministic sidecar in `arborist/qa/inspect.py: diagnose_missed_answer`. Fires only when all three clauses hold: ``` (A) answer contains a denial/absence pattern (sealed v1 list: "not mentioned", "not provided", "the evidence does not say", "does not mention", "no specific", "no evidence", "cannot determine from the provided evidence", "is not stated", "is not specified") (B) question is extraction/list shape (broad-quantifier classifier intensity ∈ {ALL, COMPREHENSIVE, OPEN_REQUEST, MANY, PLURAL} OR surface cue match: "songs by" / "works by" / "who wrote" / "what year" / "list of" / "name all" / …) (C) evidence contains candidate spans (quoted strings / title-case spans / comma-list items / year / date) within a proximity window (default 600 chars) of CLEANED subject tokens (cue and relation words like "songs", "by", "who" stripped before proximity matching — without this the guard false-triggers on generic title-case spans near the question) ``` Output is `result["answerability"]` — None when the conjunction doesn't fire, else a structured dict: ``` diagnostic_version "missed-answer-v1" answerability_warning True confidence_class "weak" | "medium" | "strong" triggered_clauses {denial, extraction_shape, candidate_proximity} denial_pattern_matched "not mentioned" extraction_cue_matched "songs by" extraction_shape "list" | "single_entity" | "open_list" answer_type "title_like" | "person" | "date" subject_tokens ["veronica", "ballestrini"] candidate_count total candidates found missed_answer_candidate_spans top-10 with offset_start / offset_end / offset_basis="evidence_object_text" / candidate_kind / nearest_subject_token / subject_proximity_chars ``` Phase 2 measurement (`bench/qa_results/phase2-sidecar-on/2026-05- 27T14-16-22Z`): 228 runs, Hermes-3-8B, claim_lattice, tail layout. **2/228 fires (0.88%), both strong confidence, both the Ballestrini regression fixture. Precision 100%, zero false positives across the 226 non-Ballestrini runs.** Phase 3 opt-in demote — `--demote-on-missed-answer` on `arborist query`/`ask`. When set, the renderer demotes `EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL` on lattice modes for strong/medium-confidence fires; lower rungs and non-lattice modes get a `· missed-answer` tail tag. Default OFF; Phase 4 default flip NO-GO until wider bench + human spot-check. Discipline: - **Sidecar, not verifier hook.** Same pattern as `arborist.qa.inspect.diagnose_*` (deflection / coherence / title-relevance) — read-only, never writes `providence_cache` or `audit_events`. - **Hash partitioning:** sidecar-mode fields (`answerability_sidecar_enabled`, `answerability_threshold`, `denial_patterns_version`, `extraction_cues_version`) fold into `governance_policy_hash` only. The Phase 3 demote flag (`answerability_demote_enabled`) ALSO folds into `verifier_policy_hash` because flipping it changes the rendered audit_mode — a verifier-output property, so the verifier hash legitimately partitions. - **Never promotes claims.** The trigger conjunction makes claim promotion structurally impossible — guard only fires on denial answers. 5F-Falsification fixture: the Ballestrini case is exactly the kind of failure that selects for adding a new falsifier into the substrate, then propagates the new fixture forward. The fixture is in `bench/qa_questions.txt` under "entity list" with a 4-line context comment pointing back here. ## Usage ```bash # Per-call override arborist query --user-payload-layout bookend "songs by veronica ballestrini" arborist query --user-payload-layout per_chunk "songs by veronica ballestrini" # Make make query Q="songs by veronica ballestrini" LAYOUT=bookend BURN=1 make query Q="songs by veronica ballestrini" LAYOUT=per_chunk BURN=1 # Policy override in Python policy = dict(DEFAULT_QUERY_POLICY) policy["user_payload_layout"] = "per_chunk" ``` ## Implementation map - `arborist/qa/prompts.py` — `format_user_payload(question, body, *, layout, evidence_label, question_label, per_chunk_marker)`. Single source of truth; raises on unknown layout. `USER_PAYLOAD_LAYOUTS` constant lists valid values. - `arborist/qa/query.py` — multi-source retrieval path. Three `_user_payload` closures (one per answer_mode) all call `format_user_payload`. Policy default added to `DEFAULT_QUERY_POLICY`. - `arborist/qa/runner.py` — single-document path. Three `_user_payload` closures all call `format_user_payload`. Policy default added to `DEFAULT_POLICY`. - `arborist/cli.py` — `--user-payload-layout` flag on both `query` and `ask` subcommands; flows into `call_policy`. - `Makefile` — `LAYOUT=` plumbing on `make query`. ## Future hardening (not yet implemented) These are followups flagged in the 2026-05-27 de-novo review. Not required to ship the opt-in policy knob, but worth doing before any default promotion or adaptive routing lands. - **Bounded reminder text.** `per_chunk` currently echoes the question verbatim before each evidence block. Long or adversarial questions can bloat the prompt and widen the instruction-injection surface. Suggested signature extension: ```python def format_user_payload( question: str, body: str, *, layout: str = "tail", ... max_reminder_chars: int = 512, # truncation cap reminder_mode: str = "text", # "text" | "qid" ) -> str: ... ``` `reminder_mode="qid"` would emit `[for qid=]` — preserves per-chunk anchor identity without re-inserting arbitrary user text N times. Trade-off: fewer lexical attention tokens. Worth A/B-benching against `reminder_mode="text"` on the filtered list-shape subset. - **Structured evidence-block injection.** Current `per_chunk` implementation uses `body.replace("\n\n=== ", ...)` — works because `render_evidence_map` is deterministic, but a structured alternative (`format_user_payload(..., evidence_blocks=[...])`) would be more robust to upstream rendering changes. Adopt if/when the evidence map's block boundary convention changes. - **Companion missed-answer falsification guard** — shipped as Ticket #000068 Phase 1+2+3. See §"Companion missed-answer guard" above and `docs/tickets/ticket-000068-*.md`. ## Roadmap Phased gating — current state in **bold**. ``` Phase 0 — ticket finalization Open layout ticket. Open companion missed-answer guard ticket (#000068). Default = tail. Phase 1 — implementation **DONE 2026-05-26 (commit 5674107).** format_user_payload helper, six callsites, CLI + Make plumbing, governance_policy_hash partitioning, USER_PAYLOAD_LAYOUTS constant, unknown-layout raises ValueError. Phase 2 — regression fixtures **DONE 2026-05-27.** Ballestrini case added to bench/qa_questions.txt under entity-list section with 4-line context comment. Pending: The Veronicas disambiguation fixture, genuine-absence fixture, quote-mode fallback fixture, governance/verifier hash partitioning unit tests. Phase 3 — bench **DONE 2026-05-27.** Three-layout sweep on curated 75-question set, claim_lattice mode, n=3, Hermes-3-8B. Result: tail/bookend are within noise; per_chunk regresses −9.78pp aggregate. See §"Bench results". Qwen-27B sanity check still open. Phase 4 — selection **NO-GO on default promotion.** Tail remains default. Pending decision: adaptive routing via the broad-quantifier classifier (per_chunk only on list/extraction queries), gated by a filtered-subset re-bench to confirm net positive within the gated class. Phase 5 — missed-answer guard (Ticket #000068) **DONE 2026-05-27.** - #000068 Phase 1 (sidecar): commit 2ab11d2. diagnose_missed_ answer in arborist/qa/inspect.py with the three-clause conjunction, 36 tests, end-to-end live verified on Ballestrini. - #000068 Phase 2 (bench): commit ec55db5. bench/qa_sweep.py instrumented to surface answerability fires aggregate; 228-run bench at bench/qa_results/phase2-sidecar-on/ shows 2 fires, both Ballestrini, both strong confidence, 100% precision. - #000068 Phase 3 (opt-in demote): commit ec55db5. --demote-on- missed-answer flag on `arborist query`/`ask`, default OFF. answerability_demote_enabled folded into _VERIFIER_POLICY_FIELDS; renderer demotes EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL for strong/medium fires. - #000068 Phase 4 (default demote-on): **NO-GO** until wider bench + human spot-check confirms low FP rate across diverse failure shapes. Per Dav1d 2026-05-27 §3.4: "a false sidecar warning is tolerable; a false audit-label demotion can damage trust in correct abstentions." ``` ## Related - `docs/qa-modes-bench.md` — three answer modes (quote / claim_lattice_pointer / claim_lattice). Layout is orthogonal to mode and folds into the same `governance_policy_hash`. - `docs/bench-maxing.md` — bench discipline. The 2026-05-27 sweep applied the 5pp signal floor against n=3, and the verdict (no default promotion) followed directly from that rule. - `docs/seven-point-program.md` — north-star directives. Layout is a D6/D7 (verifier discipline + small-model robustness) follow-up, not a new directive. The missed-answer guard (#000068, shipped 2026-05-27) extends **D2 (pointer-grounding)** by covering *unsupported absence* alongside *unsupported presence* — same falsifier discipline, new failure shape. - `docs/tickets/ticket-000068-verifier-blind-missed-answer-guard.md` — sibling ticket. Layout is the model-input lever; #000068 is the verifier-side adjunct. Read together for the full picture of the Ballestrini failure and the substrate's response to it. - 2026-05-27 de-novo reviews by Dav1dPrometheus (private working docs at `~/Downloads/response_-_user-payload-layout.txt` and `~/Downloads/RESPONSE_final_ticket-000068-verifier-blind-missed- answer-guard.txt`) — split this work into two tickets: layout (this doc) + companion missed-answer guard (#000068). Verdict, errata, hardenings, and roadmap phasing above incorporate both reviews.