docs/user-payload-layout: reflect #000068 Phase 1+2+3 shipped state

Five sections updated to match the post-2026-05-27 substrate state
(the user-payload-layout work, sibling ticket #000068, and the
2026-05-27 bench evidence are all in tree).

Verdict block (top): ADD: companion missed-answer guard -> DONE:
companion guard shipped as #000068 Phase 1+2+3 (default OFF; Phase
4 default flip NO-GO until wider bench + human spot-check).

Companion missed-answer guard section: renamed from "(proposed
sidecar)" to "(shipped 2026-05-27)". Carries the implementation
location (arborist/qa/inspect.py:diagnose_missed_answer), the full
output schema (diagnostic_version / confidence_class /
triggered_clauses / subject_tokens / missed_answer_candidate_spans
with offset_start/end/basis), the Phase 2 bench headline (2/228
fires, both Ballestrini, 100% precision, 0 FPs across 226 non-
Ballestrini), the Phase 3 demote-flag CLI surface
(--demote-on-missed-answer, default OFF), and corrected hash
discipline: Phase 1 sidecar fields fold into governance_policy_hash
only; the Phase 3 demote flag (answerability_demote_enabled) ALSO
folds into verifier_policy_hash because flipping it changes the
rendered audit_mode (a verifier-output property).

Future hardening list: "companion missed-answer falsification
guard" entry now points at #000068 instead of describing a
deterministic-sidecar to be built.

Roadmap Phase 5: "DESIGN OPEN" -> "DONE 2026-05-27" with all four
sub-phases of #000068 named individually (Phase 1 sidecar commit
2ab11d2, Phase 2 bench + Phase 3 demote commit ec55db5, Phase 4
default flip NO-GO per Dav1d 2026-05-27 §3.4). The #000068 phase
numbering is internal to that ticket; this roadmap names the
external-facing milestones.

Related links: past-tense the missed-answer guard ("when it lands"
-> "shipped 2026-05-27"); added cross-reference to
docs/tickets/ticket-000068-*.md; named both Dav1d review files by
path so a re-read can locate the inputs.

AUTOCOUNT tag (76 fixture-rows in bench/qa_questions.txt) still
matches; no test changes.

2026-05-27 10:55:38 -04:00

24 KiB

Raw Blame History

User-payload layout — where the question sits relative to evidence

A prompt-structure knob that decides where the question text appears in the final user-turn message relative to the evidence block. Pure lever on the LLM's attention budget — no change to the verifier, the audit chain, or the corpus.

Policy field: user_payload_layout ∈ {"tail", "bookend", "per_chunk"}, default "tail". CLI: --user-payload-layout on arborist query and arborist ask. Make: LAYOUT=... on make query. Folds into governance_policy_hash so each layout cache-partitions cleanly.

Verdict

GO:    static layouts shipped as opt-in policy knobs.
NO-GO: promote bookend/per_chunk as default. The n=3×75q curated
       bench (2026-05-27) found bookend statistically indistinguish-
       able from tail and per_chunk regressing −9.78pp on aggregate
       STRICT-rate against Hermes-3-8B. See §"Bench results".
DONE:  companion missed-answer falsification guard shipped as
       Ticket #000068 (Phase 1 sidecar + Phase 2 bench evidence +
       Phase 3 opt-in demote flag). Default OFF; Phase 4 default
       flip blocks on wider bench + human spot-check. See §"Verifier-
       blind missed-answer class" and docs/tickets/ticket-000068-*.md.

Layout is a real but narrow lever — useful for specific failure shapes (small-model evidence-negation under long context), not a universal default. Keep tail default; route bookend/per_chunk via the broad-quantifier classifier on query types that match the target failure mode. The verifier-side companion guard (#000068) catches the residual failure class layout doesn't fully close.

Why this exists — the Ballestrini case

Same query, same corpus, same retrieval, same evidence, same model (hermes-3-8B, https://hermes.ai.unturf.com/v1), same 22.8 KB prompt. The default tail layout produced:

Veronica Ballestrini is a country music singer and songwriter who has released several songs. However, the specific songs by her are not mentioned in the provided evidence blocks.

Evidence block E2 literally contained the song names: "Amazing", "Out There Somewhere", "Fascinated", and references to music videos for "What's Up With That" and "Don't Say". The model received an evidence chunk packed with answers and synthesized a negation.

The verifier marked the run EVIDENCE-WARRANTED because the negation literally matched no claim that needed grounding. The guard fires on false positives — claims unsupported by evidence — not on false negatives — answers the model declined to give. From the verifier's view the run was clean; from a user's view it was broken.

Root cause is the classic lost-in-the-middle / lost-at-the-front attention failure mode on small (≤8B) models. The final user message under tail was shaped:

EVIDENCE:

=== E1 (Veronica Ballestrini | primary_answer_source) ===
<span 1>

=== E2 (Veronica Ballestrini | primary_answer_source) ===
<span 2 — contains the song names>

=== E3 ... E8 ===
<spans 3-8>

---

QUESTION: songs by veronica ballestrini

The question is 28 characters at the tail of a 21,176-char evidence wall. Hermes-3-8B's attention budget loses track of which question it was asked while reading through eight evidence chunks. By the time generation begins, the model's prior is "summarize the evidence I just read" — and the evidence as a whole is mostly biographical prose, so the summary leans biographical and skirts the specific-song question.

Three layouts

`tail` (default — preserves prior cache)

EVIDENCE:
<body>
---
QUESTION: <q>

Original shape. Adequate for large models with long-context attention (Qwen-27B, Claude, GPT-4, etc.). Brittle on small models with long evidence blocks.

`bookend` (top + bottom)

QUESTION: <q>

EVIDENCE:
<body>
---
QUESTION: <q>

Question is repeated before AND after the evidence. The leading copy seeds the attention heads on the actual ask; the trailing copy re-anchors right before generation. This is the standard lost-in-the-middle mitigation (Liu et al. 2023, "Lost in the Middle: How Language Models Use Long Contexts").

Effect on Hermes-3-8B for the Ballestrini case: model goes from "specific songs are not mentioned" to extracting "Fascinated", "Don't Say", album "What I'm All About", producer Cliff Downs. Trade-off observed in this run: the model became over-eager and conflated Veronica Ballestrini ↔ The Veronicas (Australian pop rock duo whose articles were also in the top-8 retrieval), pulling "This Love", "Revolution" into the answer as if they were Ballestrini songs.

`per_chunk` (bookend + per-block reminder)

QUESTION: <q>

EVIDENCE:

[for: <q>]

=== E1 (...) ===
<span 1>

[for: <q>]

=== E2 (...) ===
<span 2>

[for: <q>]

=== E3 ... E8 ===
<spans 3-8>

---

QUESTION: <q>

Bookend, plus a one-line [for: <question>] reminder injected before each evidence block (cheap — <28 chars × N blocks on a 22 KB prompt). The question reactivates per-chunk; each chunk's attention window has the question text immediately preceding it.

Effect on Hermes-3-8B for the Ballestrini case: model correctly extracts "Amazing" as Ballestrini's single, then disambiguates The Veronicas as a separate Australian pop rock duo with their own songs ("Popular", "This Love", "Revolution", "Leave Me Alone"), and additionally surfaces Elvis Costello's "Veronica" as a third distinct entity. 6/8 evidence sources used (vs 1/8 for tail, 3/8 for bookend). No false attribution.

The per_chunk reminder is the question text verbatim, not a paraphrase — keeping it byte-identical so the model's attention heads activate on the same tokens repeatedly. Reminders are not fired for the leading block in lattice mode (the leading bookend copy already covers it) — they appear before every internal block boundary (\n\n=== ).

Where layout matters — model-size lever

Measured 2026-05-26 on the same Ballestrini query, same retrieval, three layouts × two models (n=1 burn each — directional, not defensible):

Model	tail	bookend	per_chunk
Hermes-3-8B	broken — negates evidence	works, but conflates Ballestrini ↔ The Veronicas	works, correctly disambiguates three entities
Qwen-27B	works	works (~identical output to tail)	works (~identical output to tail)

Qwen-27B is unaffected by layout on this query — its attention budget is large enough that the question stays salient through 21 KB of evidence regardless of where it appears. Hermes-3-8B is broken under tail and recovered by either bookend or per_chunk on this specific case.

Layout matters most for small models on long-context list/ extraction queries. For larger models, layout may be behaviorally near-neutral in observed output — but it is not cache-neutral or cost-neutral: every layout change shifts governance_policy_hash, changes cache_key, alters token count, and can affect latency & answer wording even when correctness is unchanged.

Bench results — 2026-05-27

Curated QA set, 76 questions (75 at sweep start + 1 Ballestrini regression added mid-session — won't affect already-run sweep results), n=3 samples per cell, claim_lattice mode, Hermes-3-8B via https://hermes.ai.unturf.com/v1, three layouts sequentially with concurrency=4. 225 runs per layout, 675 total LLM calls. Output under bench/qa_results/layout-{tail,bookend,per_chunk}/.

Layout	STRICT	HYBRID	UNGROUNDED	strict-rate	mean ratio	mean latency	Δ vs tail
tail	94	83	48	0.418	0.696	12.8s	— (control)
bookend	95	83	47	0.422	0.726	11.9s	+0.44pp (noise)
per_chunk	72	112	41	0.320	0.734	12.6s	−9.78pp (real regression)

Per the bench-maxing rule (5pp signal floor at n=3):

bookend ≈ tail. 1 STRICT delta on 225 runs is noise. mean ratio (n_verified / n_quotes) improved 0.696 → 0.726, but the strict-rate verdict is the gate. Bookend is a safe no-op aggregate — no regression, no win.
per_chunk regresses significantly. −22 STRICT, +29 HYBRID. Per-chunk reminder over-anchors the model into citing more evidence per claim:
- TOO_MANY_EVIDENCE_IDS violations: tail 20 → bookend 28 → per_chunk 54
- mean answer chars in the 32-64 KB prompt bucket: tail 720 → bookend 728 → per_chunk 1,660 (2.3× longer answers)
- WARRANT_MISSING stable across all three (~28-30), so the issue is claim-volume inflation, not deflection or warrant failure.
Per-prompt-size bucket (claim_lattice, n_runs in brackets):

bucket tail [n] bookend [n] per_chunk [n]

16-32 KB 0.40 [141] 0.41 [141] 0.34 [138]

32-64 KB 0.42 [81] 0.46 [81] 0.26 [84]

Bookend nudges +4pp on the largest-prompt bucket (marginal, ~1.5σ). Per_chunk collapses by −16pp on the same bucket. Layout's aggregate effect is opposite to what the n=1 Ballestrini anecdote suggested.

bucket	tail [n]	bookend [n]	per_chunk [n]
16-32 KB	0.40 [141]	0.41 [141]	0.34 [138]
32-64 KB	0.42 [81]	0.46 [81]	0.26 [84]

Honest read

The Ballestrini case (small-model evidence-negation under long-context tail) is a real failure mode but rare in the curated set. Most questions don't trigger that specific attention failure, so layout fixes don't move the aggregate needle. per_chunk fixes the rare case at the cost of ~10pp aggregate STRICT — a bad trade as a default. bookend is a wash.

Decision: keep tail as the default. Ship the policy field as an operator opt-in. The Ballestrini case becomes a regression fixture in bench/qa_questions.txt. Two follow-up options:

Adaptive routing: gate per_chunk behind the broad-quantifier classifier (arborist/qa/quantifier.py) so only list/extraction queries see the per-chunk reminder, narrow factoids stay on tail. The classifier already exists and folds into governance_policy_hash.
Filtered re-bench: run the same n=3 sweep on the "entity list" + "broad descriptive" + "bounded universals" subsections of qa_questions.txt only. Confirms whether per_chunk is a net positive when the query shape actually matches the failure class it targets, before wiring (1).

(1) without (2) risks installing a class-gated layout that helps the gated class but hasn't been measured net-positive even within the gated class. (2) is the prudent gate before (1).

Caveats

per_chunk broadens recall on small models — and that broadening is the regression. The per-chunk reminder activates attention on every evidence chunk, including chunks that are semantically unrelated. Good for "the answer is in chunk 7 of 8" cases; harmful when chunks 4–6 are off-topic. The Rule 8 title-mismatch verifier check catches the worst of these (drops STRICT → HYBRID), but the answer text still grows longer and more digressive. The 2026-05-27 bench confirms this is a net negative on the curated set.
Quote-mode (answer_mode=quote) has no per-chunk boundary. The body is the flat Sources: ... context string. per_chunk falls back to bookend in quote mode (same format_user_payload helper, per_chunk_marker=None for the runner's single-document path or per_chunk_marker="=== Source: " for query's multi-source context — see arborist/qa/prompts.py).
Cache scope. Flipping the layout changes governance_policy_hash → changes cache_key → all prior cached answers are bypassed on lookup. The corpus (documents, chunks, FTS5 index, audit chain) is content-addressed and unaffected. Re-ingestion is a no-op (same document_root → idempotent upsert).
Verifier untouched. verifier_policy_hash does not depend on user_payload_layout. The same hard checks (quote / span / entity / paraphrase verification, Rule 8 title-relevance, Rule 9 subject-tokens-absent, claim-count ceiling) run regardless of layout. Layout is a model-input knob, not a verification knob.

Verifier-blind missed-answer class

The Ballestrini failure under tail is not a hallucination. It is a false negative:

Evidence contains the answer.
Model says the evidence does not contain the answer.
Verifier sees no unsupported positive claim → marks run clean.
User receives a false negative under EVIDENCE-WARRANTED.

This is a verifier-blind class. The existing layered verifier (quote / span / entity / paraphrase + Rule 8 / Rule 9 / claim ceiling) guards against unsupported positive claims — it has no hook for unsupported absences. Layout fixes the attention placement that produced this specific instance, but layout alone cannot close the class — a sufficiently large prompt or an adversarial phrasing can resurface the failure under any layout.

Companion missed-answer guard — Ticket #000068 (shipped 2026-05-27)

Shipped as a deterministic sidecar in arborist/qa/inspect.py: diagnose_missed_answer. Fires only when all three clauses hold:

(A) answer contains a denial/absence pattern (sealed v1 list:
    "not mentioned", "not provided", "the evidence does not say",
    "does not mention", "no specific", "no evidence", "cannot
    determine from the provided evidence", "is not stated",
    "is not specified")
(B) question is extraction/list shape (broad-quantifier classifier
    intensity ∈ {ALL, COMPREHENSIVE, OPEN_REQUEST, MANY, PLURAL}
    OR surface cue match: "songs by" / "works by" / "who wrote" /
    "what year" / "list of" / "name all" / …)
(C) evidence contains candidate spans (quoted strings / title-case
    spans / comma-list items / year / date) within a proximity
    window (default 600 chars) of CLEANED subject tokens (cue and
    relation words like "songs", "by", "who" stripped before
    proximity matching — without this the guard false-triggers on
    generic title-case spans near the question)

Output is result["answerability"] — None when the conjunction doesn't fire, else a structured dict:

diagnostic_version            "missed-answer-v1"
answerability_warning         True
confidence_class              "weak" | "medium" | "strong"
triggered_clauses             {denial, extraction_shape,
                               candidate_proximity}
denial_pattern_matched        "not mentioned"
extraction_cue_matched        "songs by"
extraction_shape              "list" | "single_entity" | "open_list"
answer_type                   "title_like" | "person" | "date"
subject_tokens                ["veronica", "ballestrini"]
candidate_count               total candidates found
missed_answer_candidate_spans top-10 with offset_start / offset_end /
                              offset_basis="evidence_object_text" /
                              candidate_kind / nearest_subject_token /
                              subject_proximity_chars

Phase 2 measurement (bench/qa_results/phase2-sidecar-on/2026-05- 27T14-16-22Z): 228 runs, Hermes-3-8B, claim_lattice, tail layout. 2/228 fires (0.88%), both strong confidence, both the Ballestrini regression fixture. Precision 100%, zero false positives across the 226 non-Ballestrini runs.

Phase 3 opt-in demote — --demote-on-missed-answer on arborist query/ask. When set, the renderer demotes EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL on lattice modes for strong/medium-confidence fires; lower rungs and non-lattice modes get a · missed-answer tail tag. Default OFF; Phase 4 default flip NO-GO until wider bench + human spot-check.

Discipline:

Sidecar, not verifier hook. Same pattern as arborist.qa.inspect.diagnose_* (deflection / coherence / title-relevance) — read-only, never writes providence_cache or audit_events.
Hash partitioning: sidecar-mode fields (answerability_sidecar_enabled, answerability_threshold, denial_patterns_version, extraction_cues_version) fold into governance_policy_hash only. The Phase 3 demote flag (answerability_demote_enabled) ALSO folds into verifier_policy_hash because flipping it changes the rendered audit_mode — a verifier-output property, so the verifier hash legitimately partitions.
Never promotes claims. The trigger conjunction makes claim promotion structurally impossible — guard only fires on denial answers.

5F-Falsification fixture: the Ballestrini case is exactly the kind of failure that selects for adding a new falsifier into the substrate, then propagates the new fixture forward. The fixture is in bench/qa_questions.txt under "entity list" with a 4-line context comment pointing back here.

Usage

# Per-call override
arborist query --user-payload-layout bookend "songs by veronica ballestrini"
arborist query --user-payload-layout per_chunk "songs by veronica ballestrini"

# Make
make query Q="songs by veronica ballestrini" LAYOUT=bookend BURN=1
make query Q="songs by veronica ballestrini" LAYOUT=per_chunk BURN=1

# Policy override in Python
policy = dict(DEFAULT_QUERY_POLICY)
policy["user_payload_layout"] = "per_chunk"

Implementation map

arborist/qa/prompts.py — format_user_payload(question, body, *, layout, evidence_label, question_label, per_chunk_marker). Single source of truth; raises on unknown layout. USER_PAYLOAD_LAYOUTS constant lists valid values.
arborist/qa/query.py — multi-source retrieval path. Three _user_payload closures (one per answer_mode) all call format_user_payload. Policy default added to DEFAULT_QUERY_POLICY.
arborist/qa/runner.py — single-document path. Three _user_payload closures all call format_user_payload. Policy default added to DEFAULT_POLICY.
arborist/cli.py — --user-payload-layout flag on both query and ask subcommands; flows into call_policy.
Makefile — LAYOUT= plumbing on make query.

Future hardening (not yet implemented)

These are followups flagged in the 2026-05-27 de-novo review. Not required to ship the opt-in policy knob, but worth doing before any default promotion or adaptive routing lands.

Bounded reminder text. per_chunk currently echoes the question verbatim before each evidence block. Long or adversarial questions can bloat the prompt and widen the instruction-injection surface. Suggested signature extension:
```
def format_user_payload(
    question: str,
    body: str,
    *,
    layout: str = "tail",
    ...
    max_reminder_chars: int = 512,       # truncation cap
    reminder_mode: str = "text",         # "text" | "qid"
) -> str: ...
```
reminder_mode="qid" would emit [for qid=<question_hash>] — preserves per-chunk anchor identity without re-inserting arbitrary user text N times. Trade-off: fewer lexical attention tokens. Worth A/B-benching against reminder_mode="text" on the filtered list-shape subset.
Structured evidence-block injection. Current per_chunk implementation uses body.replace("\n\n=== ", ...) — works because render_evidence_map is deterministic, but a structured alternative (format_user_payload(..., evidence_blocks=[...])) would be more robust to upstream rendering changes. Adopt if/when the evidence map's block boundary convention changes.
Companion missed-answer falsification guard — shipped as Ticket #000068 Phase 1+2+3. See §"Companion missed-answer guard" above and docs/tickets/ticket-000068-*.md.

Roadmap

Phased gating — current state in bold.

Phase 0 — ticket finalization
   Open layout ticket. Open companion missed-answer guard ticket
   (#000068).
   Default = tail.

Phase 1 — implementation
   **DONE 2026-05-26 (commit 5674107).** format_user_payload helper,
   six callsites, CLI + Make plumbing, governance_policy_hash
   partitioning, USER_PAYLOAD_LAYOUTS constant, unknown-layout
   raises ValueError.

Phase 2 — regression fixtures
   **DONE 2026-05-27.** Ballestrini case added to bench/qa_questions.txt
   under entity-list section with 4-line context comment.
   Pending: The Veronicas disambiguation fixture, genuine-absence
   fixture, quote-mode fallback fixture, governance/verifier hash
   partitioning unit tests.

Phase 3 — bench
   **DONE 2026-05-27.** Three-layout sweep on curated 75-question
   set, claim_lattice mode, n=3, Hermes-3-8B. Result: tail/bookend
   are within noise; per_chunk regresses −9.78pp aggregate. See
   §"Bench results". Qwen-27B sanity check still open.

Phase 4 — selection
   **NO-GO on default promotion.** Tail remains default.
   Pending decision: adaptive routing via the broad-quantifier
   classifier (per_chunk only on list/extraction queries), gated
   by a filtered-subset re-bench to confirm net positive within
   the gated class.

Phase 5 — missed-answer guard (Ticket #000068)
   **DONE 2026-05-27.**
   - #000068 Phase 1 (sidecar): commit 2ab11d2. diagnose_missed_
     answer in arborist/qa/inspect.py with the three-clause
     conjunction, 36 tests, end-to-end live verified on Ballestrini.
   - #000068 Phase 2 (bench): commit ec55db5. bench/qa_sweep.py
     instrumented to surface answerability fires aggregate; 228-run
     bench at bench/qa_results/phase2-sidecar-on/ shows 2 fires,
     both Ballestrini, both strong confidence, 100% precision.
   - #000068 Phase 3 (opt-in demote): commit ec55db5. --demote-on-
     missed-answer flag on `arborist query`/`ask`, default OFF.
     answerability_demote_enabled folded into _VERIFIER_POLICY_FIELDS;
     renderer demotes EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL
     for strong/medium fires.
   - #000068 Phase 4 (default demote-on): **NO-GO** until wider
     bench + human spot-check confirms low FP rate across diverse
     failure shapes. Per Dav1d 2026-05-27 §3.4: "a false sidecar
     warning is tolerable; a false audit-label demotion can damage
     trust in correct abstentions."

docs/qa-modes-bench.md — three answer modes (quote / claim_lattice_pointer / claim_lattice). Layout is orthogonal to mode and folds into the same governance_policy_hash.
docs/bench-maxing.md — bench discipline. The 2026-05-27 sweep applied the 5pp signal floor against n=3, and the verdict (no default promotion) followed directly from that rule.
docs/seven-point-program.md — north-star directives. Layout is a D6/D7 (verifier discipline + small-model robustness) follow-up, not a new directive. The missed-answer guard (#000068, shipped 2026-05-27) extends D2 (pointer-grounding) by covering unsupported absence alongside unsupported presence — same falsifier discipline, new failure shape.
docs/tickets/ticket-000068-verifier-blind-missed-answer-guard.md — sibling ticket. Layout is the model-input lever; #000068 is the verifier-side adjunct. Read together for the full picture of the Ballestrini failure and the substrate's response to it.
2026-05-27 de-novo reviews by Dav1dPrometheus (private working docs at ~/Downloads/response_-_user-payload-layout.txt and ~/Downloads/RESPONSE_final_ticket-000068-verifier-blind-missed- answer-guard.txt) — split this work into two tickets: layout (this doc) + companion missed-answer guard (#000068). Verdict, errata, hardenings, and roadmap phasing above incorporate both reviews.

24 KiB Raw Blame History Unescape Escape