russell@unturf.com
e497a9501a
Five sections updated to match the post-2026-05-27 substrate state (the user-payload-layout work, sibling ticket #000068, and the 2026-05-27 bench evidence are all in tree). Verdict block (top): ADD: companion missed-answer guard -> DONE: companion guard shipped as #000068 Phase 1+2+3 (default OFF; Phase 4 default flip NO-GO until wider bench + human spot-check). Companion missed-answer guard section: renamed from "(proposed sidecar)" to "(shipped 2026-05-27)". Carries the implementation location (arborist/qa/inspect.py:diagnose_missed_answer), the full output schema (diagnostic_version / confidence_class / triggered_clauses / subject_tokens / missed_answer_candidate_spans with offset_start/end/basis), the Phase 2 bench headline (2/228 fires, both Ballestrini, 100% precision, 0 FPs across 226 non- Ballestrini), the Phase 3 demote-flag CLI surface (--demote-on-missed-answer, default OFF), and corrected hash discipline: Phase 1 sidecar fields fold into governance_policy_hash only; the Phase 3 demote flag (answerability_demote_enabled) ALSO folds into verifier_policy_hash because flipping it changes the rendered audit_mode (a verifier-output property). Future hardening list: "companion missed-answer falsification guard" entry now points at #000068 instead of describing a deterministic-sidecar to be built. Roadmap Phase 5: "DESIGN OPEN" -> "DONE 2026-05-27" with all four sub-phases of #000068 named individually (Phase 1 sidecar commit2ab11d2, Phase 2 bench + Phase 3 demote commitec55db5, Phase 4 default flip NO-GO per Dav1d 2026-05-27 §3.4). The #000068 phase numbering is internal to that ticket; this roadmap names the external-facing milestones. Related links: past-tense the missed-answer guard ("when it lands" -> "shipped 2026-05-27"); added cross-reference to docs/tickets/ticket-000068-*.md; named both Dav1d review files by path so a re-read can locate the inputs. AUTOCOUNT tag (76 fixture-rows in bench/qa_questions.txt) still matches; no test changes.
554 lines
24 KiB
Markdown
554 lines
24 KiB
Markdown
# User-payload layout — where the question sits relative to evidence
|
||
|
||
A prompt-structure knob that decides where the question text appears
|
||
in the final user-turn message relative to the evidence block. Pure
|
||
lever on the LLM's attention budget — no change to the verifier, the
|
||
audit chain, or the corpus.
|
||
|
||
Policy field: `user_payload_layout ∈ {"tail", "bookend", "per_chunk"}`,
|
||
default `"tail"`. CLI: `--user-payload-layout` on `arborist query`
|
||
and `arborist ask`. Make: `LAYOUT=...` on `make query`. Folds into
|
||
`governance_policy_hash` so each layout cache-partitions cleanly.
|
||
|
||
## Verdict
|
||
|
||
```
|
||
GO: static layouts shipped as opt-in policy knobs.
|
||
NO-GO: promote bookend/per_chunk as default. The n=3×75q curated
|
||
bench (2026-05-27) found bookend statistically indistinguish-
|
||
able from tail and per_chunk regressing −9.78pp on aggregate
|
||
STRICT-rate against Hermes-3-8B. See §"Bench results".
|
||
DONE: companion missed-answer falsification guard shipped as
|
||
Ticket #000068 (Phase 1 sidecar + Phase 2 bench evidence +
|
||
Phase 3 opt-in demote flag). Default OFF; Phase 4 default
|
||
flip blocks on wider bench + human spot-check. See §"Verifier-
|
||
blind missed-answer class" and docs/tickets/ticket-000068-*.md.
|
||
```
|
||
|
||
Layout is a real but narrow lever — useful for specific failure
|
||
shapes (small-model evidence-negation under long context), not a
|
||
universal default. Keep `tail` default; route `bookend`/`per_chunk`
|
||
via the broad-quantifier classifier on query types that match the
|
||
target failure mode. The verifier-side companion guard (#000068)
|
||
catches the residual failure class layout doesn't fully close.
|
||
|
||
## Why this exists — the Ballestrini case
|
||
|
||
Same query, same corpus, same retrieval, same evidence, same model
|
||
(`hermes-3-8B`, `https://hermes.ai.unturf.com/v1`), same 22.8 KB
|
||
prompt. The default `tail` layout produced:
|
||
|
||
> Veronica Ballestrini is a country music singer and songwriter who
|
||
> has released several songs. **However, the specific songs by her
|
||
> are not mentioned in the provided evidence blocks.**
|
||
|
||
Evidence block `E2` literally contained the song names: "Amazing",
|
||
"Out There Somewhere", "Fascinated", and references to music videos
|
||
for "What's Up With That" and "Don't Say". The model received an
|
||
evidence chunk packed with answers and synthesized a negation.
|
||
|
||
The verifier marked the run `EVIDENCE-WARRANTED` because the
|
||
negation literally matched no claim that needed grounding. The
|
||
guard fires on false positives — claims unsupported by evidence —
|
||
not on false negatives — answers the model declined to give. From
|
||
the verifier's view the run was clean; from a user's view it was
|
||
broken.
|
||
|
||
Root cause is the classic lost-in-the-middle / lost-at-the-front
|
||
attention failure mode on small (≤8B) models. The final user
|
||
message under `tail` was shaped:
|
||
|
||
```
|
||
EVIDENCE:
|
||
|
||
=== E1 (Veronica Ballestrini | primary_answer_source) ===
|
||
<span 1>
|
||
|
||
=== E2 (Veronica Ballestrini | primary_answer_source) ===
|
||
<span 2 — contains the song names>
|
||
|
||
=== E3 ... E8 ===
|
||
<spans 3-8>
|
||
|
||
---
|
||
|
||
QUESTION: songs by veronica ballestrini
|
||
```
|
||
|
||
The question is 28 characters at the tail of a 21,176-char evidence
|
||
wall. Hermes-3-8B's attention budget loses track of which question
|
||
it was asked while reading through eight evidence chunks. By the
|
||
time generation begins, the model's prior is "summarize the
|
||
evidence I just read" — and the evidence as a whole is mostly
|
||
biographical prose, so the summary leans biographical and skirts
|
||
the specific-song question.
|
||
|
||
## Three layouts
|
||
|
||
### `tail` (default — preserves prior cache)
|
||
|
||
```
|
||
EVIDENCE:
|
||
<body>
|
||
---
|
||
QUESTION: <q>
|
||
```
|
||
|
||
Original shape. Adequate for large models with long-context attention
|
||
(Qwen-27B, Claude, GPT-4, etc.). Brittle on small models with long
|
||
evidence blocks.
|
||
|
||
### `bookend` (top + bottom)
|
||
|
||
```
|
||
QUESTION: <q>
|
||
|
||
EVIDENCE:
|
||
<body>
|
||
---
|
||
QUESTION: <q>
|
||
```
|
||
|
||
Question is repeated before AND after the evidence. The leading
|
||
copy seeds the attention heads on the actual ask; the trailing copy
|
||
re-anchors right before generation. This is the standard
|
||
lost-in-the-middle mitigation (Liu et al. 2023, "Lost in the
|
||
Middle: How Language Models Use Long Contexts").
|
||
|
||
Effect on Hermes-3-8B for the Ballestrini case: model goes from
|
||
*"specific songs are not mentioned"* to extracting "Fascinated",
|
||
"Don't Say", album "What I'm All About", producer Cliff Downs.
|
||
Trade-off observed in this run: the model became over-eager and
|
||
conflated Veronica Ballestrini ↔ The Veronicas (Australian pop
|
||
rock duo whose articles were also in the top-8 retrieval), pulling
|
||
"This Love", "Revolution" into the answer as if they were
|
||
Ballestrini songs.
|
||
|
||
### `per_chunk` (bookend + per-block reminder)
|
||
|
||
```
|
||
QUESTION: <q>
|
||
|
||
EVIDENCE:
|
||
|
||
[for: <q>]
|
||
|
||
=== E1 (...) ===
|
||
<span 1>
|
||
|
||
[for: <q>]
|
||
|
||
=== E2 (...) ===
|
||
<span 2>
|
||
|
||
[for: <q>]
|
||
|
||
=== E3 ... E8 ===
|
||
<spans 3-8>
|
||
|
||
---
|
||
|
||
QUESTION: <q>
|
||
```
|
||
|
||
Bookend, plus a one-line `[for: <question>]` reminder injected
|
||
before each evidence block (cheap — `<28` chars × `N` blocks on a
|
||
22 KB prompt). The question reactivates per-chunk; each chunk's
|
||
attention window has the question text immediately preceding it.
|
||
|
||
Effect on Hermes-3-8B for the Ballestrini case: model correctly
|
||
extracts "Amazing" as Ballestrini's single, then **disambiguates**
|
||
The Veronicas as a separate Australian pop rock duo with their own
|
||
songs ("Popular", "This Love", "Revolution", "Leave Me Alone"),
|
||
and additionally surfaces Elvis Costello's "Veronica" as a third
|
||
distinct entity. 6/8 evidence sources used (vs 1/8 for tail, 3/8
|
||
for bookend). No false attribution.
|
||
|
||
The `per_chunk` reminder is the question text verbatim, not a
|
||
paraphrase — keeping it byte-identical so the model's attention
|
||
heads activate on the same tokens repeatedly. Reminders are not
|
||
fired for the leading block in lattice mode (the leading bookend
|
||
copy already covers it) — they appear before every internal block
|
||
boundary (`\n\n=== `).
|
||
|
||
## Where layout matters — model-size lever
|
||
|
||
Measured 2026-05-26 on the same Ballestrini query, same retrieval,
|
||
three layouts × two models (n=1 burn each — directional, not
|
||
defensible):
|
||
|
||
| Model | tail | bookend | per_chunk |
|
||
|--------------|---------------------------------------|----------------------------------------------------|------------------------------------------------|
|
||
| Hermes-3-8B | broken — negates evidence | works, but conflates Ballestrini ↔ The Veronicas | works, correctly disambiguates three entities |
|
||
| Qwen-27B | works | works (~identical output to tail) | works (~identical output to tail) |
|
||
|
||
Qwen-27B is unaffected by layout on this query — its attention
|
||
budget is large enough that the question stays salient through 21
|
||
KB of evidence regardless of where it appears. Hermes-3-8B is
|
||
broken under `tail` and recovered by either `bookend` or
|
||
`per_chunk` on this specific case.
|
||
|
||
Layout matters most for **small models on long-context list/
|
||
extraction queries**. For larger models, layout may be behaviorally
|
||
near-neutral in observed output — but it is **not cache-neutral or
|
||
cost-neutral**: every layout change shifts `governance_policy_hash`,
|
||
changes `cache_key`, alters token count, and can affect latency &
|
||
answer wording even when correctness is unchanged.
|
||
|
||
## Bench results — 2026-05-27
|
||
|
||
Curated QA set, <!--AUTOCOUNT:fixture-rows:bench/qa_questions.txt-->76<!--/AUTOCOUNT-->
|
||
questions (75 at sweep start + 1 Ballestrini regression added
|
||
mid-session — won't affect already-run sweep results), n=3 samples
|
||
per cell, claim_lattice mode,
|
||
Hermes-3-8B via `https://hermes.ai.unturf.com/v1`, three layouts
|
||
sequentially with concurrency=4. 225 runs per layout, 675 total
|
||
LLM calls. Output under `bench/qa_results/layout-{tail,bookend,per_chunk}/`.
|
||
|
||
| Layout | STRICT | HYBRID | UNGROUNDED | strict-rate | mean ratio | mean latency | Δ vs tail |
|
||
|-------------|--------|--------|------------|-------------|------------|--------------|------------------------|
|
||
| tail | 94 | 83 | 48 | 0.418 | 0.696 | 12.8s | — (control) |
|
||
| bookend | 95 | 83 | 47 | 0.422 | 0.726 | 11.9s | +0.44pp (noise) |
|
||
| per_chunk | 72 | 112 | 41 | 0.320 | 0.734 | 12.6s | **−9.78pp** (real regression) |
|
||
|
||
Per the bench-maxing rule (5pp signal floor at n=3):
|
||
|
||
- **bookend ≈ tail.** 1 STRICT delta on 225 runs is noise.
|
||
`mean ratio` (n_verified / n_quotes) improved 0.696 → 0.726, but
|
||
the strict-rate verdict is the gate. Bookend is a safe no-op
|
||
aggregate — no regression, no win.
|
||
- **per_chunk regresses significantly.** −22 STRICT, +29 HYBRID.
|
||
Per-chunk reminder over-anchors the model into citing more
|
||
evidence per claim:
|
||
- `TOO_MANY_EVIDENCE_IDS` violations: tail 20 → bookend 28 →
|
||
**per_chunk 54**
|
||
- mean answer chars in the 32-64 KB prompt bucket: tail 720 →
|
||
bookend 728 → **per_chunk 1,660** (2.3× longer answers)
|
||
- `WARRANT_MISSING` stable across all three (~28-30), so the
|
||
issue is **claim-volume inflation**, not deflection or
|
||
warrant failure.
|
||
- **Per-prompt-size bucket** (claim_lattice, n_runs in brackets):
|
||
|
||
| bucket | tail [n] | bookend [n] | per_chunk [n] |
|
||
|----------|----------|-------------|---------------|
|
||
| 16-32 KB | 0.40 [141] | 0.41 [141] | 0.34 [138] |
|
||
| 32-64 KB | 0.42 [81] | 0.46 [81] | 0.26 [84] |
|
||
|
||
Bookend nudges +4pp on the largest-prompt bucket (marginal,
|
||
~1.5σ). Per_chunk collapses by −16pp on the same bucket.
|
||
Layout's aggregate effect is opposite to what the n=1
|
||
Ballestrini anecdote suggested.
|
||
|
||
### Honest read
|
||
|
||
The Ballestrini case (small-model evidence-negation under
|
||
long-context tail) is a **real failure mode but rare in the
|
||
curated set**. Most questions don't trigger that specific
|
||
attention failure, so layout fixes don't move the aggregate
|
||
needle. `per_chunk` fixes the rare case at the cost of ~10pp
|
||
aggregate STRICT — a bad trade as a default. `bookend` is a wash.
|
||
|
||
**Decision**: keep `tail` as the default. Ship the policy field
|
||
as an operator opt-in. The Ballestrini case becomes a regression
|
||
fixture in `bench/qa_questions.txt`. Two follow-up options:
|
||
|
||
1. **Adaptive routing**: gate `per_chunk` behind the
|
||
broad-quantifier classifier (`arborist/qa/quantifier.py`) so
|
||
only list/extraction queries see the per-chunk reminder,
|
||
narrow factoids stay on `tail`. The classifier already exists
|
||
and folds into `governance_policy_hash`.
|
||
2. **Filtered re-bench**: run the same n=3 sweep on the
|
||
"entity list" + "broad descriptive" + "bounded universals"
|
||
subsections of `qa_questions.txt` only. Confirms whether
|
||
`per_chunk` is a net positive *when the query shape actually
|
||
matches* the failure class it targets, before wiring (1).
|
||
|
||
(1) without (2) risks installing a class-gated layout that helps
|
||
the gated class but hasn't been measured net-positive even
|
||
within the gated class. (2) is the prudent gate before (1).
|
||
|
||
## Caveats
|
||
|
||
- **`per_chunk` broadens recall on small models — and that
|
||
broadening is the regression.** The per-chunk reminder activates
|
||
attention on every evidence chunk, including chunks that are
|
||
semantically unrelated. Good for "the answer is in chunk 7 of 8"
|
||
cases; harmful when chunks 4–6 are off-topic. The Rule 8
|
||
title-mismatch verifier check catches the worst of these (drops
|
||
STRICT → HYBRID), but the answer text still grows longer and more
|
||
digressive. The 2026-05-27 bench confirms this is a net negative
|
||
on the curated set.
|
||
- **Quote-mode (`answer_mode=quote`) has no per-chunk boundary.**
|
||
The body is the flat `Sources: ...` context string. `per_chunk`
|
||
falls back to `bookend` in quote mode (same `format_user_payload`
|
||
helper, `per_chunk_marker=None` for the runner's single-document
|
||
path or `per_chunk_marker="=== Source: "` for query's multi-source
|
||
context — see `arborist/qa/prompts.py`).
|
||
- **Cache scope.** Flipping the layout changes
|
||
`governance_policy_hash` → changes `cache_key` → all prior
|
||
cached answers are bypassed on lookup. The corpus (`documents`,
|
||
`chunks`, FTS5 index, audit chain) is content-addressed and
|
||
unaffected. Re-ingestion is a no-op (same `document_root` →
|
||
idempotent upsert).
|
||
- **Verifier untouched.** `verifier_policy_hash` does not depend on
|
||
`user_payload_layout`. The same hard checks (quote / span /
|
||
entity / paraphrase verification, Rule 8 title-relevance, Rule 9
|
||
subject-tokens-absent, claim-count ceiling) run regardless of
|
||
layout. Layout is a model-input knob, not a verification knob.
|
||
|
||
## Verifier-blind missed-answer class
|
||
|
||
The Ballestrini failure under `tail` is not a hallucination. It
|
||
is a **false negative**:
|
||
|
||
```
|
||
Evidence contains the answer.
|
||
Model says the evidence does not contain the answer.
|
||
Verifier sees no unsupported positive claim → marks run clean.
|
||
User receives a false negative under EVIDENCE-WARRANTED.
|
||
```
|
||
|
||
This is a verifier-blind class. The existing layered verifier
|
||
(quote / span / entity / paraphrase + Rule 8 / Rule 9 / claim
|
||
ceiling) guards against *unsupported positive claims* — it has no
|
||
hook for *unsupported absences*. Layout fixes the attention placement
|
||
that produced this specific instance, but layout alone cannot
|
||
close the class — a sufficiently large prompt or an adversarial
|
||
phrasing can resurface the failure under any layout.
|
||
|
||
### Companion missed-answer guard — Ticket #000068 (shipped 2026-05-27)
|
||
|
||
Shipped as a deterministic sidecar in `arborist/qa/inspect.py:
|
||
diagnose_missed_answer`. Fires only when all three clauses hold:
|
||
|
||
```
|
||
(A) answer contains a denial/absence pattern (sealed v1 list:
|
||
"not mentioned", "not provided", "the evidence does not say",
|
||
"does not mention", "no specific", "no evidence", "cannot
|
||
determine from the provided evidence", "is not stated",
|
||
"is not specified")
|
||
(B) question is extraction/list shape (broad-quantifier classifier
|
||
intensity ∈ {ALL, COMPREHENSIVE, OPEN_REQUEST, MANY, PLURAL}
|
||
OR surface cue match: "songs by" / "works by" / "who wrote" /
|
||
"what year" / "list of" / "name all" / …)
|
||
(C) evidence contains candidate spans (quoted strings / title-case
|
||
spans / comma-list items / year / date) within a proximity
|
||
window (default 600 chars) of CLEANED subject tokens (cue and
|
||
relation words like "songs", "by", "who" stripped before
|
||
proximity matching — without this the guard false-triggers on
|
||
generic title-case spans near the question)
|
||
```
|
||
|
||
Output is `result["answerability"]` — None when the conjunction
|
||
doesn't fire, else a structured dict:
|
||
|
||
```
|
||
diagnostic_version "missed-answer-v1"
|
||
answerability_warning True
|
||
confidence_class "weak" | "medium" | "strong"
|
||
triggered_clauses {denial, extraction_shape,
|
||
candidate_proximity}
|
||
denial_pattern_matched "not mentioned"
|
||
extraction_cue_matched "songs by"
|
||
extraction_shape "list" | "single_entity" | "open_list"
|
||
answer_type "title_like" | "person" | "date"
|
||
subject_tokens ["veronica", "ballestrini"]
|
||
candidate_count total candidates found
|
||
missed_answer_candidate_spans top-10 with offset_start / offset_end /
|
||
offset_basis="evidence_object_text" /
|
||
candidate_kind / nearest_subject_token /
|
||
subject_proximity_chars
|
||
```
|
||
|
||
Phase 2 measurement (`bench/qa_results/phase2-sidecar-on/2026-05-
|
||
27T14-16-22Z`): 228 runs, Hermes-3-8B, claim_lattice, tail layout.
|
||
**2/228 fires (0.88%), both strong confidence, both the Ballestrini
|
||
regression fixture. Precision 100%, zero false positives across the
|
||
226 non-Ballestrini runs.**
|
||
|
||
Phase 3 opt-in demote — `--demote-on-missed-answer` on `arborist
|
||
query`/`ask`. When set, the renderer demotes
|
||
`EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL` on lattice modes for
|
||
strong/medium-confidence fires; lower rungs and non-lattice modes
|
||
get a `· missed-answer` tail tag. Default OFF; Phase 4 default flip
|
||
NO-GO until wider bench + human spot-check.
|
||
|
||
Discipline:
|
||
|
||
- **Sidecar, not verifier hook.** Same pattern as
|
||
`arborist.qa.inspect.diagnose_*` (deflection / coherence /
|
||
title-relevance) — read-only, never writes `providence_cache`
|
||
or `audit_events`.
|
||
- **Hash partitioning:** sidecar-mode fields
|
||
(`answerability_sidecar_enabled`, `answerability_threshold`,
|
||
`denial_patterns_version`, `extraction_cues_version`) fold into
|
||
`governance_policy_hash` only. The Phase 3 demote flag
|
||
(`answerability_demote_enabled`) ALSO folds into
|
||
`verifier_policy_hash` because flipping it changes the rendered
|
||
audit_mode — a verifier-output property, so the verifier hash
|
||
legitimately partitions.
|
||
- **Never promotes claims.** The trigger conjunction makes claim
|
||
promotion structurally impossible — guard only fires on denial
|
||
answers.
|
||
|
||
5F-Falsification fixture: the Ballestrini case is exactly the kind
|
||
of failure that selects for adding a new falsifier into the
|
||
substrate, then propagates the new fixture forward. The fixture is
|
||
in `bench/qa_questions.txt` under "entity list" with a 4-line
|
||
context comment pointing back here.
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
# Per-call override
|
||
arborist query --user-payload-layout bookend "songs by veronica ballestrini"
|
||
arborist query --user-payload-layout per_chunk "songs by veronica ballestrini"
|
||
|
||
# Make
|
||
make query Q="songs by veronica ballestrini" LAYOUT=bookend BURN=1
|
||
make query Q="songs by veronica ballestrini" LAYOUT=per_chunk BURN=1
|
||
|
||
# Policy override in Python
|
||
policy = dict(DEFAULT_QUERY_POLICY)
|
||
policy["user_payload_layout"] = "per_chunk"
|
||
```
|
||
|
||
## Implementation map
|
||
|
||
- `arborist/qa/prompts.py` — `format_user_payload(question, body, *,
|
||
layout, evidence_label, question_label, per_chunk_marker)`. Single
|
||
source of truth; raises on unknown layout. `USER_PAYLOAD_LAYOUTS`
|
||
constant lists valid values.
|
||
- `arborist/qa/query.py` — multi-source retrieval path. Three
|
||
`_user_payload` closures (one per answer_mode) all call
|
||
`format_user_payload`. Policy default added to
|
||
`DEFAULT_QUERY_POLICY`.
|
||
- `arborist/qa/runner.py` — single-document path. Three
|
||
`_user_payload` closures all call `format_user_payload`. Policy
|
||
default added to `DEFAULT_POLICY`.
|
||
- `arborist/cli.py` — `--user-payload-layout` flag on both `query`
|
||
and `ask` subcommands; flows into `call_policy`.
|
||
- `Makefile` — `LAYOUT=` plumbing on `make query`.
|
||
|
||
## Future hardening (not yet implemented)
|
||
|
||
These are followups flagged in the 2026-05-27 de-novo review. Not
|
||
required to ship the opt-in policy knob, but worth doing before
|
||
any default promotion or adaptive routing lands.
|
||
|
||
- **Bounded reminder text.** `per_chunk` currently echoes the
|
||
question verbatim before each evidence block. Long or
|
||
adversarial questions can bloat the prompt and widen the
|
||
instruction-injection surface. Suggested signature extension:
|
||
|
||
```python
|
||
def format_user_payload(
|
||
question: str,
|
||
body: str,
|
||
*,
|
||
layout: str = "tail",
|
||
...
|
||
max_reminder_chars: int = 512, # truncation cap
|
||
reminder_mode: str = "text", # "text" | "qid"
|
||
) -> str: ...
|
||
```
|
||
|
||
`reminder_mode="qid"` would emit `[for qid=<question_hash>]` —
|
||
preserves per-chunk anchor identity without re-inserting
|
||
arbitrary user text N times. Trade-off: fewer lexical attention
|
||
tokens. Worth A/B-benching against `reminder_mode="text"` on the
|
||
filtered list-shape subset.
|
||
|
||
- **Structured evidence-block injection.** Current `per_chunk`
|
||
implementation uses `body.replace("\n\n=== ", ...)` — works
|
||
because `render_evidence_map` is deterministic, but a structured
|
||
alternative (`format_user_payload(..., evidence_blocks=[...])`)
|
||
would be more robust to upstream rendering changes. Adopt if/when
|
||
the evidence map's block boundary convention changes.
|
||
|
||
- **Companion missed-answer falsification guard** — shipped as
|
||
Ticket #000068 Phase 1+2+3. See §"Companion missed-answer guard"
|
||
above and `docs/tickets/ticket-000068-*.md`.
|
||
|
||
## Roadmap
|
||
|
||
Phased gating — current state in **bold**.
|
||
|
||
```
|
||
Phase 0 — ticket finalization
|
||
Open layout ticket. Open companion missed-answer guard ticket
|
||
(#000068).
|
||
Default = tail.
|
||
|
||
Phase 1 — implementation
|
||
**DONE 2026-05-26 (commit 5674107).** format_user_payload helper,
|
||
six callsites, CLI + Make plumbing, governance_policy_hash
|
||
partitioning, USER_PAYLOAD_LAYOUTS constant, unknown-layout
|
||
raises ValueError.
|
||
|
||
Phase 2 — regression fixtures
|
||
**DONE 2026-05-27.** Ballestrini case added to bench/qa_questions.txt
|
||
under entity-list section with 4-line context comment.
|
||
Pending: The Veronicas disambiguation fixture, genuine-absence
|
||
fixture, quote-mode fallback fixture, governance/verifier hash
|
||
partitioning unit tests.
|
||
|
||
Phase 3 — bench
|
||
**DONE 2026-05-27.** Three-layout sweep on curated 75-question
|
||
set, claim_lattice mode, n=3, Hermes-3-8B. Result: tail/bookend
|
||
are within noise; per_chunk regresses −9.78pp aggregate. See
|
||
§"Bench results". Qwen-27B sanity check still open.
|
||
|
||
Phase 4 — selection
|
||
**NO-GO on default promotion.** Tail remains default.
|
||
Pending decision: adaptive routing via the broad-quantifier
|
||
classifier (per_chunk only on list/extraction queries), gated
|
||
by a filtered-subset re-bench to confirm net positive within
|
||
the gated class.
|
||
|
||
Phase 5 — missed-answer guard (Ticket #000068)
|
||
**DONE 2026-05-27.**
|
||
- #000068 Phase 1 (sidecar): commit 2ab11d2. diagnose_missed_
|
||
answer in arborist/qa/inspect.py with the three-clause
|
||
conjunction, 36 tests, end-to-end live verified on Ballestrini.
|
||
- #000068 Phase 2 (bench): commit ec55db5. bench/qa_sweep.py
|
||
instrumented to surface answerability fires aggregate; 228-run
|
||
bench at bench/qa_results/phase2-sidecar-on/ shows 2 fires,
|
||
both Ballestrini, both strong confidence, 100% precision.
|
||
- #000068 Phase 3 (opt-in demote): commit ec55db5. --demote-on-
|
||
missed-answer flag on `arborist query`/`ask`, default OFF.
|
||
answerability_demote_enabled folded into _VERIFIER_POLICY_FIELDS;
|
||
renderer demotes EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL
|
||
for strong/medium fires.
|
||
- #000068 Phase 4 (default demote-on): **NO-GO** until wider
|
||
bench + human spot-check confirms low FP rate across diverse
|
||
failure shapes. Per Dav1d 2026-05-27 §3.4: "a false sidecar
|
||
warning is tolerable; a false audit-label demotion can damage
|
||
trust in correct abstentions."
|
||
```
|
||
|
||
## Related
|
||
|
||
- `docs/qa-modes-bench.md` — three answer modes (quote /
|
||
claim_lattice_pointer / claim_lattice). Layout is orthogonal to
|
||
mode and folds into the same `governance_policy_hash`.
|
||
- `docs/bench-maxing.md` — bench discipline. The 2026-05-27 sweep
|
||
applied the 5pp signal floor against n=3, and the verdict (no
|
||
default promotion) followed directly from that rule.
|
||
- `docs/seven-point-program.md` — north-star directives. Layout is
|
||
a D6/D7 (verifier discipline + small-model robustness) follow-up,
|
||
not a new directive. The missed-answer guard (#000068, shipped
|
||
2026-05-27) extends **D2 (pointer-grounding)** by covering
|
||
*unsupported absence* alongside *unsupported presence* — same
|
||
falsifier discipline, new failure shape.
|
||
- `docs/tickets/ticket-000068-verifier-blind-missed-answer-guard.md`
|
||
— sibling ticket. Layout is the model-input lever; #000068 is the
|
||
verifier-side adjunct. Read together for the full picture of the
|
||
Ballestrini failure and the substrate's response to it.
|
||
- 2026-05-27 de-novo reviews by Dav1dPrometheus (private working
|
||
docs at `~/Downloads/response_-_user-payload-layout.txt` and
|
||
`~/Downloads/RESPONSE_final_ticket-000068-verifier-blind-missed-
|
||
answer-guard.txt`) — split this work into two tickets: layout
|
||
(this doc) + companion missed-answer guard (#000068). Verdict,
|
||
errata, hardenings, and roadmap phasing above incorporate both
|
||
reviews.
|