user_payload_layout: opt-in policy knob for question placement
format_user_payload helper in arborist/qa/prompts.py becomes the single
source of truth for the user-turn payload. Three layouts:
tail (default) evidence first, question at end (prior behavior)
bookend question repeated before AND after evidence — counters
lost-in-the-middle on small models (≤8B)
per_chunk bookend + a one-line [for: <q>] reminder before each
evidence block; for list/extraction queries
USER_PAYLOAD_LAYOUTS constant exported; unknown layout raises ValueError.
The six _user_payload closures in query.py (3) and runner.py (3) all
delegate to format_user_payload. Quote-mode passes per_chunk_marker=None
to fall back to bookend on flat document/sources context.
Wired through both DEFAULT_QUERY_POLICY and DEFAULT_POLICY. Folds into
governance_policy_hash (the layout changes the user-turn content the
model sees, so the policy hash partitions cleanly per layout); does NOT
fold into verifier_policy_hash (verifier rules unchanged).
Makefile gets LAYOUT_DEFAULT ?= tail and LAYOUT ?= $(LAYOUT_DEFAULT) so
operators can flip per-call (LAYOUT=bookend make query Q="...") or
session-wide (LAYOUT_DEFAULT=bookend make query Q="..."). Recommendation
matrix in the Makefile comment block above the query target encodes the
2026-05-27 bench finding.
Motivating case: the Veronica-Ballestrini "songs by" failure. Hermes-3-
8B under tail layout returned "specific songs by her are not mentioned
in the provided evidence" when evidence E2 literally contained the song
names. Same query under bookend recovered the answer (with conflation
between Veronica Ballestrini and The Veronicas); under per_chunk
recovered AND disambiguated three entities. Qwen-27B unaffected by
layout. The Ballestrini case is added to bench/qa_questions.txt as a
regression fixture under "entity list", with a 4-line comment pointing
to docs/user-payload-layout.md.
2026-05-27 bench (n=3 × 75q, claim_lattice mode, Hermes-3-8B):
tail STRICT 94/225 (0.418) — control
bookend STRICT 95/225 (0.422) — +0.44pp (noise, 5pp floor)
per_chunk STRICT 72/225 (0.320) — -9.78pp (significant regression)
Verdict: tail stays default (cache-preserving and bench-confirmed
neutral). Bookend/per_chunk available as opt-in operator knobs. Per_chunk
regresses in aggregate because the per-chunk reminder over-anchors the
model on every chunk (TOO_MANY_EVIDENCE_IDS violations rose from 20 →
54; mean answer chars in 32-64KB bucket doubled from 720 → 1660). The
Ballestrini-class failure is real but rare across the curated set; a
layout fix that helps the rare case at the cost of 10pp aggregate is a
bad default trade. Documented in full in docs/user-payload-layout.md
along with the Dav1d 2026-05-27 review framing (GO for opt-in, NO-GO
for default promotion, ADD companion missed-answer guard).
CLI changes (--user-payload-layout flag on `query` and `ask`) landed
separately in commit e5ee283 alongside the #54 busy_timeout fix.
This commit is contained in:
parent
e5ee28387e
commit
5674107c06
6 changed files with 625 additions and 9 deletions
23
Makefile
23
Makefile
|
|
@ -167,14 +167,31 @@ ANSWER_MODE ?= claim_lattice
|
|||
# tell-me-everything-about-X). Pairs cleanly with the broad-
|
||||
# quantifier reminder which is default-on for lattice modes.
|
||||
# Bench (#000008 §12.10): cap-on JSON wins +14pp on STRICT-rate.
|
||||
|
||||
# REJECT_BROAD=1 → strict reject for ALL/COMPREHENSIVE/OPEN_REQUEST
|
||||
# unbounded shapes; returns UNGROUNDED before the LLM call.
|
||||
# ALLOW_BROAD=1 → emergent search; classifier on, caps off.
|
||||
query: bootstrap ## ask the corpus a question [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N K="extra retrieval keywords" ANSWER_MODE=claim_lattice|claim_lattice_pointer|quote BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 WITNESS=1 XLANG=1 XLANG_MT=1]; JSON by default
|
||||
#
|
||||
# LAYOUT — user-payload-layout policy knob. Decides where the
|
||||
# question text sits relative to evidence in the final user-turn
|
||||
# message. See docs/user-payload-layout.md.
|
||||
# Recommendation matrix (2026-05-27 n=3 × 76q bench, Hermes-3-8B,
|
||||
# claim_lattice mode):
|
||||
# tail safe default — preserves prior cache (proven)
|
||||
# bookend safer for small-model recovery; n=3 aggregate bench
|
||||
# was a wash vs tail (+0.44pp STRICT, within 5pp floor)
|
||||
# per_chunk best on list/extraction shapes for small models
|
||||
# (the Ballestrini case) BUT regresses -9.78pp on
|
||||
# aggregate STRICT — opt-in only, never a default
|
||||
# Override per-call: LAYOUT=bookend make query Q="..."
|
||||
# Override session-wide: LAYOUT_DEFAULT=bookend make query Q="..."
|
||||
LAYOUT_DEFAULT ?= tail
|
||||
LAYOUT ?= $(LAYOUT_DEFAULT)
|
||||
query: bootstrap ## ask the corpus a question [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N K="extra retrieval keywords" ANSWER_MODE=claim_lattice|claim_lattice_pointer|quote LAYOUT=tail|bookend|per_chunk BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 WITNESS=1 XLANG=1 XLANG_MT=1]; JSON by default
|
||||
@if [ -z "$$Q" ] && [ -z "$(Q)" ]; then \
|
||||
echo "usage: make query Q=\"your question\" [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N K=\"extra retrieval keywords\" ANSWER_MODE=claim_lattice|claim_lattice_pointer|quote BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 WITNESS=1 XLANG=1 XLANG_MT=1]"; exit 2; \
|
||||
echo "usage: make query Q=\"your question\" [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N K=\"extra retrieval keywords\" ANSWER_MODE=claim_lattice|claim_lattice_pointer|quote LAYOUT=tail|bookend|per_chunk BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 WITNESS=1 XLANG=1 XLANG_MT=1]"; exit 2; \
|
||||
fi
|
||||
$(ARBORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) $(if $(JSON),--json,) $(if $(BURN),--burn,) $(if $(REPAIR),--repair,) $(if $(REPROMPTS),--repair-reprompts $(REPROMPTS),) $(if $(ANSWER_MODE),--answer-mode $(ANSWER_MODE),) $(if $(K),--retrieval-keywords "$(K)",) $(if $(BROAD),--apply-quantifier-caps,) $(if $(REJECT_BROAD),--reject-broad,) $(if $(ALLOW_BROAD),--allow-broad,) $(if $(WITNESS),--witness,) $(if $(XLANG),--crosslang-guard,) $(if $(XLANG_MT),--crosslang-translate,) "$(Q)"
|
||||
$(ARBORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) $(if $(JSON),--json,) $(if $(BURN),--burn,) $(if $(REPAIR),--repair,) $(if $(REPROMPTS),--repair-reprompts $(REPROMPTS),) $(if $(ANSWER_MODE),--answer-mode $(ANSWER_MODE),) --user-payload-layout $(LAYOUT) $(if $(K),--retrieval-keywords "$(K)",) $(if $(BROAD),--apply-quantifier-caps,) $(if $(REJECT_BROAD),--reject-broad,) $(if $(ALLOW_BROAD),--allow-broad,) $(if $(WITNESS),--witness,) $(if $(XLANG),--crosslang-guard,) $(if $(XLANG_MT),--crosslang-translate,) "$(Q)"
|
||||
|
||||
query-dry: bootstrap ## like 'make query' but skip the LLM call (dry-run) [JSON=1 BURN=1 ANSWER_MODE=... BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 XLANG=1 XLANG_MT=1]
|
||||
@if [ -z "$$Q" ] && [ -z "$(Q)" ]; then \
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue