engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs/user-payload-layout: reflect #000068 Phase 1+2+3 shipped state

Browse source 
Five sections updated to match the post-2026-05-27 substrate state
(the user-payload-layout work, sibling ticket #000068, and the
2026-05-27 bench evidence are all in tree).

Verdict block (top): ADD: companion missed-answer guard -> DONE:
companion guard shipped as #000068 Phase 1+2+3 (default OFF; Phase
4 default flip NO-GO until wider bench + human spot-check).

Companion missed-answer guard section: renamed from "(proposed
sidecar)" to "(shipped 2026-05-27)". Carries the implementation
location (arborist/qa/inspect.py:diagnose_missed_answer), the full
output schema (diagnostic_version / confidence_class /
triggered_clauses / subject_tokens / missed_answer_candidate_spans
with offset_start/end/basis), the Phase 2 bench headline (2/228
fires, both Ballestrini, 100% precision, 0 FPs across 226 non-
Ballestrini), the Phase 3 demote-flag CLI surface
(--demote-on-missed-answer, default OFF), and corrected hash
discipline: Phase 1 sidecar fields fold into governance_policy_hash
only; the Phase 3 demote flag (answerability_demote_enabled) ALSO
folds into verifier_policy_hash because flipping it changes the
rendered audit_mode (a verifier-output property).

Future hardening list: "companion missed-answer falsification
guard" entry now points at #000068 instead of describing a
deterministic-sidecar to be built.

Roadmap Phase 5: "DESIGN OPEN" -> "DONE 2026-05-27" with all four
sub-phases of #000068 named individually (Phase 1 sidecar commit
2ab11d2, Phase 2 bench + Phase 3 demote commit ec55db5, Phase 4
default flip NO-GO per Dav1d 2026-05-27 §3.4). The #000068 phase
numbering is internal to that ticket; this roadmap names the
external-facing milestones.

Related links: past-tense the missed-answer guard ("when it lands"
-> "shipped 2026-05-27"); added cross-reference to
docs/tickets/ticket-000068-*.md; named both Dav1d review files by
path so a re-read can locate the inputs.

AUTOCOUNT tag (76 fixture-rows in bench/qa_questions.txt) still
matches; no test changes.
This commit is contained in:
russell@unturf.com 2026-05-27 10:55:38 -04:00
parent ec55db513c
commit e497a9501a
No known key found for this signature in database
1 changed files with 119 additions and 61 deletions
Download patch file Download diff file Expand all files Collapse all files

180
docs/user-payload-layout.md
View file

@ -13,22 +13,24 @@ and `arborist ask`. Make: `LAYOUT=...` on `make query`. Folds into
## Verdict
```
GO: implement static layouts as opt-in policy knobs.
GO: static layouts shipped as opt-in policy knobs.
NO-GO: promote bookend/per_chunk as default. The n=3×75q curated
bench (2026-05-27) found bookend statistically indistinguish-
able from tail and per_chunk regressing −9.78pp on aggregate
STRICT-rate against Hermes-3-8B. See §"Bench results".
ADD: companion missed-answer falsification guard. The Ballestrini
failure exposes a verifier-blind false-negative class that
layout fixes attention placement for but does not fully
close. See §"Verifier-blind missed-answer class".
DONE: companion missed-answer falsification guard shipped as
Ticket #000068 (Phase 1 sidecar + Phase 2 bench evidence +
Phase 3 opt-in demote flag). Default OFF; Phase 4 default
flip blocks on wider bench + human spot-check. See §"Verifier-
blind missed-answer class" and docs/tickets/ticket-000068-*.md.
```
Layout is a real but narrow lever — useful for specific failure
shapes (small-model evidence-negation under long context), not a
universal default. Keep `tail` default; route `bookend`/`per_chunk`
via the broad-quantifier classifier on query types that match the
target failure mode.
target failure mode. The verifier-side companion guard (#000068)
catches the residual failure class layout doesn't fully close.
## Why this exists — the Ballestrini case
@ -314,53 +316,86 @@ that produced this specific instance, but layout alone cannot
close the class — a sufficiently large prompt or an adversarial
phrasing can resurface the failure under any layout.
### Companion missed-answer guard (proposed sidecar)
### Companion missed-answer guard — Ticket #000068 (shipped 2026-05-27)
The right architectural fit is a **deterministic sidecar**, no
LLM-as-judge, that fires when all three of the following hold:
Shipped as a deterministic sidecar in `arborist/qa/inspect.py:
diagnose_missed_answer`. Fires only when all three clauses hold:
```
answer contains a denial/absence pattern (e.g. "not mentioned",
"not provided", "the evidence does not say", "no specific",
"cannot determine from the provided evidence")
AND
query is an extraction shape (list / "songs by" / "works by" /
"who wrote" / "what are" / "which" / quantifier ∈ {ALL,
COMPREHENSIVE, OPEN_REQUEST} from the existing
arborist.qa.quantifier classifier)
AND
evidence contains candidate answer spans near subject tokens
(quoted strings, comma-separated title lists, title-case
spans within a proximity window of the subject's content
tokens — same proximity-clustering primitive the entity
verifier already uses)
(A) answer contains a denial/absence pattern (sealed v1 list:
"not mentioned", "not provided", "the evidence does not say",
"does not mention", "no specific", "no evidence", "cannot
determine from the provided evidence", "is not stated",
"is not specified")
(B) question is extraction/list shape (broad-quantifier classifier
intensity ∈ {ALL, COMPREHENSIVE, OPEN_REQUEST, MANY, PLURAL}
OR surface cue match: "songs by" / "works by" / "who wrote" /
"what year" / "list of" / "name all" / …)
(C) evidence contains candidate spans (quoted strings / title-case
spans / comma-list items / year / date) within a proximity
window (default 600 chars) of CLEANED subject tokens (cue and
relation words like "songs", "by", "who" stripped before
proximity matching — without this the guard false-triggers on
generic title-case spans near the question)
```
Output:
Output is `result["answerability"]` — None when the conjunction
doesn't fire, else a structured dict:
```
answerability_warning : bool
missed_answer_candidate_spans : list of (evidence_id, offset, text)
optional audit_mode demote: EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL
diagnostic_version "missed-answer-v1"
answerability_warning True
confidence_class "weak" | "medium" | "strong"
triggered_clauses {denial, extraction_shape,
candidate_proximity}
denial_pattern_matched "not mentioned"
extraction_cue_matched "songs by"
extraction_shape "list" | "single_entity" | "open_list"
answer_type "title_like" | "person" | "date"
subject_tokens ["veronica", "ballestrini"]
candidate_count total candidates found
missed_answer_candidate_spans top-10 with offset_start / offset_end /
offset_basis="evidence_object_text" /
candidate_kind / nearest_subject_token /
subject_proximity_chars
```
Phase 2 measurement (`bench/qa_results/phase2-sidecar-on/2026-05-
27T14-16-22Z`): 228 runs, Hermes-3-8B, claim_lattice, tail layout.
**2/228 fires (0.88%), both strong confidence, both the Ballestrini
regression fixture. Precision 100%, zero false positives across the
226 non-Ballestrini runs.**
Phase 3 opt-in demote — `--demote-on-missed-answer` on `arborist
query`/`ask`. When set, the renderer demotes
`EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL` on lattice modes for
strong/medium-confidence fires; lower rungs and non-lattice modes
get a `· missed-answer` tail tag. Default OFF; Phase 4 default flip
NO-GO until wider bench + human spot-check.
Discipline:
- **Sidecar, not verifier hook.** Follows the same pattern as
`arborist.qa.inspect.diagnose_*` (deflection, coherence,
title-relevance) — read-only, never writes `providence_cache` or
`audit_events`.
- **Does not fold into `verifier_policy_hash`.** If demote behavior
is wired (optional, gated), it folds into
`governance_policy_hash` (or a new `answerability_policy_hash`),
never into the verifier hash.
- **Never promotes claims.** The guard only flags possible missed
answerability; it cannot turn a HYBRID into a STRICT.
- **Sidecar, not verifier hook.** Same pattern as
`arborist.qa.inspect.diagnose_*` (deflection / coherence /
title-relevance) — read-only, never writes `providence_cache`
or `audit_events`.
- **Hash partitioning:** sidecar-mode fields
(`answerability_sidecar_enabled`, `answerability_threshold`,
`denial_patterns_version`, `extraction_cues_version`) fold into
`governance_policy_hash` only. The Phase 3 demote flag
(`answerability_demote_enabled`) ALSO folds into
`verifier_policy_hash` because flipping it changes the rendered
audit_mode — a verifier-output property, so the verifier hash
legitimately partitions.
- **Never promotes claims.** The trigger conjunction makes claim
promotion structurally impossible — guard only fires on denial
answers.
This is a 5F-Falsification fixture: the Ballestrini case is
exactly the kind of failure that selects for adding a new
falsifier into the substrate, then propagates the new fixture
forward.
5F-Falsification fixture: the Ballestrini case is exactly the kind
of failure that selects for adding a new falsifier into the
substrate, then propagates the new fixture forward. The fixture is
in `bench/qa_questions.txt` under "entity list" with a 4-line
context comment pointing back here.
## Usage
@ -431,9 +466,9 @@ any default promotion or adaptive routing lands.
would be more robust to upstream rendering changes. Adopt if/when
the evidence map's block boundary convention changes.
- **Companion missed-answer falsification guard** (see
§"Verifier-blind missed-answer class"). Deterministic sidecar
that catches the failure class layout doesn't fully close.
- **Companion missed-answer falsification guard** — shipped as
Ticket #000068 Phase 1+2+3. See §"Companion missed-answer guard"
above and `docs/tickets/ticket-000068-*.md`.
## Roadmap
@ -441,13 +476,15 @@ Phased gating — current state in **bold**.
```
Phase 0 — ticket finalization
Open layout ticket. Open companion missed-answer guard ticket.
Open layout ticket. Open companion missed-answer guard ticket
(#000068).
Default = tail.
Phase 1 — implementation
**DONE 2026-05-26.** format_user_payload helper, six callsites,
CLI + Make plumbing, governance_policy_hash partitioning,
USER_PAYLOAD_LAYOUTS constant, unknown-layout raises ValueError.
**DONE 2026-05-26 (commit 5674107).** format_user_payload helper,
six callsites, CLI + Make plumbing, governance_policy_hash
partitioning, USER_PAYLOAD_LAYOUTS constant, unknown-layout
raises ValueError.
Phase 2 — regression fixtures
**DONE 2026-05-27.** Ballestrini case added to bench/qa_questions.txt
@ -469,11 +506,25 @@ Phase 4 — selection
by a filtered-subset re-bench to confirm net positive within
the gated class.
Phase 5 — missed-answer guard
**DESIGN OPEN.** Implement deterministic sidecar (denial pattern
+ extraction query cues + candidate-span proximity). No
LLM-as-judge. Emit warnings / optional demote to
EVIDENCE-MISSED-PARTIAL.
Phase 5 — missed-answer guard (Ticket #000068)
**DONE 2026-05-27.**
- #000068 Phase 1 (sidecar): commit 2ab11d2. diagnose_missed_
answer in arborist/qa/inspect.py with the three-clause
conjunction, 36 tests, end-to-end live verified on Ballestrini.
- #000068 Phase 2 (bench): commit ec55db5. bench/qa_sweep.py
instrumented to surface answerability fires aggregate; 228-run
bench at bench/qa_results/phase2-sidecar-on/ shows 2 fires,
both Ballestrini, both strong confidence, 100% precision.
- #000068 Phase 3 (opt-in demote): commit ec55db5. --demote-on-
missed-answer flag on `arborist query`/`ask`, default OFF.
answerability_demote_enabled folded into _VERIFIER_POLICY_FIELDS;
renderer demotes EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL
for strong/medium fires.
- #000068 Phase 4 (default demote-on): **NO-GO** until wider
bench + human spot-check confirms low FP rate across diverse
failure shapes. Per Dav1d 2026-05-27 §3.4: "a false sidecar
warning is tolerable; a false audit-label demotion can damage
trust in correct abstentions."
```
## Related
@ -486,11 +537,18 @@ Phase 5 — missed-answer guard
default promotion) followed directly from that rule.
- `docs/seven-point-program.md` — north-star directives. Layout is
a D6/D7 (verifier discipline + small-model robustness) follow-up,
not a new directive. The missed-answer guard, when it lands,
would fold into D2 (pointer-grounding) by extending the
falsifier surface to cover *unsupported absence* alongside
*unsupported presence*.
- 2026-05-27 de-novo review by Dav1dPrometheus (private working
doc) — split this work into two tickets: layout (this doc) +
companion missed-answer guard. Verdict, errata, and roadmap
phasing above incorporate that review.
not a new directive. The missed-answer guard (#000068, shipped
2026-05-27) extends **D2 (pointer-grounding)** by covering
*unsupported absence* alongside *unsupported presence* — same
falsifier discipline, new failure shape.
- `docs/tickets/ticket-000068-verifier-blind-missed-answer-guard.md`
— sibling ticket. Layout is the model-input lever; #000068 is the
verifier-side adjunct. Read together for the full picture of the
Ballestrini failure and the substrate's response to it.
- 2026-05-27 de-novo reviews by Dav1dPrometheus (private working
docs at `~/Downloads/response_-_user-payload-layout.txt` and
`~/Downloads/RESPONSE_final_ticket-000068-verifier-blind-missed-
answer-guard.txt`) — split this work into two tickets: layout
(this doc) + companion missed-answer guard (#000068). Verdict,
errata, hardenings, and roadmap phasing above incorporate both
reviews.