From e497a9501abc2d471900973c796041b4b859a99a Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Wed, 27 May 2026 10:55:38 -0400 Subject: [PATCH] docs/user-payload-layout: reflect #000068 Phase 1+2+3 shipped state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five sections updated to match the post-2026-05-27 substrate state (the user-payload-layout work, sibling ticket #000068, and the 2026-05-27 bench evidence are all in tree). Verdict block (top): ADD: companion missed-answer guard -> DONE: companion guard shipped as #000068 Phase 1+2+3 (default OFF; Phase 4 default flip NO-GO until wider bench + human spot-check). Companion missed-answer guard section: renamed from "(proposed sidecar)" to "(shipped 2026-05-27)". Carries the implementation location (arborist/qa/inspect.py:diagnose_missed_answer), the full output schema (diagnostic_version / confidence_class / triggered_clauses / subject_tokens / missed_answer_candidate_spans with offset_start/end/basis), the Phase 2 bench headline (2/228 fires, both Ballestrini, 100% precision, 0 FPs across 226 non- Ballestrini), the Phase 3 demote-flag CLI surface (--demote-on-missed-answer, default OFF), and corrected hash discipline: Phase 1 sidecar fields fold into governance_policy_hash only; the Phase 3 demote flag (answerability_demote_enabled) ALSO folds into verifier_policy_hash because flipping it changes the rendered audit_mode (a verifier-output property). Future hardening list: "companion missed-answer falsification guard" entry now points at #000068 instead of describing a deterministic-sidecar to be built. Roadmap Phase 5: "DESIGN OPEN" -> "DONE 2026-05-27" with all four sub-phases of #000068 named individually (Phase 1 sidecar commit 2ab11d2, Phase 2 bench + Phase 3 demote commit ec55db5, Phase 4 default flip NO-GO per Dav1d 2026-05-27 §3.4). The #000068 phase numbering is internal to that ticket; this roadmap names the external-facing milestones. Related links: past-tense the missed-answer guard ("when it lands" -> "shipped 2026-05-27"); added cross-reference to docs/tickets/ticket-000068-*.md; named both Dav1d review files by path so a re-read can locate the inputs. AUTOCOUNT tag (76 fixture-rows in bench/qa_questions.txt) still matches; no test changes. --- docs/user-payload-layout.md | 180 ++++++++++++++++++++++++------------ 1 file changed, 119 insertions(+), 61 deletions(-) diff --git a/docs/user-payload-layout.md b/docs/user-payload-layout.md index ea7a83b..0f17da2 100644 --- a/docs/user-payload-layout.md +++ b/docs/user-payload-layout.md @@ -13,22 +13,24 @@ and `arborist ask`. Make: `LAYOUT=...` on `make query`. Folds into ## Verdict ``` -GO: implement static layouts as opt-in policy knobs. +GO: static layouts shipped as opt-in policy knobs. NO-GO: promote bookend/per_chunk as default. The n=3×75q curated bench (2026-05-27) found bookend statistically indistinguish- able from tail and per_chunk regressing −9.78pp on aggregate STRICT-rate against Hermes-3-8B. See §"Bench results". -ADD: companion missed-answer falsification guard. The Ballestrini - failure exposes a verifier-blind false-negative class that - layout fixes attention placement for but does not fully - close. See §"Verifier-blind missed-answer class". +DONE: companion missed-answer falsification guard shipped as + Ticket #000068 (Phase 1 sidecar + Phase 2 bench evidence + + Phase 3 opt-in demote flag). Default OFF; Phase 4 default + flip blocks on wider bench + human spot-check. See §"Verifier- + blind missed-answer class" and docs/tickets/ticket-000068-*.md. ``` Layout is a real but narrow lever — useful for specific failure shapes (small-model evidence-negation under long context), not a universal default. Keep `tail` default; route `bookend`/`per_chunk` via the broad-quantifier classifier on query types that match the -target failure mode. +target failure mode. The verifier-side companion guard (#000068) +catches the residual failure class layout doesn't fully close. ## Why this exists — the Ballestrini case @@ -314,53 +316,86 @@ that produced this specific instance, but layout alone cannot close the class — a sufficiently large prompt or an adversarial phrasing can resurface the failure under any layout. -### Companion missed-answer guard (proposed sidecar) +### Companion missed-answer guard — Ticket #000068 (shipped 2026-05-27) -The right architectural fit is a **deterministic sidecar**, no -LLM-as-judge, that fires when all three of the following hold: +Shipped as a deterministic sidecar in `arborist/qa/inspect.py: +diagnose_missed_answer`. Fires only when all three clauses hold: ``` -answer contains a denial/absence pattern (e.g. "not mentioned", - "not provided", "the evidence does not say", "no specific", - "cannot determine from the provided evidence") -AND -query is an extraction shape (list / "songs by" / "works by" / - "who wrote" / "what are" / "which" / quantifier ∈ {ALL, - COMPREHENSIVE, OPEN_REQUEST} from the existing - arborist.qa.quantifier classifier) -AND -evidence contains candidate answer spans near subject tokens - (quoted strings, comma-separated title lists, title-case - spans within a proximity window of the subject's content - tokens — same proximity-clustering primitive the entity - verifier already uses) +(A) answer contains a denial/absence pattern (sealed v1 list: + "not mentioned", "not provided", "the evidence does not say", + "does not mention", "no specific", "no evidence", "cannot + determine from the provided evidence", "is not stated", + "is not specified") +(B) question is extraction/list shape (broad-quantifier classifier + intensity ∈ {ALL, COMPREHENSIVE, OPEN_REQUEST, MANY, PLURAL} + OR surface cue match: "songs by" / "works by" / "who wrote" / + "what year" / "list of" / "name all" / …) +(C) evidence contains candidate spans (quoted strings / title-case + spans / comma-list items / year / date) within a proximity + window (default 600 chars) of CLEANED subject tokens (cue and + relation words like "songs", "by", "who" stripped before + proximity matching — without this the guard false-triggers on + generic title-case spans near the question) ``` -Output: +Output is `result["answerability"]` — None when the conjunction +doesn't fire, else a structured dict: ``` -answerability_warning : bool -missed_answer_candidate_spans : list of (evidence_id, offset, text) -optional audit_mode demote: EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL +diagnostic_version "missed-answer-v1" +answerability_warning True +confidence_class "weak" | "medium" | "strong" +triggered_clauses {denial, extraction_shape, + candidate_proximity} +denial_pattern_matched "not mentioned" +extraction_cue_matched "songs by" +extraction_shape "list" | "single_entity" | "open_list" +answer_type "title_like" | "person" | "date" +subject_tokens ["veronica", "ballestrini"] +candidate_count total candidates found +missed_answer_candidate_spans top-10 with offset_start / offset_end / + offset_basis="evidence_object_text" / + candidate_kind / nearest_subject_token / + subject_proximity_chars ``` +Phase 2 measurement (`bench/qa_results/phase2-sidecar-on/2026-05- +27T14-16-22Z`): 228 runs, Hermes-3-8B, claim_lattice, tail layout. +**2/228 fires (0.88%), both strong confidence, both the Ballestrini +regression fixture. Precision 100%, zero false positives across the +226 non-Ballestrini runs.** + +Phase 3 opt-in demote — `--demote-on-missed-answer` on `arborist +query`/`ask`. When set, the renderer demotes +`EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL` on lattice modes for +strong/medium-confidence fires; lower rungs and non-lattice modes +get a `· missed-answer` tail tag. Default OFF; Phase 4 default flip +NO-GO until wider bench + human spot-check. + Discipline: -- **Sidecar, not verifier hook.** Follows the same pattern as - `arborist.qa.inspect.diagnose_*` (deflection, coherence, - title-relevance) — read-only, never writes `providence_cache` or - `audit_events`. -- **Does not fold into `verifier_policy_hash`.** If demote behavior - is wired (optional, gated), it folds into - `governance_policy_hash` (or a new `answerability_policy_hash`), - never into the verifier hash. -- **Never promotes claims.** The guard only flags possible missed - answerability; it cannot turn a HYBRID into a STRICT. +- **Sidecar, not verifier hook.** Same pattern as + `arborist.qa.inspect.diagnose_*` (deflection / coherence / + title-relevance) — read-only, never writes `providence_cache` + or `audit_events`. +- **Hash partitioning:** sidecar-mode fields + (`answerability_sidecar_enabled`, `answerability_threshold`, + `denial_patterns_version`, `extraction_cues_version`) fold into + `governance_policy_hash` only. The Phase 3 demote flag + (`answerability_demote_enabled`) ALSO folds into + `verifier_policy_hash` because flipping it changes the rendered + audit_mode — a verifier-output property, so the verifier hash + legitimately partitions. +- **Never promotes claims.** The trigger conjunction makes claim + promotion structurally impossible — guard only fires on denial + answers. -This is a 5F-Falsification fixture: the Ballestrini case is -exactly the kind of failure that selects for adding a new -falsifier into the substrate, then propagates the new fixture -forward. +5F-Falsification fixture: the Ballestrini case is exactly the kind +of failure that selects for adding a new falsifier into the +substrate, then propagates the new fixture forward. The fixture is +in `bench/qa_questions.txt` under "entity list" with a 4-line +context comment pointing back here. ## Usage @@ -431,9 +466,9 @@ any default promotion or adaptive routing lands. would be more robust to upstream rendering changes. Adopt if/when the evidence map's block boundary convention changes. -- **Companion missed-answer falsification guard** (see - §"Verifier-blind missed-answer class"). Deterministic sidecar - that catches the failure class layout doesn't fully close. +- **Companion missed-answer falsification guard** — shipped as + Ticket #000068 Phase 1+2+3. See §"Companion missed-answer guard" + above and `docs/tickets/ticket-000068-*.md`. ## Roadmap @@ -441,13 +476,15 @@ Phased gating — current state in **bold**. ``` Phase 0 — ticket finalization - Open layout ticket. Open companion missed-answer guard ticket. + Open layout ticket. Open companion missed-answer guard ticket + (#000068). Default = tail. Phase 1 — implementation - **DONE 2026-05-26.** format_user_payload helper, six callsites, - CLI + Make plumbing, governance_policy_hash partitioning, - USER_PAYLOAD_LAYOUTS constant, unknown-layout raises ValueError. + **DONE 2026-05-26 (commit 5674107).** format_user_payload helper, + six callsites, CLI + Make plumbing, governance_policy_hash + partitioning, USER_PAYLOAD_LAYOUTS constant, unknown-layout + raises ValueError. Phase 2 — regression fixtures **DONE 2026-05-27.** Ballestrini case added to bench/qa_questions.txt @@ -469,11 +506,25 @@ Phase 4 — selection by a filtered-subset re-bench to confirm net positive within the gated class. -Phase 5 — missed-answer guard - **DESIGN OPEN.** Implement deterministic sidecar (denial pattern - + extraction query cues + candidate-span proximity). No - LLM-as-judge. Emit warnings / optional demote to - EVIDENCE-MISSED-PARTIAL. +Phase 5 — missed-answer guard (Ticket #000068) + **DONE 2026-05-27.** + - #000068 Phase 1 (sidecar): commit 2ab11d2. diagnose_missed_ + answer in arborist/qa/inspect.py with the three-clause + conjunction, 36 tests, end-to-end live verified on Ballestrini. + - #000068 Phase 2 (bench): commit ec55db5. bench/qa_sweep.py + instrumented to surface answerability fires aggregate; 228-run + bench at bench/qa_results/phase2-sidecar-on/ shows 2 fires, + both Ballestrini, both strong confidence, 100% precision. + - #000068 Phase 3 (opt-in demote): commit ec55db5. --demote-on- + missed-answer flag on `arborist query`/`ask`, default OFF. + answerability_demote_enabled folded into _VERIFIER_POLICY_FIELDS; + renderer demotes EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL + for strong/medium fires. + - #000068 Phase 4 (default demote-on): **NO-GO** until wider + bench + human spot-check confirms low FP rate across diverse + failure shapes. Per Dav1d 2026-05-27 §3.4: "a false sidecar + warning is tolerable; a false audit-label demotion can damage + trust in correct abstentions." ``` ## Related @@ -486,11 +537,18 @@ Phase 5 — missed-answer guard default promotion) followed directly from that rule. - `docs/seven-point-program.md` — north-star directives. Layout is a D6/D7 (verifier discipline + small-model robustness) follow-up, - not a new directive. The missed-answer guard, when it lands, - would fold into D2 (pointer-grounding) by extending the - falsifier surface to cover *unsupported absence* alongside - *unsupported presence*. -- 2026-05-27 de-novo review by Dav1dPrometheus (private working - doc) — split this work into two tickets: layout (this doc) + - companion missed-answer guard. Verdict, errata, and roadmap - phasing above incorporate that review. + not a new directive. The missed-answer guard (#000068, shipped + 2026-05-27) extends **D2 (pointer-grounding)** by covering + *unsupported absence* alongside *unsupported presence* — same + falsifier discipline, new failure shape. +- `docs/tickets/ticket-000068-verifier-blind-missed-answer-guard.md` + — sibling ticket. Layout is the model-input lever; #000068 is the + verifier-side adjunct. Read together for the full picture of the + Ballestrini failure and the substrate's response to it. +- 2026-05-27 de-novo reviews by Dav1dPrometheus (private working + docs at `~/Downloads/response_-_user-payload-layout.txt` and + `~/Downloads/RESPONSE_final_ticket-000068-verifier-blind-missed- + answer-guard.txt`) — split this work into two tickets: layout + (this doc) + companion missed-answer guard (#000068). Verdict, + errata, hardenings, and roadmap phasing above incorporate both + reviews.