Phase 2 — bench instrumentation + measurement run
bench/qa_sweep.py picks up the answerability sidecar projection per row
(answerability_fired, answerability_confidence, answerability_denial_
pattern, answerability_answer_type, answerability_candidate_count) and
aggregates per-mode (answerability_fires + S/M/W confidence breakdown)
into a new column in the markdown summary table.
Measurement run on bench/qa_results/phase2-sidecar-on/2026-05-27T14-
16-22Z (76 questions × n=3 × claim_lattice × Hermes-3-8B × tail layout,
228 runs). Headline:
sidecar fires 2/228 (0.88%)
confidence dist 2 strong / 0 medium / 0 weak
precision 100% (2/2 fires were the Ballestrini fixture)
recall on Ballestrini 2/3 across n=3 (third run model extracted
correctly -> sidecar silent,
correct behavior)
false positives 0/226 non-Ballestrini runs
verifier verdict both fires labeled STRICT by the binary
verifier (the verifier-blind class, exactly
as predicted)
Detection rule's three-clause conjunction (denial + extraction-shape +
candidate proximity near cleaned subject tokens) is operating at the
precision floor. The strong-confidence-only firing pattern is what
calibrates Phase 3's demote threshold.
Phase 3 — opt-in demote flag (default OFF per Dav1d Phase 4 NO-GO)
arborist/qa/keys.py: answerability_demote_enabled added to
_VERIFIER_POLICY_FIELDS so flipping the flag partitions cache via
verifier_policy_hash. Justification: when on, the rendered audit_mode
changes (EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL), which IS a
verifier-output property; verifier hash must move accordingly. The
other answerability_* fields stay governance-only (sidecar
diagnostic, no audit_mode mutation).
arborist/cli.py:_render_audit_label extended with answerability +
demote_enabled kwargs. Logic:
demote_triggers = (
demote_enabled
and answerability["answerability_warning"] is True
and answerability["confidence_class"] in ("strong", "medium")
)
lattice modes:
EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL (rung transition)
POINTER-LINKED / ANCHOR-WARRANTED -> "rung · missed-answer"
(tail tag; rung itself already
signals degradation)
non-lattice modes (quote/span/entity/paraphrase):
audit_mode token unchanged + "· missed-answer" tail tag
weak confidence: NEVER demotes (Phase 2 saw zero weak fires on real
failures; reserved for future expanded detection ladder)
CLI flag --demote-on-missed-answer on both `arborist query` and
`arborist ask`, default OFF. Flows into call_policy[
"answerability_demote_enabled"] and through to result[
"answerability_demote_enabled"] so the renderer reads it without
needing the policy dict.
End-to-end verified live: 4 fresh Hermes-3-8B runs with --demote-on-
missed-answer on `songs by veronica ballestrini`, all 4 rendered
EVIDENCE-MISSED-PARTIAL · via claim_lattice (Hermes hit the failure
mode in all 4, sidecar fired strong, demote logic transformed the
label).
Phase 4 (default flip to demote-on) — NO-GO per Dav1d 2026-05-27 §3.4:
"a false sidecar warning is tolerable; a false audit-label demotion
can damage trust in correct abstentions." Phase 2 precision is 100%
but n=2 fires is too few samples to claim precision floor empirically.
Default flip blocks on wider bench + human spot-check of the warnings.
Tests: 47 total (36 Phase 1 + 11 new Phase 3 covering hash partitioning
discipline + render-label projection across all four rung/confidence
matrices). Full suite 2794 passed (delta +22 from prior 2772).
Bench output (bench/qa_results/phase2-sidecar-on/) intentionally not
committed — bench/qa_results/ is gitignored per existing convention;
the ticket carries the headline numbers + path for re-inspection.
russell@unturf.com
ec55db513c