#000068 Phase 2+3: bench + opt-in demote flag for missed-answer guard
Phase 2 — bench instrumentation + measurement run
bench/qa_sweep.py picks up the answerability sidecar projection per row
(answerability_fired, answerability_confidence, answerability_denial_
pattern, answerability_answer_type, answerability_candidate_count) and
aggregates per-mode (answerability_fires + S/M/W confidence breakdown)
into a new column in the markdown summary table.
Measurement run on bench/qa_results/phase2-sidecar-on/2026-05-27T14-
16-22Z (76 questions × n=3 × claim_lattice × Hermes-3-8B × tail layout,
228 runs). Headline:
sidecar fires 2/228 (0.88%)
confidence dist 2 strong / 0 medium / 0 weak
precision 100% (2/2 fires were the Ballestrini fixture)
recall on Ballestrini 2/3 across n=3 (third run model extracted
correctly -> sidecar silent,
correct behavior)
false positives 0/226 non-Ballestrini runs
verifier verdict both fires labeled STRICT by the binary
verifier (the verifier-blind class, exactly
as predicted)
Detection rule's three-clause conjunction (denial + extraction-shape +
candidate proximity near cleaned subject tokens) is operating at the
precision floor. The strong-confidence-only firing pattern is what
calibrates Phase 3's demote threshold.
Phase 3 — opt-in demote flag (default OFF per Dav1d Phase 4 NO-GO)
arborist/qa/keys.py: answerability_demote_enabled added to
_VERIFIER_POLICY_FIELDS so flipping the flag partitions cache via
verifier_policy_hash. Justification: when on, the rendered audit_mode
changes (EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL), which IS a
verifier-output property; verifier hash must move accordingly. The
other answerability_* fields stay governance-only (sidecar
diagnostic, no audit_mode mutation).
arborist/cli.py:_render_audit_label extended with answerability +
demote_enabled kwargs. Logic:
demote_triggers = (
demote_enabled
and answerability["answerability_warning"] is True
and answerability["confidence_class"] in ("strong", "medium")
)
lattice modes:
EVIDENCE-WARRANTED -> EVIDENCE-MISSED-PARTIAL (rung transition)
POINTER-LINKED / ANCHOR-WARRANTED -> "rung · missed-answer"
(tail tag; rung itself already
signals degradation)
non-lattice modes (quote/span/entity/paraphrase):
audit_mode token unchanged + "· missed-answer" tail tag
weak confidence: NEVER demotes (Phase 2 saw zero weak fires on real
failures; reserved for future expanded detection ladder)
CLI flag --demote-on-missed-answer on both `arborist query` and
`arborist ask`, default OFF. Flows into call_policy[
"answerability_demote_enabled"] and through to result[
"answerability_demote_enabled"] so the renderer reads it without
needing the policy dict.
End-to-end verified live: 4 fresh Hermes-3-8B runs with --demote-on-
missed-answer on `songs by veronica ballestrini`, all 4 rendered
EVIDENCE-MISSED-PARTIAL · via claim_lattice (Hermes hit the failure
mode in all 4, sidecar fired strong, demote logic transformed the
label).
Phase 4 (default flip to demote-on) — NO-GO per Dav1d 2026-05-27 §3.4:
"a false sidecar warning is tolerable; a false audit-label demotion
can damage trust in correct abstentions." Phase 2 precision is 100%
but n=2 fires is too few samples to claim precision floor empirically.
Default flip blocks on wider bench + human spot-check of the warnings.
Tests: 47 total (36 Phase 1 + 11 new Phase 3 covering hash partitioning
discipline + render-label projection across all four rung/confidence
matrices). Full suite 2794 passed (delta +22 from prior 2772).
Bench output (bench/qa_results/phase2-sidecar-on/) intentionally not
committed — bench/qa_results/ is gitignored per existing convention;
the ticket carries the headline numbers + path for re-inspection.
This commit is contained in:
parent
2ab11d2e59
commit
ec55db513c
7 changed files with 317 additions and 7 deletions
|
|
@ -498,3 +498,159 @@ def test_confidence_class_in_valid_set():
|
|||
evidence=_ballestrini_evidence(),
|
||||
)
|
||||
assert result["confidence_class"] in ("weak", "medium", "strong")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 3 — opt-in demote flag + render-label projection + hash discipline
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_demote_flag_in_verifier_policy_fields():
|
||||
"""Phase 3: answerability_demote_enabled must be in the verifier
|
||||
set so flipping it changes verifier_policy_hash."""
|
||||
from arborist.qa.keys import _VERIFIER_POLICY_FIELDS
|
||||
assert "answerability_demote_enabled" in _VERIFIER_POLICY_FIELDS
|
||||
|
||||
|
||||
def test_demote_flag_changes_verifier_hash():
|
||||
"""Flipping the demote flag must partition verifier_policy_hash."""
|
||||
from arborist.qa.keys import verifier_policy_hash
|
||||
base = {"answerability_demote_enabled": False}
|
||||
flipped = {"answerability_demote_enabled": True}
|
||||
assert verifier_policy_hash(base) != verifier_policy_hash(flipped)
|
||||
|
||||
|
||||
def test_sidecar_fields_do_NOT_change_verifier_hash():
|
||||
"""The other answerability fields (governance only) must NOT
|
||||
change verifier_policy_hash."""
|
||||
from arborist.qa.keys import verifier_policy_hash
|
||||
a = {"answerability_demote_enabled": False, "denial_patterns_version": "v1"}
|
||||
b = {"answerability_demote_enabled": False, "denial_patterns_version": "v2"}
|
||||
assert verifier_policy_hash(a) == verifier_policy_hash(b)
|
||||
|
||||
|
||||
def test_sidecar_fields_DO_change_governance_hash():
|
||||
"""The sidecar enable / threshold fields fold into governance."""
|
||||
from arborist.qa.keys import governance_policy_hash
|
||||
a = {"answerability_sidecar_enabled": True}
|
||||
b = {"answerability_sidecar_enabled": False}
|
||||
assert governance_policy_hash(a) != governance_policy_hash(b)
|
||||
|
||||
|
||||
def test_render_label_demote_strong_confidence():
|
||||
"""When demote is enabled AND sidecar fired strong, label demotes."""
|
||||
from arborist.cli import _render_audit_label
|
||||
answerability = {
|
||||
"answerability_warning": True,
|
||||
"confidence_class": "strong",
|
||||
}
|
||||
label = _render_audit_label(
|
||||
"STRICT", "claim_lattice",
|
||||
violations=None,
|
||||
answerability=answerability,
|
||||
demote_enabled=True,
|
||||
)
|
||||
assert "EVIDENCE-MISSED-PARTIAL" in label
|
||||
|
||||
|
||||
def test_render_label_demote_medium_confidence():
|
||||
"""Medium confidence also demotes (per Phase 2 calibration)."""
|
||||
from arborist.cli import _render_audit_label
|
||||
answerability = {
|
||||
"answerability_warning": True,
|
||||
"confidence_class": "medium",
|
||||
}
|
||||
label = _render_audit_label(
|
||||
"STRICT", "claim_lattice",
|
||||
violations=None,
|
||||
answerability=answerability,
|
||||
demote_enabled=True,
|
||||
)
|
||||
assert "EVIDENCE-MISSED-PARTIAL" in label
|
||||
|
||||
|
||||
def test_render_label_weak_does_NOT_demote():
|
||||
"""Weak confidence never demotes the rung (Phase 2 saw zero weak
|
||||
fires on real failures; reserved for future expanded detection)."""
|
||||
from arborist.cli import _render_audit_label
|
||||
answerability = {
|
||||
"answerability_warning": True,
|
||||
"confidence_class": "weak",
|
||||
}
|
||||
label = _render_audit_label(
|
||||
"STRICT", "claim_lattice",
|
||||
violations=None,
|
||||
answerability=answerability,
|
||||
demote_enabled=True,
|
||||
)
|
||||
assert "EVIDENCE-MISSED-PARTIAL" not in label
|
||||
assert "missed-answer" not in label
|
||||
|
||||
|
||||
def test_render_label_demote_off_keeps_original():
|
||||
"""When demote flag is False, label stays as-is even on strong fire."""
|
||||
from arborist.cli import _render_audit_label
|
||||
answerability = {
|
||||
"answerability_warning": True,
|
||||
"confidence_class": "strong",
|
||||
}
|
||||
label = _render_audit_label(
|
||||
"STRICT", "claim_lattice",
|
||||
violations=None,
|
||||
answerability=answerability,
|
||||
demote_enabled=False,
|
||||
)
|
||||
assert "EVIDENCE-MISSED-PARTIAL" not in label
|
||||
assert "missed-answer" not in label
|
||||
|
||||
|
||||
def test_render_label_no_answerability_unchanged():
|
||||
"""No sidecar fire → label unchanged regardless of flag."""
|
||||
from arborist.cli import _render_audit_label
|
||||
label_off = _render_audit_label(
|
||||
"STRICT", "claim_lattice", violations=None,
|
||||
answerability=None, demote_enabled=False,
|
||||
)
|
||||
label_on = _render_audit_label(
|
||||
"STRICT", "claim_lattice", violations=None,
|
||||
answerability=None, demote_enabled=True,
|
||||
)
|
||||
assert label_off == label_on
|
||||
|
||||
|
||||
def test_render_label_quote_mode_demote_appends_tail():
|
||||
"""Non-lattice mode: demote appends '· missed-answer' tail rather
|
||||
than transitioning to MISSED-PARTIAL."""
|
||||
from arborist.cli import _render_audit_label
|
||||
answerability = {
|
||||
"answerability_warning": True,
|
||||
"confidence_class": "strong",
|
||||
}
|
||||
label = _render_audit_label(
|
||||
"STRICT", "quote",
|
||||
violations=None,
|
||||
answerability=answerability,
|
||||
demote_enabled=True,
|
||||
)
|
||||
assert "missed-answer" in label
|
||||
|
||||
|
||||
def test_render_label_lower_rung_demote_appends_tail():
|
||||
"""Lattice mode on a non-EVIDENCE-WARRANTED rung (e.g. ANCHOR-WARRANTED
|
||||
or POINTER-LINKED) gets a '· missed-answer' tail rather than a rung
|
||||
transition — the rung itself already signals degradation."""
|
||||
from arborist.cli import _render_audit_label
|
||||
answerability = {
|
||||
"answerability_warning": True,
|
||||
"confidence_class": "strong",
|
||||
}
|
||||
# HYBRID + WARRANT_MISSING → POINTER-LINKED rung
|
||||
violations = [{"kind": "WARRANT_MISSING"}]
|
||||
label = _render_audit_label(
|
||||
"HYBRID", "claim_lattice",
|
||||
violations=violations,
|
||||
answerability=answerability,
|
||||
demote_enabled=True,
|
||||
)
|
||||
assert "missed-answer" in label
|
||||
assert "EVIDENCE-MISSED-PARTIAL" not in label
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue