qa: pointer-format includes source title + chunk prefix; lock EVIDENCE-LINKED label
Two UX fixes addressing the 2026-05-01 Orwell run feedback: (1) Evidence pointer format. Old form `[E5: "<excerpt>"]` looked visually like a 1-indexed source rank — paired with a "sources (8)" list whose `[5]` slot was a different document, operators easily mis-attributed citations. New form: [E5 | Nineteen Eighty-Four | 682f0a11: "<excerpt>"] Title comes from EvidenceObject.title (URI-tail fallback when None); chunk_root prefix is the first 8 hex chars — enough to disambiguate chunks from the same source while staying compact. Renderer change in render_claim_lattice; pre-existing claim-lattice tests updated; 3 new unit tests pin the format (title-inline, URI-tail fallback, no `[E1:` collision-shape). (2) Audit-label discipline. _render_audit_label already maps STRICT → EVIDENCE-LINKED for claim_lattice* modes (verifier output stays binary, schema CHECK constraint stays); 4 new tests in test_cli_render.py pin the relabel for STRICT / HYBRID / UNGROUNDED in claim_lattice modes and the no-relabel for quote-mode where the verifier verifies pinned spans not synthesis. Net: 8 new tests, full suite 649 passed.
This commit is contained in:
parent
bf8d93caba
commit
3586eeeb0a
2 changed files with 141 additions and 5 deletions
|
|
@ -213,3 +213,67 @@ def test_render_capacity_thousand_separators():
|
|||
assert "61,550 chars" in out
|
||||
assert "60,000" in out
|
||||
assert "1,234 chars" in out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# label discipline — claim_lattice* modes get EVIDENCE-LINKED relabel
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_render_label_strict_in_claim_lattice_becomes_evidence_linked():
|
||||
"""claim_lattice / claim_lattice_pointer 'STRICT' verdicts mean
|
||||
'every pointer resolved + source_role allowed + token-coverage
|
||||
passed' — NOT semantic entailment. The display label spells out
|
||||
that distinction so users don't read STRICT as 'the answer is
|
||||
correct' on synthesis-heavy claims."""
|
||||
r = _result(
|
||||
audit_mode="STRICT",
|
||||
verifier_method="claim_lattice",
|
||||
n_quotes=1, n_verified=1,
|
||||
)
|
||||
out = _render_query_human(r, "q")
|
||||
assert "EVIDENCE-LINKED" in out
|
||||
assert "via claim_lattice" in out
|
||||
# The bare STRICT token should NOT appear on the summary line —
|
||||
# we relabeled deliberately.
|
||||
summary_line = out.splitlines()[1]
|
||||
assert "STRICT" not in summary_line
|
||||
|
||||
|
||||
def test_render_label_hybrid_in_claim_lattice_becomes_evidence_linked_partial():
|
||||
r = _result(
|
||||
audit_mode="HYBRID",
|
||||
verifier_method="claim_lattice_pointer",
|
||||
n_quotes=3, n_verified=1,
|
||||
)
|
||||
out = _render_query_human(r, "q")
|
||||
assert "EVIDENCE-LINKED-PARTIAL" in out
|
||||
assert "via claim_lattice_pointer" in out
|
||||
|
||||
|
||||
def test_render_label_ungrounded_in_claim_lattice_stays_ungrounded():
|
||||
"""UNGROUNDED already names what it means — no relabel needed."""
|
||||
r = _result(
|
||||
audit_mode="UNGROUNDED",
|
||||
verifier_method="claim_lattice",
|
||||
n_quotes=2, n_verified=0,
|
||||
)
|
||||
out = _render_query_human(r, "q")
|
||||
assert "UNGROUNDED" in out
|
||||
assert "via claim_lattice" in out
|
||||
|
||||
|
||||
def test_render_label_quote_mode_keeps_audit_mode_token():
|
||||
"""Quote / span / entity / paraphrase modes verify against
|
||||
pinned spans, not synthesis. STRICT in quote mode IS a strong
|
||||
claim about evidence units — keep the label as-is."""
|
||||
r = _result(
|
||||
audit_mode="STRICT",
|
||||
verifier_method="quote",
|
||||
n_quotes=2, n_verified=2,
|
||||
)
|
||||
out = _render_query_human(r, "q")
|
||||
assert "STRICT" in out
|
||||
assert "via quote" in out
|
||||
# Don't relabel quote-mode STRICT.
|
||||
assert "EVIDENCE-LINKED" not in out
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue