qa: pointer-format includes source title + chunk prefix; lock EVIDENCE-LINKED label

Two UX fixes addressing the 2026-05-01 Orwell run feedback:

(1) Evidence pointer format. Old form `[E5: "<excerpt>"]` looked
visually like a 1-indexed source rank — paired with a "sources (8)"
list whose `[5]` slot was a different document, operators easily
mis-attributed citations. New form:

  [E5 | Nineteen Eighty-Four | 682f0a11: "<excerpt>"]

Title comes from EvidenceObject.title (URI-tail fallback when None);
chunk_root prefix is the first 8 hex chars — enough to disambiguate
chunks from the same source while staying compact. Renderer change
in render_claim_lattice; pre-existing claim-lattice tests updated;
3 new unit tests pin the format (title-inline, URI-tail fallback,
no `[E1:` collision-shape).

(2) Audit-label discipline. _render_audit_label already maps STRICT
→ EVIDENCE-LINKED for claim_lattice* modes (verifier output stays
binary, schema CHECK constraint stays); 4 new tests in
test_cli_render.py pin the relabel for STRICT / HYBRID / UNGROUNDED
in claim_lattice modes and the no-relabel for quote-mode where the
verifier verifies pinned spans not synthesis.

Net: 8 new tests, full suite 649 passed.
This commit is contained in:
russell@unturf.com 2026-05-01 13:52:38 -04:00
parent bf8d93caba
commit 3586eeeb0a
No known key found for this signature in database
2 changed files with 141 additions and 5 deletions

View file

@ -213,3 +213,67 @@ def test_render_capacity_thousand_separators():
assert "61,550 chars" in out
assert "60,000" in out
assert "1,234 chars" in out
# ---------------------------------------------------------------------------
# label discipline — claim_lattice* modes get EVIDENCE-LINKED relabel
# ---------------------------------------------------------------------------
def test_render_label_strict_in_claim_lattice_becomes_evidence_linked():
"""claim_lattice / claim_lattice_pointer 'STRICT' verdicts mean
'every pointer resolved + source_role allowed + token-coverage
passed' — NOT semantic entailment. The display label spells out
that distinction so users don't read STRICT as 'the answer is
correct' on synthesis-heavy claims."""
r = _result(
audit_mode="STRICT",
verifier_method="claim_lattice",
n_quotes=1, n_verified=1,
)
out = _render_query_human(r, "q")
assert "EVIDENCE-LINKED" in out
assert "via claim_lattice" in out
# The bare STRICT token should NOT appear on the summary line —
# we relabeled deliberately.
summary_line = out.splitlines()[1]
assert "STRICT" not in summary_line
def test_render_label_hybrid_in_claim_lattice_becomes_evidence_linked_partial():
r = _result(
audit_mode="HYBRID",
verifier_method="claim_lattice_pointer",
n_quotes=3, n_verified=1,
)
out = _render_query_human(r, "q")
assert "EVIDENCE-LINKED-PARTIAL" in out
assert "via claim_lattice_pointer" in out
def test_render_label_ungrounded_in_claim_lattice_stays_ungrounded():
"""UNGROUNDED already names what it means — no relabel needed."""
r = _result(
audit_mode="UNGROUNDED",
verifier_method="claim_lattice",
n_quotes=2, n_verified=0,
)
out = _render_query_human(r, "q")
assert "UNGROUNDED" in out
assert "via claim_lattice" in out
def test_render_label_quote_mode_keeps_audit_mode_token():
"""Quote / span / entity / paraphrase modes verify against
pinned spans, not synthesis. STRICT in quote mode IS a strong
claim about evidence units keep the label as-is."""
r = _result(
audit_mode="STRICT",
verifier_method="quote",
n_quotes=2, n_verified=2,
)
out = _render_query_human(r, "q")
assert "STRICT" in out
assert "via quote" in out
# Don't relabel quote-mode STRICT.
assert "EVIDENCE-LINKED" not in out