engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

qa: drop manual-quote rule from pointer verifier; port G0 policy to runner

Browse source 
Removes the strict no-double-quote check (`MANUAL_QUOTE_VIOLATION`) from
verify_claim_lattice. The rule was rejecting factually correct,
source-grounded claims for cosmetic punctuation: Hermes-3-8B paraphrases
prose but copies named-quoted phrases verbatim from source (e.g.
`"Constitution State"` lifted from a Connecticut chunk). Pre-fix,
`make query Q="tell me about connecticut"` reported `0/13 verified` on
a paragraph where every claim was correct and source-supported, just
because the model preserved source quote marks.

The two checks that remain handle the cases the manual_quote rule was
nominally meant to catch:
  - claim_lattice_min_citation_coverage (Rule 5, default 0.30)
    catches lazy-anchor citations whose only overlap with the cited
    span is a single topical token
  - claim_lattice_max_pointers_per_claim (Rule 6, default 2)
    catches the encyclopedic-mega-claim where the model emits
    `[E1,E2,...,E26]` after one sentence

Bench n=3 across 8 fixed questions vs pre-G0-hardening baseline (also
n=3): pointer-mode grounded count (STRICT+HYBRID) goes 15 → 21,
UNGROUNDED 9 → 3, broad-descriptive failures (connecticut, python)
fully cured. STRICT count goes 12 → 9 because previously-bogus STRICTs
(microsoft cited transit chunks that just shared one token, supermans
lazy-anchor magnet at 32/32) are now honestly reclassified to HYBRID.

Side changes:
  - Ports max_pointers / min_citation_coverage policy fields and
    verify_claim_lattice call into runner.py so single-document `ask`
    matches multi-source `query` semantics (query.py already had them
    via b39e79b).
  - Worked Example 2 (broad-descriptive) added to pointer-mode prompt
    so "tell me about X" has a template, plus Rules 7 (anti-echo) and
    8 (max-2-pointers).
  - Two manual_quote tests in tests/test_claim_lattice.py inverted to
    document the new behavior (quotes-in-claim no longer block).
  - `_has_manual_quote` retained — still used by verify_claim_lattice_json.

459 tests pass.
This commit is contained in:
russell@unturf.com 2026-04-30 11:47:07 -04:00
parent 2601575f15
commit 224bfd6a2b
No known key found for this signature in database
4 changed files with 107 additions and 57 deletions
Download patch file Download diff file Expand all files Collapse all files

43
tests/test_claim_lattice.py
View file

@ -227,25 +227,35 @@ def test_source_role_blocked_violation():
assert v["claim_statuses"][0]["status"] == "SOURCE_ROLE_BLOCKED"
def test_strict_no_quote_rule_any_double_quote_violates():
"""Even a 4-char quoted span triggers — model is forbidden to type
quote characters at all in pointer mode."""
def test_double_quote_in_claim_text_no_longer_blocks_verification():
"""Pre-2026-04-30: any `"` in claim text was a hard MANUAL_QUOTE_VIOLATION
that blocked every pointer on the claim — even when the claim was
factually correct and source-grounded. Hermes-3-8B paraphrases prose
but copies named-quoted phrases verbatim from source (e.g.
`"Constitution State"` from a Connecticut chunk), so the rule was
rejecting good claims for cosmetic punctuation. Removed in favor
of the coverage-threshold check (Rule 5) and pointer cap (Rule 6).
The claim below would previously have failed with MANUAL_QUOTE_VIOLATION;
now it stands or falls on whether the cited evidence actually supports
it — same as any quote-free claim.
"""
em = build_evidence_map(_sample_chunks())
answer = 'The "T-rex" appears. [E1]\n'
v = verify_claim_lattice(answer, em)
assert v["audit_mode"] == "UNGROUNDED"
assert any(viol["kind"] == "MANUAL_QUOTE_VIOLATION"
for viol in v["violations"])
assert v["claim_statuses"][0]["status"] == "MANUAL_QUOTE_VIOLATION"
# No MANUAL_QUOTE_VIOLATION emitted anywhere.
assert not any(viol["kind"] == "MANUAL_QUOTE_VIOLATION"
for viol in v["violations"])
assert v["claim_statuses"][0]["status"] != "MANUAL_QUOTE_VIOLATION"
def test_curly_quotes_also_violate():
def test_curly_quotes_also_no_longer_block():
"""Mirrors the ASCII-quote case for curly typographic quotes."""
em = build_evidence_map(_sample_chunks())
answer = "The “T-rex” appears. [E1]\n"
v = verify_claim_lattice(answer, em)
assert v["audit_mode"] == "UNGROUNDED"
assert any(viol["kind"] == "MANUAL_QUOTE_VIOLATION"
for viol in v["violations"])
assert not any(viol["kind"] == "MANUAL_QUOTE_VIOLATION"
for viol in v["violations"])
def test_no_evidence_pointer_downgrades():
@ -749,7 +759,16 @@ def test_per_chunk_evidence_map_query_path(tmp_path):
"Velociraptor is a theropod. [E2]\n"
"T-rex is apex. [E3]\n"
))
policy = dict(DEFAULT_QUERY_POLICY, answer_mode="claim_lattice_pointer")
# Override the per-source chunk cap so all three chunks of this
# single test source surface as E1/E2/E3. Production default caps
# at 2 chunks per source to keep the evidence catalog small for
# broad-descriptive questions; this test is asserting the per-chunk
# mapping itself, so it opts into the unbounded path.
policy = dict(
DEFAULT_QUERY_POLICY,
answer_mode="claim_lattice_pointer",
claim_lattice_max_chunks_per_source=8,
)
result = query(
question="dinosaur paragraphs",
qa_db=qa_db,