qa: drop manual-quote rule from pointer verifier; port G0 policy to runner
Removes the strict no-double-quote check (`MANUAL_QUOTE_VIOLATION`) from
verify_claim_lattice. The rule was rejecting factually correct,
source-grounded claims for cosmetic punctuation: Hermes-3-8B paraphrases
prose but copies named-quoted phrases verbatim from source (e.g.
`"Constitution State"` lifted from a Connecticut chunk). Pre-fix,
`make query Q="tell me about connecticut"` reported `0/13 verified` on
a paragraph where every claim was correct and source-supported, just
because the model preserved source quote marks.
The two checks that remain handle the cases the manual_quote rule was
nominally meant to catch:
- claim_lattice_min_citation_coverage (Rule 5, default 0.30)
catches lazy-anchor citations whose only overlap with the cited
span is a single topical token
- claim_lattice_max_pointers_per_claim (Rule 6, default 2)
catches the encyclopedic-mega-claim where the model emits
`[E1,E2,...,E26]` after one sentence
Bench n=3 across 8 fixed questions vs pre-G0-hardening baseline (also
n=3): pointer-mode grounded count (STRICT+HYBRID) goes 15 → 21,
UNGROUNDED 9 → 3, broad-descriptive failures (connecticut, python)
fully cured. STRICT count goes 12 → 9 because previously-bogus STRICTs
(microsoft cited transit chunks that just shared one token, supermans
lazy-anchor magnet at 32/32) are now honestly reclassified to HYBRID.
Side changes:
- Ports max_pointers / min_citation_coverage policy fields and
verify_claim_lattice call into runner.py so single-document `ask`
matches multi-source `query` semantics (query.py already had them
via b39e79b).
- Worked Example 2 (broad-descriptive) added to pointer-mode prompt
so "tell me about X" has a template, plus Rules 7 (anti-echo) and
8 (max-2-pointers).
- Two manual_quote tests in tests/test_claim_lattice.py inverted to
document the new behavior (quotes-in-claim no longer block).
- `_has_manual_quote` retained — still used by verify_claim_lattice_json.
459 tests pass.
This commit is contained in:
parent
2601575f15
commit
224bfd6a2b
4 changed files with 107 additions and 57 deletions
|
|
@ -349,12 +349,7 @@ DEFAULT_QUERY_POLICY = {
|
|||
"begin a line with the word EVIDENCE or with a pointer-id "
|
||||
"prefix like E1: or E2: — those tags belong only above the "
|
||||
"QUESTION, never in your output.\n"
|
||||
"8. Do not include the double-quote character anywhere in "
|
||||
"your answer. The runtime interpolates the literal source "
|
||||
"span at display time, including its punctuation. Paraphrase "
|
||||
"any phrase that the source has wrapped in quote marks "
|
||||
"instead of copying the marks themselves.\n"
|
||||
"9. Each claim line cites at most two pointers. Lines with "
|
||||
"8. Each claim line cites at most two pointers. Lines with "
|
||||
"three or more pointers are rejected. If more than two "
|
||||
"evidence blocks support a claim, pick the two that most "
|
||||
"directly contain the claim's key terms; or split the "
|
||||
|
|
@ -365,8 +360,7 @@ DEFAULT_QUERY_POLICY = {
|
|||
"line, plain prose with bracket tags. Pointer IDs come from "
|
||||
"the EVIDENCE blocks above. At most two pointers per claim. "
|
||||
"No `EVIDENCE:` header in your answer, no `E#:` line "
|
||||
"prefix, no double-quote characters. "
|
||||
"Now answer the question on the next message."
|
||||
"prefix. Now answer the question on the next message."
|
||||
),
|
||||
"claim_lattice_allowed_source_roles": [
|
||||
"primary_answer_source",
|
||||
|
|
|
|||
|
|
@ -111,8 +111,8 @@ DEFAULT_POLICY = {
|
|||
"Answer using natural-language pointer-lines: one claim per "
|
||||
"line, followed by a bracket tag with the pointer IDs that "
|
||||
"directly support that claim.\n\n"
|
||||
"WORKED EXAMPLE\n"
|
||||
"--------------\n"
|
||||
"WORKED EXAMPLE 1 — narrow factoid\n"
|
||||
"---------------------------------\n"
|
||||
"EVIDENCE:\n\n"
|
||||
"=== E1 (Apple_Inc | primary_answer_source) ===\n"
|
||||
"Apple Inc. was founded by Steve Jobs, Steve Wozniak, and "
|
||||
|
|
@ -125,7 +125,22 @@ DEFAULT_POLICY = {
|
|||
"Steve Jobs co-founded Apple. [E1]\n"
|
||||
"Steve Wozniak co-founded Apple. [E1,E2]\n"
|
||||
"Ronald Wayne co-founded Apple. [E1]\n\n"
|
||||
"END OF EXAMPLE\n\n"
|
||||
"WORKED EXAMPLE 2 — broad descriptive\n"
|
||||
"------------------------------------\n"
|
||||
"EVIDENCE:\n\n"
|
||||
"=== E1 (Mars | primary_answer_source) ===\n"
|
||||
"Mars is the fourth planet from the Sun. It has two moons, "
|
||||
"Phobos and Deimos. Mars has a thin atmosphere of carbon "
|
||||
"dioxide. Average surface temperature is around -60 "
|
||||
"degrees Celsius.\n\n"
|
||||
"QUESTION: tell me about Mars\n\n"
|
||||
"ANSWER:\n"
|
||||
"Mars is the fourth planet from the Sun. [E1]\n"
|
||||
"Mars has two moons, Phobos and Deimos. [E1]\n"
|
||||
"Mars has a thin atmosphere of carbon dioxide. [E1]\n"
|
||||
"The average surface temperature on Mars is around -60 "
|
||||
"degrees Celsius. [E1]\n\n"
|
||||
"END OF EXAMPLES\n\n"
|
||||
"RULES (each rule says what TO do):\n"
|
||||
"1. Reference evidence by pointer ID. The runtime displays "
|
||||
"the literal source span beside each claim — referencing is "
|
||||
|
|
@ -140,13 +155,23 @@ DEFAULT_POLICY = {
|
|||
"answer is the right answer when only short evidence "
|
||||
"exists.\n"
|
||||
"6. Write each claim as one plain-prose sentence on its own "
|
||||
"line."
|
||||
"line.\n"
|
||||
"7. Your answer goes only in the ANSWER position. Never "
|
||||
"begin a line with the word EVIDENCE or with a pointer-id "
|
||||
"prefix like E1: or E2: — those tags belong only above the "
|
||||
"QUESTION, never in your output.\n"
|
||||
"8. Each claim line cites at most two pointers. Lines with "
|
||||
"three or more pointers are rejected. If more than two "
|
||||
"evidence blocks support a claim, pick the two that most "
|
||||
"directly contain the claim's key terms; or split the "
|
||||
"claim into two lines."
|
||||
),
|
||||
"claim_lattice_grounding_reminder": (
|
||||
"REMINDER: format = pointer-line — `Claim text. [E1]` per "
|
||||
"line, plain prose with bracket tags. Pointer IDs come from "
|
||||
"the EVIDENCE blocks above. Cite 1 or 2 pointers per claim. "
|
||||
"Now answer the question on the next message."
|
||||
"the EVIDENCE blocks above. At most two pointers per claim. "
|
||||
"No `EVIDENCE:` header in your answer, no `E#:` line "
|
||||
"prefix. Now answer the question on the next message."
|
||||
),
|
||||
# Allowed source roles for claim-lattice verification. Roles outside
|
||||
# this set get classified SOURCE_ROLE_BLOCKED and downgrade the
|
||||
|
|
@ -159,6 +184,23 @@ DEFAULT_POLICY = {
|
|||
"background_source",
|
||||
"unclassified",
|
||||
],
|
||||
# Hard cap on pointer ids per claim line — mirrors prompt Rule 9.
|
||||
# Lines exceeding this cap classify as SCHEMA_INVALID and the
|
||||
# verdict can no longer reach STRICT. Folds into
|
||||
# governance_policy_hash so changing the cap invalidates prior
|
||||
# cached records.
|
||||
"claim_lattice_max_pointers_per_claim": 2,
|
||||
# Minimum claim-token coverage required for the citation-overlap
|
||||
# check (Rule 6) to pass. Pre-2026-04-30 the threshold was implicit
|
||||
# at "≥1 shared token", which let through lazy-anchored claims
|
||||
# whose only overlap was a single topical word (e.g. "Yale
|
||||
# University... [E9]" cited to a highway-data span containing only
|
||||
# "Connecticut"). 0.30 means a 10-token claim needs ≥3 of its
|
||||
# content tokens to appear in the cited span. Short claims (≤3
|
||||
# content tokens) keep the old ≥1-token floor so narrow factoids
|
||||
# like "Steve Jobs co-founded Apple" still pass. Folds into
|
||||
# governance_policy_hash on change.
|
||||
"claim_lattice_min_citation_coverage": 0.30,
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -399,6 +441,12 @@ def ask(
|
|||
],
|
||||
)
|
||||
),
|
||||
max_pointers_per_claim=int(policy.get(
|
||||
"claim_lattice_max_pointers_per_claim", 2
|
||||
)),
|
||||
min_citation_coverage=float(policy.get(
|
||||
"claim_lattice_min_citation_coverage", 0.30
|
||||
)),
|
||||
)
|
||||
# Rendered prose (literal spans interpolated) is the user-facing
|
||||
# answer text — never the model's raw pointer-line output. If
|
||||
|
|
|
|||
|
|
@ -840,7 +840,7 @@ def verify_claim_lattice(
|
|||
The model wrote pointer-line prose (``Claim text. [E12]``); the
|
||||
parser pulled (claim_text, [pointer_ids]) pairs from each non-empty
|
||||
line. This verifier maps each pointer id back to its
|
||||
content-addressed evidence object and runs seven hard checks:
|
||||
content-addressed evidence object and runs six hard checks:
|
||||
|
||||
1. Parser succeeded — ``parse_status == "PARSED"`` (line had a
|
||||
bracket tag). NO_EVIDENCE_POINTER claims (prose without tag)
|
||||
|
|
@ -848,24 +848,32 @@ def verify_claim_lattice(
|
|||
2. Pointer id resolves to an entry in the runtime-built evidence
|
||||
map. No model-invented ids.
|
||||
3. Resolved entry's ``source_role`` is in ``allowed_source_roles``.
|
||||
4. Claim text contains no double-quote characters anywhere (strict
|
||||
no-quote rule — see ``_has_manual_quote``).
|
||||
5. Claim text non-empty after tag strip.
|
||||
6. Claim's content tokens textually overlap the cited evidence
|
||||
span (per-pair, lexical only — see
|
||||
``_claim_textually_overlaps_evidence``). Catches the magnet-
|
||||
chunk lazy-anchor where the model cites an evidence pointer
|
||||
whose text contains zero claim-content tokens.
|
||||
7. Pointer count per claim does not exceed ``max_pointers_per_claim``
|
||||
4. Claim text non-empty after tag strip.
|
||||
5. Claim's content tokens textually overlap the cited evidence
|
||||
span at coverage ≥ ``min_citation_coverage`` (per-pair, lexical
|
||||
only — see ``_claim_textually_overlaps_evidence``). Catches the
|
||||
magnet-chunk lazy-anchor where the model cites an evidence
|
||||
pointer whose text contains few claim-content tokens.
|
||||
6. Pointer count per claim does not exceed ``max_pointers_per_claim``
|
||||
(default 2 — matches the prompt's "1 or 2 pointers per claim"
|
||||
rule). Catches the encyclopedic-mega-claim failure where the
|
||||
model produces one giant claim line citing every pointer.
|
||||
|
||||
Removed 2026-04-30: the strict no-double-quote rule. The model
|
||||
routinely paraphrases source prose but copies named-quoted phrases
|
||||
verbatim (e.g. ``"Constitution State"`` from a Connecticut span).
|
||||
Hard-rejecting claims that contained any ``"`` char was rejecting
|
||||
factually correct, source-grounded claims for cosmetic punctuation.
|
||||
The coverage threshold (Rule 5) and pointer cap (Rule 6) carry the
|
||||
weight of catching synthetic-quote / mega-claim failures the old
|
||||
rule was meant to catch. ``_has_manual_quote`` is still defined and
|
||||
used by ``verify_claim_lattice_json``.
|
||||
|
||||
Returns a verdict in the same shape as ``verify_quotes`` + extras:
|
||||
|
||||
n_quotes total claim-pointer pairs (denominator)
|
||||
n_verified pairs where pointer resolved AND
|
||||
source_role allowed AND no manual quote
|
||||
source_role allowed AND coverage met
|
||||
AND claim text non-empty
|
||||
audit_mode STRICT / HYBRID / UNGROUNDED
|
||||
unverified_quotes claim texts that didn't reach
|
||||
|
|
@ -877,7 +885,6 @@ def verify_claim_lattice(
|
|||
{EVIDENCE_LINKED, EVIDENCE_LINKED_PARTIAL,
|
||||
UNKNOWN_EVIDENCE_ID,
|
||||
SOURCE_ROLE_BLOCKED,
|
||||
MANUAL_QUOTE_VIOLATION,
|
||||
CITATION_MISMATCH,
|
||||
NO_EVIDENCE_POINTER, SCHEMA_INVALID}
|
||||
violations structured violation records for the
|
||||
|
|
@ -980,10 +987,6 @@ def verify_claim_lattice(
|
|||
unverified.append(claim_text)
|
||||
continue
|
||||
|
||||
# Strict no-quote rule: any double-quote in claim text is a
|
||||
# MANUAL_QUOTE_VIOLATION. Block every pointer link on this claim.
|
||||
manual_quote = _has_manual_quote(claim_text)
|
||||
|
||||
per_id_results: list[dict] = []
|
||||
resolved_evidence_ids: list[str] = []
|
||||
for pid in pointer_ids:
|
||||
|
|
@ -1011,11 +1014,6 @@ def verify_claim_lattice(
|
|||
"pid": pid, "ok": False, "kind": "SOURCE_ROLE_BLOCKED",
|
||||
})
|
||||
continue
|
||||
if manual_quote:
|
||||
per_id_results.append({
|
||||
"pid": pid, "ok": False, "kind": "MANUAL_QUOTE_VIOLATION",
|
||||
})
|
||||
continue
|
||||
if not _claim_textually_overlaps_evidence(
|
||||
claim_text, obj.span, min_coverage=min_citation_coverage
|
||||
):
|
||||
|
|
@ -1037,19 +1035,10 @@ def verify_claim_lattice(
|
|||
resolved_evidence_ids.append(obj.evidence_id)
|
||||
n_pairs_verified += 1
|
||||
|
||||
if manual_quote:
|
||||
violations.append({
|
||||
"kind": "MANUAL_QUOTE_VIOLATION",
|
||||
"claim_idx": idx,
|
||||
"claim_text": claim_text,
|
||||
})
|
||||
|
||||
ok_pids = [r["pid"] for r in per_id_results if r["ok"]]
|
||||
bad_kinds = sorted({r["kind"] for r in per_id_results if not r["ok"]})
|
||||
|
||||
if manual_quote:
|
||||
status = "MANUAL_QUOTE_VIOLATION"
|
||||
elif ok_pids and not bad_kinds:
|
||||
if ok_pids and not bad_kinds:
|
||||
status = "EVIDENCE_LINKED"
|
||||
elif ok_pids:
|
||||
status = "EVIDENCE_LINKED_PARTIAL"
|
||||
|
|
|
|||
|
|
@ -227,25 +227,35 @@ def test_source_role_blocked_violation():
|
|||
assert v["claim_statuses"][0]["status"] == "SOURCE_ROLE_BLOCKED"
|
||||
|
||||
|
||||
def test_strict_no_quote_rule_any_double_quote_violates():
|
||||
"""Even a 4-char quoted span triggers — model is forbidden to type
|
||||
quote characters at all in pointer mode."""
|
||||
def test_double_quote_in_claim_text_no_longer_blocks_verification():
|
||||
"""Pre-2026-04-30: any `"` in claim text was a hard MANUAL_QUOTE_VIOLATION
|
||||
that blocked every pointer on the claim — even when the claim was
|
||||
factually correct and source-grounded. Hermes-3-8B paraphrases prose
|
||||
but copies named-quoted phrases verbatim from source (e.g.
|
||||
`"Constitution State"` from a Connecticut chunk), so the rule was
|
||||
rejecting good claims for cosmetic punctuation. Removed in favor
|
||||
of the coverage-threshold check (Rule 5) and pointer cap (Rule 6).
|
||||
|
||||
The claim below would previously have failed with MANUAL_QUOTE_VIOLATION;
|
||||
now it stands or falls on whether the cited evidence actually supports
|
||||
it — same as any quote-free claim.
|
||||
"""
|
||||
em = build_evidence_map(_sample_chunks())
|
||||
answer = 'The "T-rex" appears. [E1]\n'
|
||||
v = verify_claim_lattice(answer, em)
|
||||
assert v["audit_mode"] == "UNGROUNDED"
|
||||
assert any(viol["kind"] == "MANUAL_QUOTE_VIOLATION"
|
||||
for viol in v["violations"])
|
||||
assert v["claim_statuses"][0]["status"] == "MANUAL_QUOTE_VIOLATION"
|
||||
# No MANUAL_QUOTE_VIOLATION emitted anywhere.
|
||||
assert not any(viol["kind"] == "MANUAL_QUOTE_VIOLATION"
|
||||
for viol in v["violations"])
|
||||
assert v["claim_statuses"][0]["status"] != "MANUAL_QUOTE_VIOLATION"
|
||||
|
||||
|
||||
def test_curly_quotes_also_violate():
|
||||
def test_curly_quotes_also_no_longer_block():
|
||||
"""Mirrors the ASCII-quote case for curly typographic quotes."""
|
||||
em = build_evidence_map(_sample_chunks())
|
||||
answer = "The “T-rex” appears. [E1]\n"
|
||||
v = verify_claim_lattice(answer, em)
|
||||
assert v["audit_mode"] == "UNGROUNDED"
|
||||
assert any(viol["kind"] == "MANUAL_QUOTE_VIOLATION"
|
||||
for viol in v["violations"])
|
||||
assert not any(viol["kind"] == "MANUAL_QUOTE_VIOLATION"
|
||||
for viol in v["violations"])
|
||||
|
||||
|
||||
def test_no_evidence_pointer_downgrades():
|
||||
|
|
@ -749,7 +759,16 @@ def test_per_chunk_evidence_map_query_path(tmp_path):
|
|||
"Velociraptor is a theropod. [E2]\n"
|
||||
"T-rex is apex. [E3]\n"
|
||||
))
|
||||
policy = dict(DEFAULT_QUERY_POLICY, answer_mode="claim_lattice_pointer")
|
||||
# Override the per-source chunk cap so all three chunks of this
|
||||
# single test source surface as E1/E2/E3. Production default caps
|
||||
# at 2 chunks per source to keep the evidence catalog small for
|
||||
# broad-descriptive questions; this test is asserting the per-chunk
|
||||
# mapping itself, so it opts into the unbounded path.
|
||||
policy = dict(
|
||||
DEFAULT_QUERY_POLICY,
|
||||
answer_mode="claim_lattice_pointer",
|
||||
claim_lattice_max_chunks_per_source=8,
|
||||
)
|
||||
result = query(
|
||||
question="dinosaur paragraphs",
|
||||
qa_db=qa_db,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue