qa: 4th verifier strategy — paraphrase via token-coverage; strip trailing citations
Two changes that make the verifier accept more honest grounding without
relaxing what STRICT means.
1. Strip trailing citation parentheticals at extraction time.
`extract_quotes` and `extract_claim_spans` now drop a single trailing
`(Source: ...)` / `(citing X)` / `(https://...)` parenthetical
before returning. The Pikachu real-corpus case: model wrote the
verbatim source sentence and appended `(Source: https://...)`. The
prose IS in the corpus — only the inserted citation broke substring
match. Now substring fires; STRICT recovered.
Conservative regex: only strips when the parenthetical content
starts with a citation cue word OR contains a URL. A genuine prose
parenthetical like "Pikachu (a Pokémon species) lives in forests"
stays.
2. Paraphrase strategy as a 4th verifier method.
In the span path, items that fail substring match get a
token-coverage probe. Tokens >=4 chars from the span; if >= 85% of
them appear in the normalized base context, count the span as
paraphrase-verified. Paraphrase items contribute to n_verified;
verifier_method flips to 'paraphrase' when any soft-verified items
are present so an auditor can tell.
Gated by `_is_prose_span`: a list of proper nouns ("Keanu Reeves,
Laurence Fishburne") has no lowercase content tokens >=4 chars and
falls through to the entity strategy where proximity policy can
tight-cluster check. Real prose spans ("Pikachu is a species of
Pokémon creatures...") have multiple lowercase content tokens and
qualify for paraphrase.
Soft-signal note (from CLAUDE.md soft/hard rule): paraphrase is
heuristic, not byte-equivalence. The hard chain still records the
classification — verifier_method='paraphrase' is the explicit marker
that an audit reader can use to distinguish lexical-verbatim from
paraphrase-overlap. Quote strategy stays verbatim-only — quotes ARE
quotes; paraphrase-in-quotes is the model's mistake to flag, not
auto-promote.
Schema: verifier_method CHECK constraint expanded to include
'paraphrase'. New idempotent migration `_rebuild_providence_cache_paraphrase`
mirrors `_rebuild_providence_cache_ungrounded` (table rebuild via
BEGIN IMMEDIATE; no value translation needed). Probes existing CHECK
from sqlite_master and no-ops if already expanded.
Tests: 9 new in tests/test_verify.py — 4 trailing-citation strip
(quote+span variants, URL-only parenthetical, end-to-end STRICT
recovery), 4 paraphrase strategy (high-coverage promote, low-coverage
reject, mixed-method label, quote-strategy preservation). Default
suite still green: 326 passed, 1 skipped.
Burned the live Pikachu cache record (audit chain extended) so a
fresh `make query Q="who is pikachu?"` exercises the new verifier
end-to-end.
This commit is contained in:
parent
d3168fa236
commit
10c1cba6e0
3 changed files with 368 additions and 6 deletions
|
|
@ -82,6 +82,60 @@ _PROPER_NOUN_RE = re.compile(
|
|||
MIN_QUOTE_CHARS = 8
|
||||
MIN_SPAN_CHARS = 12
|
||||
|
||||
# Trailing parenthetical the model often appends to verbatim source
|
||||
# prose, defeating substring match even when the prose itself IS in
|
||||
# the corpus. Examples we strip:
|
||||
#
|
||||
# "...lightning-based attacks. (Source: https://...)"
|
||||
# "...invented in 1976. (citing Wikipedia)"
|
||||
# "...protagonist of the game. (see Pikachu_(character))"
|
||||
# "...released in 1997 (https://en.wikipedia.org/...)"
|
||||
#
|
||||
# Conservative regex: only strips a SINGLE trailing parenthetical at
|
||||
# end-of-string after optional whitespace, and only when the contents
|
||||
# either start with a recognized citation cue OR contain a URL. Refuses
|
||||
# to strip parentheticals that look like genuine prose (e.g.
|
||||
# "Pikachu (a Pokémon species)").
|
||||
_TRAILING_CITATION_RE = re.compile(
|
||||
r"""
|
||||
\s*\(\s*
|
||||
(?: # one of:
|
||||
(?:source|src|citing|see|ref|reference|from) # citation cue word
|
||||
\s*:?\s*[^()]* # optional content
|
||||
| # OR
|
||||
https?://\S+\s*[^()]* # URL-led parenthetical
|
||||
)
|
||||
\s*\)\s*$
|
||||
""",
|
||||
re.IGNORECASE | re.VERBOSE,
|
||||
)
|
||||
|
||||
|
||||
def _strip_trailing_citation(text: str) -> str:
|
||||
"""Drop a trailing `(Source: ...)`-like parenthetical, if any.
|
||||
|
||||
Idempotent: applies once. The model rarely chains citations, so we
|
||||
don't loop. Returns the input unchanged if no match.
|
||||
"""
|
||||
return _TRAILING_CITATION_RE.sub("", text).rstrip()
|
||||
|
||||
|
||||
# Paraphrase strategy thresholds. The 4th verifier method runs only
|
||||
# when quote/span/entity have all failed to classify; it accepts a
|
||||
# span as "paraphrase-verified" when token coverage in the source
|
||||
# context crosses ``DEFAULT_PARAPHRASE_COVERAGE`` and the span has at
|
||||
# least ``DEFAULT_PARAPHRASE_MIN_TOKENS`` content tokens (>4 chars).
|
||||
#
|
||||
# Soft-signal note: token overlap is a heuristic, NOT byte-equivalence.
|
||||
# The hard chain still records `verifier_method = 'paraphrase'` so an
|
||||
# auditor can distinguish lexical-verbatim from paraphrase-overlap.
|
||||
# A span verified via paraphrase contributes to `n_verified` but the
|
||||
# audit_mode trichotomy stays unchanged: STRICT requires all units
|
||||
# (quote/span/entity/paraphrase) to verify; HYBRID = some verify some
|
||||
# don't; UNGROUNDED = none.
|
||||
DEFAULT_PARAPHRASE_COVERAGE = 0.85
|
||||
DEFAULT_PARAPHRASE_MIN_TOKENS = 4
|
||||
|
||||
# Entity-path policies. Entity-existence in source is circumstantial, not
|
||||
# claim-level proof — the model could correctly name entities while making
|
||||
# claims around them that came from training. These four policies span the
|
||||
|
|
@ -146,6 +200,10 @@ def extract_quotes(answer_text: str) -> list[str]:
|
|||
quotes: list[str] = []
|
||||
for i in range(0, len(positions) - 1, 2):
|
||||
span = answer_text[positions[i] + 1:positions[i + 1]]
|
||||
# Strip model-appended citations BEFORE the length gate so a
|
||||
# `verbatim... (Source: https://...)` span that's >MIN long with
|
||||
# the citation but short without it doesn't bypass the test.
|
||||
span = _strip_trailing_citation(span)
|
||||
if len(span) >= MIN_QUOTE_CHARS:
|
||||
quotes.append(span)
|
||||
return quotes
|
||||
|
|
@ -165,6 +223,10 @@ def extract_claim_spans(answer_text: str) -> list[str]:
|
|||
continue
|
||||
for sent in _SENT_RE.split(stripped):
|
||||
sent = sent.strip().rstrip(".:,;")
|
||||
# Drop model-appended `(Source: ...)` parentheticals so a
|
||||
# verbatim sentence doesn't fail substring match because of
|
||||
# an inserted citation.
|
||||
sent = _strip_trailing_citation(sent)
|
||||
if len(sent) < MIN_SPAN_CHARS:
|
||||
continue
|
||||
if _is_framing(sent):
|
||||
|
|
@ -238,6 +300,103 @@ def _check_each(items: list[str], norm_ctx: str) -> tuple[list[str], list[str]]:
|
|||
return verified, unverified
|
||||
|
||||
|
||||
def _token_coverage(
|
||||
span: str,
|
||||
norm_ctx: str,
|
||||
*,
|
||||
min_token_len: int = 4,
|
||||
) -> tuple[float, int]:
|
||||
"""Fraction of meaningful tokens from `span` that appear anywhere
|
||||
in `norm_ctx`. Returns ``(coverage_fraction, content_token_count)``.
|
||||
|
||||
Tokens shorter than ``min_token_len`` chars are skipped — they're
|
||||
function words and short connectors that match almost any English
|
||||
text and don't carry topical signal.
|
||||
"""
|
||||
nspan = _normalize(span)
|
||||
tokens = [t for t in nspan.split() if len(t) >= min_token_len]
|
||||
if not tokens:
|
||||
return (0.0, 0)
|
||||
present = sum(1 for t in tokens if t in norm_ctx)
|
||||
return (present / len(tokens), len(tokens))
|
||||
|
||||
|
||||
def _is_prose_span(span: str, *, min_lowercase_content: int = 2) -> bool:
|
||||
"""Heuristic: does this span look like prose (eligible for paraphrase
|
||||
matching) vs. a list of proper nouns (better matched by entity)?
|
||||
|
||||
Counts tokens in the ORIGINAL (pre-normalize) span whose first
|
||||
letter is lowercase and whose length is >= 4 chars. A pure list
|
||||
like ``"Keanu Reeves, Laurence Fishburne"`` has zero such tokens —
|
||||
those go through the entity path. A real sentence like ``"Pikachu
|
||||
is a species of Pokémon creatures..."`` has multiple ("species",
|
||||
"creatures", "from", etc.) and qualifies for paraphrase.
|
||||
|
||||
Threshold ``min_lowercase_content=2`` lets through "the cast..."
|
||||
(1 lowercase content token: "cast") only if more lowercase prose
|
||||
is present. Lists with a leading "The" don't sneak through.
|
||||
"""
|
||||
n_lower = 0
|
||||
for tok in span.split():
|
||||
# Strip leading/trailing punctuation for the case check (so
|
||||
# "cast:" still counts as a lowercase token).
|
||||
core = tok.strip(".,;:!?\"'()[]{}")
|
||||
if len(core) < 4:
|
||||
continue
|
||||
if core[0].islower():
|
||||
n_lower += 1
|
||||
if n_lower >= min_lowercase_content:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _check_each_with_paraphrase(
|
||||
items: list[str],
|
||||
norm_ctx: str,
|
||||
*,
|
||||
paraphrase_coverage: float = DEFAULT_PARAPHRASE_COVERAGE,
|
||||
paraphrase_min_tokens: int = DEFAULT_PARAPHRASE_MIN_TOKENS,
|
||||
) -> tuple[list[str], list[str], list[str]]:
|
||||
"""Three-bucket variant of ``_check_each``.
|
||||
|
||||
For each item:
|
||||
- if its normalized form is a substring of ``norm_ctx``: STRICT
|
||||
(verbatim verified)
|
||||
- else if its meaningful-token coverage in ``norm_ctx`` is
|
||||
``>= paraphrase_coverage`` and it has at least
|
||||
``paraphrase_min_tokens`` content tokens: PARAPHRASE
|
||||
(token-overlap verified)
|
||||
- else: UNVERIFIED
|
||||
|
||||
Returns ``(strict_verified, paraphrase_verified, unverified)``.
|
||||
Order within each bucket preserves input order.
|
||||
|
||||
Soft-signal note: paraphrase verification is heuristic. Callers
|
||||
that need byte-equivalence (proof export to other peers, audit
|
||||
chain claims) must use ``_check_each`` directly.
|
||||
"""
|
||||
strict: list[str] = []
|
||||
paraphrase: list[str] = []
|
||||
unverified: list[str] = []
|
||||
for it in items:
|
||||
if _normalize(it) in norm_ctx:
|
||||
strict.append(it)
|
||||
continue
|
||||
# Paraphrase only fires for prose-shaped spans. Lists of proper
|
||||
# nouns (e.g. "Keanu Reeves, Laurence Fishburne") fall through
|
||||
# to the entity strategy where proximity policy can disambiguate
|
||||
# tight clusters from scattered mentions.
|
||||
if not _is_prose_span(it):
|
||||
unverified.append(it)
|
||||
continue
|
||||
cov, n_tok = _token_coverage(it, norm_ctx)
|
||||
if n_tok >= paraphrase_min_tokens and cov >= paraphrase_coverage:
|
||||
paraphrase.append(it)
|
||||
else:
|
||||
unverified.append(it)
|
||||
return strict, paraphrase, unverified
|
||||
|
||||
|
||||
def verify_quotes(
|
||||
answer_text: str,
|
||||
context: str,
|
||||
|
|
@ -289,17 +448,25 @@ def verify_quotes(
|
|||
"verifier_method": "quote",
|
||||
}
|
||||
|
||||
# Strategy 2: bullet/sentence spans verbatim in context.
|
||||
# Strategy 2: bullet/sentence spans. Try verbatim substring first;
|
||||
# then fall back to paraphrase (token-coverage) for items that didn't
|
||||
# substring-match. Spans verified via paraphrase contribute to
|
||||
# n_verified — the verifier_method label flips to "paraphrase" when
|
||||
# any soft-verified items are present so an auditor can tell.
|
||||
spans = extract_claim_spans(answer_text)
|
||||
if spans:
|
||||
verified, unverified = _check_each(spans, norm_ctx)
|
||||
strict, paraphrase, unverified = _check_each_with_paraphrase(
|
||||
spans, norm_ctx
|
||||
)
|
||||
verified = strict + paraphrase
|
||||
if verified:
|
||||
method = "paraphrase" if paraphrase else "span"
|
||||
return {
|
||||
"n_quotes": len(spans),
|
||||
"n_verified": len(verified),
|
||||
"audit_mode": _classify(verified, unverified),
|
||||
"unverified_quotes": unverified,
|
||||
"verifier_method": "span",
|
||||
"verifier_method": method,
|
||||
}
|
||||
|
||||
# Strategy 3: multi-word proper nouns (entity grounding) — gated by
|
||||
|
|
|
|||
|
|
@ -187,7 +187,7 @@ CREATE TABLE IF NOT EXISTS providence_cache (
|
|||
-- spans matched but multi-word proper nouns did. 'none' = no
|
||||
-- evidence at all (truly emergent).
|
||||
verifier_method TEXT NOT NULL DEFAULT 'none'
|
||||
CHECK (verifier_method IN ('quote','span','entity','none'))
|
||||
CHECK (verifier_method IN ('quote','span','entity','paraphrase','none'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_providence_root ON providence_cache(source_root);
|
||||
CREATE INDEX IF NOT EXISTS idx_providence_state ON providence_cache(falsification_state);
|
||||
|
|
@ -370,7 +370,7 @@ def _migrate_audit_mode(conn: sqlite3.Connection) -> None:
|
|||
conn.execute(
|
||||
"ALTER TABLE providence_cache ADD COLUMN verifier_method TEXT "
|
||||
"NOT NULL DEFAULT 'none' "
|
||||
"CHECK (verifier_method IN ('quote','span','entity','none'))"
|
||||
"CHECK (verifier_method IN ('quote','span','entity','paraphrase','none'))"
|
||||
)
|
||||
|
||||
# VISUAL → UNGROUNDED rename. Detect legacy CHECK by inspecting DDL.
|
||||
|
|
@ -380,6 +380,15 @@ def _migrate_audit_mode(conn: sqlite3.Connection) -> None:
|
|||
if ddl_row and "'VISUAL'" in (ddl_row[0] or ""):
|
||||
_rebuild_providence_cache_ungrounded(conn)
|
||||
|
||||
# paraphrase verifier_method addition. Detect legacy CHECK by
|
||||
# inspecting DDL — if the constraint doesn't already list
|
||||
# 'paraphrase', rebuild the table.
|
||||
ddl_row = conn.execute(
|
||||
"SELECT sql FROM sqlite_master WHERE type='table' AND name='providence_cache'"
|
||||
).fetchone()
|
||||
if ddl_row and "'paraphrase'" not in (ddl_row[0] or ""):
|
||||
_rebuild_providence_cache_paraphrase(conn)
|
||||
|
||||
conn.execute(
|
||||
"CREATE INDEX IF NOT EXISTS idx_providence_audit "
|
||||
"ON providence_cache(audit_mode)"
|
||||
|
|
@ -435,6 +444,78 @@ def _migrate_mesh_peer_chains(conn: sqlite3.Connection) -> None:
|
|||
)
|
||||
|
||||
|
||||
def _rebuild_providence_cache_paraphrase(conn: sqlite3.Connection) -> None:
|
||||
"""One-time table rebuild: expand verifier_method CHECK to include
|
||||
'paraphrase' (4th verifier strategy: token-coverage paraphrase
|
||||
matching, alongside quote/span/entity).
|
||||
|
||||
Same pattern as ``_rebuild_providence_cache_ungrounded``: SQLite
|
||||
can't modify a CHECK constraint in place, so we create a new table
|
||||
with the expanded CHECK, copy the data verbatim (no value
|
||||
translation needed — paraphrase is additive), drop the old, rename
|
||||
the new. Caller probes existing CHECK from sqlite_master and only
|
||||
invokes this when 'paraphrase' is missing.
|
||||
"""
|
||||
new_create = """
|
||||
CREATE TABLE providence_cache_new (
|
||||
cache_key TEXT PRIMARY KEY,
|
||||
source_root TEXT NOT NULL,
|
||||
document_uri TEXT NOT NULL,
|
||||
question_hash TEXT NOT NULL,
|
||||
question_text TEXT NOT NULL,
|
||||
answer_text TEXT NOT NULL,
|
||||
merkle_proof TEXT NOT NULL,
|
||||
model_profile_hash TEXT NOT NULL,
|
||||
conversation_hash TEXT NOT NULL,
|
||||
governance_policy_hash TEXT NOT NULL,
|
||||
schema_version TEXT NOT NULL,
|
||||
canonicalization_version TEXT NOT NULL,
|
||||
chunking_version TEXT NOT NULL,
|
||||
falsification_state TEXT NOT NULL DEFAULT 'live'
|
||||
CHECK (falsification_state IN ('live','failed','stale','quarantined')),
|
||||
chain TEXT NOT NULL DEFAULT 'private'
|
||||
CHECK (chain IN ('private','public')),
|
||||
audit_event_hash TEXT,
|
||||
created_at INTEGER NOT NULL,
|
||||
last_hit_at INTEGER,
|
||||
hit_count INTEGER NOT NULL DEFAULT 0,
|
||||
audit_mode TEXT NOT NULL DEFAULT 'UNGROUNDED'
|
||||
CHECK (audit_mode IN ('STRICT','HYBRID','UNGROUNDED')),
|
||||
n_quotes INTEGER NOT NULL DEFAULT 0,
|
||||
n_verified INTEGER NOT NULL DEFAULT 0,
|
||||
unverified_quotes TEXT,
|
||||
verifier_method TEXT NOT NULL DEFAULT 'none'
|
||||
CHECK (verifier_method IN ('quote','span','entity','paraphrase','none'))
|
||||
)
|
||||
"""
|
||||
conn.execute("BEGIN IMMEDIATE")
|
||||
try:
|
||||
conn.execute(new_create)
|
||||
conn.execute(
|
||||
"INSERT INTO providence_cache_new "
|
||||
"(cache_key, source_root, document_uri, question_hash, question_text, "
|
||||
" answer_text, merkle_proof, model_profile_hash, conversation_hash, "
|
||||
" governance_policy_hash, schema_version, canonicalization_version, "
|
||||
" chunking_version, falsification_state, chain, audit_event_hash, "
|
||||
" created_at, last_hit_at, hit_count, audit_mode, n_quotes, "
|
||||
" n_verified, unverified_quotes, verifier_method) "
|
||||
"SELECT "
|
||||
" cache_key, source_root, document_uri, question_hash, question_text, "
|
||||
" answer_text, merkle_proof, model_profile_hash, conversation_hash, "
|
||||
" governance_policy_hash, schema_version, canonicalization_version, "
|
||||
" chunking_version, falsification_state, chain, audit_event_hash, "
|
||||
" created_at, last_hit_at, hit_count, audit_mode, n_quotes, "
|
||||
" n_verified, unverified_quotes, verifier_method "
|
||||
"FROM providence_cache"
|
||||
)
|
||||
conn.execute("DROP TABLE providence_cache")
|
||||
conn.execute("ALTER TABLE providence_cache_new RENAME TO providence_cache")
|
||||
conn.execute("COMMIT")
|
||||
except Exception:
|
||||
conn.execute("ROLLBACK")
|
||||
raise
|
||||
|
||||
|
||||
def _rebuild_providence_cache_ungrounded(conn: sqlite3.Connection) -> None:
|
||||
"""One-time table rebuild: rename audit_mode value VISUAL → UNGROUNDED.
|
||||
|
||||
|
|
@ -472,7 +553,7 @@ def _rebuild_providence_cache_ungrounded(conn: sqlite3.Connection) -> None:
|
|||
n_verified INTEGER NOT NULL DEFAULT 0,
|
||||
unverified_quotes TEXT,
|
||||
verifier_method TEXT NOT NULL DEFAULT 'none'
|
||||
CHECK (verifier_method IN ('quote','span','entity','none'))
|
||||
CHECK (verifier_method IN ('quote','span','entity','paraphrase','none'))
|
||||
)
|
||||
"""
|
||||
conn.execute("BEGIN IMMEDIATE")
|
||||
|
|
|
|||
|
|
@ -331,6 +331,120 @@ def test_wikitext_context_strips_markup_before_substring_test():
|
|||
assert v["unverified_quotes"] == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Trailing-citation strip — model-appended `(Source: ...)` no longer breaks
|
||||
# verbatim substring match.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_extract_quotes_strips_trailing_source_parenthetical():
|
||||
"""The Pikachu real-corpus case: model wrote a verbatim quote and
|
||||
appended a citation. Strip the citation so substring match works."""
|
||||
answer = (
|
||||
'The model said: "Pikachu can store electricity in its cheeks. '
|
||||
'(Source: https://en.wikipedia.org/wiki/Pikachu)"'
|
||||
)
|
||||
quotes = extract_quotes(answer)
|
||||
assert quotes == ["Pikachu can store electricity in its cheeks."]
|
||||
|
||||
|
||||
def test_extract_claim_spans_strips_trailing_source_parenthetical():
|
||||
"""Span-strategy variant — sentence-level extraction also strips."""
|
||||
answer = (
|
||||
"Pikachu evolves from Pichu. (Source: https://example.com/p)\n"
|
||||
"Pikachu has yellow fur. (citing Wikipedia)"
|
||||
)
|
||||
spans = extract_claim_spans(answer)
|
||||
assert any("Pikachu evolves from Pichu" in s and "Source" not in s for s in spans)
|
||||
assert any("Pikachu has yellow fur" in s and "citing" not in s for s in spans)
|
||||
|
||||
|
||||
def test_extract_claim_spans_strips_url_only_parenthetical():
|
||||
"""A bare URL parenthetical (no `Source:` cue word) is also a citation."""
|
||||
answer = "The release date was 1996 (https://en.wikipedia.org/wiki/Pikachu)"
|
||||
spans = extract_claim_spans(answer)
|
||||
assert any("https" not in s for s in spans)
|
||||
|
||||
|
||||
def test_strict_recovered_with_citation_strip():
|
||||
"""End-to-end: a span with verbatim source content + appended citation
|
||||
used to flag UNGROUNDED. With the strip, it classifies STRICT."""
|
||||
context = (
|
||||
"Pikachu can store electricity in its cheeks and release it in "
|
||||
"lightning-based attacks."
|
||||
)
|
||||
answer = (
|
||||
"Pikachu can store electricity in its cheeks and release it in "
|
||||
"lightning-based attacks. (Source: https://en.wikipedia.org/wiki/Pikachu)"
|
||||
)
|
||||
v = verify_quotes(answer, context)
|
||||
assert v["audit_mode"] == "STRICT"
|
||||
assert v["unverified_quotes"] == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Paraphrase strategy (4th tier) — token-coverage fallback in span path.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_paraphrase_promotes_high_token_coverage_span():
|
||||
"""Model rewrites source content but every meaningful token is in
|
||||
the corpus. Verifier accepts as paraphrase-verified."""
|
||||
# Context contains every meaningful (>=4 char) token from the answer
|
||||
# span, just in a different sequence — that's the paraphrase case.
|
||||
context = (
|
||||
"Pikachu is a Pokémon species, one of many creatures from the "
|
||||
"Pokémon franchise developed by Game Freak."
|
||||
)
|
||||
answer = (
|
||||
"Pikachu is a species of Pokémon creatures from the Pokémon "
|
||||
"franchise."
|
||||
)
|
||||
v = verify_quotes(answer, context)
|
||||
assert v["verifier_method"] == "paraphrase"
|
||||
assert v["audit_mode"] == "STRICT"
|
||||
assert v["unverified_quotes"] == []
|
||||
|
||||
|
||||
def test_paraphrase_does_not_promote_low_token_coverage_span():
|
||||
"""A span whose content is mostly NOT in the corpus stays unverified."""
|
||||
context = "Pikachu is a Pokémon species."
|
||||
answer = "Pikachu was elected mayor of Tokyo in 1988 by aristocrats."
|
||||
v = verify_quotes(answer, context)
|
||||
assert v["audit_mode"] == "UNGROUNDED"
|
||||
|
||||
|
||||
def test_paraphrase_method_label_set_when_any_paraphrase_used():
|
||||
"""If even one span verified via paraphrase (and others via substring),
|
||||
verifier_method flips to 'paraphrase' so an auditor knows soft signals
|
||||
were involved."""
|
||||
# Answer line 1 is verbatim. Line 2 reorders the same content tokens
|
||||
# — paraphrase. Both classify as verified.
|
||||
context = (
|
||||
"Apple Inc. was founded by Steve Jobs and Steve Wozniak. "
|
||||
"Apple has its main office in Cupertino, California, from where "
|
||||
"the company manages worldwide operations."
|
||||
)
|
||||
answer = (
|
||||
"Apple Inc. was founded by Steve Jobs and Steve Wozniak.\n"
|
||||
"The company manages its worldwide operations from Cupertino."
|
||||
)
|
||||
v = verify_quotes(answer, context)
|
||||
assert v["verifier_method"] == "paraphrase"
|
||||
assert v["audit_mode"] == "STRICT"
|
||||
|
||||
|
||||
def test_paraphrase_strategy_keeps_quote_strategy_strict():
|
||||
"""Quote strategy stays verbatim-only — quotes ARE quotes, paraphrase-
|
||||
in-quotes is the model's mistake. Paraphrase fallback applies to span
|
||||
strategy only."""
|
||||
context = "The actual sentence is something specific about cats."
|
||||
answer = '"This is a totally different sentence about dogs and rats"'
|
||||
v = verify_quotes(answer, context)
|
||||
assert v["verifier_method"] == "quote"
|
||||
assert v["audit_mode"] == "UNGROUNDED"
|
||||
|
||||
|
||||
def test_wikitext_strip_does_not_rescue_genuine_hallucination():
|
||||
"""A quote the model invented stays unverified even after stripping."""
|
||||
raw_wikitext_context = (
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue