qa: JSON mode uses pointer IDs (E1, E2, …) — close hallucination loop
JSON mode previously presented content-addressed evidence_ids
(``Eed1b6e396``-style) in the EVIDENCE block headers and expected
the same in the model's claim output. Hermes-3-8B was fabricating
plausible near-miss IDs (e.g. ``E1b6e396`` when the runtime had
``Eed1b6e396``) on cross-document relationship questions →
UNKNOWN_EVIDENCE_ID → UNGROUNDED, even when the answer text was
factually correct (e.g. "Homer Simpson's boss is Mr. Burns.").
Switching the prompt-facing surface to short pointer IDs (``E1``,
``E2``, …) — same as claim_lattice_pointer mode — closes the
hallucination loop:
- Pointer IDs are short, enumerable, and fabrication-obvious.
The model can't invent ``E27`` if only ``E1``-``E10`` were shown;
out-of-range IDs read as schema violations at first glance.
- The runtime still resolves each pointer_id to its content-
addressed evidence_id internally and stores THAT in
``evidence_id_pairs`` for the cache & run-DAG. Cache_keys stay
run-stable; only the prompt-facing string changes.
- JSON schema unchanged (``evidence_ids: [str, ...]``), so vLLM
guided_json continues to constrain output shape.
Live-test verification: ``who is homer simpson's boss?`` went from
JSON-mode UNGROUNDED 0/1 (hallucinated ``E1b6e396``) to STRICT 1/1
(model emits ``E1``, resolves cleanly). Homer fixture pin removed —
runs against the JSON default now.
Files touched:
- aborist/qa/evidence.py: render_evidence_block_for_json uses
e.pointer_id instead of e.evidence_id.
- aborist/qa/verify.py: verify_claim_lattice_json switched from
evidence_map_by_evidence_id to evidence_map_by_pointer_id;
captures both pointer_ids (model-emitted) and evidence_ids
(run-stable) in claim_statuses + evidence_id_pairs.
- aborist/qa/{runner,query}.py: claim_lattice_json_system_prompt
+ grounding_reminder describe pointer IDs; example shifts from
``E........`` placeholder to ``E1``.
- tests/test_verify_json.py: stub _ev() takes pointer_id; four
fixtures updated to set it.
- tests/test_qa_quality_live.py: Homer fixture unpinned (now
runs default JSON mode and grounds). Red-fish-blue-fish
fixture pinned to pointer mode — JSON mode hits a separate
token-budget runaway on "plot of X" prose-summary shapes
(~2/3 of samples blow max_tokens with whitespace spam after
the closing brace). Different failure mode, addressed in a
later commit.
460 unit tests + 11 live fixtures pass.
This commit is contained in:
parent
b99aa8d300
commit
bb8450d402
6 changed files with 142 additions and 77 deletions
|
|
@ -184,20 +184,26 @@ def render_evidence_map(evidence: list[EvidenceObject]) -> str:
|
|||
def render_evidence_block_for_json(e: EvidenceObject) -> str:
|
||||
"""Format one evidence object for the JSON-mode LLM prompt.
|
||||
|
||||
Header carries the content-addressed ``evidence_id`` (the long hex
|
||||
handle the model will cite in its JSON output) instead of the
|
||||
prompt-facing pointer_id used by claim_lattice_pointer mode::
|
||||
2026-04-30: header uses the prompt-facing ``pointer_id`` (E1, E2,
|
||||
…) — same as claim_lattice_pointer mode — instead of the
|
||||
content-addressed ``evidence_id`` (long hex). The change closes a
|
||||
real failure mode: small models (Hermes-3-8B observed) were
|
||||
fabricating plausible-looking content-addressed IDs (e.g.
|
||||
``E1b6e396`` when the runtime had ``Eed1b6e396``) → UNKNOWN_
|
||||
EVIDENCE_ID → UNGROUNDED, even when the answer text was correct.
|
||||
Pointer IDs (``E1``-``E10``) are short, enumerable, and fabrication-
|
||||
obvious — the model can't invent ``E27`` if only ``E1``-``E10`` were
|
||||
shown.
|
||||
|
||||
=== E1f8e4c2a (Jurassic_Park_(film) | primary_answer_source) ===
|
||||
The runtime still stores content-addressed ``evidence_id`` in the
|
||||
cache & run-DAG (resolved on-the-fly in ``verify_claim_lattice_json``);
|
||||
only the prompt-facing string changes::
|
||||
|
||||
=== E1 (Jurassic_Park_(film) | primary_answer_source) ===
|
||||
<literal span text>
|
||||
|
||||
JSON mode pairs naturally with grammar-constrained inference where
|
||||
the model can be schema-pinned to emit valid evidence_id strings.
|
||||
The longer handle is out-of-distribution prose for small models —
|
||||
use claim_lattice_pointer for those.
|
||||
"""
|
||||
label = (e.title or e.document_uri.rsplit("/", 1)[-1]) or "untitled"
|
||||
return f"=== {e.evidence_id} ({label} | {e.source_role}) ===\n{e.span}"
|
||||
return f"=== {e.pointer_id} ({label} | {e.source_role}) ===\n{e.span}"
|
||||
|
||||
|
||||
def render_evidence_map_for_json(evidence: list[EvidenceObject]) -> str:
|
||||
|
|
|
|||
|
|
@ -391,11 +391,11 @@ DEFAULT_QUERY_POLICY = {
|
|||
# `claim_lattice_use_guided_json=False` disables the extra_body
|
||||
# pass for endpoints that 400 on unknown fields.
|
||||
"claim_lattice_json_system_prompt": (
|
||||
"You will see numbered EVIDENCE blocks tagged with content-"
|
||||
"addressed evidence IDs (long hex strings starting with 'E'). "
|
||||
"Answer as a single JSON object using EXACTLY this schema, "
|
||||
"with NO prose, NO markdown fences, NO preamble:\n\n"
|
||||
' {"claims":[{"text":"<claim text>","evidence_ids":["E........"]}]}\n\n'
|
||||
"You will see numbered EVIDENCE blocks tagged with short "
|
||||
"pointer IDs: E1, E2, E3, etc. Answer as a single JSON "
|
||||
"object using EXACTLY this schema, with NO prose, NO "
|
||||
"markdown fences, NO preamble:\n\n"
|
||||
' {"claims":[{"text":"<claim text>","evidence_ids":["E1"]}]}\n\n'
|
||||
"RULES:\n"
|
||||
"1. Output a single JSON object. No code fences. No commentary.\n"
|
||||
"2. `text` is plain prose with NO double-quote characters "
|
||||
|
|
@ -403,17 +403,20 @@ DEFAULT_QUERY_POLICY = {
|
|||
"spans inside the text. If you need to mention a name "
|
||||
"containing punctuation, use the source's own form without "
|
||||
"wrapping it in quotes.\n"
|
||||
"3. `evidence_ids` MUST be IDs from the EVIDENCE blocks above. "
|
||||
"Do not invent IDs. At most two IDs per claim.\n"
|
||||
"3. `evidence_ids` MUST be pointer IDs that appear verbatim "
|
||||
"in the EVIDENCE block headers above (E1, E2, …). Do NOT "
|
||||
"invent IDs — only IDs that already exist above. At most "
|
||||
"two IDs per claim.\n"
|
||||
"4. Each claim must reference at least one evidence_id.\n"
|
||||
"5. If no evidence supports a claim, omit the claim."
|
||||
),
|
||||
"claim_lattice_json_grounding_reminder": (
|
||||
"REMINDER: emit a single JSON object with the exact schema "
|
||||
'`{"claims":[{"text":"...","evidence_ids":["E........"]}]}`. '
|
||||
"No code fences, no prose preamble. Each claim references at "
|
||||
"most two evidence_ids from the blocks above. The text field "
|
||||
"must contain no double-quote characters. "
|
||||
'`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. '
|
||||
"evidence_ids are pointer IDs (E1, E2, …) that appear "
|
||||
"verbatim in the EVIDENCE block headers — never invented. "
|
||||
"At most two per claim. Text has no double-quote characters. "
|
||||
"No code fences, no preamble. "
|
||||
"Now answer the question on the next message."
|
||||
),
|
||||
"claim_lattice_use_guided_json": True,
|
||||
|
|
|
|||
|
|
@ -229,25 +229,27 @@ DEFAULT_POLICY = {
|
|||
# Lenient pre-parser in verify_claim_lattice_json keeps the path
|
||||
# survivable on inference paths without grammar guidance.
|
||||
"claim_lattice_json_system_prompt": (
|
||||
"You will see numbered EVIDENCE blocks tagged with content-"
|
||||
"addressed evidence IDs (long hex strings starting with 'E'). "
|
||||
"Answer as a single JSON object using EXACTLY this schema, "
|
||||
"with NO prose, NO markdown fences, NO preamble:\n\n"
|
||||
' {"claims":[{"text":"<claim text>","evidence_ids":["E........"]}]}\n\n'
|
||||
"You will see numbered EVIDENCE blocks tagged with short "
|
||||
"pointer IDs: E1, E2, E3, etc. Answer as a single JSON "
|
||||
"object using EXACTLY this schema, with NO prose, NO "
|
||||
"markdown fences, NO preamble:\n\n"
|
||||
' {"claims":[{"text":"<claim text>","evidence_ids":["E1"]}]}\n\n'
|
||||
"RULES:\n"
|
||||
"1. Output a single JSON object. No code fences. No commentary.\n"
|
||||
"2. `text` is plain prose with NO double-quote characters.\n"
|
||||
"3. `evidence_ids` MUST be IDs from the EVIDENCE blocks above. "
|
||||
"At most two IDs per claim.\n"
|
||||
"3. `evidence_ids` MUST be pointer IDs that appear verbatim in "
|
||||
"the EVIDENCE block headers above (E1, E2, …). Do NOT invent "
|
||||
"new IDs. At most two IDs per claim.\n"
|
||||
"4. Each claim must reference at least one evidence_id.\n"
|
||||
"5. If no evidence supports a claim, omit the claim."
|
||||
),
|
||||
"claim_lattice_json_grounding_reminder": (
|
||||
"REMINDER: emit a single JSON object with the exact schema "
|
||||
'`{"claims":[{"text":"...","evidence_ids":["E........"]}]}`. '
|
||||
"No code fences, no preamble. At most two evidence_ids per "
|
||||
"claim. Text field has no double-quote characters. "
|
||||
"Now answer the question on the next message."
|
||||
'`{"claims":[{"text":"...","evidence_ids":["E1"]}]}`. '
|
||||
"No code fences, no preamble. evidence_ids must be pointer "
|
||||
"IDs (E1, E2, …) from the EVIDENCE blocks above — do not "
|
||||
"invent IDs. At most two per claim. Text has no double-quote "
|
||||
"characters. Now answer the question on the next message."
|
||||
),
|
||||
"claim_lattice_use_guided_json": True,
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1251,7 +1251,20 @@ def verify_claim_lattice_json(
|
|||
Parses the model's JSON output (lenient pre-parser handles markdown
|
||||
fences / preamble / curly quotes / trailing commas), validates the
|
||||
schema, then runs the same hard checks as ``verify_claim_lattice``
|
||||
but on content-addressed ``evidence_id``s directly:
|
||||
but reading ``evidence_ids`` from the JSON claim objects.
|
||||
|
||||
2026-04-30: switched from content-addressed evidence_ids
|
||||
(``Eed1b6e396``) to pointer_ids (``E1``, ``E2``, …) in the prompt
|
||||
& JSON output. Hermes-3-8B was fabricating plausible content-
|
||||
addressed IDs (``E1b6e396``-style near-misses) on cross-document
|
||||
relationship questions; the verifier correctly rejected them as
|
||||
UNKNOWN_EVIDENCE_ID but the answer text was often factually
|
||||
correct, leaving us with honest UNGROUNDED on right answers.
|
||||
Pointer IDs are short, enumerable, and fabrication-obvious. The
|
||||
runtime still resolves each pointer_id to its content-addressed
|
||||
evidence_id internally and stores that in ``evidence_id_pairs``
|
||||
(cache/run-DAG continuity); only the prompt-facing surface
|
||||
changes.
|
||||
|
||||
1. JSON parses (lenient). Failure → SCHEMA_INVALID, UNGROUNDED.
|
||||
2. Top-level is ``{"claims": [...]}``.
|
||||
|
|
@ -1270,11 +1283,11 @@ def verify_claim_lattice_json(
|
|||
``"trailing_comma"``). Empty list = strict JSON parse on first try.
|
||||
"""
|
||||
from aborist.qa.evidence import (
|
||||
evidence_map_by_evidence_id as _by_eid,
|
||||
evidence_map_by_pointer_id as _by_pointer,
|
||||
render_claim_lattice as _render,
|
||||
)
|
||||
|
||||
by_eid = _by_eid(evidence_map)
|
||||
by_pointer = _by_pointer(evidence_map)
|
||||
violations: list[dict] = []
|
||||
claim_statuses: list[dict] = []
|
||||
unverified: list[str] = []
|
||||
|
|
@ -1365,14 +1378,21 @@ def verify_claim_lattice_json(
|
|||
"n_ids": len(eids), "max": max_evidence_per_claim,
|
||||
})
|
||||
|
||||
# Per-id resolution + checks.
|
||||
# Per-id resolution + checks. ``eids`` are pointer_ids
|
||||
# (E1, E2, …) emitted by the model; we resolve each to its
|
||||
# EvidenceObject and capture the content-addressed
|
||||
# ``evidence_id`` for the cache/run-DAG handle. Pointer-style
|
||||
# IDs make fabrication obvious — if only E1-E10 were shown,
|
||||
# an emitted "E27" reads as a hallucination at the schema
|
||||
# check, not as a near-miss content-addressed string.
|
||||
per_id_results = []
|
||||
verified_ids = []
|
||||
verified_pointer_ids: list[str] = []
|
||||
verified_evidence_ids: list[str] = []
|
||||
for eid in eids:
|
||||
if not isinstance(eid, str):
|
||||
per_id_results.append({"eid": str(eid), "ok": False, "kind": "SCHEMA_INVALID"})
|
||||
continue
|
||||
obj = by_eid.get(eid)
|
||||
obj = by_pointer.get(eid)
|
||||
if obj is None:
|
||||
per_id_results.append({"eid": eid, "ok": False, "kind": "UNKNOWN_EVIDENCE_ID"})
|
||||
violations.append({
|
||||
|
|
@ -1384,7 +1404,8 @@ def verify_claim_lattice_json(
|
|||
per_id_results.append({"eid": eid, "ok": False, "kind": "SOURCE_ROLE_BLOCKED"})
|
||||
violations.append({
|
||||
"kind": "SOURCE_ROLE_BLOCKED",
|
||||
"claim_idx": idx, "evidence_id": eid,
|
||||
"claim_idx": idx, "evidence_id": obj.evidence_id,
|
||||
"pointer_id": eid,
|
||||
"source_role": obj.source_role,
|
||||
})
|
||||
continue
|
||||
|
|
@ -1394,15 +1415,17 @@ def verify_claim_lattice_json(
|
|||
per_id_results.append({"eid": eid, "ok": False, "kind": "CITATION_MISMATCH"})
|
||||
violations.append({
|
||||
"kind": "CITATION_MISMATCH",
|
||||
"claim_idx": idx, "evidence_id": eid,
|
||||
"claim_idx": idx, "evidence_id": obj.evidence_id,
|
||||
"pointer_id": eid,
|
||||
"claim_text": claim_text[:200],
|
||||
})
|
||||
continue
|
||||
per_id_results.append({"eid": eid, "ok": True})
|
||||
verified_ids.append(eid)
|
||||
verified_pointer_ids.append(eid)
|
||||
verified_evidence_ids.append(obj.evidence_id)
|
||||
|
||||
n_pairs += max(1, len(eids))
|
||||
n_pairs_verified += len(verified_ids)
|
||||
n_pairs_verified += len(verified_pointer_ids)
|
||||
|
||||
if not eids:
|
||||
claim_statuses.append({
|
||||
|
|
@ -1414,9 +1437,9 @@ def verify_claim_lattice_json(
|
|||
n_pairs += 1
|
||||
continue
|
||||
|
||||
if len(verified_ids) == len(eids):
|
||||
if len(verified_pointer_ids) == len(eids):
|
||||
status = "EVIDENCE_LINKED"
|
||||
elif verified_ids:
|
||||
elif verified_pointer_ids:
|
||||
status = "EVIDENCE_LINKED_PARTIAL"
|
||||
else:
|
||||
# Pick the worst per-id reason for the claim status.
|
||||
|
|
@ -1424,17 +1447,27 @@ def verify_claim_lattice_json(
|
|||
status = kinds[0] if kinds else "UNKNOWN_EVIDENCE_ID"
|
||||
unverified.append(claim_text)
|
||||
|
||||
# claim_statuses records BOTH ids: pointer (what model wrote)
|
||||
# and content-addressed (run-stable handle). Keeps the audit
|
||||
# trail legible at both layers.
|
||||
claim_statuses.append({
|
||||
"text": claim_text,
|
||||
"evidence_ids": eids,
|
||||
"pointer_ids": list(eids),
|
||||
"evidence_ids": list(verified_evidence_ids),
|
||||
"status": status,
|
||||
"reasons": [r["kind"] for r in per_id_results if not r["ok"]],
|
||||
})
|
||||
if verified_ids:
|
||||
valid_claims.append({"text": claim_text, "evidence_ids": verified_ids})
|
||||
evidence_id_pairs.append(list(verified_ids))
|
||||
if verified_pointer_ids:
|
||||
# Renderer takes the pointer-id form (model's view) and the
|
||||
# by_pointer index; cache/run-DAG get the content-addressed
|
||||
# evidence_ids (run-stable form).
|
||||
valid_claims.append({
|
||||
"text": claim_text,
|
||||
"pointer_ids": verified_pointer_ids,
|
||||
})
|
||||
evidence_id_pairs.append(list(verified_evidence_ids))
|
||||
|
||||
rendered_text = _render(valid_claims, by_eid) if valid_claims else ""
|
||||
rendered_text = _render(valid_claims, by_pointer) if valid_claims else ""
|
||||
|
||||
if n_pairs_verified > 0 and not violations:
|
||||
audit_mode = "STRICT"
|
||||
|
|
|
|||
|
|
@ -150,8 +150,18 @@ def test_red_fish_blue_fish_identifies_seuss_book():
|
|||
Red Sea, Detroit Red Wings, marine aquarium fish list...). The right
|
||||
answer is the Dr. Seuss book ``One Fish Two Fish Red Fish Blue Fish``.
|
||||
The model should anchor on the Seuss article and the answer should
|
||||
contain a Seuss-specific token."""
|
||||
r = _ask("plot of red fish blue fish?")
|
||||
contain a Seuss-specific token.
|
||||
|
||||
Pinned to claim_lattice_pointer mode: JSON mode hits a token-
|
||||
budget runaway on "plot of X" prose-summary shapes for this
|
||||
question (~2/3 of samples produce malformed JSON with whitespace
|
||||
spam after the closing brace, blowing the lenient parser's
|
||||
bounds). Pointer mode reliably hits STRICT 1/1 at 4.5s. Until
|
||||
JSON-mode max_tokens / prompt discipline addresses the runaway,
|
||||
this fixture documents that prose-summary questions are
|
||||
pointer-mode's strength.
|
||||
"""
|
||||
r = _ask("plot of red fish blue fish?", mode="claim_lattice_pointer")
|
||||
assert _grounded(r), f"audit_mode={r.get('audit_mode')!r}"
|
||||
txt = _answer_lower(r)
|
||||
seuss_markers = ("seuss", "rhyming", "children's book", "creatures")
|
||||
|
|
@ -229,17 +239,17 @@ def test_homer_simpson_boss_is_mr_burns():
|
|||
Springfield Nuclear Power Plant). The model needs to land on
|
||||
`Mr. Burns` and cite a chunk from one of the Simpsons articles.
|
||||
|
||||
Pinned to claim_lattice_pointer mode: in JSON mode Hermes-3-8B
|
||||
sometimes hallucinates a content-addressed evidence_id (e.g.
|
||||
``Eed1b6e396``) that doesn't resolve in the runtime evidence
|
||||
map → UNKNOWN_EVIDENCE_ID → UNGROUNDED, even when the rendered
|
||||
answer text correctly names Mr. Burns. The pointer variant uses
|
||||
short numeric pointer tags the model can't fabricate, so the
|
||||
grounding sticks. This fixture pins to pointer mode to document
|
||||
that cross-document relationships are pointer-mode's strength
|
||||
until JSON mode's evidence_id discipline tightens.
|
||||
Used to be pinned to claim_lattice_pointer mode: pre-2026-04-30
|
||||
the JSON mode prompt presented content-addressed evidence_ids
|
||||
(e.g. ``Eed1b6e396``) and Hermes-3-8B was fabricating
|
||||
plausible-looking near-miss IDs (``E1b6e396``) → UNGROUNDED on
|
||||
factually correct answers. Now JSON mode uses the same short
|
||||
pointer IDs as the pointer variant (E1, E2, …) — the model
|
||||
can't fabricate ``E27`` if only ``E1``-``E10`` were shown — so the
|
||||
fixture runs against the default (JSON) mode and grounds
|
||||
cleanly.
|
||||
"""
|
||||
r = _ask("who is homer simpson's boss?", mode="claim_lattice_pointer")
|
||||
r = _ask("who is homer simpson's boss?")
|
||||
assert _grounded(r), f"audit_mode={r.get('audit_mode')!r}"
|
||||
txt = _answer_lower(r)
|
||||
assert "burns" in txt, f"answer missing 'Burns': {txt[:300]}"
|
||||
|
|
|
|||
|
|
@ -77,8 +77,19 @@ def test_lenient_raises_on_truly_broken():
|
|||
# ---------------------------------------------------------------- JSON verifier
|
||||
|
||||
|
||||
def _ev(eid: str, span: str, role: str = "primary_answer_source") -> EvidenceObject:
|
||||
"""Stub evidence object with deterministic eid for the test."""
|
||||
def _ev(
|
||||
eid: str,
|
||||
span: str,
|
||||
role: str = "primary_answer_source",
|
||||
pointer_id: str | None = None,
|
||||
) -> EvidenceObject:
|
||||
"""Stub evidence object with deterministic eid for the test.
|
||||
|
||||
JSON-mode prompt-facing surface uses ``pointer_id`` (E1, E2, …)
|
||||
since 2026-04-30; if the test wants the verifier to resolve a
|
||||
citation, it must set ``pointer_id`` explicitly. ``evidence_id``
|
||||
stays content-addressed for the cache/run-DAG handle.
|
||||
"""
|
||||
return EvidenceObject(
|
||||
evidence_id=eid,
|
||||
source_root="00" * 32,
|
||||
|
|
@ -91,19 +102,19 @@ def _ev(eid: str, span: str, role: str = "primary_answer_source") -> EvidenceObj
|
|||
source_role=role,
|
||||
text_hash="22" * 32,
|
||||
span=span,
|
||||
pointer_id=None,
|
||||
pointer_id=pointer_id,
|
||||
)
|
||||
|
||||
|
||||
def test_verify_json_strict_when_all_claims_resolve():
|
||||
evidence = [
|
||||
_ev("E1f8e4c2a", "Brachiosaurus appears in the Jurassic Park film as a herbivore."),
|
||||
_ev("E2c9d7b3f", "Velociraptor is featured prominently throughout Jurassic Park."),
|
||||
_ev("E1f8e4c2a", "Brachiosaurus appears in the Jurassic Park film as a herbivore.", pointer_id="E1"),
|
||||
_ev("E2c9d7b3f", "Velociraptor is featured prominently throughout Jurassic Park.", pointer_id="E2"),
|
||||
]
|
||||
answer = json.dumps({
|
||||
"claims": [
|
||||
{"text": "Brachiosaurus appears in the film", "evidence_ids": ["E1f8e4c2a"]},
|
||||
{"text": "Velociraptor is featured", "evidence_ids": ["E2c9d7b3f"]},
|
||||
{"text": "Brachiosaurus appears in the film", "evidence_ids": ["E1"]},
|
||||
{"text": "Velociraptor is featured", "evidence_ids": ["E2"]},
|
||||
]
|
||||
})
|
||||
v = verify_claim_lattice_json(answer, evidence)
|
||||
|
|
@ -117,12 +128,12 @@ def test_verify_json_strict_when_all_claims_resolve():
|
|||
|
||||
def test_verify_json_hybrid_when_some_unknown_evidence_id():
|
||||
evidence = [
|
||||
_ev("E1f8e4c2a", "Brachiosaurus appears in the Jurassic Park film as a herbivore."),
|
||||
_ev("E1f8e4c2a", "Brachiosaurus appears in the Jurassic Park film as a herbivore.", pointer_id="E1"),
|
||||
]
|
||||
answer = json.dumps({
|
||||
"claims": [
|
||||
{"text": "Brachiosaurus appears in the film", "evidence_ids": ["E1f8e4c2a"]},
|
||||
{"text": "Made-up claim", "evidence_ids": ["EFAKEFAKE"]},
|
||||
{"text": "Brachiosaurus appears in the film", "evidence_ids": ["E1"]},
|
||||
{"text": "Made-up claim", "evidence_ids": ["E99"]},
|
||||
]
|
||||
})
|
||||
v = verify_claim_lattice_json(answer, evidence)
|
||||
|
|
@ -140,12 +151,12 @@ def test_verify_json_ungrounded_on_schema_invalid():
|
|||
def test_verify_json_recovers_from_markdown_fence():
|
||||
"""JSON-fenced output still parses & verifies; fence fixup logged."""
|
||||
evidence = [
|
||||
_ev("E1f8e4c2a", "Brachiosaurus appears in the Jurassic Park film as a herbivore."),
|
||||
_ev("E1f8e4c2a", "Brachiosaurus appears in the Jurassic Park film as a herbivore.", pointer_id="E1"),
|
||||
]
|
||||
answer = (
|
||||
'```json\n'
|
||||
'{"claims": [{"text": "Brachiosaurus appears in the film", '
|
||||
'"evidence_ids": ["E1f8e4c2a"]}]}\n'
|
||||
'"evidence_ids": ["E1"]}]}\n'
|
||||
'```'
|
||||
)
|
||||
v = verify_claim_lattice_json(answer, evidence)
|
||||
|
|
@ -157,13 +168,13 @@ def test_verify_json_manual_quote_violation():
|
|||
"""Strict no-double-quote rule — even valid JSON with double quotes
|
||||
inside a claim's text field fails MANUAL_QUOTE_VIOLATION."""
|
||||
evidence = [
|
||||
_ev("E1f8e4c2a", "Brachiosaurus appears in the Jurassic Park film as a herbivore."),
|
||||
_ev("E1f8e4c2a", "Brachiosaurus appears in the Jurassic Park film as a herbivore.", pointer_id="E1"),
|
||||
]
|
||||
answer = json.dumps({
|
||||
"claims": [
|
||||
{
|
||||
"text": 'Brachiosaurus is "a herbivore" appears in the film',
|
||||
"evidence_ids": ["E1f8e4c2a"],
|
||||
"evidence_ids": ["E1"],
|
||||
}
|
||||
]
|
||||
})
|
||||
|
|
@ -176,11 +187,11 @@ def test_verify_json_blocks_disallowed_source_role():
|
|||
"""Evidence resolved but source_role outside the allowlist fails
|
||||
SOURCE_ROLE_BLOCKED."""
|
||||
evidence = [
|
||||
_ev("E1f8e4c2a", "Brachiosaurus content here.", role="noisy_background_source"),
|
||||
_ev("E1f8e4c2a", "Brachiosaurus content here.", role="noisy_background_source", pointer_id="E1"),
|
||||
]
|
||||
answer = json.dumps({
|
||||
"claims": [
|
||||
{"text": "Brachiosaurus appears", "evidence_ids": ["E1f8e4c2a"]},
|
||||
{"text": "Brachiosaurus appears", "evidence_ids": ["E1"]},
|
||||
]
|
||||
})
|
||||
v = verify_claim_lattice_json(answer, evidence)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue