qa: per-claim status taxonomy on verifier + repair-action plans on sidecar
Two paired enhancements (D + E from the toy-Hermes design pass):
D. Per-claim status taxonomy on `verify_quotes`.
New `claim_statuses` field on every verdict — a per-evidence-unit
list with three labels:
VERIFIED_QUOTE unit substring-matched in normalized context
(any of quote/span/entity strategies)
SUPPORTED_PARAPHRASE unit cleared the paraphrase token-coverage
threshold (≥85% topical tokens present)
UNSUPPORTED unit didn't match anything
Diagnostic labels (QUOTE_INTEGRITY_FAILED, SOURCE_MISMATCH,
FALSIFIED) stay in the sidecar / falsification machinery — the
binary-verifier discipline holds. Empty list when no evidence
was extracted at all (verifier_method='none'). Backward-compatible:
existing fields (audit_mode, n_quotes, n_verified, unverified_quotes,
verifier_method) unchanged; current callers ignore the new field.
E. Repair-action plans on sidecar diagnoses.
Each `_classify_span` diagnosis now carries a `repair` field with
a concrete suggestion the operator can act on:
synthetic_elision_inside_quote → split_into_two_quotes
(when both halves verbatim)
→ trim_to_verified_half
(when only one half verbatim)
→ remove_claim
interior_elision → include_aside_for_verbatim
(with the dropped aside text)
trailing_artifact → trim_trailing_artifact
(with the kept_prefix string)
paraphrase → downgrade_to_paraphrase
partial_paraphrase → split_or_remove
no_overlap → remove_claim
Read-only suggestions — sidecar still doesn't write to providence_cache
or audit_events. The repair stage is recommendation, not mutation.
Operator (or an automated repair pass) decides whether to act.
Human render in `aborist inspect` shows `repair: <action> (<reason>)`
under each diagnosis line.
Tests:
- verify: claim_statuses_quote_path_labels_each_unit (per-quote VERIFIED
/ UNSUPPORTED), claim_statuses_paraphrase_method_flagged,
claim_statuses_empty_when_no_evidence.
- inspect: repair_synthetic_elision_split_when_both_halves_verbatim,
repair_interior_elision_includes_aside, repair_trailing_artifact_trim,
repair_no_overlap_remove.
462 tests pass (verify +3, inspect +4).
This commit is contained in:
parent
d0a3d93836
commit
3cdebb7e2a
5 changed files with 256 additions and 2 deletions
|
|
@ -144,6 +144,67 @@ def test_classify_synthetic_elision_does_not_fire_when_source_has_brackets():
|
|||
assert out["diagnosis"] == "verbatim_in_base"
|
||||
|
||||
|
||||
def test_repair_synthetic_elision_split_when_both_halves_verbatim():
|
||||
"""Repair plan for `"prefix [...] suffix"` where source carries
|
||||
both halves verbatim: split into two quotes."""
|
||||
base = (
|
||||
"The film centers on the fictional Isla Nublar, in Costa Rica. "
|
||||
"Universal Studios acquired the rights to the novel before publication."
|
||||
)
|
||||
span = (
|
||||
"The film centers on the fictional Isla Nublar [...] Universal Studios "
|
||||
"acquired the rights to the novel"
|
||||
)
|
||||
out = _classify_span(span, _norm(base), _norm(base))
|
||||
assert out["diagnosis"] == "synthetic_elision_inside_quote"
|
||||
assert out["repair"]["action"] == "split_into_two_quotes"
|
||||
assert len(out["repair"]["quotes"]) == 2
|
||||
|
||||
|
||||
def test_repair_interior_elision_includes_aside():
|
||||
"""Repair plan for parenthetical-elision: rewrite quote to include
|
||||
the dropped aside so it becomes verbatim."""
|
||||
base = (
|
||||
"Clark Joseph Kent (middle name is also Jerome according to some "
|
||||
"versions) is a fictional character created by Jerry Siegel and Joe Shuster."
|
||||
)
|
||||
span = (
|
||||
"Clark Joseph Kent is a fictional character created by Jerry Siegel "
|
||||
"and Joe Shuster."
|
||||
)
|
||||
out = _classify_span(span, _norm(base), _norm(base))
|
||||
assert out["diagnosis"] == "interior_elision"
|
||||
assert out["repair"]["action"] == "include_aside_for_verbatim"
|
||||
assert "jerome" in out["repair"]["aside_to_restore"].lower()
|
||||
|
||||
|
||||
def test_repair_trailing_artifact_trim():
|
||||
"""Repair plan for model-appended `(Source: ...)`: trim the tail.
|
||||
Trailing-artifact probe requires ≥60 char matching prefix, so the
|
||||
test uses a long-enough prose span."""
|
||||
base = (
|
||||
"Pikachu can store electricity in its cheeks and release it in "
|
||||
"lightning-based attacks. Pikachu evolves from Pichu."
|
||||
)
|
||||
span = (
|
||||
"Pikachu can store electricity in its cheeks and release it in "
|
||||
"lightning-based attacks. (Source: https://en.wikipedia.org/wiki/Pikachu)"
|
||||
)
|
||||
out = _classify_span(span, _norm(base), _norm(base))
|
||||
assert out["diagnosis"] == "trailing_artifact"
|
||||
assert out["repair"]["action"] == "trim_trailing_artifact"
|
||||
assert "(Source:" not in out["repair"]["kept_prefix"]
|
||||
|
||||
|
||||
def test_repair_no_overlap_remove():
|
||||
"""Repair plan for full-invention spans: remove the claim."""
|
||||
base = "Pikachu is a Pokémon species."
|
||||
span = "The Roman Senate convened in 49 BC to debate Caesar's rebellion"
|
||||
out = _classify_span(span, _norm(base), _norm(base))
|
||||
assert out["diagnosis"] == "no_overlap"
|
||||
assert out["repair"]["action"] == "remove_claim"
|
||||
|
||||
|
||||
def test_classify_paraphrase_high_token_coverage():
|
||||
"""Tokens all present, sequence different — model rewrote the source."""
|
||||
base = (
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue