qa: per-claim status taxonomy on verifier + repair-action plans on sidecar

Two paired enhancements (D + E from the toy-Hermes design pass):

D. Per-claim status taxonomy on `verify_quotes`.

   New `claim_statuses` field on every verdict — a per-evidence-unit
   list with three labels:

       VERIFIED_QUOTE        unit substring-matched in normalized context
                             (any of quote/span/entity strategies)
       SUPPORTED_PARAPHRASE  unit cleared the paraphrase token-coverage
                             threshold (≥85% topical tokens present)
       UNSUPPORTED           unit didn't match anything

   Diagnostic labels (QUOTE_INTEGRITY_FAILED, SOURCE_MISMATCH,
   FALSIFIED) stay in the sidecar / falsification machinery — the
   binary-verifier discipline holds. Empty list when no evidence
   was extracted at all (verifier_method='none'). Backward-compatible:
   existing fields (audit_mode, n_quotes, n_verified, unverified_quotes,
   verifier_method) unchanged; current callers ignore the new field.

E. Repair-action plans on sidecar diagnoses.

   Each `_classify_span` diagnosis now carries a `repair` field with
   a concrete suggestion the operator can act on:

       synthetic_elision_inside_quote → split_into_two_quotes
                                        (when both halves verbatim)
                                      → trim_to_verified_half
                                        (when only one half verbatim)
                                      → remove_claim
       interior_elision               → include_aside_for_verbatim
                                        (with the dropped aside text)
       trailing_artifact              → trim_trailing_artifact
                                        (with the kept_prefix string)
       paraphrase                     → downgrade_to_paraphrase
       partial_paraphrase             → split_or_remove
       no_overlap                     → remove_claim

   Read-only suggestions — sidecar still doesn't write to providence_cache
   or audit_events. The repair stage is recommendation, not mutation.
   Operator (or an automated repair pass) decides whether to act.
   Human render in `aborist inspect` shows `repair: <action>  (<reason>)`
   under each diagnosis line.

Tests:
- verify: claim_statuses_quote_path_labels_each_unit (per-quote VERIFIED
  / UNSUPPORTED), claim_statuses_paraphrase_method_flagged,
  claim_statuses_empty_when_no_evidence.
- inspect: repair_synthetic_elision_split_when_both_halves_verbatim,
  repair_interior_elision_includes_aside, repair_trailing_artifact_trim,
  repair_no_overlap_remove.

462 tests pass (verify +3, inspect +4).
This commit is contained in:
russell@unturf.com 2026-04-29 18:37:52 -04:00
parent d0a3d93836
commit 3cdebb7e2a
No known key found for this signature in database
5 changed files with 256 additions and 2 deletions

View file

@ -144,6 +144,67 @@ def test_classify_synthetic_elision_does_not_fire_when_source_has_brackets():
assert out["diagnosis"] == "verbatim_in_base"
def test_repair_synthetic_elision_split_when_both_halves_verbatim():
"""Repair plan for `"prefix [...] suffix"` where source carries
both halves verbatim: split into two quotes."""
base = (
"The film centers on the fictional Isla Nublar, in Costa Rica. "
"Universal Studios acquired the rights to the novel before publication."
)
span = (
"The film centers on the fictional Isla Nublar [...] Universal Studios "
"acquired the rights to the novel"
)
out = _classify_span(span, _norm(base), _norm(base))
assert out["diagnosis"] == "synthetic_elision_inside_quote"
assert out["repair"]["action"] == "split_into_two_quotes"
assert len(out["repair"]["quotes"]) == 2
def test_repair_interior_elision_includes_aside():
"""Repair plan for parenthetical-elision: rewrite quote to include
the dropped aside so it becomes verbatim."""
base = (
"Clark Joseph Kent (middle name is also Jerome according to some "
"versions) is a fictional character created by Jerry Siegel and Joe Shuster."
)
span = (
"Clark Joseph Kent is a fictional character created by Jerry Siegel "
"and Joe Shuster."
)
out = _classify_span(span, _norm(base), _norm(base))
assert out["diagnosis"] == "interior_elision"
assert out["repair"]["action"] == "include_aside_for_verbatim"
assert "jerome" in out["repair"]["aside_to_restore"].lower()
def test_repair_trailing_artifact_trim():
"""Repair plan for model-appended `(Source: ...)`: trim the tail.
Trailing-artifact probe requires 60 char matching prefix, so the
test uses a long-enough prose span."""
base = (
"Pikachu can store electricity in its cheeks and release it in "
"lightning-based attacks. Pikachu evolves from Pichu."
)
span = (
"Pikachu can store electricity in its cheeks and release it in "
"lightning-based attacks. (Source: https://en.wikipedia.org/wiki/Pikachu)"
)
out = _classify_span(span, _norm(base), _norm(base))
assert out["diagnosis"] == "trailing_artifact"
assert out["repair"]["action"] == "trim_trailing_artifact"
assert "(Source:" not in out["repair"]["kept_prefix"]
def test_repair_no_overlap_remove():
"""Repair plan for full-invention spans: remove the claim."""
base = "Pikachu is a Pokémon species."
span = "The Roman Senate convened in 49 BC to debate Caesar's rebellion"
out = _classify_span(span, _norm(base), _norm(base))
assert out["diagnosis"] == "no_overlap"
assert out["repair"]["action"] == "remove_claim"
def test_classify_paraphrase_high_token_coverage():
"""Tokens all present, sequence different — model rewrote the source."""
base = (