arborist/aborist
russell@unturf.com 10c1cba6e0
qa: 4th verifier strategy — paraphrase via token-coverage; strip trailing citations
Two changes that make the verifier accept more honest grounding without
relaxing what STRICT means.

1. Strip trailing citation parentheticals at extraction time.
   `extract_quotes` and `extract_claim_spans` now drop a single trailing
   `(Source: ...)` / `(citing X)` / `(https://...)` parenthetical
   before returning. The Pikachu real-corpus case: model wrote the
   verbatim source sentence and appended `(Source: https://...)`. The
   prose IS in the corpus — only the inserted citation broke substring
   match. Now substring fires; STRICT recovered.

   Conservative regex: only strips when the parenthetical content
   starts with a citation cue word OR contains a URL. A genuine prose
   parenthetical like "Pikachu (a Pokémon species) lives in forests"
   stays.

2. Paraphrase strategy as a 4th verifier method.
   In the span path, items that fail substring match get a
   token-coverage probe. Tokens >=4 chars from the span; if >= 85% of
   them appear in the normalized base context, count the span as
   paraphrase-verified. Paraphrase items contribute to n_verified;
   verifier_method flips to 'paraphrase' when any soft-verified items
   are present so an auditor can tell.

   Gated by `_is_prose_span`: a list of proper nouns ("Keanu Reeves,
   Laurence Fishburne") has no lowercase content tokens >=4 chars and
   falls through to the entity strategy where proximity policy can
   tight-cluster check. Real prose spans ("Pikachu is a species of
   Pokémon creatures...") have multiple lowercase content tokens and
   qualify for paraphrase.

Soft-signal note (from CLAUDE.md soft/hard rule): paraphrase is
heuristic, not byte-equivalence. The hard chain still records the
classification — verifier_method='paraphrase' is the explicit marker
that an audit reader can use to distinguish lexical-verbatim from
paraphrase-overlap. Quote strategy stays verbatim-only — quotes ARE
quotes; paraphrase-in-quotes is the model's mistake to flag, not
auto-promote.

Schema: verifier_method CHECK constraint expanded to include
'paraphrase'. New idempotent migration `_rebuild_providence_cache_paraphrase`
mirrors `_rebuild_providence_cache_ungrounded` (table rebuild via
BEGIN IMMEDIATE; no value translation needed). Probes existing CHECK
from sqlite_master and no-ops if already expanded.

Tests: 9 new in tests/test_verify.py — 4 trailing-citation strip
(quote+span variants, URL-only parenthetical, end-to-end STRICT
recovery), 4 paraphrase strategy (high-coverage promote, low-coverage
reject, mixed-method label, quote-strategy preservation). Default
suite still green: 326 passed, 1 skipped.

Burned the live Pikachu cache record (audit chain extended) so a
fresh `make query Q="who is pikachu?"` exercises the new verifier
end-to-end.
2026-04-29 10:43:55 -04:00
..
distill storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
mesh mesh: per-peer audit chain-of-claims tracking on the wire 2026-04-28 19:41:25 -04:00
qa qa: 4th verifier strategy — paraphrase via token-coverage; strip trailing citations 2026-04-29 10:43:55 -04:00
search verify: layered strategies + entity policies, rename VISUAL → UNGROUNDED 2026-04-28 16:58:31 -04:00
sources crawler: skip feeds + sitemaps at ingest (they're discovery, not knowledge) 2026-04-29 05:55:28 -04:00
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
cli.py qa: 'aborist inspect' — sidecar diagnose for unverified spans 2026-04-29 10:29:06 -04:00
compress.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
document.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
evict.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
ingest.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
merkle.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
progress.py progress reporter + structured benchmark 2026-04-27 11:37:20 -04:00
snapshot.py add corpus-level snapshots: single-hash identity for the forest 2026-04-27 21:29:10 -04:00
source.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
store.py qa: 4th verifier strategy — paraphrase via token-coverage; strip trailing citations 2026-04-29 10:43:55 -04:00
wikitext.py verify: strip wikitext from context before substring matching 2026-04-28 15:48:07 -04:00