verify: strip wikitext from context before substring matching

Adds aborist/wikitext.py with to_base() — deterministic wikitext →
prose conversion via mwparserfromhell. Drops <ref>...</ref>,
[[File:...]], [[Image:...]], [[Category:...]] entirely; resolves
piped wikilinks to display text; collapses templates, formatting,
and HTML markup. extract_wikilinks() preserves the link graph for
the definition-cloud artifact (recoverable from any page on demand
without re-parsing wikitext at query time).

Wires to_base() into verify_quotes(). Without the strip, the
verifier compared the model's clean prose against [[Cloud Strife]],
[[Shinra Electric Power Company|Shinra]], etc. and falsely flagged
real source quotes as VISUAL. Concrete case from a make-query run:
'Cloud Strife, an unsociable mercenary who claims to be a former
1st Class member of Shinra's SOLDIER unit;' is verbatim in the
Final_Fantasy_VII article wikitext (modulo markup). With the strip
that quote now verifies; the genuine model hallucinations in the
same answer still flag honestly. Side-effect: 43% smaller context
size on average so less LLM token waste.

BASE_VERSION = 'wikitext-base-v1' is the algorithm pin. Bump when
the strip rules change. Soft-imported in verify.py so environments
without mwparserfromhell installed degrade gracefully (no strip,
same behavior as before this commit).

Build:
- pyproject.toml: new [wikitext] extras (mwparserfromhell>=0.6),
  pulled in by [dev]
- Makefile: chain-check / chain-check-shards targets — fast
  audit-chain integrity probe, counts dangling prev_event_hash
  references; 0 = chain intact

Tests:
- tests/test_wikitext.py: 31 tests (rules, idempotence, real-corpus
  fixture)
- tests/test_verify.py: 2 regression tests pinning the FF7 flip
  (Cloud Strife quote: VISUAL → STRICT after strip; genuine
  hallucination: stays VISUAL)
- tests/fixtures/ff7_characters_chunk0.wikitext: real chunk from a
  shard, used to validate the strip on actual Wikipedia content
This commit is contained in:
russell@unturf.com 2026-04-28 15:48:07 -04:00
parent 1ef0ac753a
commit 22a8071936
No known key found for this signature in database
7 changed files with 481 additions and 1 deletions

View file

@ -30,7 +30,8 @@ SEARCH_Q ?= computer
ingest-xml-attached ingest-abstract \
ingest-grok ingest-grok-media \
ingest-self ingest-git ingest-hg \
verify search stats test docs clean clean-db clean-data help
verify search stats test docs chain-check chain-check-shards \
clean clean-db clean-data help
all: bootstrap fetch-cur ingest-cur verify stats ## bootstrap → fetch cur → ingest cur → verify → stats
@ -143,6 +144,23 @@ verify-shards: bootstrap ## cross-shard Merkle round-trip on a random sample
analyze-shards: bootstrap ## cross-shard compression spectrum + audit integrity
$(ABORIST) --shards-dir $(SHARDS_DIR) analyze
# Audit-chain integrity probe: counts dangling prev_event_hash references.
# Faster than `analyze` and trivially scriptable. 0 = chain intact.
define CHAIN_CHECK_SQL
SELECT COUNT(*) AS chain_breaks FROM audit_events a1
LEFT JOIN audit_events a2 ON a2.event_hash = a1.prev_event_hash
WHERE a1.prev_event_hash IS NOT NULL AND a2.event_hash IS NULL
endef
export CHAIN_CHECK_SQL
chain-check: ## audit-chain break count for $(DB) (0 = intact)
@printf '%s ' "$(DB)"; sqlite3 $(DB) "$$CHAIN_CHECK_SQL"
chain-check-shards: ## audit-chain break count for every *.db in $(SHARDS_DIR)
@for db in $(SHARDS_DIR)/*.db; do \
printf '%s ' "$$db"; sqlite3 "$$db" "$$CHAIN_CHECK_SQL"; \
done
# Sequential per-shard distill (one process iterates all shards).
distill-shards: bootstrap ## distill every shard in $(SHARDS_DIR), surface -> depth=1 cores
$(ABORIST) --shards-dir $(SHARDS_DIR) distill --process first-sentence-v1 --kind surface