verify: strip wikitext from context before substring matching
Adds aborist/wikitext.py with to_base() — deterministic wikitext → prose conversion via mwparserfromhell. Drops <ref>...</ref>, [[File:...]], [[Image:...]], [[Category:...]] entirely; resolves piped wikilinks to display text; collapses templates, formatting, and HTML markup. extract_wikilinks() preserves the link graph for the definition-cloud artifact (recoverable from any page on demand without re-parsing wikitext at query time). Wires to_base() into verify_quotes(). Without the strip, the verifier compared the model's clean prose against [[Cloud Strife]], [[Shinra Electric Power Company|Shinra]], etc. and falsely flagged real source quotes as VISUAL. Concrete case from a make-query run: 'Cloud Strife, an unsociable mercenary who claims to be a former 1st Class member of Shinra's SOLDIER unit;' is verbatim in the Final_Fantasy_VII article wikitext (modulo markup). With the strip that quote now verifies; the genuine model hallucinations in the same answer still flag honestly. Side-effect: 43% smaller context size on average so less LLM token waste. BASE_VERSION = 'wikitext-base-v1' is the algorithm pin. Bump when the strip rules change. Soft-imported in verify.py so environments without mwparserfromhell installed degrade gracefully (no strip, same behavior as before this commit). Build: - pyproject.toml: new [wikitext] extras (mwparserfromhell>=0.6), pulled in by [dev] - Makefile: chain-check / chain-check-shards targets — fast audit-chain integrity probe, counts dangling prev_event_hash references; 0 = chain intact Tests: - tests/test_wikitext.py: 31 tests (rules, idempotence, real-corpus fixture) - tests/test_verify.py: 2 regression tests pinning the FF7 flip (Cloud Strife quote: VISUAL → STRICT after strip; genuine hallucination: stays VISUAL) - tests/fixtures/ff7_characters_chunk0.wikitext: real chunk from a shard, used to validate the strip on actual Wikipedia content
This commit is contained in:
parent
1ef0ac753a
commit
22a8071936
7 changed files with 481 additions and 1 deletions
20
Makefile
20
Makefile
|
|
@ -30,7 +30,8 @@ SEARCH_Q ?= computer
|
|||
ingest-xml-attached ingest-abstract \
|
||||
ingest-grok ingest-grok-media \
|
||||
ingest-self ingest-git ingest-hg \
|
||||
verify search stats test docs clean clean-db clean-data help
|
||||
verify search stats test docs chain-check chain-check-shards \
|
||||
clean clean-db clean-data help
|
||||
|
||||
all: bootstrap fetch-cur ingest-cur verify stats ## bootstrap → fetch cur → ingest cur → verify → stats
|
||||
|
||||
|
|
@ -143,6 +144,23 @@ verify-shards: bootstrap ## cross-shard Merkle round-trip on a random sample
|
|||
analyze-shards: bootstrap ## cross-shard compression spectrum + audit integrity
|
||||
$(ABORIST) --shards-dir $(SHARDS_DIR) analyze
|
||||
|
||||
# Audit-chain integrity probe: counts dangling prev_event_hash references.
|
||||
# Faster than `analyze` and trivially scriptable. 0 = chain intact.
|
||||
define CHAIN_CHECK_SQL
|
||||
SELECT COUNT(*) AS chain_breaks FROM audit_events a1
|
||||
LEFT JOIN audit_events a2 ON a2.event_hash = a1.prev_event_hash
|
||||
WHERE a1.prev_event_hash IS NOT NULL AND a2.event_hash IS NULL
|
||||
endef
|
||||
export CHAIN_CHECK_SQL
|
||||
|
||||
chain-check: ## audit-chain break count for $(DB) (0 = intact)
|
||||
@printf '%s ' "$(DB)"; sqlite3 $(DB) "$$CHAIN_CHECK_SQL"
|
||||
|
||||
chain-check-shards: ## audit-chain break count for every *.db in $(SHARDS_DIR)
|
||||
@for db in $(SHARDS_DIR)/*.db; do \
|
||||
printf '%s ' "$$db"; sqlite3 "$$db" "$$CHAIN_CHECK_SQL"; \
|
||||
done
|
||||
|
||||
# Sequential per-shard distill (one process iterates all shards).
|
||||
distill-shards: bootstrap ## distill every shard in $(SHARDS_DIR), surface -> depth=1 cores
|
||||
$(ABORIST) --shards-dir $(SHARDS_DIR) distill --process first-sentence-v1 --kind surface
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue