verify: strip wikitext from context before substring matching

Adds aborist/wikitext.py with to_base() — deterministic wikitext →
prose conversion via mwparserfromhell. Drops <ref>...</ref>,
[[File:...]], [[Image:...]], [[Category:...]] entirely; resolves
piped wikilinks to display text; collapses templates, formatting,
and HTML markup. extract_wikilinks() preserves the link graph for
the definition-cloud artifact (recoverable from any page on demand
without re-parsing wikitext at query time).

Wires to_base() into verify_quotes(). Without the strip, the
verifier compared the model's clean prose against [[Cloud Strife]],
[[Shinra Electric Power Company|Shinra]], etc. and falsely flagged
real source quotes as VISUAL. Concrete case from a make-query run:
'Cloud Strife, an unsociable mercenary who claims to be a former
1st Class member of Shinra's SOLDIER unit;' is verbatim in the
Final_Fantasy_VII article wikitext (modulo markup). With the strip
that quote now verifies; the genuine model hallucinations in the
same answer still flag honestly. Side-effect: 43% smaller context
size on average so less LLM token waste.

BASE_VERSION = 'wikitext-base-v1' is the algorithm pin. Bump when
the strip rules change. Soft-imported in verify.py so environments
without mwparserfromhell installed degrade gracefully (no strip,
same behavior as before this commit).

Build:
- pyproject.toml: new [wikitext] extras (mwparserfromhell>=0.6),
  pulled in by [dev]
- Makefile: chain-check / chain-check-shards targets — fast
  audit-chain integrity probe, counts dangling prev_event_hash
  references; 0 = chain intact

Tests:
- tests/test_wikitext.py: 31 tests (rules, idempotence, real-corpus
  fixture)
- tests/test_verify.py: 2 regression tests pinning the FF7 flip
  (Cloud Strife quote: VISUAL → STRICT after strip; genuine
  hallucination: stays VISUAL)
- tests/fixtures/ff7_characters_chunk0.wikitext: real chunk from a
  shard, used to validate the strip on actual Wikipedia content

Makefile

@@ -30,7 +30,8 @@ SEARCH_Q     ?= computer
         ingest-xml-attached ingest-abstract \
         ingest-grok ingest-grok-media \
         ingest-self ingest-git ingest-hg \
-        verify search stats test docs clean clean-db clean-data help
+        verify search stats test docs chain-check chain-check-shards \
+        clean clean-db clean-data help
 
 all: bootstrap fetch-cur ingest-cur verify stats ## bootstrap → fetch cur → ingest cur → verify → stats
 
@@ -143,6 +144,23 @@ verify-shards: bootstrap ## cross-shard Merkle round-trip on a random sample
 analyze-shards: bootstrap ## cross-shard compression spectrum + audit integrity
 	$(ABORIST) --shards-dir $(SHARDS_DIR) analyze
 
+# Audit-chain integrity probe: counts dangling prev_event_hash references.
+# Faster than `analyze` and trivially scriptable. 0 = chain intact.
+define CHAIN_CHECK_SQL
+SELECT COUNT(*) AS chain_breaks FROM audit_events a1
+ LEFT JOIN audit_events a2 ON a2.event_hash = a1.prev_event_hash
+ WHERE a1.prev_event_hash IS NOT NULL AND a2.event_hash IS NULL
+endef
+export CHAIN_CHECK_SQL
+
+chain-check: ## audit-chain break count for $(DB) (0 = intact)
+	@printf '%s ' "$(DB)"; sqlite3 $(DB) "$$CHAIN_CHECK_SQL"
+
+chain-check-shards: ## audit-chain break count for every *.db in $(SHARDS_DIR)
+	@for db in $(SHARDS_DIR)/*.db; do \
+	  printf '%s ' "$$db"; sqlite3 "$$db" "$$CHAIN_CHECK_SQL"; \
+	done
+
 # Sequential per-shard distill (one process iterates all shards).
 distill-shards: bootstrap ## distill every shard in $(SHARDS_DIR), surface -> depth=1 cores
 	$(ABORIST) --shards-dir $(SHARDS_DIR) distill --process first-sentence-v1 --kind surface

aborist/qa/verify.py

@@ -29,6 +29,16 @@ Hard rule (CLAUDE.md "soft hash vs hard hash"): every check is a lexical
 substring test under norm-v1 + lowercase canonicalization. No embeddings,
 no semantic similarity, no fuzzy alignment. The contract is "this token
 sequence either is or isn't in the context."
+
+Wikitext context is run through ``aborist.wikitext.to_base`` before the
+substring test. The corpus stores raw wikitext (so the link graph is
+recoverable from any page), but the LLM produces clean prose. Without
+the strip, every wikilink-carrying source paragraph compares as
+"different surface form" and the verifier wrongly reports VISUAL on
+genuine source-grounded quotes. With the strip, paraphrases of *markup*
+(``[[Cloud]]`` vs ``Cloud``) verify, while paraphrases of *prose* still
+flag honestly. mwparserfromhell is an optional dep; if absent, the
+strip is a no-op and verification falls back to today's behavior.
 """
 
 from __future__ import annotations
@@ -36,6 +46,11 @@ from __future__ import annotations
 import re
 import unicodedata
 
+try:
+    from aborist.wikitext import to_base as _wikitext_to_base
+except ImportError:  # pragma: no cover
+    _wikitext_to_base = None
+
 
 # Locate every double-quote character (ASCII or curly). Sequential
 # pairing in extract_quotes() turns these into intentional (open, close)
@@ -254,6 +269,11 @@ def verify_quotes(
             f"entity_policy must be one of {ENTITY_POLICIES}, got {entity_policy!r}"
         )
 
+    # Wikitext markup → plain prose. Identity if mwparserfromhell isn't
+    # installed (extras: pip install 'aborist[wikitext]').
+    if _wikitext_to_base is not None:
+        context = _wikitext_to_base(context)
+
     norm_ctx = _normalize(context)
 
     # Strategy 1: explicit double-quoted spans.

aborist/wikitext.py

@@ -0,0 +1,139 @@
+"""Wikitext → base prose conversion.
+
+Aborist stores raw MediaWiki wikitext in ``chunks.content`` so the link
+graph and original markup are recoverable from any page on demand. For
+LLM context and post-LLM faithfulness verification we need *prose* — a
+deterministic plain-text projection of the same chunk.
+
+This module provides that projection. ``to_base(raw)`` is a pure function
+of its input plus ``BASE_VERSION``: same wikitext → same prose, forever,
+as long as ``BASE_VERSION`` is unchanged.
+
+Versioning protocol
+-------------------
+Bump ``BASE_VERSION`` whenever the algorithm changes. Callers fold
+``BASE_VERSION`` into ``governance_policy_hash`` (via ``policy["base_version"]``
+in ``aborist.qa.runner`` / ``aborist.qa.query``) so a bump invalidates every
+prior providence-cache record's 8-dim cache_key on the next lookup. No
+schema migration; the next ``ask`` re-derives against fresh prose.
+
+Algorithm (wikitext-base-v1)
+----------------------------
+1. Parse with ``mwparserfromhell`` (handles nested templates, complex
+   tables, and edge cases that pure regex mangles).
+2. Drop ``<ref>...</ref>`` and self-closing ``<ref ... />`` tags. Citations
+   are not quotable claims about the topic.
+3. Drop namespace-prefixed wikilinks: ``[[File:...]]``, ``[[Image:...]]``,
+   ``[[Category:...]]``. Image params (``thumb|250px|...``) and category
+   tags are not prose; they're metadata.
+4. ``strip_code(normalize=True, collapse=True)`` — converts surviving
+   templates to empty, wikilinks to their display text, headers to bare
+   text, bold/italic markers to plain text, HTML tags to inner text,
+   HTML entities to characters, external links to anchor text.
+5. Whitespace pass: collapse runs of spaces/tabs, drop trailing space on
+   lines, collapse 3+ newlines to 2.
+
+Optional dependency. Install with ``pip install aborist[wikitext]``.
+"""
+
+from __future__ import annotations
+
+import re
+
+try:
+    import mwparserfromhell as _mw
+except ImportError as e:  # pragma: no cover
+    raise ImportError(
+        "wikitext base conversion requires extras: "
+        "pip install 'aborist[wikitext]'"
+    ) from e
+
+
+BASE_VERSION = "wikitext-base-v1"
+
+# Namespaces whose links carry no prose. ``File`` and ``Image`` are the
+# same target type (image inclusion); MediaWiki accepts both prefixes.
+# ``Category`` tags categorize a page but don't render as readable prose
+# in the article body.
+_DROP_NAMESPACES = frozenset({"file", "image", "category"})
+
+_WS_RUN = re.compile(r"[ \t]+")
+_TRAILING_WS = re.compile(r" +\n")
+_BLANK_LINES = re.compile(r"\n{3,}")
+
+
+def to_base(raw: str) -> str:
+    """Convert raw wikitext to base prose. Deterministic. Idempotent.
+
+    Empty / whitespace-only input returns ``""``. Non-wikitext input
+    (already-clean prose) round-trips unchanged modulo whitespace
+    collapsing.
+    """
+    if not raw or not raw.strip():
+        return ""
+
+    code = _mw.parse(raw)
+
+    # Drop <ref>...</ref> and self-closing <ref ... /> tags. We match on
+    # the tag name (case-insensitive) so that <REF>, <Ref>, etc. all go.
+    for tag in list(code.filter_tags()):
+        if str(tag.tag).strip().lower() == "ref":
+            try:
+                code.remove(tag)
+            except ValueError:
+                # Tag was already removed via a parent node. mwparserfromhell
+                # raises rather than no-op'ing; we swallow it.
+                pass
+
+    # Drop File: / Image: / Category: wikilinks. Image captions sometimes
+    # contain useful prose ("thumb|250px|<caption>") but the technical
+    # parameters dominate and corrupt the prose stream; cleaner to drop.
+    for link in list(code.filter_wikilinks()):
+        title = str(link.title).strip()
+        if ":" in title:
+            ns = title.split(":", 1)[0].strip().lower()
+            if ns in _DROP_NAMESPACES:
+                try:
+                    code.remove(link)
+                except ValueError:
+                    pass
+
+    base = code.strip_code(normalize=True, collapse=True)
+
+    # Whitespace normalization — keeps paragraph breaks, drops runs.
+    base = _WS_RUN.sub(" ", base)
+    base = _TRAILING_WS.sub("\n", base)
+    base = _BLANK_LINES.sub("\n\n", base)
+    return base.strip()
+
+
+def extract_wikilinks(raw: str) -> list[tuple[str, str | None]]:
+    """Return ``(target, display)`` for every wikilink in ``raw``.
+
+    ``target`` is the link target (page title) with any ``#section``
+    fragment stripped. ``display`` is the visible text if the link uses
+    ``[[Target|Display]]`` form, else ``None``.
+
+    File / Image / Category namespace links are *included* — they're the
+    very signal the link graph wants. This is the "definition cloud"
+    artifact: from any page, recover its full out-link set without
+    re-parsing wikitext at query time.
+    """
+    if not raw or not raw.strip():
+        return []
+    code = _mw.parse(raw)
+    out: list[tuple[str, str | None]] = []
+    for link in code.filter_wikilinks():
+        title = str(link.title).strip()
+        # Strip ``#section`` so two links to "Foo#bar" and "Foo#baz"
+        # collapse to one ("Foo") in the link graph. Sections are rarely
+        # the meaningful unit downstream.
+        if "#" in title:
+            title = title.split("#", 1)[0].strip()
+        if not title:
+            continue
+        text = str(link.text).strip() if link.text is not None else None
+        if text == "":
+            text = None
+        out.append((title, text))
+    return out

pyproject.toml

@@ -24,9 +24,13 @@ dependencies = [
 html = [
   "selectolax>=0.3",
 ]
+wikitext = [
+  "mwparserfromhell>=0.6",
+]
 dev = [
   "pytest>=8",
   "aborist[html]",
+  "aborist[wikitext]",
 ]
 
 [project.scripts]

tests/fixtures/ff7_characters_chunk0.wikitext

@@ -0,0 +1 @@
+{{Refimprove|date=October 2009}} {{Original research|date=July 2008}} [[File:FFVIInomuracastdesigns.JPG|thumb|250px|Tetsuya Nomura's designs of the main playable characters in the original ''Final Fantasy VII'' game. Clockwise from top right: Cait Sith, Tifa, Barret, Cloud, Aerith, Yuffie, Red XIII, Vincent, Cid.]] {{Final Fantasy characters}} ''[[Final Fantasy VII]]'', a [[console role-playing game]] developed by [[Square Co.|Square]] (now [[Square Enix]]) and originally released in 1997, features a large number of [[fictional character]]s in major and minor roles. The game follows [[protagonist]] [[Cloud Strife]], a troubled mercenary who joins with several others to stop the [[megacorporation]] [[Gaia (Final Fantasy VII)#Shinra Electric Power Company|Shinra]] from draining the life of the planet to use as an energy source. As the story progresses, conflicts escalate and the world's safety becomes the central concern as new forces emerge to challenge the original group. Cloud and his team eventually face [[Sephiroth (Final Fantasy)|Sephiroth]], the game's main [[antagonist]]. The characters and the world they inhabit (known as [[Gaia (Final Fantasy VII)|Gaia]] but originally just referred to as "The Planet") have been expanded on in a series of spin-off games and movies known as ''[[Compilation of Final Fantasy VII]]''. The original ''Final Fantasy VII'' game features nine [[playable character]]s, two of whom—[[Characters of the Final Fantasy VII series#Yuffie Kisaragi|Yuffie]] and [[Vincent Valentine|Vincent]]—are optional. These nine are viewed by many as the main characters of the ''Compilation'' series. Their roles are minor outside the original game but each character has nevertheless become extremely popular. ==Concept and creation== When looking at the story of the original ''Final Fantasy VII'', [[character creation|character designer]] [[Tetsuya Nomura]] decided it was very dark and needed characters who reflected that. Thus Cloud's original character design called for slicked back black hair with no spikes, intended to serve as a contrast to Sephiroth's long, flowing silver hair.<ref name="Khosla">{{cite web|url= http://flaregamer.com/b2article.php?p=81&more=1|title=Tetsuya Nomura 20s|publisher=FLAREgamer|author=Khosla, Sheila|year=2003 |accessdate=2006-04-13}}</ref> However, to give Cloud a unique feature that would emphasize his role in the game as the main character, Nomura changed the design to feature Cloud's now trademark shock of spiky, bright blond hair.<ref name="Khosla"/> For Tifa's design, Nomura has admitted to facing a difficult decision in choosing to give her a miniskirt or pants. With input from other members of the game's development staff, he eventually selected a dark miniskirt, contrasted by Aerith' long, pink dress.<ref name="Khosla"/> Vincent's character developed from horror researcher to detective, then to chemist, and finally to the figure of a former Turk with a tragic past. It has been explained that his crimson mantle was added to symbolize the idea of carrying a heavy weight on his shoulders associated with death. Nomura has indicated that Cid Highwind's fighting style resembles that of a [[Dragoon (character class)|Dragon Knight]], a [[character class]] chosen because his last name is the same as that of two previous Dragoon featured in the ''Final Fantasy'' series, [[Final Fantasy II#Characters|Ricard Highwind]] of ''[[Final Fantasy II]]'' and [[Kain Highwind]] of ''[[Final Fantasy IV]]''.<ref name="Khosla"/> Although the game was Nomura's favorite ''Final Fantasy'' project, he felt that ''Final Fantasy VII'' was hindered by graphical limitations, and that his designs were, consequently, very plain in comparison

tests/test_verify.py

@@ -303,3 +303,45 @@ def test_entity_path_hybrid_when_some_proper_nouns_unverified():
     assert v["audit_mode"] == "HYBRID"
     assert v["verifier_method"] == "entity"
     assert "Marlon Brando" in v["unverified_quotes"]
+
+
+# ----------------------------------------------- wikitext-base-v1 integration
+
+
+def test_wikitext_context_strips_markup_before_substring_test():
+    """Without the strip the verifier sees [[Cloud Strife]] in source and a
+    bare 'Cloud Strife' in the answer's quoted span; substring fails. With
+    aborist.wikitext.to_base wired into verify_quotes, the context becomes
+    plain prose and the model's quote verifies. This is the case from the
+    real FF7 query that prompted wiring wikitext-base-v1 into the verifier."""
+    raw_wikitext_context = (
+        "[[Cloud Strife]], an unsociable mercenary who claims to be a former "
+        "1st Class member of [[Shinra Electric Power Company|Shinra]]'s "
+        "[[SOLDIER (Final Fantasy VII)|SOLDIER]] unit;<ref>Smith 2010</ref>"
+    )
+    answer = (
+        'The protagonist is described as: "Cloud Strife, an unsociable '
+        "mercenary who claims to be a former 1st Class member of Shinra's "
+        'SOLDIER unit;"'
+    )
+    v = verify_quotes(answer, raw_wikitext_context)
+    assert v["verifier_method"] == "quote"
+    assert v["n_verified"] == 1
+    assert v["audit_mode"] == "STRICT"
+    assert v["unverified_quotes"] == []
+
+
+def test_wikitext_strip_does_not_rescue_genuine_hallucination():
+    """A quote the model invented stays unverified even after stripping."""
+    raw_wikitext_context = (
+        "[[Cloud Strife]] is the [[protagonist]] of [[Final Fantasy VII]]."
+    )
+    answer = (
+        'The story is: "Set in a dystopian world, Final Fantasy VII\'s story '
+        'centers on mercenary Cloud Strife..."'
+    )
+    v = verify_quotes(answer, raw_wikitext_context)
+    assert v["verifier_method"] == "quote"
+    assert v["n_verified"] == 0
+    assert v["audit_mode"] == "VISUAL"
+    assert len(v["unverified_quotes"]) == 1

tests/test_wikitext.py

@@ -0,0 +1,256 @@
+"""Unit tests for aborist.wikitext — wikitext-base-v1.
+
+Most tests are tiny synthetic inputs that pin one rule each. The
+``test_real_chunk_*`` tests use a real Wikipedia chunk (FF7 characters)
+pulled from a live shard so we catch realistic interactions: nested
+templates, mixed namespaces, refs inside paragraphs.
+
+The fixture file ``tests/fixtures/ff7_characters_chunk0.wikitext`` is
+chunk 0 of ``Characters_of_the_Final_Fantasy_VII_series`` and was the
+input that triggered the wikitext-base-v1 design (verifier flagged 6
+"quotes" as VISUAL because the model paraphrased the wikitext form).
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from aborist.wikitext import BASE_VERSION, extract_wikilinks, to_base
+
+
+FIXTURE = Path(__file__).parent / "fixtures" / "ff7_characters_chunk0.wikitext"
+
+
+def test_base_version_pinned():
+    """If you change BASE_VERSION you stale every cached answer. The pin
+    here is a tripwire: bumping requires intent."""
+    assert BASE_VERSION == "wikitext-base-v1"
+
+
+def test_empty_input():
+    assert to_base("") == ""
+    assert to_base("   \n\t  ") == ""
+
+
+def test_plain_prose_round_trips():
+    src = "This is plain prose with no markup."
+    assert to_base(src) == src
+
+
+def test_simple_wikilink():
+    assert to_base("Hello [[Cloud Strife]] world.") == "Hello Cloud Strife world."
+
+
+def test_piped_wikilink_uses_display_text():
+    assert to_base("Hello [[Cloud Strife|Cloud]] world.") == "Hello Cloud world."
+
+
+def test_sectioned_wikilink_uses_display_text():
+    src = "See [[Cloud Strife#weapons|the buster sword]]."
+    assert to_base(src) == "See the buster sword."
+
+
+def test_file_link_dropped_entirely():
+    src = "[[File:foo.jpg|thumb|250px|A caption with [[Cloud]] inside.]] body text"
+    out = to_base(src)
+    assert "thumb" not in out
+    assert "250px" not in out
+    assert "foo.jpg" not in out
+    assert "body text" in out
+
+
+def test_image_link_dropped_entirely():
+    src = "[[Image:bar.png|left|Some image]] body text"
+    out = to_base(src)
+    assert "left" not in out
+    assert "bar.png" not in out
+    assert "body text" in out
+
+
+def test_category_link_dropped():
+    src = "Body. [[Category:Final Fantasy VII characters]]"
+    out = to_base(src)
+    assert "Category:" not in out
+    assert "Body." in out
+
+
+def test_ref_tag_inline_dropped():
+    src = "Cloud is great.<ref>Smith 2010, p. 5.</ref> The end."
+    assert to_base(src) == "Cloud is great. The end."
+
+
+def test_ref_tag_named_dropped():
+    src = 'Cloud is great.<ref name="x">Smith 2010</ref> Reuse.<ref name="x"/> End.'
+    out = to_base(src)
+    assert "Smith" not in out
+    assert "Cloud is great." in out
+    assert "Reuse." in out
+    assert "End." in out
+
+
+def test_template_cite_dropped():
+    src = "Plain prose. {{cite book|title=Foo|author=Bar}} more."
+    out = to_base(src)
+    assert "Foo" not in out
+    assert "Plain prose." in out
+    assert "more." in out
+
+
+def test_template_infobox_dropped():
+    src = "{{Infobox character|name=Cloud|hair=blond}} Body."
+    assert to_base(src) == "Body."
+
+
+def test_bold_and_italic_preserve_text():
+    assert to_base("This is '''bold''' text.") == "This is bold text."
+    assert to_base("This is ''italic'' text.") == "This is italic text."
+
+
+def test_header_keeps_text():
+    out = to_base("== Section Title ==\nBody under it.")
+    assert "Section Title" in out
+    assert "Body under it." in out
+    assert "==" not in out
+
+
+def test_html_entities_decoded():
+    out = to_base("Tom &amp; Jerry, 50&nbsp;words.")
+    assert "Tom & Jerry" in out
+    assert "&amp;" not in out
+
+
+def test_html_tags_removed_text_kept():
+    out = to_base("Inline <small>caveat</small> text. <br/>Newline.")
+    assert "caveat" in out
+    assert "<" not in out
+    assert ">" not in out
+
+
+def test_external_link_with_text():
+    assert to_base("Visit [https://example.com our site].") == "Visit our site."
+
+
+def test_external_link_bare_dropped():
+    out = to_base("Visit [https://example.com] now.")
+    assert "https://" not in out
+    assert "Visit" in out
+    assert "now." in out
+
+
+def test_html_comment_dropped():
+    assert to_base("Visible <!-- hidden --> rest.") == "Visible rest."
+
+
+def test_idempotence():
+    """to_base(to_base(x)) == to_base(x). Critical for the verifier:
+    if context is already-clean prose (e.g. an HTML source), running
+    to_base on it must not corrupt the prose."""
+    raw = (
+        "Hello [[Cloud Strife|Cloud]]<ref>cite</ref>, see [[File:foo.jpg]] "
+        "and {{Infobox|x=1}} the end."
+    )
+    once = to_base(raw)
+    twice = to_base(once)
+    assert once == twice
+
+
+def test_nested_templates_dropped():
+    src = "{{cite |title={{lang|en|Foo}}}} body."
+    assert to_base(src) == "body."
+
+
+# ---------------------------------------------------------------------------
+# Real-corpus fixture: chunk 0 of Characters_of_the_Final_Fantasy_VII_series
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture
+def ff7_raw() -> str:
+    return FIXTURE.read_text()
+
+
+def test_real_chunk_strips_markup(ff7_raw):
+    """The chunk that triggered this design. After to_base, the named
+    characters should appear as bare prose, not wikitext."""
+    base = to_base(ff7_raw)
+    # Concrete topical content survives.
+    assert "Cloud Strife" in base
+    assert "Sephiroth" in base
+    assert "Final Fantasy VII" in base
+    # Wikitext markers are gone.
+    assert "[[" not in base
+    assert "]]" not in base
+    assert "{{" not in base
+    assert "}}" not in base
+    assert "<ref" not in base
+    assert "File:" not in base
+    assert "Category:" not in base
+
+
+def test_real_chunk_protagonist_phrase_recoverable(ff7_raw):
+    """The wikitext source has:
+        ...follows [[protagonist]] [[Cloud Strife]], a troubled mercenary...
+    After to_base this should read as continuous prose. We verify the
+    canonical FF7 pitch surfaces verbatim — the verifier can substring-test
+    against this exact form."""
+    base = to_base(ff7_raw)
+    assert "follows protagonist Cloud Strife, a troubled mercenary" in base
+
+
+def test_real_chunk_substantial_compression(ff7_raw):
+    """Sanity check: the base form should be smaller than raw (markup
+    removed) but not so small that we've lost the prose."""
+    base = to_base(ff7_raw)
+    assert len(base) < len(ff7_raw)
+    # FF7 chunk 0 is 3722 chars raw; base form is roughly 75% of that.
+    assert len(base) > len(ff7_raw) * 0.5
+
+
+# ---------------------------------------------------------------------------
+# extract_wikilinks — the link graph artifact
+# ---------------------------------------------------------------------------
+
+
+def test_extract_wikilinks_simple():
+    out = extract_wikilinks("Hello [[Cloud Strife]] world.")
+    assert out == [("Cloud Strife", None)]
+
+
+def test_extract_wikilinks_with_display():
+    out = extract_wikilinks("Hello [[Cloud Strife|Cloud]] world.")
+    assert out == [("Cloud Strife", "Cloud")]
+
+
+def test_extract_wikilinks_section_collapsed():
+    """[[Foo#bar]] and [[Foo#baz]] both collapse to ('Foo', None)."""
+    out = extract_wikilinks("See [[Foo#bar]] and [[Foo#baz|qux]].")
+    assert out == [("Foo", None), ("Foo", "qux")]
+
+
+def test_extract_wikilinks_includes_namespaces():
+    """File: / Image: / Category: are dropped from PROSE but kept in the
+    link graph — they are exactly the structural signal the graph wants."""
+    src = "[[File:x.jpg]] [[Category:Y]] [[Image:z.png|caption]] [[Cloud]]"
+    out = extract_wikilinks(src)
+    targets = [t for t, _ in out]
+    assert "File:x.jpg" in targets
+    assert "Category:Y" in targets
+    assert "Image:z.png" in targets
+    assert "Cloud" in targets
+
+
+def test_extract_wikilinks_empty_input():
+    assert extract_wikilinks("") == []
+    assert extract_wikilinks("   ") == []
+    assert extract_wikilinks("plain prose, no links.") == []
+
+
+def test_extract_wikilinks_real_chunk(ff7_raw):
+    out = extract_wikilinks(ff7_raw)
+    assert len(out) > 10
+    targets = {t for t, _ in out}
+    # A few links we expect from chunk 0.
+    assert "Cloud Strife" in targets
+    assert "Final Fantasy VII" in targets