# `aborist.wikitext`

A single function: `to_base(raw)`. Converts MediaWiki wikitext to
plain prose deterministically.

```python
from aborist.wikitext import to_base, BASE_VERSION

prose = to_base("[[The Beatles]] are an [[English rock band]] from [[Liverpool]].")
# → "The Beatles are an English rock band from Liverpool."
```

## Why it exists

The corpus stores raw wikitext (so the link graph is recoverable
on demand) but the LLM and verifier both want plain prose. Reasons:

1. **Token efficiency.** Wikipedia chunks ship to Hermes with ~43%
   fewer tokens after wikitext-strip — bigger context window for
   the same chars budget.
2. **Verbatim citation.** The model can quote source paragraphs
   verbatim instead of escaping `[[wikilinks]]`. The verifier's
   substring test then matches cleanly.
3. **Pinned identity.** `BASE_VERSION='wikitext-base-v1'` lives in
   `policy["base_version"]`, which folds into
   `governance_policy_hash`. Bumping `BASE_VERSION` invalidates
   every prior cache record on next lookup — same discipline as
   `chunking_version` and `canonicalization_version`.

## Hot-path discipline

`to_base()` runs on the assembled context **before** the LLM call
in `aborist/qa/runner.py` and `aborist/qa/query.py`, **and again
inside `verify_quotes`** so the verifier compares like-against-like.
Both sides see prose.

## Optional dependency

Backed by `mwparserfromhell`. Install via `pip install '.[wikitext]'`
to enable. Without the dep, `_wikitext_to_base = None` and
`policy["base_version"] = None` — graceful fallback leaves raw
wikitext in both context and verifier (works, just less efficient).

## Source

[`aborist/wikitext.py`](../../aborist/wikitext.py)