russell@unturf.com
326badf6d8
Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).
(1) README — label refresh:
- Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
- Verifier section spells out both layers (schema trichotomy +
display ladder), the seven hard checks of claim_lattice, and
the five anchor classes of warrant.
- Architecture tree updated: concepts/ package added, qa/
sub-modules expanded (warrant, evidence, parse_claims, dag),
verify.py described as quote/span/entity/paraphrase + claim_lattice.
- Concept overlay description updated for corpus-derived layer
(concept_relations table, link_reciprocity extractor, 1.6%
tax cite).
- Test count: 326+ → 641+.
(2) docs/diagrams/ — Graphviz dot sources:
- aborist-modules.dot — top-level package graph (substrate /
storage / sources / retrieval / qa / mesh / cli)
- query-pipeline.dot — question → cache → retrieval → LLM →
verify → render → cache write, with phase budgets
- ingest-pipeline.dot — source doc → canonicalize → chunk →
Merkle → upsert (+ optional distill)
- verifier-ladder.dot — (audit_mode, violations) → display rung
decision tree
Existing mesh-*.dot kept as-is. Makefile `make docs` target
extended to also emit .svg alongside the existing .png so the
diagrams render in markdown viewers.
(3) docs/modules/ — per-module reference pages:
- index.md (links to every diagram + every module page)
- merkle.md, document.md, store.md, ingest.md, evict.md,
sources.md, search.md, concepts.md, qa.md, distill.md,
wikitext.md
Each page is a one-screenful concise reference: what the
module is for, public API, key invariants, embedded diagrams
where useful, link to source. Mesh stays at the existing
docs/mesh.md + docs/mesh-deploy.md (already comprehensive).
Tests: 641 passed (no code change).
46 lines
1.7 KiB
Markdown
46 lines
1.7 KiB
Markdown
# `aborist.wikitext`
|
|
|
|
A single function: `to_base(raw)`. Converts MediaWiki wikitext to
|
|
plain prose deterministically.
|
|
|
|
```python
|
|
from aborist.wikitext import to_base, BASE_VERSION
|
|
|
|
prose = to_base("[[The Beatles]] are an [[English rock band]] from [[Liverpool]].")
|
|
# → "The Beatles are an English rock band from Liverpool."
|
|
```
|
|
|
|
## Why it exists
|
|
|
|
The corpus stores raw wikitext (so the link graph is recoverable
|
|
on demand) but the LLM and verifier both want plain prose. Reasons:
|
|
|
|
1. **Token efficiency.** Wikipedia chunks ship to Hermes with ~43%
|
|
fewer tokens after wikitext-strip — bigger context window for
|
|
the same chars budget.
|
|
2. **Verbatim citation.** The model can quote source paragraphs
|
|
verbatim instead of escaping `[[wikilinks]]`. The verifier's
|
|
substring test then matches cleanly.
|
|
3. **Pinned identity.** `BASE_VERSION='wikitext-base-v1'` lives in
|
|
`policy["base_version"]`, which folds into
|
|
`governance_policy_hash`. Bumping `BASE_VERSION` invalidates
|
|
every prior cache record on next lookup — same discipline as
|
|
`chunking_version` and `canonicalization_version`.
|
|
|
|
## Hot-path discipline
|
|
|
|
`to_base()` runs on the assembled context **before** the LLM call
|
|
in `aborist/qa/runner.py` and `aborist/qa/query.py`, **and again
|
|
inside `verify_quotes`** so the verifier compares like-against-like.
|
|
Both sides see prose.
|
|
|
|
## Optional dependency
|
|
|
|
Backed by `mwparserfromhell`. Install via `pip install '.[wikitext]'`
|
|
to enable. Without the dep, `_wikitext_to_base = None` and
|
|
`policy["base_version"] = None` — graceful fallback leaves raw
|
|
wikitext in both context and verifier (works, just less efficient).
|
|
|
|
## Source
|
|
|
|
[`aborist/wikitext.py`](../../aborist/wikitext.py)
|