docs: README label refresh + per-module reference + Graphviz diagrams

Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).

(1) README — label refresh:
    - Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
      four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
      EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
    - Verifier section spells out both layers (schema trichotomy +
      display ladder), the seven hard checks of claim_lattice, and
      the five anchor classes of warrant.
    - Architecture tree updated: concepts/ package added, qa/
      sub-modules expanded (warrant, evidence, parse_claims, dag),
      verify.py described as quote/span/entity/paraphrase + claim_lattice.
    - Concept overlay description updated for corpus-derived layer
      (concept_relations table, link_reciprocity extractor, 1.6%
      tax cite).
    - Test count: 326+ → 641+.

(2) docs/diagrams/ — Graphviz dot sources:
    - aborist-modules.dot — top-level package graph (substrate /
      storage / sources / retrieval / qa / mesh / cli)
    - query-pipeline.dot — question → cache → retrieval → LLM →
      verify → render → cache write, with phase budgets
    - ingest-pipeline.dot — source doc → canonicalize → chunk →
      Merkle → upsert (+ optional distill)
    - verifier-ladder.dot — (audit_mode, violations) → display rung
      decision tree
    Existing mesh-*.dot kept as-is. Makefile `make docs` target
    extended to also emit .svg alongside the existing .png so the
    diagrams render in markdown viewers.

(3) docs/modules/ — per-module reference pages:
    - index.md (links to every diagram + every module page)
    - merkle.md, document.md, store.md, ingest.md, evict.md,
      sources.md, search.md, concepts.md, qa.md, distill.md,
      wikitext.md
    Each page is a one-screenful concise reference: what the
    module is for, public API, key invariants, embedded diagrams
    where useful, link to source. Mesh stays at the existing
    docs/mesh.md + docs/mesh-deploy.md (already comprehensive).

Tests: 641 passed (no code change).

2026-05-01 23:19:01 -04:00

1.7 KiB

Raw Blame History

`aborist.wikitext`

A single function: to_base(raw). Converts MediaWiki wikitext to plain prose deterministically.

from aborist.wikitext import to_base, BASE_VERSION

prose = to_base("[[The Beatles]] are an [[English rock band]] from [[Liverpool]].")
# → "The Beatles are an English rock band from Liverpool."

Why it exists

The corpus stores raw wikitext (so the link graph is recoverable on demand) but the LLM and verifier both want plain prose. Reasons:

Token efficiency. Wikipedia chunks ship to Hermes with ~43% fewer tokens after wikitext-strip — bigger context window for the same chars budget.
Verbatim citation. The model can quote source paragraphs verbatim instead of escaping [[wikilinks]]. The verifier's substring test then matches cleanly.
Pinned identity. BASE_VERSION='wikitext-base-v1' lives in policy["base_version"], which folds into governance_policy_hash. Bumping BASE_VERSION invalidates every prior cache record on next lookup — same discipline as chunking_version and canonicalization_version.

Hot-path discipline

to_base() runs on the assembled context before the LLM call in aborist/qa/runner.py and aborist/qa/query.py, and again inside verify_quotes so the verifier compares like-against-like. Both sides see prose.

Optional dependency

Backed by mwparserfromhell. Install via pip install '.[wikitext]' to enable. Without the dep, _wikitext_to_base = None and policy["base_version"] = None — graceful fallback leaves raw wikitext in both context and verifier (works, just less efficient).

Source

aborist/wikitext.py

1.7 KiB Raw Blame History

aborist.wikitext