docs: README label refresh + per-module reference + Graphviz diagrams

Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).

(1) README — label refresh:
    - Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
      four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
      EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
    - Verifier section spells out both layers (schema trichotomy +
      display ladder), the seven hard checks of claim_lattice, and
      the five anchor classes of warrant.
    - Architecture tree updated: concepts/ package added, qa/
      sub-modules expanded (warrant, evidence, parse_claims, dag),
      verify.py described as quote/span/entity/paraphrase + claim_lattice.
    - Concept overlay description updated for corpus-derived layer
      (concept_relations table, link_reciprocity extractor, 1.6%
      tax cite).
    - Test count: 326+ → 641+.

(2) docs/diagrams/ — Graphviz dot sources:
    - aborist-modules.dot — top-level package graph (substrate /
      storage / sources / retrieval / qa / mesh / cli)
    - query-pipeline.dot — question → cache → retrieval → LLM →
      verify → render → cache write, with phase budgets
    - ingest-pipeline.dot — source doc → canonicalize → chunk →
      Merkle → upsert (+ optional distill)
    - verifier-ladder.dot — (audit_mode, violations) → display rung
      decision tree
    Existing mesh-*.dot kept as-is. Makefile `make docs` target
    extended to also emit .svg alongside the existing .png so the
    diagrams render in markdown viewers.

(3) docs/modules/ — per-module reference pages:
    - index.md (links to every diagram + every module page)
    - merkle.md, document.md, store.md, ingest.md, evict.md,
      sources.md, search.md, concepts.md, qa.md, distill.md,
      wikitext.md
    Each page is a one-screenful concise reference: what the
    module is for, public API, key invariants, embedded diagrams
    where useful, link to source. Mesh stays at the existing
    docs/mesh.md + docs/mesh-deploy.md (already comprehensive).

Tests: 641 passed (no code change).

2026-05-01 23:19:01 -04:00

2.7 KiB

Raw Blame History

`aborist.sources`

Corpus producers. Each is a Source ABC implementation that yields Document instances; the standard ingest.ingest_source(source, db) pipeline takes them from there.

The Source ABC lives in aborist/source.py:

class Source(ABC):
    @abstractmethod
    def iter_documents(self) -> Iterator[Document]: ...

Built-in sources

`wikipedia.py` — Phase III SQL dumps (the canonical bootstrap)

Streams the Wikipedia 2003 cur (current revisions) and old (revision history) SQL dumps. Hand-rolled escape-aware parser (no sqlite3 import — the dump is MySQL syntax). 4× speedup vs char-by-char loops via str.find + slicing. cProfile any change.

Default Wikipedia 2003-05-16 dump source: https://dumps.wikimedia.org/archive/2003/2003-05-16/en/. robots.txt returned 404 → no rules.

aborist/sources/wikipedia.py

`wikipedia_xml.py` — Phase IV XML dumps

Modern Wikipedia dump format (enwiki-YYYYMMDD-pages-articles.xml.bz2, enwiki-YYYYMMDD-pages-meta-history*.xml.bz2). Uses xml.etree.ElementTree.iterparse to stream-parse without loading the whole tree.

aborist/sources/wikipedia_xml.py

`html_page.py` — single-URL or URL-list HTML ingest

Robots-aware (urllib.robotparser). Uses selectolax for fast HTML parsing (CSS-selector based; ~10× faster than lxml). Pulls the main body text + every <a href> as an Edge row.

The edges rows are what the corpus-derived synonym extractor later reads — no separate crawler needed for site-internal link graphs.

Optional dep: pip install '.[html]' for selectolax + httpx.

aborist/sources/html_page.py

`crawler/` — async BFS web crawl

Verbatim lift of an AsyncWebFetcher implementation + an ingest bridge. BFS-discovers same-domain URLs from a seed, respecting robots.txt + crawl delays. Captures ETag + Last-Modified per URL into document_http_meta so a future recrawl can send conditional HEAD requests.

aborist/sources/crawler/bridge.py

`grok.py` — xAI Grok export

Reads the xAI-conversations.json data export shape. Each conversation becomes one Document; media prompts are kept inline.

aborist/sources/grok.py

`vcs.py` — git + Mercurial repositories

HEAD walk. Each commit becomes a Document (commit message + diff stat). The supersedes chain captures commit ancestry as edges.

aborist/sources/vcs.py

Source

aborist/sources/ · aborist/source.py (ABC)

2.7 KiB Raw Blame History Unescape Escape

aborist.sources

Built-in sources

wikipedia.py — Phase III SQL dumps (the canonical bootstrap)

wikipedia_xml.py — Phase IV XML dumps

html_page.py — single-URL or URL-list HTML ingest

crawler/ — async BFS web crawl

grok.py — xAI Grok export

vcs.py — git + Mercurial repositories