# `aborist.sources`

Corpus producers. Each is a `Source` ABC implementation that yields
`Document` instances; the standard `ingest.ingest_source(source, db)`
pipeline takes them from there.

The `Source` ABC lives in [`aborist/source.py`](../../aborist/source.py):

```python
class Source(ABC):
    @abstractmethod
    def iter_documents(self) -> Iterator[Document]: ...
```

## Built-in sources

### `wikipedia.py` — Phase III SQL dumps (the canonical bootstrap)

Streams the Wikipedia 2003 `cur` (current revisions) and `old`
(revision history) SQL dumps. Hand-rolled escape-aware parser
(no `sqlite3` import — the dump is MySQL syntax). 4× speedup vs
char-by-char loops via `str.find` + slicing. cProfile any change.

Default Wikipedia 2003-05-16 dump source:
`https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`.
robots.txt returned 404 → no rules.

[`aborist/sources/wikipedia.py`](../../aborist/sources/wikipedia.py)

### `wikipedia_xml.py` — Phase IV XML dumps

Modern Wikipedia dump format (`enwiki-YYYYMMDD-pages-articles.xml.bz2`,
`enwiki-YYYYMMDD-pages-meta-history*.xml.bz2`). Uses `xml.etree.ElementTree.iterparse`
to stream-parse without loading the whole tree.

[`aborist/sources/wikipedia_xml.py`](../../aborist/sources/wikipedia_xml.py)

### `html_page.py` — single-URL or URL-list HTML ingest

Robots-aware (`urllib.robotparser`). Uses `selectolax` for fast
HTML parsing (CSS-selector based; ~10× faster than `lxml`). Pulls
the main body text + every `<a href>` as an `Edge` row.

The `edges` rows are what the corpus-derived synonym extractor
later reads — no separate crawler needed for site-internal link
graphs.

Optional dep: `pip install '.[html]'` for `selectolax` + `httpx`.

[`aborist/sources/html_page.py`](../../aborist/sources/html_page.py)

### `crawler/` — async BFS web crawl

Verbatim lift of an `AsyncWebFetcher` implementation + an `ingest`
bridge. BFS-discovers same-domain URLs from a seed, respecting
`robots.txt` + crawl delays. Captures ETag + Last-Modified per URL
into `document_http_meta` so a future recrawl can send conditional
HEAD requests.

[`aborist/sources/crawler/bridge.py`](../../aborist/sources/crawler/bridge.py)

### `grok.py` — xAI Grok export

Reads the `xAI-conversations.json` data export shape. Each
conversation becomes one `Document`; media prompts are kept
inline.

[`aborist/sources/grok.py`](../../aborist/sources/grok.py)

### `vcs.py` — git + Mercurial repositories

HEAD walk. Each commit becomes a `Document` (commit message + diff
stat). The supersedes chain captures commit ancestry as edges.

[`aborist/sources/vcs.py`](../../aborist/sources/vcs.py)

## Source

[`aborist/sources/`](../../aborist/sources/) ·
[`aborist/source.py`](../../aborist/source.py) (ABC)