russell@unturf.com
f90b7c69f0
Drops in for any dated snapshot in dumps.wikimedia.org/archive — pages-articles.xml.bz2 and pages-meta-current.xml.bz2 (cur snapshot, single-revision-per-page) plus pages-meta-history.xml.bz2 (multi_revision=True). abstract.xml feed yields pre-distilled summaries at ~1/100th the chunk volume. Streams .bz2/.gz directly via iterparse with bounded memory; same shard / resume contract as the SQL source. Falls back to title-prefix namespace filtering when older export schemas omit the per-page <ns> element (e.g. enwiki 20101011). Makefile defaults target enwiki 20101011 (6.2 GB). Override with WP_XML_YEAR / WP_XML_MONTH / WP_XML_DATE / WP_XML_LANG to fetch any other archived snapshot. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_concepts.py | ||
| test_distill.py | ||
| test_distill_recursive.py | ||
| test_evict.py | ||
| test_grok_source.py | ||
| test_html_source.py | ||
| test_ingest.py | ||
| test_merkle.py | ||
| test_qa.py | ||
| test_query.py | ||
| test_resume.py | ||
| test_tfidf.py | ||
| test_versioned_ingest.py | ||
| test_wikipedia_old.py | ||
| test_wikipedia_xml.py | ||