engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

README: document the Phase IV XML ingest path

Browse source 
Adds a new section after the 2003 Wikipedia / distillation block that
covers Phase IV (2006+) XML dumps: fetch-xml / ingest-xml-attached /
ingest-abstract targets, the WP_XML_YEAR/MONTH/DATE knobs for picking
any dated snapshot in dumps.wikimedia.org/archive, plus a pointer at
the abstract.xml feed as a cheap-coverage alternative to the full
pages-articles dump (~5-10 GB instead of ~95 GB on disk for enwiki
2010-11). Existing 2003 section retitled to flag the format split.
This commit is contained in:
russell@unturf.com 2026-04-27 17:36:03 -04:00
parent d25c0fe66f
commit 5c2eced7db
No known key found for this signature in database
1 changed files with 30 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

32
README.md
View file

@ -84,9 +84,9 @@ Creates `.venv/`, installs the package in editable mode with the `[dev,html]` ex
After bootstrap, every workflow lives behind a `make` target. Run `make help` to list them.
## Data: Wikipedia 2003-05-16
## Data: Wikipedia 2003-05-16 (Phase III SQL dump)
The 2003 dataset lives at <https://dumps.wikimedia.org/archive/2003/2003-05-16/en/> as three files:
The 2003 dataset lives at <https://dumps.wikimedia.org/archive/2003/2003-05-16/en/> as three files. This is a MySQL extended-INSERT format dump; for XML-format dumps from 2006 onward see the Phase IV section below.
| file | size | what |
|---|---|---|
@ -137,6 +137,34 @@ make distill-shards-tfidf-parallel # TF-IDF cores, per-shard
Run both — they generate independent cores per source. TF-IDF cores let neologisms (like a personal term that never appears in any title) match retrievals via keyword overlap.
## Data: Wikipedia 2010-11 (and other Phase IV snapshots)
In 2006 MediaWiki swapped its dumps from MySQL `INSERT INTO cur` syntax to XML. Aborist reads both — the SQL path above for 2003-2005 cur dumps, and a streaming XML path for any dated snapshot in <https://dumps.wikimedia.org/archive/>. The largest single snapshot in that archive is enwiki 2010-11-08:
| file | size | what |
|---|---|---|
| `enwiki-20101011-pages-articles.xml.bz2` | 6.2 GB | latest revision of every main-namespace article on 2010-11-08 (~3.4M pages, ~1.9M after redirects) |
| `enwiki-20101011-abstract.xml` | 2.9 GB | first-paragraph abstracts only — pre-distilled summaries at ~1/100th the chunk volume |
Other useful dated snapshots in the archive: 2006-07 (1.8 GB), 2006-12 (1.9 GB), 2010-03 (varies by language). All work via the same source class.
```
# defaults target enwiki 20101011 (2010-11)
make fetch-xml # ~6.2 GB compressed download
make ingest-xml-attached SHARDS=4 # ~2 hours sharded; ~95 GB on disk after
# pick any other snapshot by overriding the date variables
make fetch-xml WP_XML_YEAR=2006 WP_XML_MONTH=2006-07 WP_XML_DATE=20061104
# abstract feed (one-paragraph summaries, full coverage at ~5-10 GB total)
make fetch-abstract
make ingest-abstract
```
The XML source streams `.xml.bz2` directly via `iterparse` with bounded memory (each `<page>` is processed and cleared). Same shard / resume / Merkle contract as the SQL source. Title-prefix namespace filtering kicks in for older export schemas that omit per-page `<ns>`.
To ingest historical revisions instead of just the current snapshot, point `WP_XML` at a `pages-meta-history.xml.bz2` file and use `make ingest-xml-history` — the source emits one Document per revision and aborist's prior-doc detection chains them with `supersedes` edges.
## Data: personal Grok export
If you have an xAI data-export bundle, point `GROK_EXPORT` at its root directory (the one containing `ttl/30d/export_data/<user-id>/`):