russell@unturf.com
a056acfb5b
The 2003-05-16 archive ships three files:
20030516_cur_tablesql.bz2 82 MB current snapshot (single revision/page)
old_tablesqlbz2.1 640 MiB \
old_tablesqlbz2.2 252 MiB / split halves of old (full revision
history). Concatenate before bzcat.
Generalize the parser:
WikipediaSqlDump(table='cur'|'old') — shared statement parser, single
column-position contract for the
first 4 fields (id/ns/title/text)
WikipediaCurDump — back-compat wrapper, table='cur'
WikipediaOldDump — new, table='old'; old has no is_redirect, every
revision is real
Old rows surface old_id and old_timestamp via Document.extra so a
downstream pass can sort revisions chronologically before re-ingesting
through the supersedes-edge path.
Makefile gains:
fetch-cur / fetch-old / fetch (both)
ingest-cur / ingest-old / ingest (cur default)
WP_OLD target concatenates the two split parts
CLI ingest --source now accepts wikipedia_cur or wikipedia_old.
Smoke (real dump): 5 revisions of "AtlasShrugged/Companies" yielded
correctly with old_id=2..10, timestamps from January 2002.
25 lines
811 B
Python
25 lines
811 B
Python
"""Source implementations. Add a new corpus = add a new module here."""
|
|
|
|
from aborist.sources.wikipedia import (
|
|
WikipediaCurDump,
|
|
WikipediaOldDump,
|
|
WikipediaSqlDump,
|
|
)
|
|
|
|
__all__ = ["WikipediaCurDump", "WikipediaOldDump", "WikipediaSqlDump"]
|
|
|
|
# HtmlPageSource has optional deps (selectolax). Surface import errors only
|
|
# when callers actually request it.
|
|
try:
|
|
from aborist.sources.html_page import HtmlPageSource # noqa: F401
|
|
__all__.append("HtmlPageSource")
|
|
except ImportError: # pragma: no cover
|
|
pass
|
|
|
|
# HtmlPageSource has optional deps (httpx + selectolax). Surface import errors
|
|
# only when callers actually request it.
|
|
try:
|
|
from aborist.sources.html_page import HtmlPageSource # noqa: F401
|
|
__all__.append("HtmlPageSource")
|
|
except ImportError: # pragma: no cover
|
|
pass
|