arborist/aborist/sources
russell@unturf.com d2ceba201e
crawler: live stderr heartbeat during BFS + ingest (rate-limited)
A no-cap crawl could take many seconds with no feedback. Wires the
existing aborist.progress.Progress into both phases so stderr shows
heartbeats every 2s (Progress's default interval).

Discovery phase: prints prefix='crawl', counts discovered URLs,
shows queue depth as the secondary number ('inserted' slot in the
Progress format — works fine, semantically "still to do").

Ingest phase: prefix='ingest', total_estimate=len(urls) so the user
sees percent + ETA. Threads through ingest_source's existing
progress= parameter.

Plus three banner lines to stderr at phase boundaries (start crawl,
end discovery, start ingest) so even sub-2s crawls show signs of
life. All flushed via Progress's flush=True path; stderr is
line-buffered by default so this works without PYTHONUNBUFFERED.

Tests stay green (Progress goes to stderr, pytest captures only the
stdout summary). 19 bridge tests + 273 default suite, all passing.
2026-04-28 21:05:45 -04:00
..
crawler crawler: live stderr heartbeat during BFS + ingest (rate-limited) 2026-04-28 21:05:45 -04:00
__init__.py add git and mercurial repo sources for self-play 2026-04-27 18:17:39 -04:00
grok.py add Grok export source: conversations + media posts 2026-04-27 13:49:42 -04:00
html_page.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
vcs.py add git and mercurial repo sources for self-play 2026-04-27 18:17:39 -04:00
wikipedia.py fix _iter_insert_statements: persist scan_pos across chunks 2026-04-27 12:15:47 -04:00
wikipedia_xml.py add Phase IV Wikipedia XML + abstract sources 2026-04-27 17:24:28 -04:00