arborist/aborist/sources/crawler
russell@unturf.com d2ceba201e
crawler: live stderr heartbeat during BFS + ingest (rate-limited)
A no-cap crawl could take many seconds with no feedback. Wires the
existing aborist.progress.Progress into both phases so stderr shows
heartbeats every 2s (Progress's default interval).

Discovery phase: prints prefix='crawl', counts discovered URLs,
shows queue depth as the secondary number ('inserted' slot in the
Progress format — works fine, semantically "still to do").

Ingest phase: prefix='ingest', total_estimate=len(urls) so the user
sees percent + ETA. Threads through ingest_source's existing
progress= parameter.

Plus three banner lines to stderr at phase boundaries (start crawl,
end discovery, start ingest) so even sub-2s crawls show signs of
life. All flushed via Progress's flush=True path; stderr is
line-buffered by default so this works without PYTHONUNBUFFERED.

Tests stay green (Progress goes to stderr, pytest captures only the
stdout summary). 19 bridge tests + 273 default suite, all passing.
2026-04-28 21:05:45 -04:00
..
__init__.py crawler: verbatim lift from agents.ai.unturf.com (off by default) 2026-04-28 20:23:36 -04:00
async_web_fetcher.py crawler: verbatim lift from agents.ai.unturf.com (off by default) 2026-04-28 20:23:36 -04:00
bridge.py crawler: live stderr heartbeat during BFS + ingest (rate-limited) 2026-04-28 21:05:45 -04:00
web_fetch.py crawler: verbatim lift from agents.ai.unturf.com (off by default) 2026-04-28 20:23:36 -04:00