arborist/aborist
russell@unturf.com d2ceba201e
crawler: live stderr heartbeat during BFS + ingest (rate-limited)
A no-cap crawl could take many seconds with no feedback. Wires the
existing aborist.progress.Progress into both phases so stderr shows
heartbeats every 2s (Progress's default interval).

Discovery phase: prints prefix='crawl', counts discovered URLs,
shows queue depth as the secondary number ('inserted' slot in the
Progress format — works fine, semantically "still to do").

Ingest phase: prefix='ingest', total_estimate=len(urls) so the user
sees percent + ETA. Threads through ingest_source's existing
progress= parameter.

Plus three banner lines to stderr at phase boundaries (start crawl,
end discovery, start ingest) so even sub-2s crawls show signs of
life. All flushed via Progress's flush=True path; stderr is
line-buffered by default so this works without PYTHONUNBUFFERED.

Tests stay green (Progress goes to stderr, pytest captures only the
stdout summary). 19 bridge tests + 273 default suite, all passing.
2026-04-28 21:05:45 -04:00
..
distill storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
mesh mesh: per-peer audit chain-of-claims tracking on the wire 2026-04-28 19:41:25 -04:00
qa qa: pipe wikitext-base-v1 into LLM context (was verifier-only) 2026-04-28 17:08:49 -04:00
search verify: layered strategies + entity policies, rename VISUAL → UNGROUNDED 2026-04-28 16:58:31 -04:00
sources crawler: live stderr heartbeat during BFS + ingest (rate-limited) 2026-04-28 21:05:45 -04:00
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
cli.py crawler: live stderr heartbeat during BFS + ingest (rate-limited) 2026-04-28 21:05:45 -04:00
compress.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
document.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
evict.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
ingest.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
merkle.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
progress.py progress reporter + structured benchmark 2026-04-27 11:37:20 -04:00
snapshot.py add corpus-level snapshots: single-hash identity for the forest 2026-04-27 21:29:10 -04:00
source.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
store.py crawler: bridge to ingest + fast recrawl-check via conditional HEAD 2026-04-28 20:34:54 -04:00
wikitext.py verify: strip wikitext from context before substring matching 2026-04-28 15:48:07 -04:00