aborist/progress.py — stdlib-only rate-limited stderr reporter.
Periodic lines (default every 2s) showing:
elapsed | seen | inserted | docs/s now | docs/s avg | percent | ETA
Wired into ingest_source via a `progress` parameter; CLI default-on
with --quiet to suppress and --total-estimate N to enable percent/ETA.
Demo on a 5k-doc ingest with --progress-interval 0.8:
[ 1s] | 200 seen | 200 new | 193 now | ( 193 avg) docs/s | 4.0% | ETA 24s
[ 7s] | 3,200 seen | 3,200 new | 458 now | ( 424 avg) docs/s | 64.0% | ETA 3s
[ 12s] | 5,001 seen | 5,000 new | 392 now | ( 413 avg) docs/s | 100.0% | ETA 0s
bench/run.sh + `make bench` — reproducible 5000-doc workload through
three configs:
serial single process, single SQLite
parallel-shared N shards, one shared SQLite (WAL serialized)
attached N shards, per-shard SQLite (true parallel writes)
Output is a one-shot table plus CSV at /tmp/aborist-bench/results.csv
so the ratchet is visible as we keep optimizing. Override via
BENCH_DOCS=N and SHARDS=N.
Latest baseline (this commit, on this machine):
config wall_s docs docs/s
serial 11.74 5000 425.7
parallel-shared 10.39 5000 481.1
attached 7.99 5000 625.5
53 tests still passing.
russell@unturf.com
649aeec79a