engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

phase 1b: --shard rank/total + parallel Makefile targets

Browse source 
Workers each parse the full SQL stream but only emit rows whose index
satisfies (idx % total == rank). N processes cooperate against a
single shared SQLite file; WAL+synchronous=NORMAL handles writer
contention via BEGIN IMMEDIATE serialization.

Implementation chose OS-level processes over in-process mp.Queue: the
gain ceiling is identical (writer is the bottleneck either way) and
the surface area is one new flag plus stride-filter logic instead of
a queue-based pipeline with picklable workers.

Makefile gains:
  ingest-cur-parallel SHARDS=4
  ingest-old-parallel SHARDS=4

Benchmark on 20k cur articles:
  serial            55.3 s   361 docs/s   1.0x CPU usage
  4-way parallel    43.1 s   464 docs/s   ~3.0x CPU usage   1.28x wall

Both runs landed 19,999 unique documents — correctness verified.

The write-lock is now the bottleneck; further gains require Phase 2
attach-forever sharding (each shard owns its own SQLite file, queries
fan out via ATTACH DATABASE, zero merge cost).
This commit is contained in:
russell@unturf.com 2026-04-27 11:10:25 -04:00
parent c81c1c0ff0
commit d0857fb43d
No known key found for this signature in database
3 changed files with 53 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

17
Makefile
View file

@ -74,7 +74,22 @@ ingest-cur: bootstrap fetch-cur ## ingest INGEST_LIMIT cur articles
ingest-old: bootstrap fetch-old ## ingest INGEST_LIMIT old (history) revisions
$(ABORIST) --db $(DB) ingest --source wikipedia_old --path $(WP_OLD) --limit $(INGEST_LIMIT)
ingest: ingest-cur ## default ingest = cur (use ingest-old or both for full)
# Spawn N parallel shard processes. They parallelize parser CPU and
# serialize SQLite writes through WAL. Override SHARDS=N (default 4).
SHARDS ?= 4
ingest-cur-parallel: bootstrap fetch-cur ## ingest cur with SHARDS=N (default 4) parallel processes
@for i in $$(seq 0 $$(($(SHARDS) - 1))); do \
$(ABORIST) --db $(DB) ingest --source wikipedia_cur --path $(WP_CUR) \
--shard $$i/$(SHARDS) & \
done; wait
ingest-old-parallel: bootstrap fetch-old ## ingest old history with SHARDS=N parallel processes
@for i in $$(seq 0 $$(($(SHARDS) - 1))); do \
$(ABORIST) --db $(DB) ingest --source wikipedia_old --path $(WP_OLD) \
--shard $$i/$(SHARDS) & \
done; wait
ingest: ingest-cur ## default ingest = cur (use ingest-old or *-parallel for full)
verify: bootstrap ## round-trip Merkle proofs for VERIFY_N random documents
$(ABORIST) --db $(DB) verify -n $(VERIFY_N)