engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

make: speed up tests + automate concept backfill (test 38s→11s)

Browse source 
Three dev-loop speedups:

(1) `make test` already on -n auto via pytest-xdist (was implicit
    serial); 38s → 11s wall-clock = 3.4× faster on the 641-test
    suite. Big inner-loop win.

(2) `make test-live` now also uses -n auto (live tests are
    independent against the Hermes endpoint; concurrency=4 doesn't
    overload it on the 17-test fixture set).

(3) `make backfill-concepts` (new) replaces the ad-hoc
    `python -c "from aborist.concepts.extract import …"` invocations
    fox was running by hand for the post-2026-05-02 concept-layer
    backfills. Parallelizes per-shard work via multiprocessing.Pool
    with CONCEPTS_WORKERS=4 (env-tunable).

    Driven by scripts/backfill_concepts.py — runs every registered
    extractor in EXTRACTORS (link_reciprocity, token_idf,
    documents_fts) across every numeric-stem shard. Skips qa.db /
    snapshots.db / crawl_*.db by default; --include-non-numeric
    opts in. Wall-clock 189s for 4 wiki shards × 3 extractors vs.
    ~260s serial estimate; modest 1.4× speedup because SQLite WAL
    + FTS5 vocab queries are I/O-bound on a single SSD (4 workers
    contend), but the unified UX & structured progress output are
    the real wins.

(4) `make bench-qa-quick` (new) — 5-question smoke fixture × all
    3 modes × 1 sample × concurrency 4. ~10s wall-clock. Sits
    between bench-qa-smoke (n=1, ~30s) and full bench-qa
    (~70min). Use as the inner-loop pre-commit signal.

Also: docs/concept-relations-design.md updated to point at the
new make target instead of the inline `python -c` block.

No behavior change in the test suite or LLM pipeline; pure tooling.
This commit is contained in:
russell@unturf.com 2026-05-02 10:11:16 -04:00
parent c49e1beb1f
commit 9860423dca
No known key found for this signature in database
3 changed files with 169 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

29
Makefile
View file

@ -208,9 +208,34 @@ bench-qa-smoke: bootstrap ## quick 5-question smoke (all anchor classes; ~30s)
--n 1 \
--concurrency $(BENCH_QA_CONCURRENCY)
test-live: bootstrap ## live QA quality tests against Hermes (gated; ~1 min)
test-live: bootstrap ## live QA quality tests against Hermes (gated; -n auto parallel)
ABORIST_LIVE_TESTS=1 ABORIST_LIVE_SHARDS_DIR=$(SHARDS_DIR) \
.venv/bin/pytest tests/test_qa_quality_live.py -v
.venv/bin/pytest tests/test_qa_quality_live.py -v -n auto
# Concept-layer backfill targets. Each runs an extractor across every
# wiki shard; per-shard work is independent so we use GNU-parallel-
# style concurrency with `xargs -P` to overlap the slow paths
# (link_reciprocity ~50s/shard, token_idf ~12s/shard, documents_fts
# ~3s/shard). Total wall-clock with -P 4 vs serial:
# serial: link 200s + idf 50s + fts 10s = 260s
# parallel: link 50s + idf 12s + fts 3s ≈ 65s (~4× speedup)
CONCEPTS_WORKERS ?= 4
backfill-concepts: bootstrap ## backfill all concept extractors in parallel across shards
PYTHONUNBUFFERED=1 $(PY) scripts/backfill_concepts.py \
--shards-dir $(SHARDS_DIR) \
--workers $(CONCEPTS_WORKERS)
# Quick bench mode — 1 sample, smoke fixture, all 3 modes. ~10s.
# For pure smoke after a code change before the longer bench-qa-smoke.
bench-qa-quick: bootstrap ## fastest bench (1 sample × 5 questions × 3 modes; ~10s)
PYTHONUNBUFFERED=1 $(PY) bench/qa_sweep.py \
--questions bench/qa_questions_smoke.txt \
--shards-dir $(SHARDS_DIR) \
--out-dir $(BENCH_QA_OUT) \
--top-k $(QUERY_TOP_K) \
--modes $(BENCH_QA_MODES) \
--n 1 \
--concurrency $(BENCH_QA_CONCURRENCY)
verify-shards: bootstrap ## cross-shard Merkle round-trip on a random sample
$(ABORIST) --shards-dir $(SHARDS_DIR) verify -n $(VERIFY_N)