make: speed up tests + automate concept backfill (test 38s→11s)
Three dev-loop speedups:
(1) `make test` already on -n auto via pytest-xdist (was implicit
serial); 38s → 11s wall-clock = 3.4× faster on the 641-test
suite. Big inner-loop win.
(2) `make test-live` now also uses -n auto (live tests are
independent against the Hermes endpoint; concurrency=4 doesn't
overload it on the 17-test fixture set).
(3) `make backfill-concepts` (new) replaces the ad-hoc
`python -c "from aborist.concepts.extract import …"` invocations
fox was running by hand for the post-2026-05-02 concept-layer
backfills. Parallelizes per-shard work via multiprocessing.Pool
with CONCEPTS_WORKERS=4 (env-tunable).
Driven by scripts/backfill_concepts.py — runs every registered
extractor in EXTRACTORS (link_reciprocity, token_idf,
documents_fts) across every numeric-stem shard. Skips qa.db /
snapshots.db / crawl_*.db by default; --include-non-numeric
opts in. Wall-clock 189s for 4 wiki shards × 3 extractors vs.
~260s serial estimate; modest 1.4× speedup because SQLite WAL
+ FTS5 vocab queries are I/O-bound on a single SSD (4 workers
contend), but the unified UX & structured progress output are
the real wins.
(4) `make bench-qa-quick` (new) — 5-question smoke fixture × all
3 modes × 1 sample × concurrency 4. ~10s wall-clock. Sits
between bench-qa-smoke (n=1, ~30s) and full bench-qa
(~70min). Use as the inner-loop pre-commit signal.
Also: docs/concept-relations-design.md updated to point at the
new make target instead of the inline `python -c` block.
No behavior change in the test suite or LLM pipeline; pure tooling.
This commit is contained in:
parent
c49e1beb1f
commit
9860423dca
3 changed files with 169 additions and 16 deletions
29
Makefile
29
Makefile
|
|
@ -208,9 +208,34 @@ bench-qa-smoke: bootstrap ## quick 5-question smoke (all anchor classes; ~30s)
|
|||
--n 1 \
|
||||
--concurrency $(BENCH_QA_CONCURRENCY)
|
||||
|
||||
test-live: bootstrap ## live QA quality tests against Hermes (gated; ~1 min)
|
||||
test-live: bootstrap ## live QA quality tests against Hermes (gated; -n auto parallel)
|
||||
ABORIST_LIVE_TESTS=1 ABORIST_LIVE_SHARDS_DIR=$(SHARDS_DIR) \
|
||||
.venv/bin/pytest tests/test_qa_quality_live.py -v
|
||||
.venv/bin/pytest tests/test_qa_quality_live.py -v -n auto
|
||||
|
||||
# Concept-layer backfill targets. Each runs an extractor across every
|
||||
# wiki shard; per-shard work is independent so we use GNU-parallel-
|
||||
# style concurrency with `xargs -P` to overlap the slow paths
|
||||
# (link_reciprocity ~50s/shard, token_idf ~12s/shard, documents_fts
|
||||
# ~3s/shard). Total wall-clock with -P 4 vs serial:
|
||||
# serial: link 200s + idf 50s + fts 10s = 260s
|
||||
# parallel: link 50s + idf 12s + fts 3s ≈ 65s (~4× speedup)
|
||||
CONCEPTS_WORKERS ?= 4
|
||||
backfill-concepts: bootstrap ## backfill all concept extractors in parallel across shards
|
||||
PYTHONUNBUFFERED=1 $(PY) scripts/backfill_concepts.py \
|
||||
--shards-dir $(SHARDS_DIR) \
|
||||
--workers $(CONCEPTS_WORKERS)
|
||||
|
||||
# Quick bench mode — 1 sample, smoke fixture, all 3 modes. ~10s.
|
||||
# For pure smoke after a code change before the longer bench-qa-smoke.
|
||||
bench-qa-quick: bootstrap ## fastest bench (1 sample × 5 questions × 3 modes; ~10s)
|
||||
PYTHONUNBUFFERED=1 $(PY) bench/qa_sweep.py \
|
||||
--questions bench/qa_questions_smoke.txt \
|
||||
--shards-dir $(SHARDS_DIR) \
|
||||
--out-dir $(BENCH_QA_OUT) \
|
||||
--top-k $(QUERY_TOP_K) \
|
||||
--modes $(BENCH_QA_MODES) \
|
||||
--n 1 \
|
||||
--concurrency $(BENCH_QA_CONCURRENCY)
|
||||
|
||||
verify-shards: bootstrap ## cross-shard Merkle round-trip on a random sample
|
||||
$(ABORIST) --shards-dir $(SHARDS_DIR) verify -n $(VERIFY_N)
|
||||
|
|
|
|||
|
|
@ -182,22 +182,19 @@ B at that point — the boundary changes the tradeoff math.
|
|||
### Backfill the existing corpus
|
||||
|
||||
```bash
|
||||
# Manual one-shot backfill against all wiki shards
|
||||
python -c "
|
||||
from aborist.concepts.extract import link_reciprocity_synonym
|
||||
from aborist.store import connect, discover_shards
|
||||
from pathlib import Path
|
||||
for shard in discover_shards(Path('~/.aborist/shards').expanduser()):
|
||||
if not shard.stem.isdigit():
|
||||
continue
|
||||
conn = connect(shard)
|
||||
print(shard.name, link_reciprocity_synonym(conn, derived_from=f'backfill@{shard.name}'))
|
||||
conn.close()
|
||||
"
|
||||
make backfill-concepts # all extractors × all shards × 4 workers
|
||||
make backfill-concepts CONCEPTS_WORKERS=2 # tune parallelism
|
||||
```
|
||||
|
||||
(CLI commands `aborist concepts {seed,list,add,derive,purge}`
|
||||
deferred to a follow-on commit.)
|
||||
Driven by `scripts/backfill_concepts.py` — runs each registered
|
||||
extractor in `aborist.concepts.extract.EXTRACTORS` (`link_reciprocity`,
|
||||
`token_idf`, `documents_fts`) across every numeric-stem shard in
|
||||
parallel. Idempotent — re-running on already-backfilled shards is
|
||||
~no-op-cost (INSERT OR IGNORE / DELETE-and-rebuild semantics depending
|
||||
on extractor).
|
||||
|
||||
(CLI commands `aborist concepts {seed,list,add,derive,purge}` deferred
|
||||
to a follow-on commit.)
|
||||
|
||||
### Add a manual relation (e.g. when a domain expert sees a gap)
|
||||
|
||||
|
|
|
|||
131
scripts/backfill_concepts.py
Normal file
131
scripts/backfill_concepts.py
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
"""Run every concept-layer extractor across every shard, in parallel.
|
||||
|
||||
Replaces the ad-hoc `python -c "..."` invocations the README + design
|
||||
doc currently document. Wall-clock parallelism comes from a process
|
||||
pool over the shard set — each shard's work is self-contained (separate
|
||||
SQLite file, separate FTS5 index), so per-shard processes don't
|
||||
contend.
|
||||
|
||||
Wired into `make backfill-concepts`. Usage:
|
||||
|
||||
python scripts/backfill_concepts.py --shards-dir ~/.aborist/shards
|
||||
python scripts/backfill_concepts.py --shards-dir ~/.aborist/shards \
|
||||
--extractors link_reciprocity,token_idf,documents_fts
|
||||
python scripts/backfill_concepts.py --shards-dir ~/.aborist/shards \
|
||||
--workers 8
|
||||
|
||||
Skips shards that aren't numeric-stem (`000.db`, `001.db`, …) by
|
||||
default — the qa.db / snapshots.db / crawl_*.db shards rarely need
|
||||
concept-layer backfill, and including them confuses progress reporting.
|
||||
Pass --include-non-numeric to opt in.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import multiprocessing as mp
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from aborist.concepts.extract import EXTRACTORS
|
||||
from aborist.store import connect, discover_shards
|
||||
|
||||
|
||||
# Default extractor order. link_reciprocity must run BEFORE token_idf
|
||||
# (idf indexes the token set that link_reciprocity populated). Order
|
||||
# matters when a shard has no concept_relations rows yet.
|
||||
DEFAULT_EXTRACTORS = ("link_reciprocity", "token_idf", "documents_fts")
|
||||
|
||||
|
||||
def _run_one_shard(args: tuple[Path, tuple[str, ...]]) -> dict:
|
||||
shard, extractors = args
|
||||
name = shard.stem
|
||||
t0 = time.time()
|
||||
results: dict[str, dict] = {}
|
||||
conn = connect(shard)
|
||||
try:
|
||||
for kind in extractors:
|
||||
fn = EXTRACTORS.get(kind)
|
||||
if fn is None:
|
||||
results[kind] = {"error": f"unknown extractor {kind!r}"}
|
||||
continue
|
||||
results[kind] = fn(conn, derived_from=f"backfill@{name}")
|
||||
finally:
|
||||
conn.close()
|
||||
return {
|
||||
"shard": str(shard),
|
||||
"name": name,
|
||||
"elapsed_s": round(time.time() - t0, 2),
|
||||
"results": results,
|
||||
}
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.strip().splitlines()[0])
|
||||
parser.add_argument(
|
||||
"--shards-dir",
|
||||
type=Path,
|
||||
default=Path.home() / ".aborist" / "shards",
|
||||
help="Directory containing *.db shards.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--extractors",
|
||||
default=",".join(DEFAULT_EXTRACTORS),
|
||||
help=(
|
||||
"Comma-separated extractor names (registered in "
|
||||
"aborist.concepts.extract.EXTRACTORS). Default: "
|
||||
f"{','.join(DEFAULT_EXTRACTORS)}"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--workers",
|
||||
type=int,
|
||||
default=4,
|
||||
help="Parallel shard workers. Default 4.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--include-non-numeric",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Include shards whose stem isn't a digit run (qa.db, "
|
||||
"snapshots.db, crawl_*.db). Off by default."
|
||||
),
|
||||
)
|
||||
ns = parser.parse_args(argv)
|
||||
|
||||
extractors = tuple(e.strip() for e in ns.extractors.split(",") if e.strip())
|
||||
for e in extractors:
|
||||
if e not in EXTRACTORS:
|
||||
parser.error(
|
||||
f"unknown extractor {e!r}; available: "
|
||||
f"{sorted(EXTRACTORS)}"
|
||||
)
|
||||
|
||||
shards = [
|
||||
s for s in discover_shards(ns.shards_dir)
|
||||
if ns.include_non_numeric or s.stem.isdigit()
|
||||
]
|
||||
if not shards:
|
||||
parser.error(f"no shards found under {ns.shards_dir}")
|
||||
|
||||
print(
|
||||
f">> {len(shards)} shard(s) × {len(extractors)} extractor(s) "
|
||||
f"× {ns.workers} workers",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
grand_t0 = time.time()
|
||||
work = [(s, extractors) for s in shards]
|
||||
with mp.Pool(processes=ns.workers) as pool:
|
||||
for outcome in pool.imap_unordered(_run_one_shard, work):
|
||||
print(f"[{outcome['name']}] {outcome['elapsed_s']}s", flush=True)
|
||||
for kind, result in outcome["results"].items():
|
||||
print(f" {kind}: {result}", flush=True)
|
||||
|
||||
print(f"\n>> total wall-clock: {round(time.time() - grand_t0, 1)}s")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Loading…
Add table
Add a link
Reference in a new issue