arborist/docs
russell@unturf.com 9860423dca
make: speed up tests + automate concept backfill (test 38s→11s)
Three dev-loop speedups:

(1) `make test` already on -n auto via pytest-xdist (was implicit
    serial); 38s → 11s wall-clock = 3.4× faster on the 641-test
    suite. Big inner-loop win.

(2) `make test-live` now also uses -n auto (live tests are
    independent against the Hermes endpoint; concurrency=4 doesn't
    overload it on the 17-test fixture set).

(3) `make backfill-concepts` (new) replaces the ad-hoc
    `python -c "from aborist.concepts.extract import …"` invocations
    fox was running by hand for the post-2026-05-02 concept-layer
    backfills. Parallelizes per-shard work via multiprocessing.Pool
    with CONCEPTS_WORKERS=4 (env-tunable).

    Driven by scripts/backfill_concepts.py — runs every registered
    extractor in EXTRACTORS (link_reciprocity, token_idf,
    documents_fts) across every numeric-stem shard. Skips qa.db /
    snapshots.db / crawl_*.db by default; --include-non-numeric
    opts in. Wall-clock 189s for 4 wiki shards × 3 extractors vs.
    ~260s serial estimate; modest 1.4× speedup because SQLite WAL
    + FTS5 vocab queries are I/O-bound on a single SSD (4 workers
    contend), but the unified UX & structured progress output are
    the real wins.

(4) `make bench-qa-quick` (new) — 5-question smoke fixture × all
    3 modes × 1 sample × concurrency 4. ~10s wall-clock. Sits
    between bench-qa-smoke (n=1, ~30s) and full bench-qa
    (~70min). Use as the inner-loop pre-commit signal.

Also: docs/concept-relations-design.md updated to point at the
new make target instead of the inline `python -c` block.

No behavior change in the test suite or LLM pipeline; pure tooling.
2026-05-02 10:11:16 -04:00
..
diagrams docs: README label refresh + per-module reference + Graphviz diagrams 2026-05-01 23:19:01 -04:00
modules docs: README label refresh + per-module reference + Graphviz diagrams 2026-05-01 23:19:01 -04:00
bench-maxing.md docs(claude): compress + attribute — split bench-maxing, add docs index 2026-05-01 14:12:37 -04:00
concept-relations-design.md make: speed up tests + automate concept backfill (test 38s→11s) 2026-05-02 10:11:16 -04:00
cti-architecture.md arch: layer-cake docs + verifier_policy_hash + old-maps principle 2026-05-01 12:10:29 -04:00
mesh-deploy.md docs: mesh-deploy runbook for two-host gossip setup 2026-04-28 17:24:37 -04:00
mesh.md docs: mesh multiplayer protocol with dot diagrams 2026-04-28 12:54:36 -04:00
naming-deferral.md arch: layer-cake docs + verifier_policy_hash + old-maps principle 2026-05-01 12:10:29 -04:00
qa-modes-bench-2026-04-30.md docs(qa): final stop-sequence bench row, day-journey net summary 2026-04-30 16:32:41 -04:00
qa-modes-bench-2026-05-02.md bench: 2026-05-02 journal + CLAUDE.md headline refresh 2026-05-02 08:50:45 -04:00
reference-frame-failure-class.md qa: phrase-pattern retrieval route closes the reference-frame failure class 2026-05-01 13:53:03 -04:00
self-reference-distillation-design.md arch: layer-cake docs + verifier_policy_hash + old-maps principle 2026-05-01 12:10:29 -04:00
self-reference-thought-chains-design.md qa: self-reference thought chains — STRICT-as-fact substrate 2026-05-01 10:16:47 -04:00
seven-point-program.md qa(frame): #000002 land — reference-frame polarity contract (D3 → ✓) 2026-05-01 20:18:13 -04:00
test-coverage-audit-2026-05-01.md arch: layer-cake docs + verifier_policy_hash + old-maps principle 2026-05-01 12:10:29 -04:00
ticket-000001-retrieval-keywords-audit-gap.md qa(provenance): #000001 land — retrieval_plan_hash binds D4 2026-05-01 19:26:01 -04:00
ticket-000002-reference-frame-polarity-contract.md qa(frame): #000002 land — reference-frame polarity contract (D3 → ✓) 2026-05-01 20:18:13 -04:00
ticket-000003-anchor-class-warrant.md qa(warrant): #000003 land — anchor-class generalization (D6 → ✓) 2026-05-01 19:16:06 -04:00
ticket-000004-directive-coverage-in-bench.md docs: open #000004 (closed/retroactive) + #000005; update program status 2026-05-01 16:31:44 -04:00
ticket-000005-label-ladder-migration.md qa(label): #000005 land — four-rung ladder migration (D7 sharpened) 2026-05-01 19:20:47 -04:00
TICKETS.md docs(concepts): design reference + 1.6% storage-tax rationale 2026-05-01 21:39:09 -04:00
verifier-semantic-gap-design.md docs(qa): design proposal for verifier semantic-gap NLI sidecar 2026-04-30 19:50:05 -04:00