2 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
8d6961fcc1
|
aborist/arborist
modified: .gitlab-ci.yml modified: bench/qa_questions.txt modified: bench/qa_sweep.py modified: bench/run.sh modified: docs/TICKETS.md modified: docs/_source/README.md modified: docs/_source/_ext/makefile_targets.py modified: docs/_source/api/cli.rst modified: docs/_source/api/distill.rst modified: docs/_source/api/mesh.rst modified: docs/_source/api/qa.rst modified: docs/_source/api/retrieval.rst modified: docs/_source/api/storage.rst modified: docs/_source/api/substrate.rst modified: docs/_source/concepts.rst modified: docs/_source/conf.py modified: docs/_source/cookbook.rst modified: docs/_source/index.rst modified: docs/_source/license.rst modified: docs/_source/quickstart.rst modified: docs/bench-maxing.md modified: docs/benchmarks.md modified: docs/cti-architecture.md modified: docs/diagrams/aborist-modules.dot modified: docs/diagrams/aborist-modules.svg modified: docs/diagrams/mesh-data-flow.dot modified: docs/diagrams/mesh-epoch-lifecycle.dot modified: docs/diagrams/mesh-epoch-lifecycle.svg modified: docs/diagrams/mesh-group-decisions.dot modified: docs/diagrams/mesh-group-decisions.svg modified: docs/diagrams/mesh-identity-stack.dot modified: docs/diagrams/mesh-secret-envelope.dot modified: docs/mesh.md modified: docs/qa-modes-bench.md modified: docs/seven-point-program.md modified: docs/tickets/ticket-000001-retrieval-keywords-audit-gap.md modified: docs/tickets/ticket-000002-reference-frame-polarity-contract.md modified: docs/tickets/ticket-000003-anchor-class-warrant.md modified: docs/tickets/ticket-000005-label-ladder-migration.md modified: docs/tickets/ticket-000006-bench-emergent-findings.md modified: docs/tickets/ticket-000007-query-layer-hyphen-fold.md modified: docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md modified: docs/tickets/ticket-000009-quantifier-preflight-dag-binding.md modified: docs/tickets/ticket-000010-metacognition-preflight-guard.md modified: docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md modified: scripts/backfill_concepts.py modified: scripts/bench_emergent.py modified: tests/crawler/test_async_web_fetcher.py modified: tests/crawler/test_bridge.py modified: tests/crawler/test_web_fetch.py modified: tests/test_bench_qa_sweep.py modified: tests/test_burn.py modified: tests/test_burn_doc.py modified: tests/test_claim_lattice.py modified: tests/test_cli_render.py modified: tests/test_compress.py modified: tests/test_concepts.py modified: tests/test_dag.py modified: tests/test_directives.py modified: tests/test_distill.py modified: tests/test_distill_recursive.py modified: tests/test_evict.py modified: tests/test_frame.py modified: tests/test_grok_source.py modified: tests/test_html_source.py modified: tests/test_ingest.py modified: tests/test_inspect.py modified: tests/test_journal.py modified: tests/test_keys.py modified: tests/test_llm_context_base.py modified: tests/test_merkle.py modified: tests/test_mesh.py modified: tests/test_mesh_aead.py modified: tests/test_mesh_chain.py modified: tests/test_mesh_cli.py modified: tests/test_mesh_cli_pull.py modified: tests/test_mesh_wire.py modified: tests/test_mesh_wire_e2e.py modified: tests/test_metacognition.py modified: tests/test_migration_audit_mode.py modified: tests/test_providence_source.py modified: tests/test_qa.py modified: tests/test_qa_quality_live.py modified: tests/test_quantifier_caps.py modified: tests/test_quantifier_classifier.py modified: tests/test_quantifier_phase4.py modified: tests/test_quantifier_reminder.py modified: tests/test_query.py modified: tests/test_reclassify.py modified: tests/test_repair.py modified: tests/test_resume.py modified: tests/test_snapshot.py modified: tests/test_soft_preflight.py modified: tests/test_tfidf.py modified: tests/test_vcs_source.py modified: tests/test_verify.py modified: tests/test_verify_json.py modified: tests/test_versioned_ingest.py modified: tests/test_warrant.py modified: tests/test_wikipedia_old.py modified: tests/test_wikipedia_xml.py modified: tests/test_wikitext.py |
|||
|
9860423dca
|
make: speed up tests + automate concept backfill (test 38s→11s)
Three dev-loop speedups:
(1) `make test` already on -n auto via pytest-xdist (was implicit
serial); 38s → 11s wall-clock = 3.4× faster on the 641-test
suite. Big inner-loop win.
(2) `make test-live` now also uses -n auto (live tests are
independent against the Hermes endpoint; concurrency=4 doesn't
overload it on the 17-test fixture set).
(3) `make backfill-concepts` (new) replaces the ad-hoc
`python -c "from aborist.concepts.extract import …"` invocations
fox was running by hand for the post-2026-05-02 concept-layer
backfills. Parallelizes per-shard work via multiprocessing.Pool
with CONCEPTS_WORKERS=4 (env-tunable).
Driven by scripts/backfill_concepts.py — runs every registered
extractor in EXTRACTORS (link_reciprocity, token_idf,
documents_fts) across every numeric-stem shard. Skips qa.db /
snapshots.db / crawl_*.db by default; --include-non-numeric
opts in. Wall-clock 189s for 4 wiki shards × 3 extractors vs.
~260s serial estimate; modest 1.4× speedup because SQLite WAL
+ FTS5 vocab queries are I/O-bound on a single SSD (4 workers
contend), but the unified UX & structured progress output are
the real wins.
(4) `make bench-qa-quick` (new) — 5-question smoke fixture × all
3 modes × 1 sample × concurrency 4. ~10s wall-clock. Sits
between bench-qa-smoke (n=1, ~30s) and full bench-qa
(~70min). Use as the inner-loop pre-commit signal.
Also: docs/concept-relations-design.md updated to point at the
new make target instead of the inline `python -c` block.
No behavior change in the test suite or LLM pipeline; pure tooling.
|
