ticket #000039 §13.5: amend test plan with CLI subprocess coverage

Phase 1 plan §13.5 named four test files (3 unit + 1 integration)
but did NOT name CLI subprocess tests for the three new CLI
surfaces §13.4 introduces:

  - arborist ingest --embed (flag on existing subcommand)
  - arborist query --retrieval={fts5|vec|hybrid} (new flag)
  - arborist vec rebuild (new subcommand)

Per docs/calculator-test-patterns.md §6 (codified earlier today
in commit 0725eb4 from the three-module pattern bench): import-only
tests miss argparse + main() drift. Yesterday's substrate refactor
caught this hazard three times — fork_score.py import (85be5eb),
Makefile bench-fork-score target (209d670), .gitlab-ci.yml job
+ script (b320e27). Each fix was 1-3 lines, but each had been
shipped to main + would have surfaced as a noisy CI failure on
next pipeline run.

§13.5 now adds three CLI subprocess test files:

  - tests/test_cli_vec_rebuild.py
  - tests/test_cli_ingest_embed_flag.py
  - tests/test_cli_query_retrieval_flag.py

Each gated via pytest.importorskip("sqlite_vec") so they skip
cleanly when [vec] extras absent. Pattern matches fox's
test_cli_baseline_runs_clean / test_cli_invalid_input_exits_2
in tests/test_t3_bound_calculator.py (the exemplar for
calculator-style CLI tests).

§13.7 size estimate revised: 4 test files → 7 test files (+3 CLI
subprocess), 250 → 400 test LOC. CLI subprocess tests are
~30-40 LOC each (boilerplate + tmp_path + subprocess.run +
JSON parse). Phase 1 total grows from ~550+250 → ~550+400 LOC.

Doc-only edit; doesn't unblock or block fox's §13.8 four
decisions — the test-plan addition is mechanical discipline,
not a scope change. Phase 1 still gates on the four §13.8
decisions before any code lands.

Cross-ref: docs/calculator-test-patterns.md §6 (CLI subprocess
pattern) + the three substrate-rename defect commits caught by
that pattern in retrospect (85be5eb / 209d670 / b320e27).
This commit is contained in:
russell@unturf.com 2026-05-10 13:19:53 -04:00
parent 10b81e187d
commit 7ea0fdf2b1
No known key found for this signature in database

View file

@ -735,6 +735,31 @@ Unit tests:
fields fold into `governance_policy_hash`; flipping any one
invalidates prior cached records.
CLI surface tests (per
``docs/calculator-test-patterns.md`` §6 — subprocess invocation
catches argparse + main() drift the import-only tests miss; this
hazard surfaced three times during the 2026-05-10 substrate
rename refactor in yesterday's commit chain, so we pin the
pattern):
- `tests/test_cli_vec_rebuild.py` — subprocess invocation of
``arborist vec rebuild --db <tmp_path/test.db> --batch 4``
against a 12-chunk synthetic shard; assert exit 0, JSON
status output if any, row-count parity in
``chunk_vecs``. Skipped via ``pytest.importorskip`` when
``[vec]`` extras absent.
- `tests/test_cli_ingest_embed_flag.py` — subprocess invocation
of ``arborist ingest --source html --embed --author X``
against a tmp shard; assert ``--embed`` flag wires through
to the embedder; row count after ingest matches; vec rows
match chunk rows 1:1.
- `tests/test_cli_query_retrieval_flag.py` — subprocess
invocation of ``arborist query --retrieval=vec`` and
``--retrieval=hybrid`` and ``--retrieval=fts5`` against a
pre-embedded shard fixture; assert all three exit 0 and
produce structurally-distinct retrieval candidate sets
(vec ≠ fts5 on at least one query).
Integration tests (gated on `[vec]` extras installed; skipped
otherwise):
- `tests/test_vec_smoke_integration.py` — runs the §13.2 smoke on
@ -754,14 +779,18 @@ otherwise):
- New files: 2 (embed.py, search/vec.py) ~ 200 LOC.
- Patches: 4 (store.py, query.py, cli.py, Makefile) ~ 100 LOC.
- Tests: 4 files ~ 250 LOC + fixtures.
- Tests: 7 files ~ 400 LOC + fixtures (4 unit + 3 CLI subprocess
per the §13.5 amendment 2026-05-10 + 1 integration). The CLI
subprocess tests are ~30-40 LOC each — boilerplate + tmp_path
+ subprocess.run + JSON parse.
- Phase-1 doc append in `docs/`: this §13 + a journey-note bench
result file.
- pyproject.toml: 1 stanza.
Total: ~550 LOC + ~250 test LOC. Single substantial commit if all
Total: ~550 LOC + ~400 test LOC. Single substantial commit if all
tests pass + smoke succeeds; otherwise broken into the natural
gates (smoke → embed.py → vec.py → integration tests → bench).
gates (smoke → embed.py → vec.py → CLI subprocess tests →
integration tests → bench).
### 13.8 Decisions fox needs to make to unblock Phase 1