fix(test): keep real-CLI smoke tests out of the ordinary test target (#664) (#667)

* fix(test): keep real-CLI smoke tests out of the ordinary test target (#664)

* docs(architecture): fix stale make-test composition claim

make test now excludes tools/tests/test_cli_smoke.py (#664); update
the quality-gates description so it no longer claims real-CLI smoke
tests run as part of make test.

Author: wshobson

.github/workflows/validate.yml

@@ -153,7 +153,7 @@ jobs:
         run: uv sync --all-extras
 
       - name: Run tools test suite
-        run: uv run pytest -q ../../tools/tests/
+        run: uv run pytest -q ../../tools/tests/ --ignore=../../tools/tests/test_cli_smoke.py
 
   multi-harness-generate:
     name: Cross-harness generation + validation

ARCHITECTURE.md

@@ -72,7 +72,7 @@ Three mechanical gates, each runnable as a make target and wired into CI:
 
 1. **`make validate`** — structural validation of every generated artifact. Errors block CI; warnings advisory.
 2. **`make garden`** — drift detection (dead links, stale artifacts, oversize skills, marketplace orphans). Sorted by severity with per-kind summary.
-3. **`make test`** — full pytest suite (386 tests: adapters + validators + gardener + real-source + round-trip + real-CLI smoke).
+3. **`make test`** — pytest suite (adapters + validators + gardener + real-source + round-trip). Real-CLI smoke tests are excluded; run them separately via `make smoke-test`.
 
 CI workflow: [`.github/workflows/validate.yml`](.github/workflows/validate.yml) runs all three on every PR, plus a `cli-smoke-test` job that installs OpenCode + Gemini and exercises them against the generated artifacts.
 

Makefile

@@ -195,7 +195,7 @@ garden:
 
 # Full pytest suite — plugin-eval framework + tools/ adapters/validators/gardener.
 test:
-	uv run $(EVAL_PROJECT) pytest -q plugins/plugin-eval/ tools/tests/
+	uv run $(EVAL_PROJECT) pytest -q plugins/plugin-eval/ tools/tests/ --ignore=tools/tests/test_cli_smoke.py
 
 # Real-CLI smoke test. Generates artifacts (if not present), then invokes whichever
 # of opencode / gemini / codex / claude are on PATH. Per-CLI tests skip gracefully