Skip to content

Commit fa841dc

Browse files
authored
test(smoke): add deterministic composition probes to the e2e smoke tier
The deterministic tier covered bash/script/bun/uv/timeout and two trigger rules. It did not cover join semantics against a skipped upstream, until_bash termination, output_type, or fan-out at all — so the composition primitives merged in #2223/#2224/#2467 had no unattended regression test. Four workflows, no AI nodes, all assertions in bash: e2e-joins trigger_rule all_done + none_failed_min_one_success against a SKIPPED upstream, output_type, and a loop_group terminating on until_bash e2e-echo-child one bash node; the cheapest possible fan-out child e2e-fanout-alldone fan_out over a literal list with one failing child; all_done must aggregate it as {error,status} e2e-fanout-allsuccess same list, join: all_success — must FAIL the node. Wired as a negative test, so a zero exit is the regression and CI inverts the assertion Each runs in seconds and costs nothing, which is what makes them viable on every push rather than as a deliberate exercise. Verified with --no-worktree, the form CI uses: the first two exit 0, the third exits 1. Also adds rasmus-tests/ for the AI-driven probes these were derived from. Those need a funded provider and minutes per run, so they stay out of CI — t8-cascade additionally needs a concurrent abandon to observe cascade-cancel and cannot run unattended without a driver script. The split is now readable from either side: unattended work lives in test-workflows/, subscription work in rasmus-tests/. Two engine findings came out of running them: #2494 and #2495.
1 parent 3173489 commit fa841dc

15 files changed

Lines changed: 565 additions & 0 deletions

.archon/config.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,11 @@ worktree:
33

44
docs:
55
path: packages/docs-web/src/content/docs
6+
7+
aliases:
8+
# Cheap model for the t1–t4 engine-primitive test workflows. Every test workflow
9+
# references `@mini` rather than a literal model id, so retargeting them all is a
10+
# one-line edit here — and so per-run tier/alias rebinding (#2481) can reach them.
11+
'@mini':
12+
provider: pi
13+
model: minimax/MiniMax-M3
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# rasmus-tests
2+
3+
Test workflows running on Rasmus's config.
4+
5+
They exercise engine primitives against real GitHub issues, so they need a funded
6+
provider and take minutes per run — that is why they are not in CI. The deterministic
7+
counterparts that *are* in CI live in `../test-workflows/` (`e2e-joins`,
8+
`e2e-fanout-alldone`, `e2e-fanout-allsuccess`).
9+
10+
| Workflow | Exercises |
11+
| --- | --- |
12+
| `t1-fix-issue` | structured output · `when:` · `cancel:` · `loop_group` self-heal via `$LOOP_PREV` |
13+
| `t2-fix-issue-include` + `t2-review-block` | `include:` · `with:``$INPUTS` at load · namespacing |
14+
| `t3-triage-fanout` + `t3-probe-issue` | `fan_out` over a runtime list · `max_parallel` · `parent_run_id` |
15+
| `t4-subrun` | `workflow:` child run · `input:` forwarding · output threading |
16+
| `t8-cascade` + `t8-slow-child` | cascade-cancel — needs a concurrent `archon workflow abandon` |
17+
18+
All AI nodes resolve through the `@mini` alias in `.archon/config.yaml`.
Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
name: t1-fix-issue
2+
description: |
3+
ENGINE-PRIMITIVE TEST — not a quality workflow. Prompts are deliberately minimal;
4+
the point is to exercise the engine, not to produce good output.
5+
6+
Proves: bash node (stdout as output) · structured output + strict `.field` access ·
7+
`when:` on a structured field · `cancel:` as a first-class outcome · `loop_group`
8+
self-heal, where BOTH the validation failure and the review findings feed the next
9+
implement iteration via `$LOOP_PREV` · `until:` signal · `max_iterations` bound.
10+
11+
NO ARTIFACTS. Nothing writes to $ARTIFACTS_DIR. Every hand-off between nodes is
12+
`$node.output`. This is the by-reference-vs-by-value experiment (#2123, #1848) run
13+
as a workflow: if in-and-out alone is sufficient here, that is evidence.
14+
15+
Usage: archon workflow run t1-fix-issue "<issue-number>"
16+
model: '@mini'
17+
18+
nodes:
19+
- id: fetch
20+
depends_on: []
21+
bash: |
22+
set -euo pipefail
23+
gh issue view "$ARGUMENTS" --json number,title,body,comments
24+
25+
- id: assess
26+
depends_on: [fetch]
27+
prompt: |
28+
$fetch.output
29+
30+
Read the code this issue refers to. Is the problem real and worth fixing now?
31+
output_format:
32+
type: object
33+
properties:
34+
verdict:
35+
type: string
36+
enum: ['fix', 'skip']
37+
reason:
38+
type: string
39+
required: [verdict, reason]
40+
41+
- id: bail
42+
depends_on: [assess]
43+
when: "$assess.output.verdict == 'skip'"
44+
cancel: 'Not worth fixing — $assess.output.reason'
45+
46+
- id: build
47+
depends_on: [assess]
48+
when: "$assess.output.verdict == 'fix'"
49+
loop_group:
50+
until: 'BUILD-CLEAN'
51+
max_iterations: 3
52+
nodes:
53+
- id: implement
54+
depends_on: []
55+
prompt: |
56+
Fix this issue with the smallest change that works, then commit.
57+
Do not write any report files — your reply is the only output that matters.
58+
59+
$fetch.output
60+
61+
Findings from the previous attempt (empty on the first iteration):
62+
$LOOP_PREV.review.output
63+
64+
- id: check
65+
depends_on: [implement]
66+
bash: |
67+
# This node must ALWAYS exit 0. A failing body node fails the whole
68+
# loop_group immediately — which would kill the very loop that exists to
69+
# fix the failure. So the gate REPORTS a verdict and `review` acts on it.
70+
# (Fail-fast is right for a linear DAG and wrong inside a self-heal loop.)
71+
set -o pipefail
72+
ok=1
73+
echo '=== type-check ==='
74+
bun run type-check 2>&1 | tail -30 || ok=0
75+
# `bun --filter`, never a bare `bun test <dir>`. A single invocation over a
76+
# whole package runs every file in ONE process, where conflicting
77+
# `mock.module()` calls poison each other — 616 spurious failures on an
78+
# unmodified tree. Each package's own `test` script carries the splits that
79+
# avoid it (see CLAUDE.md, "Test isolation").
80+
for p in $(git diff --name-only "$BASE_BRANCH"...HEAD | grep '^packages/' | cut -d/ -f2 | sort -u); do
81+
echo "=== tests: @archon/$p ==="
82+
bun --filter "@archon/$p" test 2>&1 | tail -20 || ok=0
83+
done
84+
if [ "$ok" = 1 ]; then echo 'VALIDATION: PASS'; else echo 'VALIDATION: FAIL'; fi
85+
86+
- id: review
87+
depends_on: [check]
88+
prompt: |
89+
Validation output — type-check, plus the tests of every package the diff touches:
90+
$check.output
91+
92+
Review the working-tree diff against $BASE_BRANCH.
93+
94+
The validation above is AUTHORITATIVE and you cannot overrule it. If it ends
95+
in `VALIDATION: FAIL`, do not reply BUILD-CLEAN no matter how the diff reads —
96+
say what failed and what to change.
97+
98+
If it ends in `VALIDATION: PASS` and you have no findings of your own, reply
99+
with exactly: BUILD-CLEAN
100+
Otherwise list what to fix, in one short paragraph. That text is fed back
101+
to the implementer verbatim on the next iteration.
102+
103+
- id: open-pr
104+
depends_on: [build]
105+
bash: |
106+
set -euo pipefail
107+
git push -u origin HEAD
108+
gh pr create --base "$BASE_BRANCH" --title "fix: issue $ARGUMENTS" --body "$(cat <<'ARCHON_EOF'
109+
Automated fix for issue $ARGUMENTS.
110+
111+
## Assessment
112+
$assess.output.reason
113+
114+
## Final review pass
115+
$build.output
116+
ARCHON_EOF
117+
)"
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
name: t2-fix-issue-include
2+
description: |
3+
ENGINE-PRIMITIVE TEST — t1 with the review step pulled in by reference instead of
4+
written inline. Same task, different composition.
5+
6+
Proves: `include:` load-time inlining · `with:` inputs and the `$INPUTS.<name>`
7+
macro (#2467, merged 5 Aug) · namespacing — the block's internal `$gather.output`
8+
ref must survive being renamed to `review__gather` · `$includeId.output` resolving
9+
to the block's terminal node.
10+
11+
Worth running once with a required input REMOVED, to confirm it fails the LOAD
12+
rather than reaching a model.
13+
14+
Usage: archon workflow run t2-fix-issue-include "<issue-number>"
15+
model: '@mini'
16+
17+
nodes:
18+
- id: fetch
19+
depends_on: []
20+
bash: |
21+
set -euo pipefail
22+
gh issue view "$ARGUMENTS" --json number,title,body
23+
24+
- id: implement
25+
depends_on: [fetch]
26+
prompt: |
27+
Fix this issue with the smallest change that works, then commit.
28+
29+
$fetch.output
30+
31+
- id: review
32+
depends_on: [implement]
33+
include: t2-review-block
34+
with:
35+
base: dev
36+
focus: correctness of the fix, and nothing else
37+
38+
- id: summary
39+
depends_on: [review]
40+
prompt: |
41+
Review findings:
42+
$review.output
43+
44+
State in one line whether this change is ready to open as a PR.
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
name: t2-review-block
2+
description: |
3+
ENGINE-PRIMITIVE TEST — a shared building block, not a standalone workflow.
4+
Authored once, pulled into any workflow that needs a review pass via `include:`.
5+
6+
Takes two load-time inputs: `$INPUTS.base` (branch to diff against) and
7+
`$INPUTS.focus` (what the reviewer should care about). Both are substituted at
8+
DISCOVERY time, before anything executes — a missing one fails the load.
9+
model: '@mini'
10+
11+
nodes:
12+
- id: gather
13+
depends_on: []
14+
bash: |
15+
set -euo pipefail
16+
git diff "$INPUTS.base"...HEAD --stat
17+
echo '---'
18+
git diff "$INPUTS.base"...HEAD | head -400
19+
20+
- id: critique
21+
depends_on: [gather]
22+
prompt: |
23+
$gather.output
24+
25+
Review this diff. Focus on: $INPUTS.focus
26+
Reply with your findings in one short paragraph, or "no findings".
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: t3-probe-issue
2+
description: |
3+
ENGINE-PRIMITIVE TEST — the fan-out child. Probes ONE issue and reports a triage
4+
verdict. Read-only: it reads the issue, reads the code, reads direction.md, and
5+
writes nothing. No labels are applied — the parent reports proposals to a human.
6+
7+
Runs standalone too: archon workflow run t3-probe-issue "<issue-number>"
8+
model: '@mini'
9+
10+
# Required for the fan-out in t3-triage-fanout. Concurrent children on one shared
11+
# checkout take a path-exclusive lock, so without this the engine refuses to spawn
12+
# them at all (correctly — a lock-cancelled child is not recoverable by resume,
13+
# #2180). This workflow only reads the issue, the code and direction.md.
14+
mutates_checkout: false
15+
16+
nodes:
17+
- id: issue
18+
depends_on: []
19+
bash: |
20+
set -euo pipefail
21+
gh issue view "$ARGUMENTS" --json number,title,body,comments,labels
22+
23+
- id: probe
24+
depends_on: [issue]
25+
prompt: |
26+
$issue.output
27+
28+
Check the issue's claims against the actual code, then check it against the
29+
project direction in `.archon/maintainer-standup/direction.md`.
30+
31+
Pick one label:
32+
should-fix — real, in scope, worth doing
33+
should-close — not real, already fixed, or out of scope per direction.md
34+
needs-human — you cannot tell without a maintainer's judgement
35+
output_format:
36+
type: object
37+
properties:
38+
issue:
39+
type: string
40+
label:
41+
type: string
42+
enum: ['should-fix', 'should-close', 'needs-human']
43+
reason:
44+
type: string
45+
required: [issue, label, reason]
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: t3-triage-fanout
2+
description: |
3+
ENGINE-PRIMITIVE TEST — fan-out over unlabelled issues, one child run each.
4+
5+
Proves: `fan_out` over a runtime item list (#2224, merged 5 Aug) · `max_parallel` ·
6+
`join: all_done` · child runs with `parent_run_id` · terminal outputs threading back
7+
into the parent as `$node.output`.
8+
9+
READ-ONLY BY DESIGN. No child writes anything, so no child needs isolation — this is
10+
the common fan-out shape, and the one that shows isolation is not a prerequisite.
11+
Labels are proposed to a human, never applied.
12+
13+
The assertions that matter here are negative: a child returning `needs-human` must
14+
not fail the parent, and abandoning the parent must cascade-cancel the children.
15+
16+
Usage: archon workflow run t3-triage-fanout ""
17+
model: '@mini'
18+
19+
nodes:
20+
- id: find
21+
depends_on: []
22+
bash: |
23+
set -euo pipefail
24+
gh issue list --state open --search 'no:label' --limit 3 --json number \
25+
| jq -c '[.[].number | tostring]'
26+
27+
- id: probe-each
28+
depends_on: [find]
29+
workflow: t3-probe-issue
30+
input: '$find.output'
31+
fan_out:
32+
# 2, not 3 — with max_parallel == item count nothing ever queues, so the
33+
# concurrency bound would go untested. 3 items through a width of 2 forces
34+
# one child to wait for a slot.
35+
max_parallel: 2
36+
items: '$find.output'
37+
join: all_done
38+
39+
- id: triage
40+
depends_on: [probe-each]
41+
prompt: |
42+
$probe-each.output
43+
44+
List each issue with its proposed label and a one-line reason.
45+
End with the single label you are least confident about, and why.
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
name: t4-subrun
2+
description: |
3+
ENGINE-PRIMITIVE TEST — the minimal sub-run. A parent that does nothing except
4+
start one child and use its result.
5+
6+
Proves: `workflow:` as a runtime child run (#2121 phase 2) · its own run row with
7+
`parent_run_id` · `input:` forwarding as the child's `$ARGUMENTS` · the child's
8+
terminal output threading back as `$node.output`.
9+
10+
Deliberately trivial. If this fails, the sub-run mechanism is what failed — there
11+
is nothing else in the graph to blame.
12+
13+
Usage: archon workflow run t4-subrun "<issue-number>"
14+
model: '@mini'
15+
16+
nodes:
17+
- id: delegate
18+
depends_on: []
19+
workflow: t3-probe-issue
20+
input: '$ARGUMENTS'
21+
22+
- id: report
23+
depends_on: [delegate]
24+
prompt: |
25+
$delegate.output
26+
27+
State in one line what the child run concluded.
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
name: t8-cascade
2+
description: |
3+
ENGINE-PRIMITIVE TEST — cascade-cancel. Fans out three sleeping children, so the
4+
parent can be abandoned mid-flight and the children observed.
5+
6+
Pass condition: abandoning the parent leaves every non-terminal child `cancelled`,
7+
not orphaned `running`. Nothing here spends a model call.
8+
9+
Usage: archon workflow run t8-cascade "" --detach # then: archon workflow abandon <id>
10+
mutates_checkout: false
11+
12+
nodes:
13+
- id: spread
14+
depends_on: []
15+
workflow: t8-slow-child
16+
fan_out:
17+
items: '["a", "b", "c"]'
18+
max_parallel: 3
19+
join: all_done
20+
21+
- id: after
22+
depends_on: [spread]
23+
bash: echo 'should not be reached if the parent was abandoned'
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
name: t8-slow-child
2+
description: |
3+
ENGINE-PRIMITIVE TEST — a child that takes long enough to interrupt. One bash node
4+
that sleeps, so cascade-cancel can be observed without spending a model call.
5+
mutates_checkout: false
6+
7+
nodes:
8+
- id: wait
9+
depends_on: []
10+
timeout: 120000
11+
bash: |
12+
echo "child starting: $ARGUMENTS"
13+
sleep 60
14+
echo "child finished: $ARGUMENTS"

0 commit comments

Comments
 (0)