Skip to content

Commit 908e12f

Browse files
authored
fix(graph): forward RenderConfig.exclude to test tasks for all test behaviors (#2850)
## Description `RenderConfig.exclude` was only applied to the **aggregate test task** created for `TestBehavior.AFTER_ALL`. For `AFTER_EACH` / `BUILD`, the per-model and detached (multiple-parents) test tasks ignored it — so an exclusion such as `exclude=["resource_type:unit_test"]` did **not** stop unit tests from running, and users had to repeat the exclusion in `operator_args` (e.g. `dbt_cmd_flags`) as a workaround. ### Fix - `create_test_task_metadata()` now forwards `render_config.exclude` in the **node** (per-model / detached) branch too, via a small `forward_render_exclude_to_test()` helper. - `generate_task_or_group()` passes `render_config` through to the `AFTER_EACH` test task (it previously wasn't, which is why the node branch never saw the exclusion). Exclusions are purely additive, so forwarding them is safe. `select` / `selector` are **intentionally** still only forwarded for `AFTER_ALL` — combining a global selection with the per-model selection would require resolving dbt's union/intersection set semantics, which Cosmos does not attempt. This limitation is now documented in `selecting-excluding.rst`. ### Behavior (validated by the new integration test, `MULTIPLE_PARENTS_TEST_DBT_PROJECT` with `exclude=["resource_type:unit_test"]`) | Test task | `--exclude` before | `--exclude` after | |---|---|---| | `combined_model.test` (parent of detached test) | `custom_test_…` | `resource_type:unit_test custom_test_…` | | `model_a.test` (parent) | `custom_test_…` | `resource_type:unit_test custom_test_…` | | `model_b.test` (not a parent) | _(none)_ | `resource_type:unit_test` | | detached multi-parent test | _(none)_ | `resource_type:unit_test` | ## Related Issue(s) Closes #1763 Relates to #1865 (already closed) ## Context This is a smaller **re-implementation of #2006** (by @anyapriya, which went stale) against the **current, refactored** graph code. The two pieces #2006 carried — the `detached_from_parent: dict[str, list[DbtNode]]` type-hint fix and avoiding render-config mutation — are already on `main` (the latter via the `_convert_list_to_str` helper), so this PR only needs the exclude-forwarding. Credit to @anyapriya for the original fix and tests, which this preserves. ## Breaking Change? No. ## Checklist - [x] I have made corresponding changes to the documentation - [x] I have added tests that prove my fix is effective (unit tests in `tests/airflow/test_graph.py`; integration test in `tests/test_converter.py`) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent cafb34d commit 908e12f

4 files changed

Lines changed: 259 additions & 9 deletions

File tree

cosmos/airflow/graph.py

Lines changed: 38 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,15 @@ def calculate_leaves(tasks_ids: list[str], nodes: dict[str, DbtNode]) -> list[st
128128
return leaves
129129

130130

131+
def _split_exclude(value: list[str] | str | None) -> list[str]:
132+
"""Normalize an ``exclude`` value (space-separated str, list, or None) to a list of items."""
133+
if isinstance(value, str):
134+
return value.split()
135+
if isinstance(value, list):
136+
return list(value)
137+
return []
138+
139+
131140
def exclude_detached_tests_if_needed(
132141
node: DbtNode,
133142
task_args: dict[str, str],
@@ -140,21 +149,32 @@ def exclude_detached_tests_if_needed(
140149
"""
141150
if detached_from_parent is None:
142151
detached_from_parent = {}
143-
current_exclude = task_args.get("exclude")
144-
# Handle both list[str] (legacy) and str formats for backward compatibility
145-
if isinstance(current_exclude, list):
146-
exclude_items = current_exclude
147-
elif isinstance(current_exclude, str):
148-
exclude_items = current_exclude.split() if current_exclude else []
149-
else:
150-
exclude_items = []
152+
exclude_items = _split_exclude(task_args.get("exclude"))
151153
tests_detached_from_this_node: list[DbtNode] = detached_from_parent.get(node.unique_id, [])
152154
for test_node in tests_detached_from_this_node:
153155
exclude_items.append(test_node.resource_name.split(".")[0])
154156
if exclude_items:
155157
task_args["exclude"] = _convert_list_to_str(exclude_items) or ""
156158

157159

160+
def forward_render_exclude_to_test(task_args: dict[str, Any], render_config: RenderConfig | None) -> None:
161+
"""
162+
Forward ``RenderConfig.exclude`` to a node command that runs tests, in-place — a per-model
163+
(AFTER_EACH / detached) test task, or the inline ``dbt build`` command under TestBehavior.BUILD.
164+
165+
Exclusions are purely additive: the render-level excludes are unioned with any exclude already
166+
present in ``task_args`` (e.g. supplied via ``operator_args``), preserving both. ``select`` /
167+
``selector`` are intentionally not forwarded here.
168+
See https://github.com/astronomer/astronomer-cosmos/issues/1763.
169+
"""
170+
if render_config is None or not render_config.exclude:
171+
return
172+
existing_items = _split_exclude(task_args.get("exclude"))
173+
render_items = _split_exclude(render_config.exclude)
174+
merged = existing_items + [item for item in render_items if item not in existing_items]
175+
task_args["exclude"] = _convert_list_to_str(merged)
176+
177+
158178
def _override_profile_if_needed(task_kwargs: dict[str, Any], profile_kwargs_override: dict[str, Any]) -> None:
159179
"""
160180
Changes in-place the profile configuration if it needs to be overridden.
@@ -189,8 +209,10 @@ def create_test_task_metadata(
189209
:param execution_mode: The Cosmos execution mode we're aiming to run the dbt task at (e.g. local)
190210
:param task_args: Arguments to be used to instantiate an Airflow Task
191211
:param on_warning_callback: A callback function called on warnings with additional Context variables “test_names”
192-
and “test_results” of type List.
212+
and “test_results” of type List.
193213
:param node: If the test relates to a specific node, the node reference
214+
:param render_config: The RenderConfig for the dbt project. Its ``exclude`` is forwarded to the test task (for
215+
every test behavior); its ``select`` / ``selector`` are applied only to the TestBehavior.AFTER_ALL task.
194216
:param detached_from_parent: Dictionary that maps node ids and their children tests that should be run detached
195217
:returns: The metadata necessary to instantiate the source dbt node as an Airflow task.
196218
"""
@@ -212,6 +234,8 @@ def create_test_task_metadata(
212234
else: # tested with node.resource_type == DbtResourceType.SEED or DbtResourceType.SNAPSHOT
213235
task_args["select"] = node.resource_name
214236

237+
forward_render_exclude_to_test(task_args, render_config)
238+
215239
extra_context = {"dbt_node_config": node.context_dict}
216240
task_owner = node.owner
217241
elif render_config is not None: # TestBehavior.AFTER_ALL
@@ -430,6 +454,10 @@ def create_task_metadata( # noqa: C901
430454
if test_indirect_selection != TestIndirectSelection.EAGER:
431455
args["indirect_selection"] = test_indirect_selection.value
432456
args["on_warning_callback"] = on_warning_callback
457+
# Under BUILD, tests run inline with ``dbt build`` (there is no separate per-model test task),
458+
# so forward the render-level exclude here too — otherwise an exclusion such as
459+
# exclude=["resource_type:unit_test"] would not be honored for BUILD. See #1763.
460+
forward_render_exclude_to_test(args, render_config)
433461
exclude_detached_tests_if_needed(node, args, detached_from_parent)
434462
task_id, args = _get_task_id_and_args(
435463
node=node,
@@ -678,6 +706,7 @@ def generate_task_or_group(
678706
test_indirect_selection,
679707
task_args=task_args,
680708
node=node,
709+
render_config=render_config,
681710
on_warning_callback=on_warning_callback,
682711
detached_from_parent=detached_from_parent,
683712
enable_owner_inheritance=render_config.enable_owner_inheritance,

docs/guides/translate_dbt_to_airflow/selecting-excluding.rst

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,34 @@ Examples:
145145
)
146146
)
147147
148+
How ``exclude`` and ``select`` interact with test tasks
149+
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
150+
151+
``RenderConfig.exclude`` is passed through to the generated **test tasks for every test
152+
behavior** (``AFTER_EACH``, ``BUILD`` and ``AFTER_ALL``). This means an exclusion such as
153+
``exclude=["resource_type:unit_test"]`` is applied to the ``dbt test`` commands too, so you can
154+
exclude a specific test or a whole resource type without also having to repeat it in
155+
``operator_args``. Exclusions are purely additive, which makes this safe to forward.
156+
157+
.. code-block:: python
158+
159+
from cosmos.airflow.dag import DbtDag
160+
from cosmos.config import RenderConfig
161+
162+
jaffle_shop = DbtDag(
163+
render_config=RenderConfig(
164+
exclude=[
165+
"resource_type:unit_test"
166+
], # excluded from both the run and the test tasks
167+
)
168+
)
169+
170+
``RenderConfig.select`` / ``RenderConfig.selector``, on the other hand, are forwarded to the test
171+
task **only** for ``TestBehavior.AFTER_ALL``. They are intentionally not forwarded to the per-model
172+
``AFTER_EACH`` / ``BUILD`` test tasks, because combining a global selection with the per-model
173+
selection would require resolving dbt's union / intersection set semantics, which Cosmos does not
174+
attempt. Those test tasks are already scoped to their own model's selection.
175+
148176
Using ``selector``
149177
~~~~~~~~~~~~~~~~~~
150178
.. note::

tests/airflow/test_graph.py

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -827,6 +827,30 @@ def test_create_task_metadata_ephemeral_model_disabled_renders_dbt_build_in_buil
827827
assert metadata.operator_class == "cosmos.operators.local.DbtBuildLocalOperator"
828828

829829

830+
def test_create_task_metadata_build_mode_forwards_render_config_exclude():
831+
"""Under TestBehavior.BUILD, tests run inline with `dbt build`, so RenderConfig.exclude must be
832+
forwarded to the build command — otherwise e.g. exclude=["resource_type:unit_test"] would not
833+
take effect for BUILD. See https://github.com/astronomer/astronomer-cosmos/issues/1763."""
834+
model_node = DbtNode(
835+
unique_id=f"{DbtResourceType.MODEL.value}.my_project.model_a",
836+
resource_type=DbtResourceType.MODEL,
837+
depends_on=[],
838+
path_base=Path("base_path"),
839+
original_file_path=Path("models/model_a.sql"),
840+
fqn=["my_project", "model_a"],
841+
)
842+
metadata = create_task_metadata(
843+
model_node,
844+
execution_mode=ExecutionMode.LOCAL,
845+
args={},
846+
dbt_dag_task_group_identifier="",
847+
render_config=RenderConfig(test_behavior=TestBehavior.BUILD, exclude=["resource_type:unit_test"]),
848+
)
849+
assert metadata.operator_class == "cosmos.operators.local.DbtBuildLocalOperator"
850+
assert metadata.arguments["select"] == "fqn:my_project.model_a"
851+
assert metadata.arguments["exclude"] == "resource_type:unit_test"
852+
853+
830854
def test_create_task_metadata_ephemeral_empty_operator_inherits_owner():
831855
"""The ephemeral EmptyOperator inherits the dbt model owner when owner inheritance is enabled (default)."""
832856
metadata = create_task_metadata(
@@ -1313,6 +1337,106 @@ def test_create_test_task_metadata(node_type, node_unique_id, test_indirect_sele
13131337
}
13141338

13151339

1340+
@pytest.mark.parametrize(
1341+
"exclude,expected_exclude",
1342+
[
1343+
(["resource_type:unit_test"], "resource_type:unit_test"),
1344+
(["resource_type:unit_test", "tag:skip_in_tests"], "resource_type:unit_test tag:skip_in_tests"),
1345+
],
1346+
)
1347+
def test_create_test_task_metadata_forwards_render_config_exclude_to_node_test(exclude, expected_exclude):
1348+
"""RenderConfig.exclude must reach per-model (AFTER_EACH) / detached test tasks, not only AFTER_ALL.
1349+
1350+
See https://github.com/astronomer/astronomer-cosmos/issues/1763.
1351+
"""
1352+
sample_node = DbtNode(
1353+
unique_id=f"{DbtResourceType.MODEL.value}.my_folder.node_name",
1354+
resource_type=DbtResourceType.MODEL,
1355+
depends_on=[],
1356+
path_base=Path("."),
1357+
original_file_path=Path("."),
1358+
tags=[],
1359+
config={},
1360+
)
1361+
metadata = create_test_task_metadata(
1362+
test_task_name="test",
1363+
execution_mode=ExecutionMode.LOCAL,
1364+
test_indirect_selection=TestIndirectSelection.EAGER,
1365+
task_args={"task_arg": "value"},
1366+
node=sample_node,
1367+
render_config=RenderConfig(exclude=exclude),
1368+
)
1369+
assert metadata.arguments["select"] == "node_name"
1370+
assert metadata.arguments["exclude"] == expected_exclude
1371+
1372+
1373+
def test_create_test_task_metadata_without_render_config_exclude_preserves_existing_exclude():
1374+
"""An empty RenderConfig.exclude must not clobber an exclude supplied through operator/task args."""
1375+
sample_node = DbtNode(
1376+
unique_id=f"{DbtResourceType.MODEL.value}.my_folder.node_name",
1377+
resource_type=DbtResourceType.MODEL,
1378+
depends_on=[],
1379+
path_base=Path("."),
1380+
original_file_path=Path("."),
1381+
tags=[],
1382+
config={},
1383+
)
1384+
metadata = create_test_task_metadata(
1385+
test_task_name="test",
1386+
execution_mode=ExecutionMode.LOCAL,
1387+
test_indirect_selection=TestIndirectSelection.EAGER,
1388+
task_args={"exclude": "tag:my_custom_exclude"},
1389+
node=sample_node,
1390+
render_config=RenderConfig(exclude=[]),
1391+
)
1392+
assert metadata.arguments["exclude"] == "tag:my_custom_exclude"
1393+
1394+
1395+
def test_create_test_task_metadata_unions_render_config_exclude_with_existing_exclude():
1396+
"""A render-level exclude must be unioned with (not overwrite) an operator/task-args exclude."""
1397+
sample_node = DbtNode(
1398+
unique_id=f"{DbtResourceType.MODEL.value}.my_folder.node_name",
1399+
resource_type=DbtResourceType.MODEL,
1400+
depends_on=[],
1401+
path_base=Path("."),
1402+
original_file_path=Path("."),
1403+
tags=[],
1404+
config={},
1405+
)
1406+
metadata = create_test_task_metadata(
1407+
test_task_name="test",
1408+
execution_mode=ExecutionMode.LOCAL,
1409+
test_indirect_selection=TestIndirectSelection.EAGER,
1410+
task_args={"exclude": "tag:foo"},
1411+
node=sample_node,
1412+
render_config=RenderConfig(exclude=["resource_type:unit_test"]),
1413+
)
1414+
# Both the operator-supplied and render-level exclusions are preserved (additive).
1415+
assert metadata.arguments["exclude"] == "tag:foo resource_type:unit_test"
1416+
1417+
1418+
def test_create_test_task_metadata_does_not_duplicate_overlapping_excludes():
1419+
"""An exclude present in both task args and RenderConfig must appear only once."""
1420+
sample_node = DbtNode(
1421+
unique_id=f"{DbtResourceType.MODEL.value}.my_folder.node_name",
1422+
resource_type=DbtResourceType.MODEL,
1423+
depends_on=[],
1424+
path_base=Path("."),
1425+
original_file_path=Path("."),
1426+
tags=[],
1427+
config={},
1428+
)
1429+
metadata = create_test_task_metadata(
1430+
test_task_name="test",
1431+
execution_mode=ExecutionMode.LOCAL,
1432+
test_indirect_selection=TestIndirectSelection.EAGER,
1433+
task_args={"exclude": "resource_type:unit_test"},
1434+
node=sample_node,
1435+
render_config=RenderConfig(exclude=["resource_type:unit_test", "tag:foo"]),
1436+
)
1437+
assert metadata.arguments["exclude"] == "resource_type:unit_test tag:foo"
1438+
1439+
13161440
@pytest.mark.parametrize(
13171441
"input,expected", [("snake_case", "SnakeCase"), ("snake_case_with_underscores", "SnakeCaseWithUnderscores")]
13181442
)

tests/test_converter.py

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -377,6 +377,75 @@ def test_converter_creates_dag_with_test_with_multiple_parents():
377377
)
378378

379379

380+
@pytest.mark.integration
381+
def test_converter_passes_render_config_exclude_to_test_tasks():
382+
"""
383+
RenderConfig.exclude must be forwarded to the generated test tasks for every test behavior,
384+
not only TestBehavior.AFTER_ALL — i.e. exclude=["resource_type:unit_test"] reaches the
385+
``dbt test`` command of each per-model / detached test task.
386+
387+
See https://github.com/astronomer/astronomer-cosmos/issues/1763
388+
and https://github.com/astronomer/astronomer-cosmos/issues/1865.
389+
"""
390+
project_config = ProjectConfig(dbt_project_path=MULTIPLE_PARENTS_TEST_DBT_PROJECT)
391+
execution_config = ExecutionConfig(execution_mode=ExecutionMode.LOCAL)
392+
render_config = RenderConfig(should_detach_multiple_parents_tests=True, exclude=["resource_type:unit_test"])
393+
profile_config = ProfileConfig(
394+
profile_name="default",
395+
target_name="dev",
396+
profile_mapping=PostgresUserPasswordProfileMapping(
397+
conn_id="example_conn",
398+
profile_args={"schema": "public"},
399+
disable_event_tracking=True,
400+
),
401+
)
402+
with DAG("sample_dag", start_date=datetime(2024, 4, 16)) as dag:
403+
converter = DbtToAirflowConverter(
404+
dag=dag,
405+
project_config=project_config,
406+
profile_config=profile_config,
407+
execution_config=execution_config,
408+
render_config=render_config,
409+
)
410+
tasks = converter.tasks_map
411+
412+
assert len(converter.tasks_map) == 4
413+
414+
# The render-level exclusion is prepended to each test command; for parents of the detached
415+
# test the detached test name is then appended (exclusions are additive).
416+
args = tasks["model.my_dbt_project.combined_model"].children["combined_model.test"].build_cmd({})[0]
417+
assert args[1:] == [
418+
"test",
419+
"--select",
420+
"combined_model",
421+
"--exclude",
422+
"resource_type:unit_test custom_test_combined_model_combined_model_",
423+
]
424+
425+
args = tasks["model.my_dbt_project.model_a"].children["model_a.test"].build_cmd({})[0]
426+
assert args[1:] == [
427+
"test",
428+
"--select",
429+
"model_a",
430+
"--exclude",
431+
"resource_type:unit_test custom_test_combined_model_combined_model_",
432+
]
433+
434+
# model_b is not a parent of the detached test, so it only carries the render-level exclusion
435+
args = tasks["model.my_dbt_project.model_b"].children["model_b.test"].build_cmd({})[0]
436+
assert args[1:] == ["test", "--select", "model_b", "--exclude", "resource_type:unit_test"]
437+
438+
# The detached (multiple-parents) test task also honors the render-level exclusion
439+
args = tasks["test.my_dbt_project.custom_test_combined_model_combined_model_.c6e4587380"].build_cmd({})[0]
440+
assert args[1:] == [
441+
"test",
442+
"--select",
443+
"custom_test_combined_model_combined_model_",
444+
"--exclude",
445+
"resource_type:unit_test",
446+
]
447+
448+
380449
@pytest.mark.integration
381450
def test_converter_creates_dag_with_test_with_multiple_parents_with_should_detach_multiple_parents_tests_false():
382451
"""

0 commit comments

Comments
 (0)