Releases: astronomer/astronomer-cosmos
v1.15.0a1
Features
- Add
ExecutionMode.WATCHER_KUBERNETESmode with test handling by @vricciardulli in #2529 and #2543 - Support dbt docs on Kubernetes via
DbtDocsS3KubernetesOperatorby @jx2lee in #2058 - Add
use_tarballoption to theupload_to_gcp_gscallback by @TTMichaelA in #2497 - Add AWS and Azure tarball upload options by @TTMichaelA in #2553
- Expose
freshness_callbackinExecutionConfig.setup_operator_argsforExecutionMode.WATCHERby @pankajastro in #2586 - Allow templating dbt deps installation via
ExecutionConfigby @pankajastro in #2728 - Add configurable seed rendering behavior by @pankajkoti in #2755
- Render ephemeral dbt models as
EmptyOperatorby default by @pankajkoti in #2759 - Support per-node callbacks via string import paths by @highlyavailable in #2781
- Map Snowflake
query_tagfrom connection extras to dbt profile by @pankajastro in #2650 - Add Snowflake profile mapping for non-encrypted private key file by @pankajastro in #2660
- Export Clickhouse and encrypted Snowflake mappings from
cosmos.profilesby @pankajastro in #2668 - Forward Snowflake
insecure_modefrom connection Extra into dbt profile by @toor11 in #2744 - Extend Snowflake profile mappings with
authenticator,client_session_keep_alive,host, andportby @Aaditya-git in #2748 - Support
extra__google_cloud_platform__projectinGoogleCloudServiceAccountDictProfileMappingby @jroachgolf84 in #2626 - Enable the orjson parser for the whole project (experimental) by @corsettigyg in #2552
- Warn when users pass output-only template fields to local operators by @goingforstudying-ctrl in #2737
Enhancements
- Replace
openlineage-airflowwithapache-airflow-providers-openlineageby @tatiana in #2753 - Refactor handling of watcher producer-only dbt flags by @tatiana in #2717
- Disclose the
--no-static-parserflag inExecutionMode.LOCALby @pankajkoti in #2714 - Refactor
_default_freshness_callbackto return a list of(id, state)tuples by @pankajastro in #2572 - Restore dynamically resolved user-provided node types by @jroachgolf84 in #2608
- Promote inline watcher status collections to named constants by @pankajastro in #2671
- Align
WATCHER_KUBERNETESproducer defaulttask_idwithPRODUCER_WATCHER_TASK_IDby @tatiana in #2794 - Centralize version-aware
EmptyOperatorimports in a compatibility module by @pankajkoti in #2758 - Move
safe_copyout ofcosmos.cacheintocosmos.fsby @pankajkoti in #2715 - Refactor seed checksum hashing and caching helpers by @pankajkoti in #2784
- Avoid O(n^2) string accumulation in
store_compiled_sqlby @pankajastro in #2751 - Use GA versions for
propagate_logsinprovider_infoby @pankajastro in #2712 - Use
cosmos.log.get_loggerconsistently by @pankajastro in #2670 - Use lazy logging instead of f-strings in logger calls by @pankajastro in #2672 and #2680
- Ban eager module-level imports of optional providers via ruff by @tatiana in #2743
- Remove unused
# type: ignorecomments and re-enablewarn_unused_ignoresby @pankajastro in #2710 and #2763 - Drop the Python <3.9 fallback for
functools.cacheby @pankajastro in #2732 - Remove stale Airflow 2.3 comment on
registry_conn_iddefault by @pankajastro in #2766 - Remove an outdated TODO on watcher terminal-status handling by @pankajastro in #2697
- Unify operator logging on
self.logby @pankajastro in #2681 - Centralise
resource_nameextraction from dbtunique_idby @pankajkoti in #2687 - Tighten
# type: ignorecomments to specify error codes by @pankajastro in #2711
Bug Fixes
- Fix
example_cosmos_sourcesto use a dbt project with sources by @jroachgolf84 in #2614 - Unify Postgres
profiles.ymlacross dbt example projects by @yeoreums in #2501 - Pin
dbt-core<2.0in virtualenv example DAGs to avoid alpha pickup by @pankajastro in #2752
Docs
- Create a glossary stub by @lzdanski in #2461
- Improve the glossary: fix structure, add Cosmos-specific terms by @pankajastro in #2618
- Add a top-level FAQ page by @tatiana in #2635
- Align the FAQ page with the documentation style guide by @tatiana in #2667
- Update docs for
DbtDocsS3KubernetesOperatorby @jx2lee in #2575 - Add docs for
ExecutionMode.WATCHERand thedepends_on_pastlimitation by @tatiana in #2602 - Document watcher mode dbt 1.5+ requirement by @tatiana in #2700
- Improve clarity for
watcher_dbt_execution_queueby @jroachgolf84 in #2705 - Document
compiled_sqlandfreshnessas output-only template fields by @00yhj22-debug in #2719 - Document seed rendering behavior by @pankajkoti in #2757
- Document tradeoffs of remote cache and disabling the dbt ls cache by @pankajkoti in #2722
- Document the LOCAL vs WATCHER benchmark on the Helm chart by @pankajkoti in #2688
- Enhance S3 dbt docs documentation by @jroachgolf84 in #2765
- Restore memory-optimised imports docs for Cosmos < 1.14.0 by @pankajkoti in #2604
- Update dbt and Airflow Spark URLs in profile mapping docstrings by @pankajastro in #2695
- Remove a duplicate
node_conversion_by_task_groupentry in render config docs by @pankajkoti in #2770 - Capitalize Docker as a product name in prose by @pankajkoti in #2623
- Normalize heading underlines across docs by @pankajkoti in #2641, #2655, #2656, and #2663
- Add language hints to docs code-block directives by @pankajastro in #2686
- Add the Cosmos 1.14.2 row to the watcher-retry behavior history by @tatiana in #2742
Others
- Reduce integration test CI time from ~30 min to ~16 min by @pankajkoti in #2562
- Speed up Airflow 3.1+ integration tests by caching
InProcessExecutionAPIby @pankajkoti in #2547 - Carve dbt-loom tests into a dedicated CI job by @pankajastro in #2761
- Re-enable the cross-project dbt ls example DAG in the dbt-loom CI job by @pankajkoti in #2775
- Move test jobs that need no secrets to an unprivileged
pull_requestworkflow by @pankajkoti in #2791 - Remove the Authorize gate from CI jobs that don't use secrets by @pankajkoti in #2567
- Avoid running tests when PRs from forks only change docs by @tatiana in #2583
- Forward extra args to the test-integration hatch script by @tatiana in #2682
- Add
release-*branches forpull_request_targetto run CI on release branches by @pankajkoti in #2603 - Pin pre-commit hooks and GitHub Actions to commit SHAs by @pankajkoti in #2610, #2611, #2612, and #2613
- Set
persist-credentials: falsefor checkout in the actionlint workflow by @pankajkoti in #2793 - Stop the stale bot from auto-closing PRs by @pankajkoti in #2746
- Make local
hatch run docs:buildfail on Sphinx warnings by @pankajkoti in #2639 - Skip the watcher gateway test on Airflow 3.0 by @tatiana in #2607
- Add an example DAG for seed rendering behavior by @pankajkoti in #2756
- Add integration tests for ephemeral models rendered as empty operators by @pankajkoti in #2760
- Improve coverage for
DbtRunAirflowAsyncBigqueryOperatorby @pankajastro in #2733 - Add test coverage for
DbtModel._extract_configbranches by @pankajastro in #2779 - Extract project conventions into
AGENTS.md; slimCLAUDE.mdby @tatiana in #2703 - Document the AI agent attribution convention by @tatiana in #2718
- Teach AI agents to use pre-commit before committing and pushing by @tatiana in #2702
- Instruct Claude Code to read
AGENTS.mdfor commands before running them by @pankajkoti in #2774 - Document docs build and serve commands, and the docs style guide, in
CLAUDE.mdby @pankajkoti in #2637 and #2665 - Bump black from 26.3.1 to 26.5.1 by @pankajkoti in #2727
- Bump codecov-action to v6.0.2 by @pankajkoti in #2786
- Dependency updates by @dependabot in #2541, #2555, #2571, #2578, #2579, #2582, #2588, #2589, #2609, #2621, #2627, #2651, #2674, #2707, #2721, #2724, #2725, #2730, #2745, #2783, #2789, and #2790
Milestone: Cosmos 1.15.0
v1.14.2
Behaviour Changes
These changes adjust observable behaviour of the ExecutionMode.WATCHER execution mode.
None of them break the public Cosmos API, but users relying on undocumented internals
(graph wiring assertions, XCom backup Variable names, retry-on-recovery semantics, or
retry log format) should review before upgrading.
ExecutionMode.WATCHER+depends_on_past=True: when the producer task has
depends_on_past=True(typically set viadefault_args), the producer-done gateway
task insideDbtTaskGroupis now wired downstream of every consumer task, in addition
to the producer. This is required so thatwait_for_downstreamgating behaves
correctly across DAG runs and the task group acts as a single unit that must fully
succeed before the next run starts. Users withdepends_on_past=False(the default)
see no topology change. See #2615.ExecutionMode.WATCHERdownstream retry on upstream recovery: dbt models that were
skipped after an upstream-failure event are now retried in the same DAG run when the
upstream task succeeds on retry. Previously these models remained skipped for the run.
See #2684.ExecutionMode.WATCHERconsumer-retry log format: the consumer's fallbackdbt
invocation no longer inherits the producer's internal--log-format jsonflag, so
retry task logs now default to dbt's normal text format. Users who relied on JSON output
in retry logs can opt in viaoperator_args={"dbt_cmd_flags": ["--log-format", "json"]}.
See #2713.ExecutionMode.WATCHERXCom-backup Variable key scheme: the per-model XCom backup
Variable key now includes the full task-group path and sanitises disallowed characters
(+/:) fromrun_id. External monitoring or cleanup scripts that match the
old key pattern will need updating. See #2629 and #2683.
Bug Fixes
- Sanitize disallowed characters from XCom backup variable key by @MichaelRBlack in #2629
- Prevent watcher producers from colliding on one XCom-backup key by @tatiana in #2683
- Retry watcher downstream models on upstream-failure recovery by @tatiana in #2684
- Fix
ExecutionMode.WATCHERinteraction withdepends_on_pastby @johnhoran in #2615 - Strip
--log-formatfrom producer flags on watcher consumer retry by @tatiana in #2713 - Fix duplicate
deferrablekwarg inDbtRunAirflowAsyncBigqueryOperatorby @pankajastro in #2616 - Fix dbt docs iframe
srcmissing deployment path prefix by @pankajastro in #2640 - Defer
TaskInstanceimport in cluster policy to fix Sentry init crash by @pankajastro in #2662 - Restore type hints broken by lazy imports in
cosmos/__init__.pyby @pankajastro in #2647 - Fix
ExecutionMode.WATCHERnon-dbt stdout being suppressed from logs by @pankajastro in #2654 - Fix test sensor retry behaviour in
ExecutionMode.WATCHERby @pankajkoti in #2658 - Fix watcher fallback selector for versioned dbt models by @pankajkoti in #2659
- Break out of iframe from Airflow 2 dbt Docs 404 link by @pankajastro in #2685
Docs
- Document source freshness aware execution for
ExecutionMode.WATCHERby @pankajastro in #2617 - Add reference docs for
DbtRunLocalOperator,DbtTestLocalOperator,DbtSnapshotLocalOperatorandDbtBuildLocalOperatorby @pankajastro in #2643 - Add watcher retry behaviour history documentation by @tatiana in #2600
- Add Apache Airflow® trademark on first prominent mention by @pankajkoti in #2624
- Sentence-case section headings by @pankajkoti in #2630
- Use
-for bullet points by @pankajkoti in #2631 - Drop decorative separator lines by @pankajkoti in #2632
- Normalize heading underlines in
docs/guides/anddocs/index.rstby @pankajkoti in #2664 - Fix broken cross-directory doc links by @pankajastro in #2694
- Fix broken external links in hand-written docs by @pankajastro in #2696
- Document support for Airflow 3.2 in the compatibility policy by @pankajastro in #2652
- Refresh the dbt/Airflow conflicts table to match the compatibility policy by @pankajastro in #2653
- Document incremental model limitation for
ExecutionMode.AIRFLOW_ASYNCby @pankajastro in #2642
Others
- Import
ParamValidationErrorfromairflow.sdkto silence deprecation warning by @pankajastro in #2645 - Import
DAGfromairflow.sdkto silence deprecation warning by @pankajastro in #2644 - Enforce docs style guide via pre-commit hook by @pankajkoti and @tatiana in #2633
- Add Airflow 3.2 to the test matrix in
CLAUDE.mdby @pankajastro in #2646 - Document the lazy-logging standard in
CLAUDE.mdby @pankajastro in #2679 - Extract watcher XCom-key helpers and inline single-use bindings by @pankajastro in #2673
- Remove leftover
scripts/airflow3directory by @pankajastro in #2661 - Fix
altered_jaffle_shopseed-dep CTE references by @pankajastro in #2690 - Skip Airflow 3.0 integration test stuck on
example_watcher_with_freshnessby @pankajastro in #2692 - Fix typo "constrantis" → "constraints" in tests env comment by @pankajastro in #2669
v1.14.2a4
Pre-release alpha of Cosmos 1.14.2.
Refresh of
a3with eleven additional milestone PRs (nine from @pankajastro, two from @pankajkoti). All applied cleanly on top of the a3 cherry-picks. See the release PR for cherry-pick provenance and manual conflict resolution: #2708.
Bug Fixes
- Sanitize disallowed characters from XCom backup variable key by @MichaelRBlack in #2629
- Prevent watcher producers from colliding on one XCom-backup key by @tatiana in #2683
- Retry watcher downstream models on upstream-failure recovery by @tatiana in #2684
- Fix
ExecutionMode.WATCHERinteraction withdepends_on_pastby @johnhoran in #2615 - Fix duplicate
deferrablekwarg inDbtRunAirflowAsyncBigqueryOperatorby @pankajastro in #2616 - Fix dbt docs iframe
srcmissing deployment path prefix by @pankajastro in #2640 - Defer
TaskInstanceimport in cluster policy to fix Sentry init crash by @pankajastro in #2662 - Restore type hints broken by lazy imports in
cosmos/__init__.pyby @pankajastro in #2647 - Fix
ExecutionMode.WATCHERnon-dbt stdout being suppressed from logs by @pankajastro in #2654 - Fix test sensor retry behaviour in
ExecutionMode.WATCHERby @pankajkoti in #2658 - Fix watcher fallback selector for versioned dbt models by @pankajkoti in #2659
Docs
- Document source freshness aware execution for
ExecutionMode.WATCHERby @pankajastro in #2617 - Add reference docs for
DbtRunLocalOperator,DbtTestLocalOperator,DbtSnapshotLocalOperatorandDbtBuildLocalOperatorby @pankajastro in #2643 - Add watcher retry behaviour history documentation by @tatiana in #2600
- Add Apache Airflow® trademark on first prominent mention by @pankajkoti in #2624
- Sentence-case section headings by @pankajkoti in #2630
- Use
-for bullet points by @pankajkoti in #2631 - Drop decorative separator lines by @pankajkoti in #2632
- Normalize heading underlines in
docs/guides/anddocs/index.rstby @pankajkoti in #2664 - Fix broken cross-directory doc links by @pankajastro in #2694
- Fix broken external links in hand-written docs by @pankajastro in #2696
- Document support for Airflow 3.2 in the compatibility policy by @pankajastro in #2652
- Refresh the dbt/Airflow conflicts table to match the compatibility policy by @pankajastro in #2653
Others
- Import
ParamValidationErrorfromairflow.sdkto silence deprecation warning by @pankajastro in #2645 - Import
DAGfromairflow.sdkto silence deprecation warning by @pankajastro in #2644 - Enforce docs style guide via pre-commit hook by @pankajkoti and @tatiana in #2633
- Add Airflow 3.2 to the test matrix in
CLAUDE.mdby @pankajastro in #2646 - Document the lazy-logging standard in
CLAUDE.mdby @pankajastro in #2679 - Extract watcher XCom-key helpers and inline single-use bindings by @pankajastro in #2673
- Remove leftover
scripts/airflow3directory by @pankajastro in #2661 - Fix
altered_jaffle_shopseed-dep CTE references by @pankajastro in #2690 - Skip Airflow 3.0 integration test stuck on
example_watcher_with_freshnessby @pankajastro in #2692 - Fix typo "constrantis" → "constraints" in tests env comment by @pankajastro in #2669
Milestone: Cosmos 1.14.2
Full changelog: astronomer-cosmos-v1.14.2a3...astronomer-cosmos-v1.14.2a4
v1.14.2a3
Pre-release alpha of Cosmos 1.14.2.
First alpha cut from
release-1.14with the full milestone applied (previous alphasa1/a2were point fixes off feature branches). See the release PR for cherry-pick provenance and manual conflict resolution: #2708.
Bug Fixes
- Sanitize disallowed characters from XCom backup variable key by @MichaelRBlack in #2629
- Prevent watcher producers from colliding on one XCom-backup key by @tatiana in #2683
- Retry watcher downstream models on upstream-failure recovery by @tatiana in #2684
- Fix
ExecutionMode.WATCHERinteraction withdepends_on_pastby @johnhoran in #2615 - Fix duplicate
deferrablekwarg inDbtRunAirflowAsyncBigqueryOperatorby @pankajastro in #2616 - Fix dbt docs iframe
srcmissing deployment path prefix by @pankajastro in #2640 - Defer
TaskInstanceimport in cluster policy to fix Sentry init crash by @pankajastro in #2662 - Restore type hints broken by lazy imports in
cosmos/__init__.pyby @pankajastro in #2647 - Fix
ExecutionMode.WATCHERnon-dbt stdout being suppressed from logs by @pankajastro in #2654
Docs
- Document source freshness aware execution for
ExecutionMode.WATCHERby @pankajastro in #2617 - Add reference docs for
DbtRunLocalOperator,DbtTestLocalOperator,DbtSnapshotLocalOperatorandDbtBuildLocalOperatorby @pankajastro in #2643 - Add watcher retry behaviour history documentation by @tatiana in #2600
- Add Apache Airflow® trademark on first prominent mention by @pankajkoti in #2624
- Sentence-case section headings by @pankajkoti in #2630
- Use
-for bullet points by @pankajkoti in #2631 - Drop decorative separator lines by @pankajkoti in #2632
- Normalize heading underlines in
docs/guides/anddocs/index.rstby @pankajkoti in #2664 - Fix broken cross-directory doc links by @pankajastro in #2694
- Fix broken external links in hand-written docs by @pankajastro in #2696
Others
- Import
ParamValidationErrorfromairflow.sdkto silence deprecation warning by @pankajastro in #2645 - Import
DAGfromairflow.sdkto silence deprecation warning by @pankajastro in #2644 - Enforce docs style guide via pre-commit hook by @pankajkoti and @tatiana in #2633
Milestone: Cosmos 1.14.2
Full changelog: astronomer-cosmos-v1.14.1...astronomer-cosmos-v1.14.2a3
v1.14.2a2
v1.14.2a1
v1.14.1
Bug Fixes
- Fix
ExecutionMode.WATCHERproducer retry behaviour by @tatiana in #2559 - Prevent watcher producer skip propagating to downstream tasks via gateway task by @johnhoran and @tatiana in #2597
- Keep watcher sensor polling when producer is still running by @pankajkoti in #2592
- Fix circular import error in Cosmos plugin discovery under Astro Runtime by @tatiana in #2538
- Fix
CosmosRichLoggercrash onNonelog message by @tatiana in #2540 - Enable inlets and outlets using dbt Fusion on Airflow 3 by @ichirotakami in #2561
- Fix incorrectly skipped source downstream tasks in
ExecutionMode.WATCHERby @pankajastro in #2563 - Fix duplicate logs in
dbt buildwhen source freshness is enabled by @pankajastro in #2564 - Warn and normalize when
source_rendering_behavior=Noneis passed by @pankajastro in #2570 - Gracefully handle
Variable.set()failures on Astro Remote Execution by @hkc-8010 in #2573 - Skip malformed YAML selectors instead of failing entirely by @YourRoyalLinus in #2577
Docs
- Update watcher test behavior docs for Cosmos 1.14.0 by @tatiana in #2549
- Add redirect for moved partial-parsing docs page by @tatiana in #2550
- Document
ExecutionMode.WATCHERanddepends_on_pastlimitation by @tatiana in #2602 - Restore memory-optimised imports docs for Cosmos < 1.14.0 by @pankajkoti in #2604
Others
- Speed up Airflow 3.1+ integration tests by caching InProcessExecutionAPI by @pankajkoti in #2547
- Improve stability of cache hash unit tests by @tatiana in #2539
- Fix mypy 1.20.0 type check failures by @pankajkoti in #2546
- Fix CI failures caused by docs build memory exhaustion by @pankajkoti in #2580
- Fix dbt Fusion broken integration tests by @tatiana in #2581
- Fix flaky
cosmos_manifest_selectors_exampleDAG in CI by @pankajkoti in #2593 - Reduce pre-commit autoupdate frequency PRs by @tatiana in #2544
- Bump
reviewdog/action-actionlintfrom 1.71.0 to 1.72.0 by @dependabot in #2542 - Skip watcher gateway test on Airflow 3.0 by @tatiana in #2607