Skip to content

Releases: astronomer/astronomer-cosmos

v1.15.0a1

12 Jun 14:56

Choose a tag to compare

v1.15.0a1 Pre-release
Pre-release

Features

  • Add ExecutionMode.WATCHER_KUBERNETES mode with test handling by @vricciardulli in #2529 and #2543
  • Support dbt docs on Kubernetes via DbtDocsS3KubernetesOperator by @jx2lee in #2058
  • Add use_tarball option to the upload_to_gcp_gs callback by @TTMichaelA in #2497
  • Add AWS and Azure tarball upload options by @TTMichaelA in #2553
  • Expose freshness_callback in ExecutionConfig.setup_operator_args for ExecutionMode.WATCHER by @pankajastro in #2586
  • Allow templating dbt deps installation via ExecutionConfig by @pankajastro in #2728
  • Add configurable seed rendering behavior by @pankajkoti in #2755
  • Render ephemeral dbt models as EmptyOperator by default by @pankajkoti in #2759
  • Support per-node callbacks via string import paths by @highlyavailable in #2781
  • Map Snowflake query_tag from connection extras to dbt profile by @pankajastro in #2650
  • Add Snowflake profile mapping for non-encrypted private key file by @pankajastro in #2660
  • Export Clickhouse and encrypted Snowflake mappings from cosmos.profiles by @pankajastro in #2668
  • Forward Snowflake insecure_mode from connection Extra into dbt profile by @toor11 in #2744
  • Extend Snowflake profile mappings with authenticator, client_session_keep_alive, host, and port by @Aaditya-git in #2748
  • Support extra__google_cloud_platform__project in GoogleCloudServiceAccountDictProfileMapping by @jroachgolf84 in #2626
  • Enable the orjson parser for the whole project (experimental) by @corsettigyg in #2552
  • Warn when users pass output-only template fields to local operators by @goingforstudying-ctrl in #2737

Enhancements

  • Replace openlineage-airflow with apache-airflow-providers-openlineage by @tatiana in #2753
  • Refactor handling of watcher producer-only dbt flags by @tatiana in #2717
  • Disclose the --no-static-parser flag in ExecutionMode.LOCAL by @pankajkoti in #2714
  • Refactor _default_freshness_callback to return a list of (id, state) tuples by @pankajastro in #2572
  • Restore dynamically resolved user-provided node types by @jroachgolf84 in #2608
  • Promote inline watcher status collections to named constants by @pankajastro in #2671
  • Align WATCHER_KUBERNETES producer default task_id with PRODUCER_WATCHER_TASK_ID by @tatiana in #2794
  • Centralize version-aware EmptyOperator imports in a compatibility module by @pankajkoti in #2758
  • Move safe_copy out of cosmos.cache into cosmos.fs by @pankajkoti in #2715
  • Refactor seed checksum hashing and caching helpers by @pankajkoti in #2784
  • Avoid O(n^2) string accumulation in store_compiled_sql by @pankajastro in #2751
  • Use GA versions for propagate_logs in provider_info by @pankajastro in #2712
  • Use cosmos.log.get_logger consistently by @pankajastro in #2670
  • Use lazy logging instead of f-strings in logger calls by @pankajastro in #2672 and #2680
  • Ban eager module-level imports of optional providers via ruff by @tatiana in #2743
  • Remove unused # type: ignore comments and re-enable warn_unused_ignores by @pankajastro in #2710 and #2763
  • Drop the Python <3.9 fallback for functools.cache by @pankajastro in #2732
  • Remove stale Airflow 2.3 comment on registry_conn_id default by @pankajastro in #2766
  • Remove an outdated TODO on watcher terminal-status handling by @pankajastro in #2697
  • Unify operator logging on self.log by @pankajastro in #2681
  • Centralise resource_name extraction from dbt unique_id by @pankajkoti in #2687
  • Tighten # type: ignore comments to specify error codes by @pankajastro in #2711

Bug Fixes

  • Fix example_cosmos_sources to use a dbt project with sources by @jroachgolf84 in #2614
  • Unify Postgres profiles.yml across dbt example projects by @yeoreums in #2501
  • Pin dbt-core<2.0 in virtualenv example DAGs to avoid alpha pickup by @pankajastro in #2752

Docs

Others

Milestone: Cosmos 1.15.0

v1.14.2

21 May 12:20

Choose a tag to compare

Behaviour Changes

These changes adjust observable behaviour of the ExecutionMode.WATCHER execution mode.
None of them break the public Cosmos API, but users relying on undocumented internals
(graph wiring assertions, XCom backup Variable names, retry-on-recovery semantics, or
retry log format) should review before upgrading.

  • ExecutionMode.WATCHER + depends_on_past=True: when the producer task has
    depends_on_past=True (typically set via default_args), the producer-done gateway
    task inside DbtTaskGroup is now wired downstream of every consumer task, in addition
    to the producer. This is required so that wait_for_downstream gating behaves
    correctly across DAG runs and the task group acts as a single unit that must fully
    succeed before the next run starts. Users with depends_on_past=False (the default)
    see no topology change. See #2615.
  • ExecutionMode.WATCHER downstream retry on upstream recovery: dbt models that were
    skipped after an upstream-failure event are now retried in the same DAG run when the
    upstream task succeeds on retry. Previously these models remained skipped for the run.
    See #2684.
  • ExecutionMode.WATCHER consumer-retry log format: the consumer's fallback dbt
    invocation no longer inherits the producer's internal --log-format json flag, so
    retry task logs now default to dbt's normal text format. Users who relied on JSON output
    in retry logs can opt in via operator_args={"dbt_cmd_flags": ["--log-format", "json"]}.
    See #2713.
  • ExecutionMode.WATCHER XCom-backup Variable key scheme: the per-model XCom backup
    Variable key now includes the full task-group path and sanitises disallowed characters
    (+ / :) from run_id. External monitoring or cleanup scripts that match the
    old key pattern will need updating. See #2629 and #2683.

Bug Fixes

  • Sanitize disallowed characters from XCom backup variable key by @MichaelRBlack in #2629
  • Prevent watcher producers from colliding on one XCom-backup key by @tatiana in #2683
  • Retry watcher downstream models on upstream-failure recovery by @tatiana in #2684
  • Fix ExecutionMode.WATCHER interaction with depends_on_past by @johnhoran in #2615
  • Strip --log-format from producer flags on watcher consumer retry by @tatiana in #2713
  • Fix duplicate deferrable kwarg in DbtRunAirflowAsyncBigqueryOperator by @pankajastro in #2616
  • Fix dbt docs iframe src missing deployment path prefix by @pankajastro in #2640
  • Defer TaskInstance import in cluster policy to fix Sentry init crash by @pankajastro in #2662
  • Restore type hints broken by lazy imports in cosmos/__init__.py by @pankajastro in #2647
  • Fix ExecutionMode.WATCHER non-dbt stdout being suppressed from logs by @pankajastro in #2654
  • Fix test sensor retry behaviour in ExecutionMode.WATCHER by @pankajkoti in #2658
  • Fix watcher fallback selector for versioned dbt models by @pankajkoti in #2659
  • Break out of iframe from Airflow 2 dbt Docs 404 link by @pankajastro in #2685

Docs

Others

v1.14.2a4

20 May 15:05

Choose a tag to compare

v1.14.2a4 Pre-release
Pre-release

Pre-release alpha of Cosmos 1.14.2.

Refresh of a3 with eleven additional milestone PRs (nine from @pankajastro, two from @pankajkoti). All applied cleanly on top of the a3 cherry-picks. See the release PR for cherry-pick provenance and manual conflict resolution: #2708.

Bug Fixes

  • Sanitize disallowed characters from XCom backup variable key by @MichaelRBlack in #2629
  • Prevent watcher producers from colliding on one XCom-backup key by @tatiana in #2683
  • Retry watcher downstream models on upstream-failure recovery by @tatiana in #2684
  • Fix ExecutionMode.WATCHER interaction with depends_on_past by @johnhoran in #2615
  • Fix duplicate deferrable kwarg in DbtRunAirflowAsyncBigqueryOperator by @pankajastro in #2616
  • Fix dbt docs iframe src missing deployment path prefix by @pankajastro in #2640
  • Defer TaskInstance import in cluster policy to fix Sentry init crash by @pankajastro in #2662
  • Restore type hints broken by lazy imports in cosmos/__init__.py by @pankajastro in #2647
  • Fix ExecutionMode.WATCHER non-dbt stdout being suppressed from logs by @pankajastro in #2654
  • Fix test sensor retry behaviour in ExecutionMode.WATCHER by @pankajkoti in #2658
  • Fix watcher fallback selector for versioned dbt models by @pankajkoti in #2659

Docs

Others

Milestone: Cosmos 1.14.2

Full changelog: astronomer-cosmos-v1.14.2a3...astronomer-cosmos-v1.14.2a4

v1.14.2a3

20 May 06:45

Choose a tag to compare

v1.14.2a3 Pre-release
Pre-release

Pre-release alpha of Cosmos 1.14.2.

First alpha cut from release-1.14 with the full milestone applied (previous alphas a1 / a2 were point fixes off feature branches). See the release PR for cherry-pick provenance and manual conflict resolution: #2708.

Bug Fixes

  • Sanitize disallowed characters from XCom backup variable key by @MichaelRBlack in #2629
  • Prevent watcher producers from colliding on one XCom-backup key by @tatiana in #2683
  • Retry watcher downstream models on upstream-failure recovery by @tatiana in #2684
  • Fix ExecutionMode.WATCHER interaction with depends_on_past by @johnhoran in #2615
  • Fix duplicate deferrable kwarg in DbtRunAirflowAsyncBigqueryOperator by @pankajastro in #2616
  • Fix dbt docs iframe src missing deployment path prefix by @pankajastro in #2640
  • Defer TaskInstance import in cluster policy to fix Sentry init crash by @pankajastro in #2662
  • Restore type hints broken by lazy imports in cosmos/__init__.py by @pankajastro in #2647
  • Fix ExecutionMode.WATCHER non-dbt stdout being suppressed from logs by @pankajastro in #2654

Docs

Others

Milestone: Cosmos 1.14.2

Full changelog: astronomer-cosmos-v1.14.1...astronomer-cosmos-v1.14.2a3

v1.14.2a2

11 May 16:48

Choose a tag to compare

v1.14.2a2 Pre-release
Pre-release

Test: Forward non-dbt stdout to task log in WATCHER mode in #2654

v1.14.2a1

06 May 14:02

Choose a tag to compare

v1.14.2a1 Pre-release
Pre-release

Test: Fix dbt docs iframe src missing deployment path prefix PR: #2640

v1.14.1

23 Apr 14:36

Choose a tag to compare

Bug Fixes

  • Fix ExecutionMode.WATCHER producer retry behaviour by @tatiana in #2559
  • Prevent watcher producer skip propagating to downstream tasks via gateway task by @johnhoran and @tatiana in #2597
  • Keep watcher sensor polling when producer is still running by @pankajkoti in #2592
  • Fix circular import error in Cosmos plugin discovery under Astro Runtime by @tatiana in #2538
  • Fix CosmosRichLogger crash on None log message by @tatiana in #2540
  • Enable inlets and outlets using dbt Fusion on Airflow 3 by @ichirotakami in #2561
  • Fix incorrectly skipped source downstream tasks in ExecutionMode.WATCHER by @pankajastro in #2563
  • Fix duplicate logs in dbt build when source freshness is enabled by @pankajastro in #2564
  • Warn and normalize when source_rendering_behavior=None is passed by @pankajastro in #2570
  • Gracefully handle Variable.set() failures on Astro Remote Execution by @hkc-8010 in #2573
  • Skip malformed YAML selectors instead of failing entirely by @YourRoyalLinus in #2577

Docs

  • Update watcher test behavior docs for Cosmos 1.14.0 by @tatiana in #2549
  • Add redirect for moved partial-parsing docs page by @tatiana in #2550
  • Document ExecutionMode.WATCHER and depends_on_past limitation by @tatiana in #2602
  • Restore memory-optimised imports docs for Cosmos < 1.14.0 by @pankajkoti in #2604

Others

v1.14.1a4

23 Apr 11:49

Choose a tag to compare

v1.14.1a4 Pre-release
Pre-release

Cut from release PR: #2601

v1.14.1a3

23 Apr 10:42

Choose a tag to compare

v1.14.1a3 Pre-release
Pre-release

Cut from release PR: #2601

v1.14.1a2

21 Apr 10:51

Choose a tag to compare

v1.14.1a2 Pre-release
Pre-release
  • Fix watcher producer retries behaviour by @tatiana in #2559
    • Fixes ExecutionMode.WATCHER_KUBERNETES as well

#2559