Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions macros/create_udfs.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{% macro create_udfs() %}

create schema if not exists {{target.schema}};

{{ create_udf_business_days() }};
{{ create_udf_business_hours() }};

{% endmacro %}
30 changes: 29 additions & 1 deletion macros/macros.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,4 +123,32 @@ This macro only seeks to add or update the tags which are specified in dbt. It w
If you need this behaviour, it usually comes naturally as dbt drops and recreates tables/views for most materializations.
If you are using the incremental materialization, be aware of this limitation.

{% enddocs %}
{% enddocs %}

{% docs create_udfs %}

Create_udfs is a macro that does exactly as it says, calls the individiual
UDFs that are nested in macros/udfs and makes them available to use in
your dbt transformations within Snowflake.

In order to facilitate the use of UDFs within your project, you must add
the following to your dbt_project.yml file;

on-run-start:
- '{{ create_udfs() }}'

Users must have the proper permissions to create UDFs, orchestrated
through the following command as an owner/administrator:
GRANT USAGE ON LANGUAGE PLPYTHONU TO <user>

Why/motivation:
While many complex transformations are possible with just SQL, creating
UDFs allows a project to begin encorporating python packages
to make certain transformations incredibly easy and efficient.
Additionally, leveraging UDFs via dbt macros allows for:
- the ability to version control,
- the ability to read transformations from your project without needing
to interact with Snowflake
- to maintain separate dev/prod versions of the UDFs.

{% enddocs %}
5 changes: 4 additions & 1 deletion macros/macros.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,7 @@ macros:
description: '{{ doc("apply_meta_as_tags") }}'
arguments:
- name: results
description: The on-run-end context object
description: The on-run-end context object

- name: create_udfs
description: '{{ doc("create_udfs") }}'
28 changes: 28 additions & 0 deletions macros/udfs/business_days.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{% macro create_udf_business_days() %}

create or replace function {{target.schema}}.udf_business_days(date1 TIMESTAMP_NTZ, date2 TIMESTAMP_NTZ, country STRING)
returns int
language python
runtime_version = 3.8
packages = ('numpy','holidays')
handler = 'business_days_py'
as

$$
import numpy as np
import holidays as holidays

def business_days_py(start_date, end_date, country='US'):
Comment thread
cris-seaton marked this conversation as resolved.
Outdated

# calculate business days between two dates
start_date = start_date.date()
end_date = end_date.date()
years = [*range(1990,2030)]

US_holiday_list = list(holidays.(years=years))
Comment thread
cris-seaton marked this conversation as resolved.
Outdated

return np.busday_count(start_date, end_date, holidays=US_holiday_list)

$$

{% endmacro %}
27 changes: 27 additions & 0 deletions macros/udfs/business_days.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
version: 2

macros:
- name: create_udf_business_days
description: >
This macro creates a UDF within the target schema that can be used to
calculate business days (excluding US holidays) between two date fields
Comment thread
cris-seaton marked this conversation as resolved.
Outdated
in a Snowflake table. The macro mearly creates the UDF within Snowflake,
and is called by the create_udfs macro. You must ensure that create_udfs
is called on-run-start in dbt_project.yml.

To utilize the UDF within your dbt project, proper syntax is:

select
*,
udf_business_days(start_date, end_date) as business_days_elapsed
from sample_table

Notes:
- The fields passed MUST be type == date.
- This UDF only considers holidays between Jan 1 1990 and Dec 31 2030.
- This UDF hard codes holiday date range (above) and holiday date country
(as US). Modify the macro to apply to different countries, define
regional (state or province) holidays or expand the date range.

docs:
show: false
51 changes: 51 additions & 0 deletions macros/udfs/business_hours.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
{% macro create_udf_business_hours() %}

create or replace function {{target.schema}}.udf_business_hours(datetime1 TIMESTAMP_NTZ, datetime2 TIMESTAMP_NTZ)
returns int
Comment thread
cris-seaton marked this conversation as resolved.
Outdated
language python
runtime_version = 3.8
packages = ('numpy','holidays')
handler = 'business_hours_py'
as

$$
from datetime import datetime
from datetime import timedelta
import holidays
import numpy as np

def business_hours_py(datetime1, datetime2):
Comment thread
cris-seaton marked this conversation as resolved.
Outdated

years = [*range(2000,2030)]

US_holiday_list = list(holidays.US(years=years))
days = np.busday_count(
datetime1.date(),
datetime2.date(),
holidays = US_holiday_list)
complete_days_hours = (days-1) * 8
Comment thread
cris-seaton marked this conversation as resolved.
Outdated

# Jump forward to the next non-holiday
while datetime1 in US_holiday_list or datetime1.weekday() in [5,6]:
datetime1 = datetime1 + timedelta(days=1)
datetime1 = datetime1.replace(hour=9,minute=0,second=0)
Comment thread
cris-seaton marked this conversation as resolved.
Outdated

# Jump back to the last non-holiday
while datetime2 in US_holiday_list or datetime2.weekday() in [5,6]:
datetime2 = datetime2 - timedelta(days=1)
datetime2 = datetime2.replace(hour=9,minute=0,second=0)
Comment thread
cris-seaton marked this conversation as resolved.
Outdated

# Hard coded opening and closing times
biz_opened = datetime1.replace(hour=9, minute=0, second=0)
biz_closed = datetime1.replace(hour=17, minute=0, second=0)

if datetime1.date() == datetime2.date():
return round((datetime2 - datetime1).seconds/60/60,2)
else:
duration_day_zero = round((biz_closed - datetime1 ).seconds/60/60, 2)
duration_day_n = round((datetime2 - biz_opened).seconds/60/60, 2)
return round(duration_day_zero + duration_day_n + complete_days_hours, 2)

$$

{% endmacro %}