feat(duckdb): Handle WEEK, WEEK(day), ISOWEEK transpilation for DATE_DIFF & other Date/Time functions #6354

fivetran-amrutabhimsenayachit · 2025-11-19T04:13:19Z

Issue:
BigQuery's DATE_DIFF with WEEK uses Sunday-start (US convention), while ISOWEEK uses Monday-start (ISO standard). DuckDB only supports 'week' with Monday-start semantics. Direct transpilation produces incorrect results because they count different boundaries.

Before :

Bigquery:
bq --project_id fivetran-wild-west query --use_legacy_sql=false "SELECT DATE_DIFF(DATE '2017-12-18', '2017-12-17', WEEK) AS week_diff, DATE_DIFF(DATE '2017-12-18', '2017-12-17', WEEK(MONDAY)) AS week_weekday_diff, DATE_DIFF(DATE '2017-12-18', '2017-12-17', ISOWEEK) AS isoweek_diff"
+-----------+-------------------+--------------+
| week_diff | week_weekday_diff | isoweek_diff |
+-----------+-------------------+--------------+
|         0 |                 1 |            1 |
+-----------+-------------------+--------------+

Transpilation:
python3 -c "import sqlglot; print(sqlglot.transpile(\"SELECT DATE_DIFF(DATE '2017-12-18', '2017-12-17', WEEK) AS week_diff, DATE_DIFF(DATE '2017-12-18', '2017-12-17', WEEK(MONDAY)) AS week_weekday_diff, DATE_DIFF(DATE '2017-12-18', '2017-12-17', ISOWEEK) AS isoweek_diff\", read='bigquery', write='duckdb')[0])"
-->
SELECT DATE_DIFF('WEEK', CAST('2017-12-17' AS DATE), CAST('2017-12-18' AS DATE)) AS week_diff, DATE_DIFF(WEEK(MONDAY), CAST('2017-12-17' AS DATE), CAST('2017-12-18' AS DATE)) AS week_weekday_diff, DATE_DIFF('ISOWEEK', CAST('2017-12-17' AS DATE), CAST('2017-12-18' AS DATE)) AS isoweek_diff


Duckdb:
duckdb -c "SELECT DATE_DIFF('WEEK', CAST('2017-12-17' AS DATE), CAST('2017-12-18' AS DATE)) AS week_diff, DATE_DIFF(WEEK(MONDAY), CAST('2017-12-17' AS DATE), CAST('2017-12-18' AS DATE)) AS week_weekday_diff, DATE_DIFF('ISOWEEK', CAST('2017-12-17' AS DATE), CAST('2017-12-18' AS DATE)) AS isoweek_diff"
Binder Error:
Referenced column "MONDAY" not found in FROM clause!

LINE 1: ...), CAST('2017-12-18' AS DATE)) AS week_diff, DATE_DIFF(WEEK(MONDAY), CAST('2017-12-17' AS DATE), CAST('2017-12-18' AS...

After:

Bigquery:
bq --project_id fivetran-wild-west query --use_legacy_sql=false "SELECT DATE_DIFF(DATE '2017-12-18', '2017-12-17', WEEK) AS week_diff, DATE_DIFF(DATE '2017-12-18', '2017-12-17', WEEK(MONDAY)) AS week_weekday_diff, DATE_DIFF(DATE '2017-12-18', '2017-12-17', ISOWEEK) AS isoweek_diff"
+-----------+-------------------+--------------+
| week_diff | week_weekday_diff | isoweek_diff |
+-----------+-------------------+--------------+
|         0 |                 1 |            1 |
+-----------+-------------------+--------------+

Transpilation:
sqlglot % python3 -c "import sqlglot; print(sqlglot.transpile(\"SELECT DATE_DIFF(DATE '2017-12-18', '2017-12-17', WEEK) AS week_diff, DATE_DIFF(DATE '2017-12-18', '2017-12-17', WEEK(MONDAY)) AS week_weekday_diff, DATE_DIFF(DATE '2017-12-18', '2017-12-17', ISOWEEK) AS isoweek_diff\", read='bigquery', write='duckdb')[0])"
-->
SELECT DATE_DIFF('DAY', DATE_TRUNC('week', CAST('2017-12-17' AS DATE) + INTERVAL '1' DAY), DATE_TRUNC('week', CAST('2017-12-18' AS DATE) + INTERVAL '1' DAY)) // 7 AS week_diff, DATE_DIFF('DAY', DATE_TRUNC('week', CAST('2017-12-17' AS DATE)), DATE_TRUNC('week', CAST('2017-12-18' AS DATE))) // 7 AS week_weekday_diff, DATE_DIFF('DAY', DATE_TRUNC('week', CAST('2017-12-17' AS DATE)), DATE_TRUNC('week', CAST('2017-12-18' AS DATE))) // 7 AS isoweek_diff

DuckdB:
sqlglot % duckdb -c "SELECT DATE_DIFF('DAY', DATE_TRUNC('week', CAST('2017-12-17' AS DATE) + INTERVAL '1' DAY), DATE_TRUNC('week', CAST('2017-12-18' AS DATE) + INTERVAL '1' DAY)) // 7 AS week_diff, DATE_DIFF('DAY', DATE_TRUNC('week', CAST('2017-12-17' AS DATE)), DATE_TRUNC('week', CAST('2017-12-18' AS DATE))) // 7 AS week_weekday_diff, DATE_DIFF('DAY', DATE_TRUNC('week', CAST('2017-12-17' AS DATE)), DATE_TRUNC('week', CAST('2017-12-18' AS DATE))) // 7 AS isoweek_diff"
┌───────────┬───────────────────┬──────────────┐
│ week_diff │ week_weekday_diff │ isoweek_diff │
│   int64   │       int64       │    int64     │
├───────────┼───────────────────┼──────────────┤
│     0     │         1         │      1       │
└───────────┴───────────────────┴──────────────┘

Logic:

Apply Normalization:

WeekStart(SUNDAY) → "WEEK"
WeekStart(MONDAY) → "ISOWEEK"
WeekStart(other) → "WEEK(day)"

Convert canonical WeekStart to DuckDB's DATE_TRUNC approach

Calculate the shift

Sunday (day 0):    shift = 1 - 0 = +1   (shift forward 1 day)
Monday (day 1):    shift = 1 - 1 = 0    (no shift needed!)
Tuesday (day 2):   shift = 1 - 2 = -1   (shift backward 1 day)
Wednesday (day 3): shift = 1 - 3 = -2   (shift backward 2 days)
Thursday (day 4):  shift = 1 - 4 = -3   (shift backward 3 days)
Friday (day 5):    shift = 1 - 5 = -4   (shift backward 4 days)
Saturday (day 6):  shift = 1 - 6 = -5   (shift backward 5 days)

Shift the dates

Eg: Jan 6 - 5 days = Jan 1 (Monday)
      Jan 13 - 5 days = Jan 8 (Monday)

Apply DATE_TRUNC

DATE_TRUNC('week', Jan 1) = Jan 1 (Monday)
DATE_TRUNC('week', Jan 8) = Jan 8 (Monday)

Calculate DATE_DIFF

DATE_DIFF('day', Jan 1, Jan 8) = 7 days
7 / 7 = 1 week

Edge Case:

` bq query --nouse_legacy_sql "SELECT DATE_DIFF(DATE '2024-01-01', DATE '2024-01-15', WEEK(SUNDAY)) as Weeks"                                                       
+-------+
| Weeks |
+-------+
|    -2 |
+-------+
sqlglot % python -c "import sqlglot; print(sqlglot.transpile(\"SELECT DATE_DIFF(DATE '2024-01-01', DATE '2024-01-15', WEEK(SUNDAY)) as Weeks\", read='bigquery', write='duckdb')[0])"

SELECT DATE_DIFF('DAY', DATE_TRUNC('WEEK', CAST('2024-01-15' AS DATE) + INTERVAL '1' DAY), DATE_TRUNC('WEEK', CAST('2024-01-01' AS DATE) + INTERVAL '1' DAY)) // 7 AS Weeks
 sqlglot % duckdb -c "SELECT DATE_DIFF('DAY', DATE_TRUNC('WEEK', CAST('2024-01-15' AS DATE) + INTERVAL '1' DAY), DATE_TRUNC('WEEK', CAST('2024-01-01' AS DATE) + INTERVAL '1' DAY)) // 7 AS Weeks"
┌───────┐
│ Weeks │
│ int64 │
├───────┤
│  -2   │
└───────┘`

sqlglot/dialects/bigquery.py

toriwei · 2025-11-20T22:46:18Z

tests/dialects/test_bigquery.py

+        self.validate_all(
+            "DATE_DIFF('2024-01-15', '2024-01-08', WEEK(MONDAY))",
+            write={
+                "bigquery": "DATE_DIFF('2024-01-15', '2024-01-08', ISOWEEK)",


is it expected that the BigQuery sql gets changed roundtrip WEEK(MONDAY) --> ISOWEEK? can/should it remain the same?

We can preserve WEEK(MONDAY) on bigquery side.

georgesittas · 2025-11-25T12:55:37Z

tests/dialects/test_duckdb.py

                # All of these should remain as is, they don't have synonyms
                self.validate_identity(f"EXTRACT({part} FROM foo)")

+    def test_date_diff_week(self):


What "interesting" path do these tests cover? It doesn't feel very relevant / useful at first glance.

I agree, it just covers basic date_diff function tests. Since we are focussing on transpilation, I can remove these tests as there are validate_all tests in bigquery already.

tobymao · 2025-11-25T17:24:08Z

sqlglot/dialects/dialect.py

+        "THURSDAY": 4,
+        "FRIDAY": 5,
+        "SATURDAY": 6,
+        "SUNDAY": 0,


can you move this to the top of the list?

i'm a bit confused, if monday is 1 and sunday is 0, why is sunday last?

Yeah, it makes sense to move sunday at the top.

tobymao · 2025-11-25T17:29:41Z

sqlglot/dialects/dialect.py

        "CENTURIES": "CENTURY",
    }

+    WEEK_UNIT_SEMANTICS = {


what do these numbers 0 and 1 mean?

These are mapping of week unit names to (start_day, dow_number) tuples, where dow_number is the numeric day-of-week value (0=Sunday, 1=Monday, etc.)

tobymao · 2025-11-25T17:31:24Z

sqlglot/dialects/dialect.py

+        unit_name = unit.this.upper() if isinstance(unit.this, str) else str(unit.this)
+        # Only handle ISOWEEK/WEEKISO variants, not plain WEEK
+        if unit_name in ("ISOWEEK", "WEEKISO"):
+            week_info = Dialect.WEEK_UNIT_SEMANTICS.get(unit_name)


why are you calling the constant here, it can just be WEEK_UNIT_SEMANTICS, or if you want it overriadable, you need to call it on the instance of dialect

tobymao · 2025-11-25T17:32:08Z

sqlglot/dialects/dialect.py

+                    If False, return just day_name string for serialization
+
+    Returns:
+        - If include_dow=False: day_name (e.g., "SUNDAY")


why not just always return the day of the week? doesn't it make it simpler?

tobymao · 2025-11-25T17:33:05Z

sqlglot/dialects/duckdb.py

+    Handles:
+    - Var('WEEK') -> ('SUNDAY', 0)  # BigQuery default
+    - Var('ISOWEEK') -> ('MONDAY', 1)
+    - Week(Var('day')) -> ('day', dow)


what happens if the day is in a column and not a constant string value?

When week start day is dynamic (from a column), week_start will be None and we fall back to standard DATE_DIFF, since compile-time offsets would be required.

tobymao · 2025-11-25T17:33:39Z

sqlglot/dialects/duckdb.py

    return arg


+def _extract_week_start_day(unit: t.Optional[exp.Expression]) -> t.Optional[t.Tuple[str, int]]:


what is the point of this function? it doesn't seem necessary

Yeah, makes sense, will remove the redundant wrapper and call extract_week_unit_info() directly.

tobymao · 2025-11-25T17:33:56Z

sqlglot/dialects/duckdb.py

+    """
+    if start_dow == 1:
+        # No shift needed for Monday-based weeks (ISO standard)
+        return exp.Anonymous(this="DATE_TRUNC", expressions=[exp.Literal.string("week"), date_expr])


why are you creating an anonymous function?

I ll replace it with DATE_TRUNC

…DIFF function

fivetran-amrutabhimsenayachit requested review from fivetran-BradfordPaskewitz and georgesittas November 19, 2025 04:13

fivetran-amrutabhimsenayachit self-assigned this Nov 19, 2025

georgesittas reviewed Nov 19, 2025

View reviewed changes

sqlglot/dialects/bigquery.py Outdated Show resolved Hide resolved

fivetran-amrutabhimsenayachit force-pushed the RD-1050264-transpile-big-querys-date-diff-date-function-to-duck-db branch from 9996310 to bd2097a Compare November 20, 2025 18:08

fivetran-amrutabhimsenayachit requested a review from georgesittas November 20, 2025 18:12

fivetran-amrutabhimsenayachit changed the title ~~feat(duckdb): Handle WEEK, WEEK(day), ISOWEEK transpilation for DATE_DIFF function~~ feat(duckdb): Handle WEEK, WEEK(day), ISOWEEK transpilation for DATE_DIFF & other Date/Time functions Nov 20, 2025

toriwei reviewed Nov 20, 2025

View reviewed changes

georgesittas reviewed Nov 25, 2025

View reviewed changes

tobymao reviewed Nov 25, 2025

View reviewed changes

fivetran-amrutabhimsenayachit added 6 commits November 26, 2025 12:06

feat(duckdb): Handle WEEK, WEEK(day), ISOWEEK transpilation for DATE_…

ffcb075

…DIFF function

fix(duckdb): Added generic implementation

0e173c5

feat(duckdb): Preserve WEEK(MONDAY) got bigquery

1c1bf3f

fix: Applied formatting

76386d2

feat(duckdb): Addressed review comments

dba3bb8

fix: updated tests

592107b

fivetran-amrutabhimsenayachit force-pushed the RD-1050264-transpile-big-querys-date-diff-date-function-to-duck-db branch from ea1b9b4 to 592107b Compare November 26, 2025 19:05

fivetran-amrutabhimsenayachit requested a review from georgesittas November 26, 2025 19:11

		return arg


		def _extract_week_start_day(unit: t.Optional[exp.Expression]) -> t.Optional[t.Tuple[str, int]]:

feat(duckdb): Handle WEEK, WEEK(day), ISOWEEK transpilation for DATE_DIFF & other Date/Time functions #6354

Are you sure you want to change the base?

feat(duckdb): Handle WEEK, WEEK(day), ISOWEEK transpilation for DATE_DIFF & other Date/Time functions #6354

Conversation

fivetran-amrutabhimsenayachit commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tobymao Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fivetran-amrutabhimsenayachit commented Nov 19, 2025 •

edited

Loading

tobymao Nov 25, 2025 •

edited

Loading