GH-48593: [C++] C++20: use standard calendar / timezone APIs by rok · Pull Request #48601 · apache/arrow

rok · 2025-12-19T22:57:19Z

Rationale for this change

Switch to std::chrono for MSVC to be able to use the system-provided timezone automatically on Windows.

What changes are included in this PR?

This adds chrono_internal.h that uses C++20 std::chrono timezone/calendar APIs on compilers with support (MSVC only for now) and falls back to vendored date.h otherwise.

Are these changes tested?

Partially tested locally and partially to be tested on CI.

Are there any user-facing changes?

Yes, Windows users will no longer need to install the IANA tzdb (see instructions here and here). We possibly have tzdb download set up in CI too and should update it appropriately.

GitHub Issue: [C++] C++20: use standard calendar / timezone APIs #48593

github-actions · 2025-12-19T22:57:42Z

⚠️ GitHub issue #48593 has been automatically assigned in GitHub to PR creator.

rok · 2025-12-23T18:43:17Z

It seems that std::chrono on GCC (14.3.0, 15.2.0) potentially has a bug that triggers some of our tests. Meanwhile std::chrono on MSVC 19.44 ( 14.44) appears to be pass them and is correct or at least consistent with vendored date.h. I would therefore advise we only switch to std::chrono on MSVC for now as that gives us the most benefit anyway (users no longer have to deal with the tz db).

Below is the explanation and reproduction of the bug.

// GCC libstdc++ DST bug reproduction
//
// The AN to AS Transition Bug
// ---------------------------
// Source https://github.com/eggert/tz/blob/c37fbc3249c1a1334948b38f3bca47dee5c11dd1/australasia#L165-L192
// Australia/Broken_Hill used the AN (New South Wales) rules until 2000, then
// switched to AS (South Australia) rules. Under AN rules, DST started the last
// Sunday of October and ended the last Sunday of March. For the 1999-2000
// summer, DST started October 31, 1999 and would end March 26, 2000. February
// 29, 2000 falls squarely within this DST period, so the correct offset should
// be 9:30 base + 1:00 DST = 10:30 (630 minutes).
//
// Why GCC's Data is Wrong
// -----------------------
// When libstdc++ processes the zone transition from AN rules to AS rules (which
// happens in year 2000), it appears to lose or reset the DST state inherited
// from the AN rules. Instead of recognizing that DST is still active from the
// October 1999 transition, it reports offset=570 (just the 9:30 base) with
// save=0. The inconsistency is evident: it returns abbrev="ACDT" (daylight
// time) but the offset and save values indicate standard time. The AN rules
// clearly show DST should be active until the last Sunday of March 2000.
//
// Compile: g++ -std=c++20 -o gcc_dst_bug gcc_libstdcxx_dst_bug.cpp
// Expected: 630 (10:30 = 9:30 base + 1:00 DST)
// Actual:   570 (9:30 = base only, DST missing)

#include <chrono>
#include <iostream>

int main() {
  using namespace std::chrono;
  auto* tz = locate_zone("Australia/Broken_Hill");
  auto info = tz->get_info(sys_days{2000y / February / 29d} + 23h + 23min + 23s);
  std::cout << duration_cast<minutes>(info.offset).count() << "\n";
}

rok · 2025-12-23T18:50:08Z

@pitrou

cpp/src/arrow/compute/kernels/scalar_temporal_test.cc

cpp/src/arrow/util/chrono_internal.h

pitrou · 2026-01-05T09:15:39Z

It seems that std::chrono on GCC (14.3.0, 15.2.0) potentially has a bug that triggers some of our tests.

Is the bug reported somewhere? If not, can you do that?

pitrou · 2026-01-05T09:16:12Z

We possibly have tzdb download set up in CI too and should update it appropriately.

Yes, I think we should do so. There are also a bunch of code snippets in the C++ and Python codebase that could be removed, IIRC.

rok · 2026-01-05T16:48:06Z

Is the bug reported somewhere? If not, can you do that?

It seems to be related to a known issue, I added comment explaining our case.

cpp/src/arrow/util/chrono_internal.h

diff --git c/python/pyarrow/tests/test_compute.py i/python/pyarrow/tests/test_compute.py index 3975a03..08bb6b2 100644 --- c/python/pyarrow/tests/test_compute.py +++ i/python/pyarrow/tests/test_compute.py @@ -2367,15 +2367,17 @@ def _compare_strftime_strings_on_windows(result, expected): # instead of timezone abbreviations (e.g. "CET") # apache#48767 - p = "(UTC|GMT[+-][0-9]+)$" + # Match timezone suffixes: UTC, GMT offsets (GMT+1, GMT-5), or abbreviations (CET, CEST) + p = "(UTC|GMT[+-]?[0-9]*|[A-Z]{2,5})$" - ends_with_offset = pc.match_substring_regex(result, p) - all_end_with_offset = pc.all(ends_with_offset, skip_nulls=True).as_py() - assert all_end_with_offset, "All timezone values should be GMT offset format "\ - f"or UTC \nActual: {result}" + ends_with_tz = pc.match_substring_regex(result, p) + all_end_with_tz = pc.all(ends_with_tz, skip_nulls=True).as_py() + assert all_end_with_tz, "All timezone values should be GMT offset format, "\ + f"UTC, or timezone abbreviation\nActual: {result}" result_substring = pc.replace_substring_regex(result, pattern=p, replacement="") - assert expected.starts_with(result_substring), \ + expected_substring = pc.replace_substring_regex(expected, pattern=p, replacement="") + assert result_substring.equals(expected_substring), \ f"Expected: {expected}, \nActual: {result} " \ "\nNote: tz suffix is not being compared" diff --git c/python/pyarrow/util.py i/python/pyarrow/util.py index 71ec865..123e226 100644 --- c/python/pyarrow/util.py +++ i/python/pyarrow/util.py @@ -244,6 +244,7 @@ def _download_requests(url, out_path): with open(out_path, 'wb') as f: f.write(response.content) + # TODO(apacheGH-48593): Remove when libc++ supports std::chrono timezone # apache#48593 def download_tzdata_on_windows():

Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>

rok · 2026-02-27T19:28:14Z

@github-actions crossbow submit test-r-macos-as-cran

github-actions · 2026-02-27T19:30:26Z

Revision: 9e87d57

Submitted crossbow builds: ursacomputing/crossbow @ actions-a1f6d9bce5

Task	Status
test-r-macos-as-cran

rok · 2026-02-27T19:31:46Z

I've tried a couple of eebases rebasing on commit before e37c516 succeeded. Trying rebase on 59e0ba6 now. I hope there's not too much to workaround.

…ty issues (apache#49223) Now that we have CI for it, check on other issues with C++20 compatibility on CRAN. I know that the code in apache#49105 is likely problematic Resolves: apache#49287 ### Rationale for this change ### What changes are included in this PR? ### Are these changes tested? ### Are there any user-facing changes? * GitHub Issue: apache#49287 Authored-by: Jonathan Keane <jkeane@gmail.com> Signed-off-by: Nic Crane <thisisnic@gmail.com>

rok · 2026-02-27T20:21:47Z

@github-actions crossbow submit test-r-macos-as-cran

github-actions · 2026-02-27T20:23:58Z

Revision: c6caa3c

Submitted crossbow builds: ursacomputing/crossbow @ actions-10e324e6df

Task	Status
test-r-macos-as-cran

rok · 2026-02-28T00:11:24Z

@jonkeane I rebased to the commit before #49298 and cherrypicked #49223 commit. CRAN CI is now passing. So this PR by itself is above board for CRAN I suppose?

rok · 2026-02-28T00:12:16Z

@pitrou since this won't cause issues with CRAN shall we proceed with review?

github-actions bot added Component: C++ awaiting committer review Awaiting committer review labels Dec 19, 2025

rok force-pushed the cpp20_use_chrono branch 9 times, most recently from 4283740 to d82f990 Compare December 23, 2025 18:14

rok marked this pull request as ready for review December 23, 2025 18:46

rok changed the title ~~GH-48593: [Draft][C++] C++20: use standard calendar / timezone APIs~~ GH-48593: [C++] C++20: use standard calendar / timezone APIs Dec 23, 2025

pitrou reviewed Jan 5, 2026

View reviewed changes

cpp/src/arrow/compute/kernels/scalar_temporal_test.cc Outdated Show resolved Hide resolved

cpp/src/arrow/util/chrono_internal.h Outdated Show resolved Hide resolved

cpp/src/arrow/util/chrono_internal.h Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting committer review Awaiting committer review awaiting changes Awaiting changes labels Jan 5, 2026

pitrou reviewed Jan 5, 2026

View reviewed changes

cpp/src/arrow/util/chrono_internal.h Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Jan 5, 2026

rok requested review from assignUser, jonkeane and kou as code owners January 5, 2026 17:48

rok and others added 21 commits February 27, 2026 20:27

remove appveyor references

4af3b3a

Remove more appveyor references

d650046

fix timestamp check

c34e4e1

try checking full string

1e5b734

compare whole timestamp strings on windows

14f77f1

temporarily enable problematic test

e2be635

simplify macro

7e1f050

disable rounding test on windows and rename user-agent

66965c0

lint

3793b88

don't doctest

b60b141

minor changes and tzdata to latest

deb0a2a

iana db to latest

1f2d930

exclude a problematic timestamp

d31e6d4

comment out another timestamp, remove NOLINT

01632d7

reverting nolint

9a06bca

exclude two timestamps

182e44e

fixing one-off error

7af3c22

Update python/pyarrow/tests/test_compute.py

2ec0a46

Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>

provide TZDIR for ORC

c730059

forgot import sys

9e87d57

rok force-pushed the cpp20_use_chrono branch from 863e56f to 9e87d57 Compare February 27, 2026 19:28

Conversation

rok commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Dec 19, 2025

Uh oh!

rok commented Dec 23, 2025

Uh oh!

rok commented Dec 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pitrou commented Jan 5, 2026

Uh oh!

pitrou commented Jan 5, 2026

Uh oh!

rok commented Jan 5, 2026

Uh oh!

Uh oh!

rok commented Feb 27, 2026

Uh oh!

github-actions bot commented Feb 27, 2026

Uh oh!

rok commented Feb 27, 2026

Uh oh!

rok commented Feb 27, 2026

Uh oh!

github-actions bot commented Feb 27, 2026

Uh oh!

rok commented Feb 28, 2026

Uh oh!

rok commented Feb 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

rok commented Dec 19, 2025 •

edited

Loading