GH-44421: [Python] Add configuration for building & testing free-threaded wheels on Windows by lysnikolaou · Pull Request #44804 · apache/arrow

lysnikolaou · 2024-11-21T11:25:26Z

Rationale for this change

There's no blockers anymore for building Windows wheels for the free-threaded build, so this PR adds the necessary configuration to do that.

What changes are included in this PR?

Add jobs to build free-threaded wheels on Windows
Move VCPKG-related stuff to a reusable base job

Are these changes tested?

No tests required.

Are there any user-facing changes?

No.

GitHub Issue: [Python][Packaging] Enable free-threaded wheels on Windows #44421

github-actions · 2024-11-21T11:25:53Z

⚠️ GitHub issue #44421 has been automatically assigned in GitHub to PR creator.

lysnikolaou · 2024-11-26T13:53:14Z

If someone could trigger the new jobs, that'd be really helpful!

raulcd · 2024-11-27T09:36:26Z

@github-actions crossbow submit -g wheel-windows-*

github-actions · 2024-11-27T09:37:40Z

Invalid group(s) {'wheel-windows-*'}. Must be one of {'wheel', 'cuda', 'example-cpp', 'linux', 'homebrew', 'fuzz', 'integration', 'cpp', 'example-python', 'nightly-release', 'verify-rc-source-linux', 'c-glib', 'example', 'python', 'nightly', 'conan', 'verify-rc-wheels', 'packaging', 'r', 'verify-rc-source', 'verify-rc', 'nightly-packaging', 'nightly-tests', 'vcpkg', 'java', 'linux-arm64', 'verify-rc-binaries', 'conda', 'test', 'verify-rc-source-macos', 'ruby', 'verify-rc-jars', 'linux-amd64'}
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/12047528593

raulcd · 2024-11-27T09:40:58Z

@github-actions crossbow submit wheel-windows-*

github-actions · 2024-11-27T09:43:17Z

Revision: 78d8f9e1f3649a9769046917c54f8bf6df379f6f

Submitted crossbow builds: ursacomputing/crossbow @ actions-86ba33e477

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

raulcd

Thanks for the PR, I triggered the CI jobs

ci/docker/python-wheel-windows-test-vs2019.dockerfile

lysnikolaou · 2024-11-27T15:23:12Z

Unfortunately, while the free-threaded wheel is built correctly, py -3.13t -c "import pyarrow" fails. No error shown. Does anyone know how I could inspect stderr to see why this fails?

lysnikolaou · 2024-11-27T15:36:49Z

Also, the failure on 3.9 is that py cannot be found when running py -%PYTHON% setup.py bdist_wheel || exit /B 1, which seems odd.

raulcd · 2024-12-02T11:51:27Z

Unfortunately, while the free-threaded wheel is built correctly, py -3.13t -c "import pyarrow" fails. No error shown. Does anyone know how I could inspect stderr to see why this fails?

I haven't tested and I am unsure if this works but maybe we can try this (https://stackoverflow.com/a/21912169):
py -3.13t -c "import pyarrow" || echo ERROR && exit /B 1 instead of py -3.13t -c "import pyarrow" || exit /B 1

lysnikolaou · 2024-12-02T13:21:21Z

The Windows failure is probably related to pypa/setuptools#4662 and should be fixed in the next setuptools release.

pitrou · 2024-12-04T13:57:39Z

The Windows failure is probably related to pypa/setuptools#4662 and should be fixed in the next setuptools release.

Hmm, that one should be fixed in CPython AFAIU: python/cpython#111650

pitrou · 2024-12-04T13:59:43Z

Ah, scratch that, the rationale is actually here:
https://github.com/pypa/distutils/pull/310/files

pitrou · 2024-12-04T14:02:42Z

Unfortunately, while the free-threaded wheel is built correctly, py -3.13t -c "import pyarrow" fails. No error shown. Does anyone know how I could inspect stderr to see why this fails?

Well, stderr should already be shown by default, so it's weird that nothing gets displayed.

pitrou · 2024-12-18T17:31:56Z

@github-actions crossbow submit wheel-windows-*

pitrou · 2024-12-18T17:32:23Z

Given setuptools does not seem in a hurry to merge the distutils fix, I've tried to push a manual workaround in this PR.

github-actions · 2024-12-18T17:34:22Z

Revision: 83a76e31e1272824d84ca97e4e513f12d7c73f03

Submitted crossbow builds: ursacomputing/crossbow @ actions-5bfc22ac30

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

pitrou · 2024-12-18T19:59:10Z

Well, building the free-threaded wheel failed with:

LINK : fatal error LNK1104: cannot open file 'python313t.lib' [C:\arrow\python\build\temp.win-amd64-cpython-313t\arrow_python.vcxproj]

Edit: I've checked: that file exists in a Python 3.13 Windows install.

pitrou · 2024-12-18T20:26:02Z

@github-actions crossbow submit wheel-windows-cp313*

pitrou · 2025-01-06T16:20:55Z

@github-actions crossbow submit wheel-windows*

github-actions · 2025-01-06T16:23:26Z

Revision: b3ce08c75c66571a7eb84f627b4e585a2d0ade6f

Submitted crossbow builds: ursacomputing/crossbow @ actions-ec43532048

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

.pre-commit-config.yaml

ci/docker/python-free-threaded-wheel-windows-test-vs2019.dockerfile

ci/docker/python-free-threaded-wheel-windows-vs2019.dockerfile

ci/docker/python-wheel-windows-test-vs2019-base.dockerfile

docker-compose.yml

kou · 2025-01-07T00:18:19Z

ci/docker/python-wheel-windows-vs2019-base.dockerfile

Can we use python-wheel-windows-test-vs2019-base.dockerfile as the base image?

Why would we? That would make things more complicated.

Because I felt that python-wheel-windows-{,test-}vs2019-base.dockerfile have many duplication. But I re-read them now, there are not so much duplication. Sorry.

pitrou · 2025-01-07T09:32:58Z

@github-actions crossbow submit wheel-windows*

github-actions · 2025-01-07T09:35:22Z

Revision: 764437b2d222f6b7f769629b387cbbddb27673e3

Submitted crossbow builds: ursacomputing/crossbow @ actions-f31cf1d015

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

pitrou · 2025-01-07T10:57:28Z

Updated CI still green, this is ready for review again @kou

kou

+1

kou · 2025-01-07T20:42:10Z

.env

Can we avoid the a suffix?

Suggested change

PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2025-01-06

PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2025-01-06a

PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2025-01-08

PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2025-01-08

kou · 2025-01-07T20:44:34Z

.pre-commit-config.yaml

Does this match python-wheel-windows-vs2019-base.dockerfile?

Probably not, but this is asking for more and more things that are outside of the scope of this PR...

Also, the diagnostics emitted by hadolint don't seem terribly useful, especially when we have to silence most of them.

kou · 2025-01-07T20:47:22Z

ci/docker/python-free-threaded-wheel-windows-test-vs2019.dockerfile

Could you add a comment that explains what is DL3006?
A link to https://github.com/hadolint/hadolint/wiki/DL3006 and the summary in the page ("Always tag the version of an image explicitly.") will be enough.

Uh, sure. But is hadolint actually useful? It seems terribly limited.

Is there any useful hadolint alternative? I'm OK with migrating to it from hadolint.

I have no idea. I can add a comment in the meantime.

Ok, I've added comments. But this is polluting the source code with pointless comments and making it less readable.

kou · 2025-01-07T20:50:14Z

ci/scripts/python_wheel_windows_build.bat

Ah, I missed it.

pitrou · 2025-01-08T07:45:41Z

@github-actions crossbow submit wheel-windows*

github-actions · 2025-01-08T07:48:02Z

Revision: f212df9

Submitted crossbow builds: ursacomputing/crossbow @ actions-ad58f7ee92

Task	Status
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp39-cp39-amd64

pitrou · 2025-01-08T10:01:39Z

Ok, CI is green again. Do the comments look good to you @kou ?

kou

+1

lysnikolaou requested review from assignUser, jonkeane, kou and raulcd as code owners November 21, 2024 11:25

github-actions bot added the awaiting review Awaiting review label Nov 21, 2024

raulcd reviewed Nov 27, 2024

View reviewed changes

ci/docker/python-wheel-windows-test-vs2019.dockerfile Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting review Awaiting review awaiting changes Awaiting changes labels Nov 27, 2024

pitrou force-pushed the windows-free-threaded-wheels branch from 95a0a53 to 83a76e3 Compare December 18, 2024 17:31

github-actions bot added the Component: Python label Dec 18, 2024

github-actions bot removed the Component: Python label Jan 6, 2025

kou reviewed Jan 7, 2025

View reviewed changes

pitrou force-pushed the windows-free-threaded-wheels branch from 4c9872d to 097c9c6 Compare January 7, 2025 09:29

github-actions bot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Jan 7, 2025

kou approved these changes Jan 7, 2025

View reviewed changes

github-actions bot added awaiting merge Awaiting merge awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting merge Awaiting merge labels Jan 7, 2025

lysnikolaou and others added 3 commits January 8, 2025 08:38

Add configuration for building & testing free-threaded wheels on Windows

9bcae89

Lint more Dockerfiles

a439631

Add comment

f212df9

pitrou force-pushed the windows-free-threaded-wheels branch from 764437b to f212df9 Compare January 8, 2025 07:42

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jan 8, 2025

kou approved these changes Jan 8, 2025

View reviewed changes

kou merged commit 4ede48c into apache:main Jan 8, 2025
36 of 37 checks passed

kou removed the awaiting change review Awaiting change review label Jan 8, 2025

github-actions bot added the awaiting merge Awaiting merge label Jan 8, 2025

kou mentioned this pull request Jan 8, 2025

[Python][Packaging] Enable free-threaded wheels on Windows #44421

Closed

lysnikolaou deleted the windows-free-threaded-wheels branch January 8, 2025 13:52

-PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2025-01-06
-PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2025-01-06a
+PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2025-01-08
+PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2025-01-08

Conversation

lysnikolaou commented Nov 21, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Nov 21, 2024

Uh oh!

lysnikolaou commented Nov 26, 2024

Uh oh!

raulcd commented Nov 27, 2024

Uh oh!

github-actions bot commented Nov 27, 2024

Uh oh!

raulcd commented Nov 27, 2024

Uh oh!

github-actions bot commented Nov 27, 2024

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lysnikolaou commented Nov 27, 2024

Uh oh!

lysnikolaou commented Nov 27, 2024

Uh oh!

raulcd commented Dec 2, 2024

Uh oh!

lysnikolaou commented Dec 2, 2024

Uh oh!

pitrou commented Dec 4, 2024

Uh oh!

pitrou commented Dec 4, 2024

Uh oh!

pitrou commented Dec 4, 2024

Uh oh!

pitrou commented Dec 18, 2024

Uh oh!

pitrou commented Dec 18, 2024

Uh oh!

github-actions bot commented Dec 18, 2024

Uh oh!

pitrou commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pitrou commented Dec 18, 2024

Uh oh!

pitrou commented Jan 6, 2025

Uh oh!

github-actions bot commented Jan 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou commented Jan 7, 2025

Uh oh!

github-actions bot commented Jan 7, 2025

Uh oh!

pitrou commented Jan 7, 2025

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

lysnikolaou commented Nov 21, 2024 •

edited by github-actions bot

Loading

pitrou commented Dec 18, 2024 •

edited

Loading

pitrou Jan 7, 2025 •

edited

Loading