[mlir] Inconsistency when searching for Python3 and Python

[dbabokin]

I've just stepped into a trap caused by these lines of CMake code:

llvm-project/mlir/cmake/modules/MLIRDetectPythonEnv.cmake

Lines 25 to 28 in b7279ed

	# It's a little silly to detect Python a second time, but nanobind's cmake
	# code looks for Python_ not Python3_.
	find_package(Python ${LLVM_MINIMUM_PYTHON_VERSION}
	COMPONENTS Interpreter ${_python_development_component} REQUIRED)

I was building torch-mlir on my Macbook and I was following the official build guidelines. On my system I have Python3.13 and Python3.11, but I was running in VirtualEnv with Python3.11 enabled. And CMake behaved really weirdly. It looked for Python three times, every time it was looking for Python3 twice and once for Python (that's expected). While it consistently found Python3 with version 3.11, the Python was found as 3.11 at first, but then as 3.13 (really unexpected).

Then if I was rerunning cmake with exactly the same command line (which succeeded on the first pass), it crashed with the following message:

CMake Error at /Users/dbabokin/ox-workspaces/torch-mlir/.venv_torch_mlir2/lib/python3.11/site-packages/cmake/data/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
  Could NOT find Python (missing: Development.Module) (found suitable version
  "3.11.11", minimum required is "3.8")
Call Stack (most recent call first):
  /Users/dbabokin/ox-workspaces/torch-mlir/.venv_torch_mlir2/lib/python3.11/site-packages/cmake/data/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE)
  /Users/dbabokin/ox-workspaces/torch-mlir/.venv_torch_mlir2/lib/python3.11/site-packages/cmake/data/share/cmake-3.31/Modules/FindPython/Support.cmake:4002 (find_package_handle_standard_args)
  /Users/dbabokin/ox-workspaces/torch-mlir/.venv_torch_mlir2/lib/python3.11/site-packages/cmake/data/share/cmake-3.31/Modules/FindPython.cmake:631 (include)
  /Users/dbabokin/ox-workspaces/torch-mlir/externals/llvm-project/mlir/cmake/modules/MLIRDetectPythonEnv.cmake:27 (find_package)
  /Users/dbabokin/ox-workspaces/torch-mlir/externals/llvm-project/mlir/CMakeLists.txt:196 (mlir_configure_python_dev_packages)

Running cmake again was successful, one more time - failing.

Debugging the problem I found out that the root cause was that I passed -DPython3_FIND_VIRTUALENV=ONLY to CMake, but not -DPython_FIND_VIRTUALENV=ONLY!!!

So that resolved my problem and in this sense it was a user error. But I think it really makes sense to check that Python3_FIND_VIRTUALENV and Python_FIND_VIRTUALENV are defined (or not defined) in the same way and warn user otherwise.

Tagging @makslevental who is the last one who touched find_package(Python ...) code.

[llvmbot]

@llvm/issue-subscribers-mlir

Author: Dmitry Babokin (dbabokin)

I've just stepped into a trap caused by these lines of CMake code: https://github.com/llvm/llvm-project/blob/b7279ed5b3ae3e7b0fd61e0f08c86fb1958f0b6f/mlir/cmake/modules/MLIRDetectPythonEnv.cmake#L25-L28

I was building torch-mlir on my Macbook and I was following the official build guidelines. On my system I have Python3.13 and Python3.11, but I was running in VirtualEnv with Python3.11 enabled. And CMake behaved really weirdly. It looked for Python three times, every time it was looking for Python3 twice and once for Python (that's expected). While it consistently found Python3 with version 3.11, the Python was found as 3.11 at first, but then as 3.13 (really unexpected).

Then if I was rerunning cmake with exactly the same command line (which succeeded on the first pass), it crashed with the following message:

CMake Error at /Users/dbabokin/ox-workspaces/torch-mlir/.venv_torch_mlir2/lib/python3.11/site-packages/cmake/data/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:233 (message):
  Could NOT find Python (missing: Development.Module) (found suitable version
  "3.11.11", minimum required is "3.8")
Call Stack (most recent call first):
  /Users/dbabokin/ox-workspaces/torch-mlir/.venv_torch_mlir2/lib/python3.11/site-packages/cmake/data/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE)
  /Users/dbabokin/ox-workspaces/torch-mlir/.venv_torch_mlir2/lib/python3.11/site-packages/cmake/data/share/cmake-3.31/Modules/FindPython/Support.cmake:4002 (find_package_handle_standard_args)
  /Users/dbabokin/ox-workspaces/torch-mlir/.venv_torch_mlir2/lib/python3.11/site-packages/cmake/data/share/cmake-3.31/Modules/FindPython.cmake:631 (include)
  /Users/dbabokin/ox-workspaces/torch-mlir/externals/llvm-project/mlir/cmake/modules/MLIRDetectPythonEnv.cmake:27 (find_package)
  /Users/dbabokin/ox-workspaces/torch-mlir/externals/llvm-project/mlir/CMakeLists.txt:196 (mlir_configure_python_dev_packages)

Running cmake again was successful, one more time - failing.

Debugging the problem I found out that the root cause was that I passed -DPython3_FIND_VIRTUALENV=ONLY to CMake, but not -DPython_FIND_VIRTUALENV=ONLY!!!

So that resolved my problem and in this sense it was a user error. But I think it really makes sense to check that Python3_FIND_VIRTUALENV and Python_FIND_VIRTUALENV are defined (or not defined) in the same way and warn user otherwise.

Tagging @makslevental who is the last one who touched find_package(Python ...) code.

[makslevental]

This thing is a constant source of frustration (which is why that CMake section is already hairy enough). I don't think anyone has any urge to try to root cause it so personally I think layering more bandaids on it is absolutely fine (i.e., I'll take that PR if you send it).

[dbabokin]

I'll take that PR if you send it

See #126329