Modin not compatible with pandas

[rgoubet]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

I'm currently unable to run Snowpark with pandas in an Anaconda environment.

Creating the environment:

conda create -n sf -c conda-forge python=3.11 ipykernel pandas modin snowflake-snowpark-python[modin]

Next, if I run this

import snowflake.snowpark.modin.plugin

I get the following error:

RuntimeError: The pandas version installed (2.2.3) does not match the supported pandas version in Snowpark pandas (2.2.1). Run `pip install "snowflake-snowpark-python[modin]"` to resolve.

It works if I remove snowflake-snowpark-python with conda and reinstall with pip, but this is not recommended.

Installed packages

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
asn1crypto                1.5.1              pyhd8ed1ab_0    conda-forge
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
aws-c-auth                0.8.0                hcd8ed7f_7    conda-forge
aws-c-cal                 0.8.0                he70792b_1    conda-forge
aws-c-common              0.10.0               hb9d3cd8_0    conda-forge
aws-c-compression         0.3.0                hba2fe39_1    conda-forge
aws-c-event-stream        0.5.0                h127f702_4    conda-forge
aws-c-http                0.9.0                h8a7d7e2_5    conda-forge
aws-c-io                  0.15.1               h2a50c78_1    conda-forge
aws-c-mqtt                0.11.0               hd25e75f_5    conda-forge
aws-c-s3                  0.7.0                h858c4ad_7    conda-forge
aws-c-sdkutils            0.2.1                hba2fe39_0    conda-forge
aws-checksums             0.2.0                hba2fe39_1    conda-forge
aws-crt-cpp               0.29.3               hbc793f2_2    conda-forge
aws-sdk-cpp               1.11.407             h5cd358a_9    conda-forge
azure-core-cpp            1.14.0               h5cfcd09_0    conda-forge
azure-identity-cpp        1.10.0               h113e628_0    conda-forge
azure-storage-blobs-cpp   12.13.0              h3cf044e_1    conda-forge
azure-storage-common-cpp  12.8.0               h736e048_1    conda-forge
azure-storage-files-datalake-cpp 12.12.0              ha633028_1    conda-forge
bokeh                     3.6.1              pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py311hfdbb021_2    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.34.3               heb4867d_0    conda-forge
ca-certificates           2024.8.30            hbcca054_0    conda-forge
certifi                   2024.8.30          pyhd8ed1ab_0    conda-forge
cffi                      1.17.1          py311hf29c0ef_0    conda-forge
charset-normalizer        3.4.0              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
comm                      0.2.2              pyhd8ed1ab_0    conda-forge
contourpy                 1.3.0           py311hd18a35c_2    conda-forge
cryptography              43.0.3          py311hafd3f86_0    conda-forge
cytoolz                   1.0.0           py311h9ecbd09_1    conda-forge
dask                      2024.8.0           pyhd8ed1ab_0    conda-forge
dask-core                 2024.8.0           pyhd8ed1ab_0    conda-forge
dask-expr                 1.1.10             pyhd8ed1ab_0    conda-forge
debugpy                   1.8.8           py311hfdbb021_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
distributed               2024.8.0           pyhd8ed1ab_0    conda-forge
exceptiongroup            1.2.2              pyhd8ed1ab_0    conda-forge
executing                 2.1.0              pyhd8ed1ab_0    conda-forge
filelock                  3.16.1             pyhd8ed1ab_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
fsspec                    2024.10.0          pyhff2d567_0    conda-forge
gflags                    2.2.2             h5888daf_1005    conda-forge
glog                      0.7.1                hbabe93e_0    conda-forge
h2                        4.1.0              pyhd8ed1ab_0    conda-forge
hpack                     4.0.0              pyh9f0ad1d_0    conda-forge
hyperframe                6.0.1              pyhd8ed1ab_0    conda-forge
idna                      3.10               pyhd8ed1ab_0    conda-forge
importlib-metadata        8.5.0              pyha770c72_0    conda-forge
importlib_metadata        8.5.0                hd8ed1ab_0    conda-forge
ipykernel                 6.29.5             pyh3099207_0    conda-forge
ipython                   8.29.0             pyh707e725_0    conda-forge
jedi                      0.19.2             pyhff2d567_0    conda-forge
jinja2                    3.1.4              pyhd8ed1ab_0    conda-forge
jupyter_client            8.6.3              pyhd8ed1ab_0    conda-forge
jupyter_core              5.7.2              pyh31011fe_1    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
lcms2                     2.16                 hb7c19ff_0    conda-forge
ld_impl_linux-64          2.43                 h712a8e2_2    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20240722.0      cxx17_h5888daf_1    conda-forge
libarrow                  18.0.0           h3e543c6_4_cpu    conda-forge
libarrow-acero            18.0.0           h5888daf_4_cpu    conda-forge
libarrow-dataset          18.0.0           h5888daf_4_cpu    conda-forge
libarrow-substrait        18.0.0           h5c8f2c3_4_cpu    conda-forge
libblas                   3.9.0           25_linux64_openblas    conda-forge
libbrotlicommon           1.1.0                hb9d3cd8_2    conda-forge
libbrotlidec              1.1.0                hb9d3cd8_2    conda-forge
libbrotlienc              1.1.0                hb9d3cd8_2    conda-forge
libcblas                  3.9.0           25_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcurl                   8.10.1               hbbe4b11_0    conda-forge
libdeflate                1.22                 hb9d3cd8_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.6.4                h5888daf_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    14.2.0               h77fa898_1    conda-forge
libgcc-ng                 14.2.0               h69a702a_1    conda-forge
libgfortran               14.2.0               h69a702a_1    conda-forge
libgfortran5              14.2.0               hd5240d6_1    conda-forge
libgomp                   14.2.0               h77fa898_1    conda-forge
libgoogle-cloud           2.31.0               h804f50b_0    conda-forge
libgoogle-cloud-storage   2.31.0               h0121fbd_0    conda-forge
libgrpc                   1.67.1               hc2c308b_0    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblapack                 3.9.0           25_linux64_openblas    conda-forge
libnghttp2                1.64.0               h161d5f1_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.28          pthreads_h94d23a6_1    conda-forge
libparquet                18.0.0           h6bd9018_4_cpu    conda-forge
libpng                    1.6.44               hadc24fc_0    conda-forge
libprotobuf               5.28.2               h5b01275_0    conda-forge
libre2-11                 2024.07.02           hbbce691_1    conda-forge
libsodium                 1.0.20               h4ab18f5_0    conda-forge
libsqlite                 3.47.0               hadc24fc_1    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx                 14.2.0               hc0a3c3a_1    conda-forge
libstdcxx-ng              14.2.0               h4852527_1    conda-forge
libthrift                 0.21.0               h0e7cc3e_0    conda-forge
libtiff                   4.7.0                he137b08_1    conda-forge
libutf8proc               2.8.0                h166bdaf_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.4.0                hd590300_0    conda-forge
libxcb                    1.17.0               h8a09558_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.13.4               h064dc61_2    conda-forge
libzlib                   1.3.1                hb9d3cd8_2    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4                       4.3.3           py311h2cbdf9a_1    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
markupsafe                3.0.2           py311h2dc5d0c_0    conda-forge
matplotlib-inline         0.1.7              pyhd8ed1ab_0    conda-forge
modin                     0.32.0               hd8ed1ab_1    conda-forge
modin-core                0.32.0             pyhd8ed1ab_1    conda-forge
modin-dask                0.32.0               hd8ed1ab_1    conda-forge
msgpack-python            1.1.0           py311hd18a35c_0    conda-forge
ncurses                   6.5                  he02047a_1    conda-forge
nest-asyncio              1.6.0              pyhd8ed1ab_0    conda-forge
numpy                     1.26.4          py311h64a7726_0    conda-forge
openjpeg                  2.5.2                h488ebb8_0    conda-forge
openssl                   3.3.2                hb9d3cd8_0    conda-forge
orc                       2.0.2                he039a57_2    conda-forge
packaging                 24.1               pyhd8ed1ab_0    conda-forge
pandas                    2.2.3           py311h7db5c69_1    conda-forge
parso                     0.8.4              pyhd8ed1ab_0    conda-forge
partd                     1.4.2              pyhd8ed1ab_0    conda-forge
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    11.0.0          py311h49e9ac3_0    conda-forge
pip                       24.3.1             pyh8b19718_0    conda-forge
platformdirs              4.3.6              pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.48             pyha770c72_0    conda-forge
psutil                    6.1.0           py311h9ecbd09_0    conda-forge
pthread-stubs             0.4               hb9d3cd8_1002    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.3              pyhd8ed1ab_0    conda-forge
pyarrow                   18.0.0          py311h38be061_1    conda-forge
pyarrow-core              18.0.0          py311h4854187_1_cpu    conda-forge
pyarrow-hotfix            0.6                pyhd8ed1ab_0    conda-forge
pycparser                 2.22               pyhd8ed1ab_0    conda-forge
pygments                  2.18.0             pyhd8ed1ab_0    conda-forge
pyjwt                     2.9.0              pyhd8ed1ab_1    conda-forge
pyopenssl                 24.2.1             pyhd8ed1ab_2    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.10         hc5c86c4_3_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-tzdata             2024.2             pyhd8ed1ab_0    conda-forge
python_abi                3.11                    5_cp311    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.2           py311h9ecbd09_1    conda-forge
pyzmq                     26.2.0          py311h7deb3e3_3    conda-forge
re2                       2024.07.02           h77b4e00_1    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.32.3             pyhd8ed1ab_0    conda-forge
s2n                       1.5.7                hd3e8b83_0    conda-forge
setuptools                75.3.0             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.2.1                ha2e4443_0    conda-forge
snowflake-connector-python 3.12.2          py311h7db5c69_1    conda-forge
snowflake-snowpark-python 1.24.0          py311h38be061_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
tblib                     3.0.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tomlkit                   0.13.2             pyha770c72_0    conda-forge
toolz                     1.0.0              pyhd8ed1ab_0    conda-forge
tornado                   6.4.1           py311h9ecbd09_1    conda-forge
traitlets                 5.14.3             pyhd8ed1ab_0    conda-forge
typing-extensions         4.12.2               hd8ed1ab_0    conda-forge
typing_extensions         4.12.2             pyha770c72_0    conda-forge
tzdata                    2024b                hc8b5060_0    conda-forge
urllib3                   2.2.3              pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
wheel                     0.45.0             pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.11               hb9d3cd8_1    conda-forge
xorg-libxdmcp             1.1.5                hb9d3cd8_0    conda-forge
xyzservices               2024.9.0           pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zeromq                    4.3.5                h3b0a872_6    conda-forge
zict                      3.0.0              pyhd8ed1ab_0    conda-forge
zipp                      3.21.0             pyhd8ed1ab_0    conda-forge
zstandard                 0.23.0          py311hbc35293_1    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

Environment info

active environment : sf
    active env location : /home/user/miniconda/envs/sf
            shell level : 2
       user config file : /home/user/.condarc
 populated config files : /home/user/.condarc
          conda version : 24.9.2
    conda-build version : not installed
         python version : 3.12.7.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=zen2
                          __conda=24.9.2=0
                          __glibc=2.40=0
                          __linux=6.11.0=0
                          __unix=0=0
       base environment : /home/user/miniconda  (writable)
      conda av data dir : /home/user/miniconda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/user/miniconda/pkgs
                          /home/user/.conda/pkgs
       envs directories : /home/user/miniconda/envs
                          /home/user/.conda/envs
               platform : linux-64
             user-agent : conda/24.9.2 requests/2.32.3 CPython/3.12.7 Linux/6.11.0-9-generic ubuntu/24.10 glibc/2.40 solver/libmamba conda-libmamba-solver/24.9.0 libmambapy/1.5.10 aau/0.4.4 c/. s/. e/.
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

[rgoubet]

Additionally, if I pin the pandas version to 2.2.1, I get a different error:

conda create -n sf -c conda-forge python=3.11 ipykernel pandas=2.2.1 snowflake-snowpark-python[modin]

In [1]: import snowflake.snowpark.modin.plugin

ImportError: The Modin version installed (0.32.0) does not match the supported Modin version in Snowpark pandas (0.28.1). Run `pip install "snowflake-snowpark-python[modin]"` to resolve.

[sfc-gh-joshi]

Hi @rgoubet, as the message says, you'll need to pin a modin version as well (modin==0.28.1 for Snowpark Python 1.24.0). If you do this, you shouldn't need to pin the pandas version, as modin will automatically pick a compatible one for you (see https://docs.snowflake.com/en/developer-guide/snowpark/python/pandas-on-snowflake#installation).

FYI: unlike pip, conda installs don't support the square-bracket syntax (snowflake-snowpark-python[modin] isn't any different from snowflake-snowpark-python).

[rgoubet]

This didn't work for me:

conda create -n sf -c conda-forge python=3.11 ipykernel modin==0.28.1 snowflake-snowpark-python

import snowflake.snowpark.modin.plugin

ImportError: The Modin version installed (0.28.1) does not match the supported Modin version in Snowpark pandas (0.30.1). Run `pip install "snowflake-snowpark-python[modin]"` to resolve.

[sfc-gh-joshi]

We just made a new release of Snowpark Python (1.25.0) that upgrades the modin version, so you'll actually now need to pin modin==0.30.1.

[rgoubet]

OK thanks, that worked. It would be helpful to set the dependencies in the conda packages so as to avoid pinning them. This would allow conda to keep the environment up to date.

Thank you!