conda package dependency regression on urllib3 #1280

michaelaye · 2025-01-27T10:40:02Z

Describe the bug
Updating from

aiobotocore 2.16.1
botocore 1.35.88

on osx-arm64

forces downgrade of urllib3 from 2.x to 1.x

 conda install botocore=1.36 aiobotocore=2.19
Channels:
 - conda-forge
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/maye/miniforge3/envs/botocore_test

  added / updated specs:
    - aiobotocore=2.19
    - botocore=1.36


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    aiobotocore-2.19.0         |     pyhd8ed1ab_0          66 KB  conda-forge
    botocore-1.36.3            |pyge310_1234567_0         7.2 MB  conda-forge
    urllib3-1.26.19            |     pyhd8ed1ab_0         112 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         7.4 MB

The following packages will be UPDATED:

  aiobotocore                           2.16.1-pyhd8ed1ab_0 --> 2.19.0-pyhd8ed1ab_0 
  botocore                        1.35.88-pyge310_1234567_0 --> 1.36.3-pyge310_1234567_0 

The following packages will be DOWNGRADED:

  urllib3                                2.3.0-pyhd8ed1ab_0 --> 1.26.19-pyhd8ed1ab_0

reproduce

conda create -n botocore_test python=3.12 botocore=1.35.88 aiobotocore=2.16.1 -y
conda activate botocore_test
conda install botocore=1.36.3 aiobotocore=2.19.0

Checklist

I have reproduced in environment where pip check passes without errors
I have provided pip freeze results
I have provided sample code or detailed way to reproduce
I have tried the same code in botocore to ensure this is an aiobotocore specific issue
I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

aiobotocore @ file:///home/conda/feedstock_root/build_artifacts/aiobotocore_1735325162500/work
aiohappyeyeballs @ file:///home/conda/feedstock_root/build_artifacts/aiohappyeyeballs_1733331917983/work
aiohttp @ file:///Users/runner/miniforge3/conda-bld/aiohttp_1734596985522/work
aioitertools @ file:///home/conda/feedstock_root/build_artifacts/aioitertools_1735329051909/work
aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1734342155601/work
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1737819173731/work
botocore @ file:///home/conda/feedstock_root/build_artifacts/botocore_1735284346359/work
Brotli @ file:///Users/runner/miniforge3/conda-bld/brotli-split_1725267563793/work
cffi @ file:///Users/runner/miniforge3/conda-bld/cffi_1725560578465/work
frozenlist @ file:///Users/runner/miniforge3/conda-bld/frozenlist_1737645287563/work
h2 @ file:///home/conda/feedstock_root/build_artifacts/h2_1733298745555/work
hpack @ file:///home/conda/feedstock_root/build_artifacts/hpack_1737618293087/work
hyperframe @ file:///home/conda/feedstock_root/build_artifacts/hyperframe_1737618333194/work
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1733211830134/work
jmespath @ file:///home/conda/feedstock_root/build_artifacts/jmespath_1733229141657/work
multidict @ file:///Users/runner/miniforge3/conda-bld/multidict_1729065492473/work
propcache @ file:///Users/runner/miniforge3/conda-bld/propcache_1737635647354/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/bld/rattler-build_pycparser_1733195786/work
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1733217236728/work
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1733215673016/work
setuptools==75.8.0
six @ file:///home/conda/feedstock_root/build_artifacts/six_1733380938961/work
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1733188668063/work
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1734859416348/work
wheel==0.45.1
wrapt @ file:///Users/runner/miniforge3/conda-bld/wrapt_1736869483984/work
yarl @ file:///Users/runner/miniforge3/conda-bld/yarl_1737575870898/work
zstandard==0.23.0

Environment:

Python Version: 3.12
OS name and version: darwin sequoia 15.2 on M3 chip

The text was updated successfully, but these errors were encountered:

thehesiod · 2025-01-28T05:50:13Z

@jakob-keller you added this in https://github.com/aio-libs/aiobotocore/pull/1246/files. Again, the only reason we use this library is because we need to match what botocore does. We should not list these types of dependencies explicitly, they should be removed and tracked instead perhaps via a unittest.

jakob-keller · 2025-01-28T06:37:42Z

@jakob-keller you added this in https://github.com/aio-libs/aiobotocore/pull/1246/files. Again, the only reason we use this library is because we need to match what botocore does. We should not list these types of dependencies explicitly, they should be removed and tracked instead perhaps via a unittest.

At the moment, we use urllib3 directly:

aiobotocore/aiobotocore/endpoint.py

Line 16 in f13f103

from urllib3.response import HTTPHeaderDict

I will check if we can change the import into an indirect one (from botocore). That should work in this case and make the direct urllib3 dependency unnecessary.

jakob-keller · 2025-01-28T07:36:21Z

BTW: This issue is about 2.16.1 which does not include #1246:

aiobotocore/pyproject.toml

Line 32 in a14bdb4

dependencies = [

IMO, the result of the conda environment must be influenced by some other factor. I am pretty sure, this is not an issue at all, in particular not with aiobotocore.

@michaelaye: Could you please elaborate why you believe there is an issue at all?

jakob-keller · 2025-01-28T07:46:12Z

@jakob-keller you added this in https://github.com/aio-libs/aiobotocore/pull/1246/files. Again, the only reason we use this library is because we need to match what botocore does. We should not list these types of dependencies explicitly, they should be removed and tracked instead perhaps via a unittest.

At the moment, we use urllib3 directly:

aiobotocore/aiobotocore/endpoint.py

Line 16 in f13f103

from urllib3.response import HTTPHeaderDict
I will check if we can change the import into an indirect one (from botocore). That should work in this case and make the direct urllib3 dependency unnecessary.

I checked and am convinced that everything is as it should be with regards to that dependency: #1281

Again, this is unrelated to this issue which refers to 2.16.1.

michaelaye · 2025-01-28T10:15:25Z

did you see my reproduce section? This issue is about upgrading from 2.16.1 to anything newer.

Since 2.18 the requirement of

urllib3 >=1.25.4,<1.27

is explicitly set in the conda package:

❯ conda search aiobotocore=2.18 --info
Loading channels: done
aiobotocore 2.18.0 pyhd8ed1ab_0
-------------------------------
file name   : aiobotocore-2.18.0-pyhd8ed1ab_0.conda
name        : aiobotocore
version     : 2.18.0
build       : pyhd8ed1ab_0
build number: 0
size        : 66 KB
license     : Apache-2.0
subdir      : noarch
url         : https://conda.anaconda.org/conda-forge/noarch/aiobotocore-2.18.0-pyhd8ed1ab_0.conda
md5         : 6fee4b76792b913ba69c62e4899ea130
timestamp   : 2025-01-18 23:38:27 UTC
dependencies: 
  - aiohttp >=3.9.2,<4.0.0
  - aioitertools >=0.5.1,<1.0.0
  - botocore >=1.36.0,<1.36.2
  - jmespath >=0.7.1,<2.0.0
  - multidict >=6.0.0,<7.0.0
  - python >=3.9
  - python-dateutil >=2.1,<3.0.0
  - urllib3 >=1.25.4,<1.27
  - wrapt >=1.10.10,<2.0.0

 conda search aiobotocore=2.19 --info
Loading channels: done
aiobotocore 2.19.0 pyhd8ed1ab_0
-------------------------------
file name   : aiobotocore-2.19.0-pyhd8ed1ab_0.conda
name        : aiobotocore
version     : 2.19.0
build       : pyhd8ed1ab_0
build number: 0
size        : 66 KB
license     : Apache-2.0
subdir      : noarch
url         : https://conda.anaconda.org/conda-forge/noarch/aiobotocore-2.19.0-pyhd8ed1ab_0.conda
md5         : 547a232f11563ea73394e998f2c9a4b3
timestamp   : 2025-01-24 08:02:04 UTC
dependencies: 
  - aiohttp >=3.9.2,<4.0.0
  - aioitertools >=0.5.1,<1.0.0
  - botocore >=1.36.0,<1.36.4
  - jmespath >=0.7.1,<2.0.0
  - multidict >=6.0.0,<7.0.0
  - python >=3.9
  - python-dateutil >=2.1,<3.0.0
  - urllib3 >=1.25.4,<1.27
  - wrapt >=1.10.10,<2.0.0

botocore offers builds with and without urllib3=2 allowed, don't know why, but aiobotocore strictly forbids urllib3=2.x since aiobotocore version 2.18, which is the issue.

jakob-keller · 2025-01-28T10:22:36Z

did you see my reproduce section? This issue is about upgrading from 2.16.1 to anything newer.

Sorry, I did not. My bad!

Since 2.18 the requirement of

urllib3 >=1.25.4,<1.27

is explicitly set in the conda package:

That is certainly not intended. We ship to PyPI only and don't face that issue outside of conda. I will need to understand how conda packages are built and why they contain incorrect dependency metadata.

michaelaye · 2025-01-28T10:27:40Z

ah, it's already being discussed here, the feedstock repo is for creating the conda packages on the conda-forge channel.

conda-forge/aiobotocore-feedstock#101

maybe you could help with the arguments there due to your insight? for example, they are arguing if the direct setting could be left away and just let botocore decide on the requirements? I cannot judge if that's okay.

jakob-keller · 2025-01-28T10:43:49Z

ah, it's already being discussed here, the feedstock repo is for creating the conda packages on the conda-forge channel.

conda-forge/aiobotocore-feedstock#101

maybe you could help with the arguments there due to your insight? for example, they are arguing if the direct setting could be left away and just let botocore decide on the requirements? I cannot judge if that's okay.

Sure, hope this helps: conda-forge/aiobotocore-feedstock#101 (comment)

thehesiod · 2025-01-28T19:05:57Z

@jakob-keller I think we're in the clear as our urllib3 deps: https://github.com/aio-libs/aiobotocore/blob/master/pyproject.toml#L39-L40 match that of the corresponding botocore: https://github.com/boto/botocore/blob/1.36.0/setup.cfg#L8-L9 so this is ok. However, can we put some comments saying when bumping botocore deps these now need to be kept in sync as well?

jakob-keller · 2025-01-28T19:14:28Z

However, can we put some comments saying when bumping botocore deps these now need to be kept in sync as well?

Yes, that could be useful. Probably just reword the existing comment:

aiobotocore/pyproject.toml

Line 35 in f13f103

    
           "botocore >= 1.36.0, < 1.36.4", # NOTE: When updating, always keep `project.optional-dependencies` aligned

thehesiod · 2025-01-28T19:17:40Z

yep! specifically which modules to take into account

jakob-keller added the question Further information is requested label Jan 28, 2025

jakob-keller mentioned this issue Jan 28, 2025

Drop direct urllib3 dependency #1281

Closed

jakob-keller closed this as not planned Won't fix, can't repro, duplicate, stale Jan 28, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

conda package dependency regression on urllib3 #1280

conda package dependency regression on urllib3 #1280

michaelaye commented Jan 27, 2025

thehesiod commented Jan 28, 2025

jakob-keller commented Jan 28, 2025

jakob-keller commented Jan 28, 2025

jakob-keller commented Jan 28, 2025

michaelaye commented Jan 28, 2025

jakob-keller commented Jan 28, 2025

michaelaye commented Jan 28, 2025

jakob-keller commented Jan 28, 2025

thehesiod commented Jan 28, 2025

jakob-keller commented Jan 28, 2025

thehesiod commented Jan 28, 2025

conda package dependency regression on urllib3 #1280

conda package dependency regression on urllib3 #1280

Comments

michaelaye commented Jan 27, 2025

reproduce

thehesiod commented Jan 28, 2025

jakob-keller commented Jan 28, 2025

jakob-keller commented Jan 28, 2025

jakob-keller commented Jan 28, 2025

michaelaye commented Jan 28, 2025

jakob-keller commented Jan 28, 2025

michaelaye commented Jan 28, 2025

jakob-keller commented Jan 28, 2025

thehesiod commented Jan 28, 2025

jakob-keller commented Jan 28, 2025

thehesiod commented Jan 28, 2025