pip with truststore and proxy can use incorrect ssl_context

[schribl]

Description

Disclaimer:
I am not 100% sure if this issue is a bug in pip or rather requests or some other of the vendored library, but I am happy to work together to determine the right location and file the bug there.

Setup:
Linux with HTTP_PROXY and HTTPS_PROXY set and pip 25.0.1 on Python 3.10.

Issue:
When running python -m pip install <package> I receive following error messages:

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)'))': /simple/scipy/

Investigations on my side:
When setting SSL_CERT_FILE to cacert.pem it works.

Not being really happy with this workaround I did a debug session on pip and came up with this trail of events.

The truststore ssl_context is constructed here and then passed to the PipSession.
https://github.com/pypa/pip/blob/25.0.1/src/pip/_internal/cli/index_command.py#L95

Inside the PipSession either the HTTPAdapter or the CacheControlAdapter are used:
https://github.com/pypa/pip/blob/25.0.1/src/pip/_internal/network/session.py#L294

For this case consider the no cache case, the issue happens in both cases. The HTTPAdapter here is a combination of the _SSLContextAdapterMixin in the same file and the request HTTPAdapter. The ssl_context is passed in the constructor to this, the requests class does not use this kwargs. It is used in the _SSLContextAdapterMixin and stored for later use when init_poolmanager is called.

The init_poolmanger is called from requests in the constructor:
https://github.com/pypa/pip/blob/25.0.1/src/pip/_vendor/requests/adapters.py#L222

This set up self.poolmanager of the Adapter to a poolmanager that will use the ssl_context. This is done by saving it ìnside the urllib3 PoolManager see here:
https://github.com/pypa/pip/blob/25.0.1/src/pip/_vendor/urllib3/poolmanager.py#L173

When this adapter is used in send it will call get_connection_with_tls_context
https://github.com/pypa/pip/blob/25.0.1/src/pip/_vendor/requests/adapters.py#L633

Now we are approaching the issue location when in this location we use build_connection_pool_key_attributes to get the pool_kwargs. This will use the self.poolmanager (that was constructed before) in this location to get the kwargs:
https://github.com/pypa/pip/blob/25.0.1/src/pip/_vendor/requests/adapters.py#L444

The logic in the lines here
https://github.com/pypa/pip/blob/25.0.1/src/pip/_vendor/requests/adapters.py#L104-L120
will check the given PoolManager if it has a ssl_context set and not define this as part of the kwargs (e.g. no the default ssl_context). But when returning to the previous location in the case of a set proxy we will "overwrite" the used PoolManager with a ProxyManager:
https://github.com/pypa/pip/blob/25.0.1/src/pip/_vendor/requests/adapters.py#L527

As the ssl_context is not part of the pool_kwargs it will not be considered for the ProxyManager and there is no ssl_context set even so verify is True.

I did a quick sanity check by adding this command before the line 527 (where conn is created with the ProxyManger).

pool_kwargs.update(self.poolmanager.connection_pool_kwargs)

And while this is not a proper workaround it still tests if when the ssl_context is propagated everything works. And with this change the same command in the same environment is working properly.

Expected behavior

Pip will use truststore and system certificates and succesfully verify the PyPI certificates.

pip version

25.0.1

Python version

3.10

OS

Linux

How to Reproduce

I am not 100% sure I have completely understood how to reproduce it. So multiple requirements are:

• Use pip together with truststore
• Use a proxy

But I am pretty sure this are not sufficient to reproduce and there is another requirement on the OpenSSL configuration used or similar. I will continue to investigate this and if I find more details add them here.

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

Thanks for reporting.

FYI @sethmlarson , I'm planning to take a look, but does this sound right to you?

Pip has very little ability to verify truststore behavior, other than logically reasoning about the code.

[sethmlarson]

Definitely take a look, it sounds like there's indeed an issue where the ssl_context is being dropped.

[schribl]

@sethmlarson I soon will have only very limited internet access for a couple of weeks so in case you have questions it might take me some time to answer.