feature: better pip crossplatform resolves

[saikonen]

for correctly resolving transient dependencies for cross-platform package resolves, where dependencies are conditional on platform/os such as

"nvidia-cuda-runtime-cu12 (==12.4.127) ; platform_system == \"Linux\" and platform_machine == \"x86_64\"",

This intends to solve an issue with packages such as the default torch where resolving on a mac, but deploying onto a linux-64 machine will result in a broken environment due to the conditional dependencies not getting bundled up.

[saikonen]

note that for testing this, both .pip/metadata and conda.manifest local files for the involved flow need to most likely be cleared

[romain-intel]

I am not sure this is sufficient in all cases. Here is the default_environment that is used:

    return {
        "implementation_name": implementation_name,
        "implementation_version": iver,
        "os_name": os.name,
        "platform_machine": platform.machine(),
        "platform_release": platform.release(),
        "platform_system": platform.system(),
        "platform_version": platform.version(),
        "python_full_version": platform.python_version(),
        "platform_python_implementation": platform.python_implementation(),
        "python_version": ".".join(platform.python_version_tuple()[:2]),
        "sys_platform": sys.platform,
    }

I think we need to override:

• sys.platform
• platform.version()

as well. The others relate to posix versus non posix (and I think we are always posix) and cpython or not (and I think we also only do cpython)

[romain-intel]

and platform.release()

[romain-intel]

Not tested yet (working on it) but maybe something like this:

import json
import os
import platform
import sys

_override_mappings = {
    "platform_machine": platform.machine,
    "platform_release": platform.release,
    "platform_system": platform.system,
    "platform_version": platform.version,
    "sys_platform": sys.platform,
}

overrides = os.environ.get("PIP_CUSTOMIZE_OVERRIDES")
if overrides:
    overrides_dict = json.loads(overrides)
    for key, override_value in overrides_dict.items():
        if key in _override_mappings:
            if callable(_override_mappings[key]):
                _override_mappings[key] = lambda _v=override_value: _v
            else:
                _override_mappings[key] = override_value

[romain-intel]

OK, that didn't work and then I realized that if I override sys.platform, bad things happen. I still think we need to override it to handle as many markers as possible.

I ended up going with the other approach. Something like this:

import json
import os
import pip._vendor.packaging.markers as _markers


_overrides = os.environ.get("PIP_CUSTOMIZE_OVERRIDES")
if _overrides:
    _orig_default_environment = _markers.default_environment
    _overrides_dict = json.loads(_overrides)

    def _wrap_default_environment():
        result = _orig_default_environment()
        result.update(_overrides_dict)
        return result

    _markers.default_environment = _wrap_default_environment

and that seems to work. Just FYI.

[saikonen]

opened #2675 to address this, along with resolving an issue cross-platform resolving on a mac when using metaflow-dev

what are you using as a source of info for platform.version/platform.release/sys.platform when it comes to the target system? I'm not sure if any/all of these are available currently at least on the OSS side