raise supported python to match zarr>=3 by colonesej · Pull Request #1027 · ecmwf/earthkit-data

colonesej · 2026-06-22T11:04:17Z

Description

When installing locally using uv dependencies cannot be solved because zarr>=3 requires python>=3.11. This raises because ek-data is claiming to support python>=3.10, python<3.11 which is not true for the zarr dependency. This raises by using uv run because it installs the dev dependency group which points to mostly all of the dependencies. On the other hand, It is true that:

zarr is an optional dependency
uv is not the officially supported build system

but

pip may be already solving for python 3.11, unless one installs optional dependencies partially or in multiple steps. This may result in a broken environment
the required python definition is actually misleading.
python 3.10 is reaching end-of-life in October 2026

optional dependency -based tests

geopandas is an optional dependency but there is no guardrails in the testing suite for the case it is missing.
Created a NO_GEOPANDAS global variable at earthkit.data.utils.testing following what's done with other optional dependencies.
As above but for pyodc

Local tests

the following two local tests are failing. Both are marked to be skipped on GITHUB runs. Maybe it is expected that they fail then!? Posting the error report for reference

========================================================================================= test session starts ==========================================================================================
platform darwin -- Python 3.13.13, pytest-9.1.1, pluggy-1.6.0 -- /Users/mojp/.cache/uv/builds-v0/.tmpm1b6Xi/bin/python
cachedir: .pytest_cache
rootdir: /Users/mojp/github/earthkit-data
configfile: pytest.ini
testpaths: tests
plugins: cov-7.1.0, timeout-2.4.0, pyfakefs-6.2.0, reraise-2.1.2, forked-1.6.0
collected 2 items / 6 skipped                                                                                                                                                                          
run-last-failure: rerun previous 2 failures (skipped 218 files)

Exception: eckit::codec::InvalidRecord: version not found in record /Users/mojp/.local/share/eckit/geo/grid/icon/17643da2574959b644d254a3cd6e2bc0-b0699f374c63d05028c18c12f80a48f4.ek                  
Request for file:/Users/mojp/.local/share/eckit/geo/grid/icon/17643da2574959b644d254a3cd6e2bc0-b0699f374c63d05028c18c12f80a48f4.ek?key=version&offset=0 was not completed.
ECCODES ERROR   :  GridSpec: Exception thrown (eckit::codec::InvalidRecord: version not found in record /Users/mojp/.local/share/eckit/geo/grid/icon/17643da2574959b644d254a3cd6e2bc0-b0699f374c63d05028c18c12f80a48f4.ek)
FAILED
tests/xr_engine/test_xr_engine_grid.py::test_xr_engine_grid_core[icon_ch1.grib2-dims19-coords19-False-True-False] FAILED

=============================================================================================== FAILURES ===============================================================================================
_____________________________________________ test_grib_latlon_various_grids_2[icon_ch1.grib2-expected_shape0-expected_lat0-expected_lon0-expected_area0] ______________________________________________

filename = 'icon_ch1.grib2', expected_shape = (1147980,), expected_lat = [50.24130630493164, 50.23910903930664, 50.23666763305664, 50.22787857055664]
expected_lon = [17.710683465003967, 17.689199090003967, 17.702382683753967, 17.704824090003967], expected_area = [50.6, -0.9, 41.9, 17.8]

    @pytest.mark.skipif(IN_GITHUB, reason="Skipping test on GitHub CI")
    @pytest.mark.parametrize(
        "filename,expected_shape, expected_lat, expected_lon, expected_area",
        [
            (
                "icon_ch1.grib2",
                (1147980,),
                [50.24130630493164, 50.23910903930664, 50.23666763305664, 50.22787857055664],
                [17.710683465003967, 17.689199090003967, 17.702382683753967, 17.704824090003967],
                [50.6, -0.9, 41.9, 17.8],
            ),
            (
                "eORCA025_T.grib",
                (
                    1442,
                    1207,
                ),
                [-89.5, -89.5, -89.5, -89.5],
                [72.75, 73.0, 73.25, 73.5],
                [90.0, 0.0, -90.0, 360.0],
            ),
        ],
    )
    def test_grib_latlon_various_grids_2(filename, expected_shape, expected_lat, expected_lon, expected_area):
        fl = earthkit.data.from_source("url", earthkit_remote_test_data_file("xr_engine/grid", filename)).to_fieldlist()
    
>       lat, lon = fl[0].geography.latlons()
                   ^^^^^^^^^^^^^^^

tests/grib/test_grib_geography.py:411: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/earthkit/data/core/field.py:580: in geography
    return self._components[_GEOGRAPHY].component
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/earthkit/data/field/handler/core.py:187: in __getattr__
    return getattr(self._handler, name)
                   ^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/earthkit/utils/decorators/_thread_handlers.py:54: in __get__
    value = self.method(instance)
            ^^^^^^^^^^^^^^^^^^^^^
src/earthkit/data/field/grib/core.py:34: in _handler
    return self.BUILDER.build(self.handle)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

handle = <earthkit.data.readers.grib.handle.ManagedGribHandle object at 0x116ba3a10>

    @staticmethod
    def build(handle):
        from earthkit.data.field.handler.geography import GeographyFieldComponentHandler
    
        grid_type = handle.get("gridType", None)
    
        # Spectral data is handled differently right now
        if grid_type == "sh":
            from earthkit.data.field.component.geography import SpectralGeography
    
            shape = (handle.get("numberOfDataPoints", None),)
            component = SpectralGeography(shape=shape)
        # "unstructured" grids are handled differently.
        # Currently, ecCodes cannot generate lat-lon for these grids, so we need to
        # rely on the gridSpec to reconstruct the grid with the eckit-geo Grid object.
        # If the gridSpec is not available, we cannot handle these grids.
        elif grid_type == "unstructured_grid":
            if not ECKIT_GRID_SUPPORT.has_ecc_grid_spec or not ECKIT_GRID_SUPPORT.has_grid:
                raise ValueError(
                    (
                        "GribGeographyBuilder: cannot use unstructured grid because eckit-geo"
                        " grid support is not available in ecCodes"
                    )
                )
            from earthkit.data.field.component.geography import GridsSpecBasedGeography
    
            grid_spec = handle.get("gridSpec", default=None)
            if isinstance(grid_spec, str) and grid_spec != "":
                component = GridsSpecBasedGeography(grid_spec)
            else:
>               raise ValueError(
                    (
                        "GribGeographyBuilder: cannot use unstructured grid because gridSpec"
                        "  is not available in the handle"
                    )
                )
E               ValueError: GribGeographyBuilder: cannot use unstructured grid because gridSpec  is not available in the handle

src/earthkit/data/field/grib/geography.py:233: ValueError
------------------------------------------------------------------------------------------ Captured log call -------------------------------------------------------------------------------------------
ERROR    earthkit.data.field.grib.core:core.py:36 GribGeographyBuilder: cannot use unstructured grid because gridSpec  is not available in the handle
Traceback (most recent call last):
  File "/Users/mojp/github/earthkit-data/src/earthkit/data/field/grib/core.py", line 34, in _handler
    return self.BUILDER.build(self.handle)
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/Users/mojp/github/earthkit-data/src/earthkit/data/field/grib/geography.py", line 233, in build
    raise ValueError(
    ...<4 lines>...
    )
ValueError: GribGeographyBuilder: cannot use unstructured grid because gridSpec  is not available in the handle
______________________________________________________________ test_xr_engine_grid_core[icon_ch1.grib2-dims19-coords19-False-True-False] _______________________________________________________________

allow_holes = False, lazy_load = True, file = 'icon_ch1.grib2', dims = {'values': 1147980}, coords = {'latitude': ['values', 1147980], 'longitude': ['values', 1147980]}, distinct_ll = False

    @pytest.mark.cache
    @pytest.mark.skipif(IN_GITHUB, reason="Skipping test on GitHub CI")
    # @pytest.mark.parametrize("allow_holes", [False, True])
    # @pytest.mark.parametrize("lazy_load", [True, False])
    @pytest.mark.parametrize("allow_holes", [False])
    @pytest.mark.parametrize("lazy_load", [True])
    @pytest.mark.parametrize(
        "file,dims,coords,distinct_ll",
        # grid_list(files=["eORCA025_T.grib"]),
        grid_list(),
    )
    def test_xr_engine_grid_core(allow_holes, lazy_load, file, dims, coords, distinct_ll):
        ds = from_source("url", earthkit_remote_test_data_file("xr_engine", "grid", file)).to_fieldlist()
    
        a = ds.to_xarray(profile="mars", allow_holes=allow_holes, lazy_load=lazy_load)
        assert a is not None
    
        for k, v in dims.items():
>           assert a.dims[k] == v
E           assert 0 == 1147980

tests/xr_engine/test_xr_engine_grid.py:55: AssertionError
=========================================================================================== warnings summary ===========================================================================================
.venv/lib/python3.13/site-packages/_pytest/python.py:124
  /Users/mojp/github/earthkit-data/.venv/lib/python3.13/site-packages/_pytest/python.py:124: PytestRemovedIn10Warning: Passing a non-Collection iterable to parametrize is deprecated.
  Test: tests/xr_engine/test_xr_engine_grid.py::test_xr_engine_grid_core, argvalues type: generator
  Please convert to a list or tuple.
  See https://docs.pytest.org/en/stable/deprecations.html#parametrize-iterators
    metafunc.parametrize(*marker.args, **marker.kwargs, _param_mark=marker)

tests/xr_engine/test_xr_engine_grid.py::test_xr_engine_grid_core[icon_ch1.grib2-dims19-coords19-False-True-False]
  /Users/mojp/github/earthkit-data/src/earthkit/data/core/caching.py:475: DeprecationWarning: The default datetime adapter is deprecated as of Python 3.12; see the sqlite3 documentation for suggested replacement recipes
    db.execute(

tests/xr_engine/test_xr_engine_grid.py::test_xr_engine_grid_core[icon_ch1.grib2-dims19-coords19-False-True-False]
  /Users/mojp/github/earthkit-data/tests/xr_engine/test_xr_engine_grid.py:55: FutureWarning: The return type of `Dataset.dims` will be changed to return a set of dimension names in future, in order to be more consistent with `DataArray.dims`. To access a mapping from dimension names to lengths, please use `Dataset.sizes`.
    assert a.dims[k] == v

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================================================= short test summary info ========================================================================================
FAILED tests/grib/test_grib_geography.py::test_grib_latlon_various_grids_2[icon_ch1.grib2-expected_shape0-expected_lat0-expected_lon0-expected_area0] - ValueError: GribGeographyBuilder: cannot use unstructured grid because gridSpec  is not available in the handle
FAILED tests/xr_engine/test_xr_engine_grid.py::test_xr_engine_grid_core[icon_ch1.grib2-dims19-coords19-False-True-False] - assert 0 == 1147980

Contributor Declaration

By opening this pull request, I affirm the following:

All authors agree to the Contributor License Agreement.
The code follows the project's coding standards.
I have performed self-review and added comments where needed.
I have added or updated tests to verify that my changes are effective and functional.
I have run all existing tests and confirmed they pass.

codecov-commenter · 2026-06-22T11:55:48Z

Codecov Report

❌ Patch coverage is 69.69697% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.39%. Comparing base (a0d6492) to head (7532d29).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
tests/high_level_object/test_hl_odb_core.py	50.00%	1 Missing and 1 partial ⚠️
tests/odb/test_odb_pandas.py	50.00%	1 Missing and 1 partial ⚠️
tests/odb/test_odb_url.py	50.00%	1 Missing and 1 partial ⚠️
tests/readers/test_geojson_reader.py	33.33%	1 Missing and 1 partial ⚠️
tests/readers/test_shapefile_reader.py	50.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1027      +/-   ##
===========================================
- Coverage    82.43%   82.39%   -0.04%     
===========================================
  Files          237      237              
  Lines        16683    16707      +24     
  Branches       806      811       +5     
===========================================
+ Hits         13752    13766      +14     
- Misses        2694     2699       +5     
- Partials       237      242       +5

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

raise supported python to match zarr>=3

f557231

colonesej requested a review from sandorkertesz June 22, 2026 11:04

colonesej added 2 commits June 22, 2026 12:05

handle optional dependencies in tests

689c295

code quality hooks

7532d29

sandorkertesz reviewed Jun 22, 2026

View reviewed changes

Comment thread pyproject.toml

colonesej closed this Jun 22, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

raise supported python to match zarr>=3#1027

raise supported python to match zarr>=3#1027
colonesej wants to merge 3 commits into
developfrom
fix/clarify_supported_python

colonesej commented Jun 22, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jun 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

colonesej commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

optional dependency -based tests

Local tests

Contributor Declaration

Uh oh!

codecov-commenter commented Jun 22, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

colonesej commented Jun 22, 2026 •

edited

Loading