Merge main into develop

.gitignore

@@ -220,6 +220,13 @@ docs/source/experimental/*.pp*
 docs/source/experimental/*.png*
 docs/source/experimental/*.nc*
 docs/source/experimental/*.geojson*
+ocs/_build/
+docs/source/tutorials/*.tar
+docs/source/tutorials/*.zip
+docs/source/tutorials/_*
+docs/source/tutorials/earthkit_use_cases/*.grib
+docs/source/tutorials/*/_*
+
 
 autodocs/
 

README.md

@@ -10,7 +10,7 @@
     <img src="https://github.com/ecmwf/codex/raw/refs/heads/main/ESEE/foundation_badge.svg" alt="ECMWF Software EnginE">
   </a>
   <a href="https://github.com/ecmwf/codex/raw/refs/heads/main/Project Maturity">
-    <img src="https://github.com/ecmwf/codex/raw/refs/heads/main/Project Maturity/incubating_badge.svg" alt="Maturity Level">
+    <img src="https://github.com/ecmwf/codex/raw/refs/heads/main/Project Maturity/graduated_badge.svg" alt="Maturity Level">
   </a>
   <!-- <a href="https://codecov.io/gh/ecmwf/earthkit-data">
     <img src="https://codecov.io/gh/ecmwf/earthkit-data/branch/main/graph/badge.svg" alt="Code Coverage">
@@ -35,7 +35,7 @@
 </p>
 
 > \[!IMPORTANT\]
-> This software is **Incubating** and subject to ECMWF's guidelines on [Software Maturity](https://github.com/ecmwf/codex/raw/refs/heads/main/Project%20Maturity).
+> This software is **Graduated** and subject to ECMWF's guidelines on [Software Maturity](https://github.com/ecmwf/codex/raw/refs/heads/main/Project%20Maturity).
 
 **earthkit-data** is a format-agnostic interface for geospatial data with a focus on meteorology and
 climate science. It is the data handling component of [earthkit](https://github.com/ecmwf/earthkit).

docs/source/concepts/caching.rst

@@ -40,7 +40,7 @@ The cache location can be read and modified with Python (see the details below).
 
 .. tip::
 
-   See the :ref:`/how-tos/misc/cache.ipynb` notebook for examples.
+   See the :ref:`/tutorials/misc/cache.ipynb` notebook for examples.
 
 .. note::
 

docs/source/concepts/config.rst

@@ -11,8 +11,8 @@ The configuration can be accessed and modified from Python. The configuration op
 
 See the following notebooks for examples:
 
-    - :ref:`/how-tos/misc/config.ipynb`
-    - :ref:`/how-tos/misc/config_env_vars.ipynb`
+    - :ref:`/tutorials/misc/config.ipynb`
+    - :ref:`/tutorials/misc/config_env_vars.ipynb`
 
 
 .. _config_get:
@@ -138,7 +138,7 @@ Finally, unset the environment variable and check the config value again, which
 
 See also the following notebook:
 
-    - :ref:`/how-tos/misc/config_env_vars.ipynb`
+    - :ref:`/tutorials/misc/config_env_vars.ipynb`
 
 
 .. _config_table:

docs/source/concepts/data.rst

@@ -87,7 +87,7 @@ To access the stream we need to convert the data into a stream fieldlist (GRIB)
 .. code-block:: python
 
     >>> import earthkit.data as ekd
-    >>> url = "https://sites.ecmwf.int/repository/earthkit-data/how-tos/test4.grib"
+    >>> url = "https://sites.ecmwf.int/repository/earthkit-data/tutorials/test4.grib"
     >>> ds = ekd.from_source("url", url, stream=True)
     >>> fl = ds.to_fieldlist()
     >>> for f in fl:
@@ -102,10 +102,10 @@ To access the stream we need to convert the data into a stream fieldlist (GRIB)
 Examples
 ////////////////
 
-    - :ref:`/how-tos/source/data_from_stream.ipynb`
-    - :ref:`/how-tos/source/file_stream.ipynb`
-    - :ref:`/how-tos/source/fdb.ipynb`
-    - :ref:`/how-tos/source/url_stream.ipynb`
+    - :ref:`/tutorials/source/data_from_stream.ipynb`
+    - :ref:`/tutorials/source/file_stream.ipynb`
+    - :ref:`/tutorials/source/fdb.ipynb`
+    - :ref:`/tutorials/source/url_stream.ipynb`
 
 
 Special cases

docs/source/concepts/encoders/encoders.rst

@@ -0,0 +1,158 @@
+.. _encoders:
+
+Encoders
+========
+
+An **encoder** converts data into a serialisable format that can be written to a
+:ref:`target <targets>`. Encoders are most commonly used implicitly through
+:func:`~earthkit.data.to_target`, but can also be created and used directly.
+
+.. note::
+
+   The encoders are still under development. The most advanced encoder is the :ref:`grib_encoder`, which can handle a variety of data types. Currently, the other encoders only support trivial encoding of data into a serialisable format. For example, the NetCDF encoder simply calls :py:func:`xarray.to_netcdf` on an xarray Dataset/DataArray.
+
+Creating an encoder
+-------------------
+
+Use :func:`~earthkit.data.encoders.create_encoder` to instantiate an encoder by name:
+
+.. code-block:: python
+
+    >>> from earthkit.data import create_encoder
+    >>> enc = create_encoder("grib")
+    >>> enc = create_encoder("grib", template=my_field, shortName="2t")
+
+.. py:function:: create_encoder(name, *args, **kwargs)
+
+   Create an encoder identified by *name*.
+
+   :param str name: Name of an encoder (see table below).
+   :param args: Positional arguments forwarded to the encoder constructor.
+   :param kwargs: Keyword arguments forwarded to the encoder constructor.
+   :rtype: Encoder
+
+
+Common constructor parameters
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+All encoders accept the following keyword arguments in their constructors:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 80
+
+   * - Parameter
+     - Description
+   * - ``template``
+     - A default template object used as the basis for encoding. The exact accepted types depend on the encoder.
+   * - ``metadata``
+     - A dictionary of default metadata applied to every encoded item. Metadata supplied
+       directly to :meth:`Encoder.encode` is merged on top, with the per-call values taking
+       precedence.
+   * - ``**kwargs``
+     - Any additional keyword arguments are interpreted as metadata keys and merged into
+       ``metadata``.
+
+
+Encoding data
+-------------
+
+All encoders expose an :meth:`encode` method:
+
+.. code-block:: python
+
+    >>> enc = create_encoder("grib", template=field)
+    >>> result = enc.encode(values=arr, shortName="2t", step=6)
+    >>> result.to_file(open("out.grib", "wb"))
+
+.. py:method:: Encoder.encode(data=None, values=None, metadata={}, template=None, check_nans=False, missing_value=9999, target=None, **kwargs)
+
+   Encode *data* and return an :class:`EncodedData` object.
+
+   :param data: Earthkit data object to encode (Field, FieldList, xarray Dataset/DataArray, …).
+   :param values: Raw values array to encode.
+   :param dict metadata: Per-call metadata merged on top of the encoder's default metadata.
+   :param template: Per-call template overriding the encoder's default template.
+   :param bool check_nans: Replace ``NaN`` values with *missing_value* before encoding.
+   :param missing_value: Replacement value used when *check_nans* is ``True``.
+   :param target: Target object; some encoders use this to optimise the output format.
+   :rtype: EncodedData
+
+
+Automatic encoder selection
+----------------------------
+
+When writing to a file target without specifying an encoder explicitly, the encoder is
+selected automatically from the file extension:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 70
+
+   * - Extension(s)
+     - Encoder
+   * - ``.grib``, ``.grb``, ``.grib1``, ``.grib2``, ``.grb1``, ``.grb2``
+     - ``grib``
+   * - ``.nc``, ``.nc3``, ``.nc4``, ``.netcdf``
+     - ``netcdf``
+   * - ``.tiff``, ``.tif``
+     - ``geotiff``
+   * - ``.bufr``
+     - ``bufr``
+   * - ``.odb``
+     - ``odb``
+
+
+Available encoders
+-----------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 16 52 32
+
+   * - Name
+     - Description
+     - Class
+   * - ``grib``
+     - Encode data as GRIB. Supports Fields, FieldLists, and NumPy arrays. Requires a
+       template or sufficient metadata to construct a valid GRIB message.
+     - :py:class:`~earthkit.data.encoders.grib.GribEncoder`
+   * - ``netcdf``
+     - Encode data as NetCDF via xarray. Accepts Fields, FieldLists, and xarray
+       Datasets/DataArrays.
+     - :py:class:`~earthkit.data.encoders.netcdf.NetCDFEncoder`
+   * - ``geotiff``
+     - Encode data as GeoTIFF. Requires geo-referenced raster data.
+     - :py:class:`~earthkit.data.encoders.geotiff.GeoTIFFEncoder`
+   * - ``csv``
+     - Encode tabular data as comma-separated values.
+     - :py:class:`~earthkit.data.encoders.csv.CSVEncoder`
+   * - ``bufr``
+     - Encode data as BUFR (Binary Universal Form for the Representation of meteorological
+       data).
+     - :py:class:`~earthkit.data.encoders.bufr.BufrEncoder`
+   * - ``odb``
+     - Encode data as ODB (Observational DataBase) format.
+     - :py:class:`~earthkit.data.encoders.odb.ODBEncoder`
+   * - ``zarr``
+     - Encode data as a Zarr array store.
+     - :py:class:`~earthkit.data.encoders.zarr.ZarrEncoder`
+   * - ``covjson``
+     - Encode data as CoverageJSON (CovJSON).
+     - :py:class:`~earthkit.data.encoders.covjson.CovJsonEncoder`
+   * - ``geojson``
+     - Encode data as GeoJSON.
+     - :py:class:`~earthkit.data.encoders.geojson.GeoJsonEncoder`
+   * - ``pp``
+     - Encode data as Met Office PP (UM binary) format.
+     - :py:class:`~earthkit.data.encoders.pp.PPEncoder`
+   * - ``text``
+     - Encode data as plain text.
+     - :py:class:`~earthkit.data.encoders.text.TextEncoder`
+
+
+
+Examples
+--------
+
+- :ref:`/tutorials/target/grib_encoder.ipynb`