Pelagos-py provides a flexible, modular pipeline framework for defining, executing, and visualising multi-step data-processing workflows for oceanographic data.
Each pipeline is composed of a series of steps that are automatically built from a central user-defined YAML configuration file. As Pelagos-py only depends on the config file to construct a pipeline, processing can be easily reproduced by others through sharing of configs.
The documentation for this package is available here.
Please note that the documentation is still under construction.
The Pipeline class orchestrates the flow of data through a sequence of modular "steps."
Each step performs a specific processing task (e.g., data loading, quality control, profile detection, export).
Key characteristics:
| Component | Description |
|---|---|
| Configuration-driven | Users define the workflow in a YAML file describing each step, its parameters, and diagnostics options. |
| Built-in Quality Control | Pre-built quality control tests can be specified in the config by the user to flag bad data. |
| Diagnostics | Where possible, data can be visualized to see the effect of each component of the pipeline. |
The easiest way to install Pelagos-py is via pip:
pip install pelagos-py
Alternatively, you can install directly from the source using git:
git clone [https://github.com/NOC-OBG-Autonomy/pelagos-py.git](https://github.com/NOC-OBG-Autonomy/pelagos-py.git)
cd pelagos-py
# create/activate a virtual environment
pip install -e .
See Getting Started for more details.
Import the 'Pipeline' class and create a pipeline, either by letting pelagos-py build a config for your OG1 file or from a config you already have:
from pelagos_py import Pipeline
# Build a config for the file from the default template, saved as glider.yaml next to it
pipeline = Pipeline.make_config("glider.nc")
# Same, but ask about each choice (e.g. what to do about missing PAR) instead of using the defaults
pipeline = Pipeline.make_config("glider.nc", ask=True)
# Or load a config you already have, e.g. glider.yaml after editing it
pipeline = Pipeline.load_config("my_pipeline.yaml")To try it without your own data, get_demo_file downloads a demo OG1 file into examples/data/OG1 (only the first time) and returns its path; get_demo_file() lists the demos. examples/python/run_demo.py does this end to end.
from pelagos_py import Pipeline, get_demo_file
pipeline = Pipeline.make_config(get_demo_file("nelson_646_r"))Running the pipeline executes each step defined by the config in order
results = pipeline.run()Steps can optionally include diagnostic plots or summaries by setting:
diagnostics: true
The entire pipeline configuration can be exported to a YAML file for reproducibility:
pipeline.export_config("exported_pipeline.yaml")An example YAML configuration for a simple pipeline. See examples/notebooks/pipeline_demo.ipynb for a full demo.
# Pipeline Configuration
pipeline:
name: Example CTD Processing Pipeline
description: A pipeline for processing CTD data
steps:
- name: Load OG1
parameters:
file_path: ../examples/data/OG1/Nelson_646_R.nc # Path to the input NetCDF file
diagnostics: false
- name: Derive CTD
parameters:
to_derive: [
DEPTH,
PRAC_SALINITY,
ABS_SALINITY,
CONS_TEMP,
DENSITY
]
diagnostics: false
- name: "Data Export"
parameters:
export_format: "netcdf"
output_path: "../examples/data/OG1/Nelson_646_R_Processed.nc"
A full breakdown can be found here: Developer Guide.
Copyright 2025-2026 The National Oceanography Centre and contributors.