reusable-workflows

A collection of reusable GitHub workflows for ECMWF repositories.

Workflows

• ci.yml: Continuous Integration workflow for ecbuild/CMake-based projects
• ci-hpc.yml: Continuous Integration workflow for ecbuild/CMake-based projects on HPC
• ci-python.yml: Continuous Integration and Continuous Deployment workflow for Python-based projects
• ci-node.yml: Continuous Integration workflow for NodeJS-based projects
• docs.yml: Workflow for testing Sphinx-based documentation
• build-and-push-image.yml: Build and optionally push a Docker image to a container registry
• qa-precommit-run.yml: Runs the pre-commit hooks on all files server-side as a QA drop-in.
• qa-pytest-pyproject.yml: Runs pytest after a pyproject.toml install with a markdown report.
• ci-rust.yml: Continuous Integration workflow for Rust crates and workspaces
• publish-rust-crate.yml: Workflow for publishing Rust crates to crates.io
• sync.yml: Workflow for syncing a Git repository

Composite Actions

• configure-git-auth: Configure git to use a PAT for accessing private GitHub repositories

Samples

Supported Operating Systems

• Linux
• macOS

ci.yml

Usage

jobs:
  # Calls a reusable CI workflow to build & test current repository.
  #   It will pull in all needed dependencies and produce a code coverage report on success.
  #   In case the job fails, a message will be posted to a Microsoft Teams channel.
  ci:
    name: ci
    uses: ecmwf/reusable-workflows/.github/workflows/ci.yml@v1
    with:
      codecov_upload: true
      notify_teams: true
      build_package_inputs: |
        dependencies: |
          ecmwf/ecbuild
          ecmwf/eckit
        dependency_branch: develop
    secrets:
      incoming_webhook: ${{ secrets.MS_TEAMS_INCOMING_WEBHOOK }}

Inputs

skip_matrix_jobs

A list of matrix jobs to skip. Job names should be the full form of <compiler>@<platform>. Default: '' Type: string

deps_cache_key

Dependency cache key to restore from. Note that the key should be platform agnostic, as the <compiler>@<platform> suffix will be automatically appended. Upon extraction, a file called .env from the cache root directory will be loaded into the build environment, if it exists. Default: '' Type: string

deps_cache_path

Optional dependency cache path to restore to, falls back to ${{ runner.temp }}/deps. Will be considered only if deps_cache_key is supplied. Default: '' Type: string

codecov_upload

Whether to generate and upload code coverage to codecov service for main branches. Default: false Type: boolean

notify_teams

Whether to notify about workflow status via Microsoft Teams. Note that you must supply incoming_webhook secret if you switch on this feature. Default: false Type: boolean

repository

The source repository name, in case it differs from the current one. Repository names should follow the standard Github owner/name format. Default: ${{ github.repository }} Type: string

ref

The source repository reference, in case it differs from the current one. Default: ${{ github.ref }} Type: string

build_package_inputs

Optional inputs for the build-package action, provided as a YAML object value. Default: '' Type: string

Secrets

incoming_webhook

Public URL of the Microsoft Teams incoming webhook. To get the value, make sure that channel in Teams has the appropriate connector set up. It will only be used if notify_teams input is switched on. Example: https://webhook.office.com/webhookb2/...

ci-hpc.yml

Usage

jobs:
  ci-hpc:
    name: ci-hpc
    uses: ecmwf/reusable-workflows/.github/workflows/ci-hpc.yml@v2
    with:
      name-prefix: metkit-
      build-inputs: |
        --package: ecmwf/metkit@develop
        --modules: |
          ecbuild
          ninja
        --dependencies: |
          ecmwf/eccodes@develop
          ecmwf/eckit@develop
        --parallel: 64
    secrets: inherit\

Inputs

name-prefix

Job name prefix. Usually the package name. Suitable when building multiple packages within one workflow to easily differentiate between them. Default: '' Type: string

build-inputs

Inputs for build-package-hpc action. Default: '' Type: string

dev-runner

Whether to build using development runner which contains latest version of build-package-hpc. Default: '' Type: string

ci-python.yml

Usage

jobs:
  # Calls a reusable CI workflow to qa, test & deploy the current repository.
  #   It will pull in all needed dependencies and produce a code coverage report on success.
  #   If all checks were successful and a new release tag pushed, the package will be published on PyPI.
  #   In case the job fails, a message will be posted to a Microsoft Teams channel.
  ci:
    name: ci
    uses: ecmwf/reusable-workflows/.github/workflows/ci-python.yml@v1
    with:
      codecov_upload: true
      notify_teams: true
      build_package_inputs: |
        dependencies: |
          ecmwf/ecbuild
          ecmwf/eckit
          ecmwf/odc
        dependency_branch: develop
        self_build: false
    secrets:
      pypi_username: ${{ secrets.PYPI_USERNAME }}
      pypi_password: ${{ secrets.PYPI_PASSWORD }}
      incoming_webhook: ${{ secrets.MS_TEAMS_INCOMING_WEBHOOK }}

Inputs

skip_matrix_jobs

A list of matrix jobs to skip. Job names should be the full form of <compiler>@<platform>. Default: '' Type: string

codecov_upload

Whether to generate and upload code coverage to codecov service for main branches. Default: false Type: boolean

notify_teams

Whether to notify about workflow status via Microsoft Teams. Note that you must supply incoming_webhook secret if you switch on this feature. Default: false Type: boolean

python_version

The version of Python binary to use. Default: '3.9' Type: ring

upload_extra_artifact

Upload a file or directory specified in this argument as an artifact. Default: none Type: string

repository

The source repository name. Repository names should follow the standard Github owner/name format. Default: ${{ github.repository }} Type: string

ref

The source repository reference. Default: ${{ github.ref }} Type: string

build_package_inputs

Optional inputs for the build-package action, provided as a YAML object value. Default: '' Type: string

Secrets

pypi_username

Username of the PyPI account. The account must have sufficient permissions to deploy the current project. Example: MyUsername

pypi_password

Password of the PyPI account. Example: MyPassword

incoming_webhook

Public URL of the Microsoft Teams incoming webhook. To get the value, make sure that channel in Teams has the appropriate connector set up. It will only be used if notify_teams input is switched on. Example: https://webhook.office.com/webhookb2/...

ci-node.yml

Usage

jobs:
  # Calls a reusable CI NodeJS workflow to qa & test & deploy the current repository.
  #   It will install dependencies and produce a code coverage report on success.
  #   In case the job fails, a message will be posted to a Microsoft Teams channel.
  ci:
    name: ci
    uses: ecmwf/reusable-workflows/.github/workflows/ci-node.yml@v1
    with:
      codecov_upload: true
      notify_teams: true
    secrets:
      incoming_webhook: ${{ secrets.MS_TEAMS_INCOMING_WEBHOOK }}

Inputs

skip_matrix_jobs

A list of matrix jobs to skip. Job names should be the form of <platform>. Default: '' Type: string

codecov_upload

Whether to generate and upload code coverage to codecov service for main branches. Default: false Type: boolean

notify_teams

Whether to notify about workflow status via Microsoft Teams. Note that you must supply incoming_webhook secret if you switch on this feature. Default: false Type: boolean

self_build

Whether to build from currently checked out repository or not. Default: true Type: boolean

self_test

Whether to run tests from currently checked out repository or not. Default: true Type: boolean

node_version

The version of NodeJS interpreter to use. Default: '16' Type: string

repository

The source repository name. Repository names should follow the standard Github owner/name format. Default: ${{ github.repository }} Type: string

ref

The source repository reference. Default: ${{ github.ref }} Type: string

Secrets

incoming_webhook

Public URL of the Microsoft Teams incoming webhook. To get the value, make sure that channel in Teams has the appropriate connector set up. It will only be used if notify_teams input is switched on. Example: https://webhook.office.com/webhookb2/...

ci-rust.yml

Continuous Integration workflow for Rust crates and Cargo workspaces. Runs cargo fmt --check, cargo clippy, cargo doc and cargo test against a configurable matrix of toolchains, runner images and workspace directories. Default clippy invocation enforces the ECMWF Rust lint set: -W clippy::all -W clippy::pedantic -W clippy::nursery -W clippy::unwrap_used -A clippy::module_name_repetitions -A clippy::missing_errors_doc -D warnings.

Usage

on:
  push:
    branches: [main]
  pull_request:

jobs:
  ci:
    uses: ecmwf/reusable-workflows/.github/workflows/ci-rust.yml@v2
    with:
      system-dependencies: cmake

Multi-axis matrix example:

jobs:
  ci:
    uses: ecmwf/reusable-workflows/.github/workflows/ci-rust.yml@v2
    with:
      system-dependencies: cmake
      toolchains: |
        stable
        1.90.0
      workspaces: |
        .
        crates/subcrate
    secrets:
      private_repos_token: ${{ secrets.GH_REPO_READ_TOKEN }}

Inputs

skip_matrix_jobs

Newline-separated list of matrix cells to skip. Names follow <toolchain>@<os>@<workspace>. Default: '' Type: string

toolchains

Newline-separated list of Rust toolchains for the test matrix. Default: stable Type: string

os-list

Newline-separated list of runner images for the test matrix. Default: ubuntu-latest Type: string

workspaces

Newline-separated list of working directories to test. Use . for a single-workspace repo. Default: . Type: string

manifest-path

Path to Cargo.toml (relative to each workspace). Default: Cargo.toml Type: string

features

Feature selection expression passed to clippy and test. Default: --all-features Type: string

locked

Add --locked to cargo clippy and cargo test to enforce Cargo.lock. Default is false because library crates (the dominant consumer pattern) typically do not commit Cargo.lock; set true for binary/app crates that do. Default: false Type: boolean

system-dependencies

Space-separated apt packages installed before cargo runs (Linux runners only). Default: '' Type: string

clippy-args

Arguments appended to cargo clippy. Default is the ECMWF Rust enforcement set (see above). Pass an empty string to drop denials. Default: '-- -W clippy::all -W clippy::pedantic -W clippy::nursery -W clippy::unwrap_used -A clippy::module_name_repetitions -A clippy::missing_errors_doc -D warnings' Type: string

test-args

Extra args appended to cargo test. Default: --workspace Type: string

doc-args

Extra args appended to cargo doc. Default: --workspace Type: string

rustdocflags

Value exported as RUSTDOCFLAGS during cargo doc. Default: -D warnings Type: string

run-doc

Run cargo doc as part of the qa job. Default: true Type: boolean

run-integration

Run a separate integration test job on the main branch only. Default: false Type: boolean

integration-test-args

Args passed to cargo test in the integration job (e.g. --test foo -- --ignored --test-threads=1). Default: '' Type: string

integration-workspace

Working directory for the integration job. Default: . Type: string

repository

The source repository name, in case it differs from the current one. Default: ${{ github.repository }} Type: string

ref

The source repository reference. Default: ${{ github.ref }} Type: string

notify_teams

Notify a Microsoft Teams channel of the workflow status. Default: false Type: boolean

Secrets

private_repos_token

GitHub token used to rewrite ssh://git@github.com URLs to HTTPS for private repository dependencies. Also exports CARGO_NET_GIT_FETCH_WITH_CLI=true. Example: ${{ secrets.GH_REPO_READ_TOKEN }}

incoming_webhook

Microsoft Teams incoming webhook URL. Only used if notify_teams is true.

Composite action

The same fmt/clippy/test/doc pipeline is also exposed as a composite action at ecmwf/reusable-workflows/ci-rust@v2 for use inside custom workflows:

jobs:
  custom:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: ecmwf/reusable-workflows/ci-rust@v2
        with:
          manifest-path: crates/foo/Cargo.toml
          run-doc: 'false'

build-and-push-image.yml

Build and Push Usage

jobs:
  build-image:
    uses: ecmwf/reusable-workflows/.github/workflows/build-and-push-image.yml@v2
    with:
      registry: ghcr.io
      image_repository: my-org/my-app
      environment_file: environments/prod.env
      build_args: |
        environment=prod
    secrets:
      registry_username: ${{ secrets.REGISTRY_USERNAME }}
      registry_password: ${{ secrets.REGISTRY_PASSWORD }}

Build and Push Inputs

• repository: Source repository name. Default: ${{ github.repository }}. Type: string.
• ref: Source repository reference. Default: ${{ github.ref }}. Type: string.
• registry: Container registry hostname. Type: string.
• image_repository: Image repository path relative to the registry, without the tag. Type: string.
• image_tag: Image tag to publish. Defaults to the checked out commit SHA when unset. Default: ''. Type: string.
• environment_file: Optional path to a file containing environment variables to load during the build. Default: ''. Type: string.
• context: Docker build context. Default: '.'. Type: string.
• dockerfile: Path to the Dockerfile, relative to the repository root. Default: './Dockerfile'. Type: string.
• platforms: Target platforms for the Docker build. Default: 'linux/amd64'. Type: string.
• push: Whether to push the built image to the registry. Default: true. Type: boolean.
• build_args: Optional newline-separated Docker build arguments in key=value form. Default: ''. Type: string.

Build and Push Outputs

• image: Fully qualified image reference including tag.
• image_tag: Image tag used for the build.

Build and Push Secrets

• registry_username: Optional username for registry authentication.
• registry_password: Optional password or token for registry authentication.

docs.yml

Usage

jobs:
  # Calls a reusable CI workflow to build & check the documentation in the current repository.
  #   It will install required system dependencies and test Read the Docs build process.
  docs:
    name: docs
    uses: ecmwf/reusable-workflows/.github/workflows/docs.yml@v1
    with:
      system_dependencies: doxygen pandoc

Inputs

requirements_path

Path of the requirements.txt file which includes all dependencies needed for building of the documentation, relative to the repository root. Default: docs/requirements.txt Type: string

docs_path

Path of the documentation directory, relative to the repository root. Default: docs Type: string

system_dependencies

Optional list of system dependencies to install via apt command, separated by spaces. Note that each dependency must be available via standard Ubuntu 20.04 package repositories. Default: '' Type: string

python_version

The version of Python binary to use. Default: '3.9' Type: ring

repository

The source repository name, in case it differs from the current one. Repository names should follow the standard Github owner/name format. Default: ${{ github.repository }} Type: string

ref

The source repository reference, in case it differs from the current one. Default: ${{ github.ref }} Type: string

qa-precommit

Usage

on:
  push:
  pull_request_target:
    types: [opened, synchronize, reopened]

jobs:
  quality:
    uses: ecmwf/reusable-workflows/.github/workflows/qa-precommit-run.yml@v2
    with:
      skip-hooks: "no-commit-to-branch"

Inputs

python-version

Optional, pin to use a specific Python version Default: '3.x' Type: string Example: '3.9'

skip-hooks

Optional, a comma-separated string of pre-commit hooks to skip. Default: '' Type: string Example: 'no-commit-to-branch,black'

qa-pytest-pyproject

Usage

on:
  push:
  pull_request_target:
    types: [opened, synchronize, reopened]

jobs:
  checks:
    strategy:
      matrix:
        python-version: ["3.9", "3.10"]
    uses: ecmwf/reusable-workflows/.github/workflows/qa-pytest-pyproject.yml@v2
    with:
      python-version: ${{ matrix.python-version }}

Inputs

python-version

Optional, the version of Python binary to use.

Default: '3.x' Type: string Example: '3.9'

optional-dependencies

Optional, optional dependencies to install from package to be tested.

Default: 'all,tests' Type: string Example: dev,tests,third_dependency_group

install-dependencies

Optional, perform or skip dependency installation.

This enables the external installation of dependencies.

Default: true Type: boolean Example: false

skip-tests

Optional, the pytest marks to pass to the -m command flag.

Expands in pytest -v -m "${{ inputs.skip-tests }}" to coordinate marked tests.

Default: '' Type: string Example: 'no gpu'

job-summary

Default: false Type: boolean Example: true

emoji

Default: true Type: boolean Example: false

custom-pytest

Default: 'pytest' Type: string Example: 'poetry run pytest'

publish-rust-crate.yml

Usage

on:
  release:
    types: [published]
  workflow_dispatch:

jobs:
  publish:
    uses: ecmwf/reusable-workflows/.github/workflows/publish-rust-crate.yml@v2
    with:
      manifest-path: Cargo.toml
      dry-run: false
    secrets:
      cargo_registry_token: ${{ secrets.CARGO_REGISTRY_TOKEN }}

Inputs

manifest-path

Path to Cargo.toml for the crate to publish. Default: 'Cargo.toml' Type: string

toolchain

Rust toolchain to install. Default: 'stable' Type: string

dry-run

Run cargo publish in dry-run mode only. Default: false Type: boolean

run-checks

Run cargo fmt, cargo clippy and cargo test before publishing. Default: true Type: boolean

locked

Use --locked flag for cargo clippy and cargo test to enforce Cargo.lock. Default: true Type: boolean

cargo-publish-args

Extra arguments appended to cargo publish. Default: '' Type: string

environment

GitHub deployment environment for publish protection. When set, the publish job will require the configured environment approvals and rules before running. Default: '' Type: string Example: 'release'

Secrets

cargo_registry_token

Required crates.io API token used by cargo publish. Example: ${{ secrets.CARGO_REGISTRY_TOKEN }}

sync.yml

Usage

# Controls when the workflow will run
on:
  # Trigger the workflow on all pushes
  push:
    branches:
      - "**"
    tags:
      - "**"

  # Trigger the workflow when a branch or tag is deleted
  delete: ~

jobs:
  # Calls a reusable CI workflow to sync the current with a remote repository.
  #   It will correctly handle addition of any new and removal of existing Git objects.
  sync:
    name: sync
    uses: ecmwf/reusable-workflows/.github/workflows/sync.yml@v1
    secrets:
      target_repository: ${{ secrets.BITBUCKET_REPOSITORY }}
      target_username: ${{ secrets.BITBUCKET_USERNAME }}
      target_token: ${{ secrets.BITBUCKET_PAT }}

Inputs

sync_repository_inputs

Optional inputs for the sync-repository action, provided as a YAML object value. Note that some values may be overwritten by provided secrets with same name. Default: '' Type: string

Secrets

source_token

The user access token with read access to the source repository, must be URL-encoded. Default: github.token Example: ...

target_repository

Required The name of the target repository. Example: project-name/repo-name

target_username

Required The user login with write access to the target repository, must be URL-encoded. Example: user

target_token

Required The user access token with write access to the target repository, must be URL-encoded. Example: ...

label-pr.yml

Manages labels on public pull requests. contributor label is added when a PR from public fork is opened. Removes label approved-for-ci when pull request HEAD changes.

Usage

on:
  # trigger the pull request is opened or pushed to
  pull_request_target:
    types: [opened, synchronize]

jobs:
  label:
    uses: ecmwf/reusable-workflows/.github/workflows/pr-label.yml@v2

configure-git-auth

A composite action that configures git to authenticate with GitHub using a Personal Access Token (PAT). It rewrites both HTTPS and SSH github.com URLs so that all subsequent git operations in the same job use token-based HTTPS authentication. This is useful when jobs need to install dependencies from private GitHub repositories.

Usage

steps:
  - uses: actions/checkout@v4
  - uses: ecmwf/reusable-workflows/configure-git-auth@v2
    with:
      token: ${{ secrets.MY_GITHUB_PAT }}
  # subsequent steps can now access private GitHub repos

Inputs

token

GitHub PAT or token for private repository access. Required: true

Development

Install Dependencies

npm install

Lint Code

npm run lint

Action Pinning

This repository uses pinact to pin GitHub Actions and reusable workflows to their commit SHA for security. Actions from the ecmwf organisation and files under sync-files/ are excluded from pinning.

Running pinact

To pin actions in workflow files and documentation:

pinact run

To update pinned actions to their latest versions (skipping versions released less than 7 days ago):

pinact run -u --min-age 7

For installation instructions, see the pinact documentation.

Licence

This software is licensed under the terms of the Apache License Version 2.0 which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.

In applying this licence, ECMWF does not waive the privileges and immunities granted to it by virtue of its status as an intergovernmental organisation nor does it submit to any jurisdiction.