Placeholders in config files via templating engine (e.g., jinja2)

[GeigerJ2]

The use case for this, at least for me, was being able to use the infrastructure we have for the tests (config YAML, scripts, ICON namelists, etc.) also to run examples manually via CLI. For this, some settings (e.g., the Computer) should be configurable without having to edit the YAML file, so the same config.yaml can be used for differently parametrized setups. Locally, I currently have this using environment variables as placeholders and then run.sh scripts to run workflows defined in tests/, e.g. my current DYAMOND dev-setup:

#!/bin/bash
# Wrapper script to run the DYAMOND_cwd01 workflow via CLI
# This sets the required environment variables and runs sirocco run

# Get the directory where this script is located
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Set environment variables
# Note: Paths must be relative to the config file location for Sirocco validation
export SIROCCO_COMPUTER="santis-async-ssh"
export SIROCCO_SCRIPTS_DIR="scripts"  # Relative to config file
export SLURM_ACCOUNT="cwd01"
# TODO: Passing via YAML config is overwritten by default value from CLI option
# Hence, need to pass explicitly via CLI here
export WINDOW_SIZE="0"

echo "Running DYAMOND workflow with:"
echo "  Computer: $SIROCCO_COMPUTER"
echo "  Config: ${SCRIPT_DIR}/config/config.yaml"
echo "  SLURM account: $SLURM_ACCOUNT"
echo ""

# Run sirocco with the config
sirocco run "${SCRIPT_DIR}/config/config.yaml" "--window-size" "${WINDOW_SIZE}" "$@"

and then in the Sirocco config.yaml:

# ...
tasks:
  - prepare_input:
      plugin: shell
      computer: ${SIROCCO_COMPUTER}
      account: ${SLURM_ACCOUNT}
      walltime: 00:05:00
      nodes: 1
      ntasks_per_node: 4
      cpus_per_task: 72
      path: ./scripts/prepare_input.sh
      command: ./prepare_input.sh --pool={PORT::data_pool}
# ...