Action for generate CI matrix

[samuelcolvin]

I maintain three packages that use maturin-action, and it's great.

The hardest part is setting up CI to generate all the right binaries.

I propose we create a separate action to generate the github actions matrix with the right variables set for maturin-action to use to generate binaries.

The list-python-dependencies package, and in particular it's readme shows the workflow for this:

• separate job to generate matrix info
• the action outputs a pile of JSON defining the jobs to run and variables required for each job, specifically you need to write to the file defined by the GITHUB_OUTPUT env var - code example here
• the first job sets outputs: to include that output
• second job defines matrix to read that JSON - in the example case PYTHON_DEPENDENCY_CASE: ${{ fromJSON(needs.find_dependency_cases.outputs.PYTHON_DEPENDENCY_CASES) }}
• you then define the maturin-action in the steps for that job

We could have switches to exclude/skip some targets, much like cibuildwheel offers, and a switch for abi3 etc.

A big advantages of this would be that without any extra work from maintainers, more and more targets could be added as maturin-action supports them - or remove if there are known issues.

Example current matrix sections:

• rtoml
• watchfiles (abi3)
• pydantic-core

Demo of how this might finally look in a workflow yaml file:

jobs:
  generate_build_matrix:
    runs-on: ubuntu-latest

    outputs:
      MATURIN_MATRIX: ${{ steps.generate-matrix.outputs.MATURIN_MATRIX }}

    steps:
      - uses: actions/checkout@v3
      - uses: pyo3/maturin-generate-matrix-action@main
        id: generate-matrix
        with:
          exclude: 'win-pypy-*'

  build:
    needs:
      - generate_build_matrix

    strategy:
      # not sure if this actually works or if you need to use a nested matrix namepace as the package above does
      matrix: ${{ fromJSON(needs.generate_build_matrix.outputs.MATURIN_MATRIX) }}

    name: ${{ matrix.name }}
    runs-on: ${{ matrix.runs-on }}

    steps:
      - uses: actions/checkout@v3

      - name: set up python
        uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.python-version }}
          architecture: ${{ matrix.python-architecture }}
          
      - run: ...install stuff...

      - name: build
        uses: PyO3/maturin-action@v1
        with:
          command: ${{ matrix.command }}
          target: ${{ matrix.target }}
          manylinux: ${{ matrix.manylinux }}
          container: ${{ matrix.container }}
          args: ${{ matrix.args }}
          # on one combined line:
          matrix_action_config: ${{ matrix.matrix-action-config }}

      - run: twine check --strict ${{ matrix.out-dir }}/*

      - uses: actions/upload-artifact@v3
        with:
          name: pypi_files
          path: ${{ matrix.out-dir }}

[samuelcolvin]

@messense do you think you will be able to work on this?

Otherwise, there's a chance the pydantic team might be able to.

@davidhewitt do you have and thoughts on this?

[messense]

do you think you will be able to work on this?

I'm a bit busy recently, help appreciated!

[djc]

I think this is done now? At least there's a generate-ci command advertised in the README.

[davidhewitt]

I think generate-ci creates a full github.yml file to commit to the repo. It think we want something similar that you can call in your github action to generate just the build matrix.