dynamically test with all supported Python versions

[zacharyburnett]

uses fill: to fill in toxenvs py311-xdist, py312-xdist, py313-xdist, py314-xdist (reading requires-python: from pyproject.toml)

analogous to spacetelescope/romancal#2256

Tasks

• If you have a specific reviewer in mind, tag them.
• add a build milestone, i.e. Build 12.0 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see changelog readme for instructions)
    • if your change breaks step-level or public API (as defined in the docs), also add a changes/<PR#>.breaking.rst news fragment
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.58%. Comparing base (6f21fa1) to head (db8eb1f).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10428      +/-   ##
==========================================
- Coverage   86.59%   86.58%   -0.01%     
==========================================
  Files         374      374              
  Lines       40471    40459      -12     
==========================================
- Hits        35047    35033      -14     
- Misses       5424     5426       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[zacharyburnett]

these run devdeps tests on the latest available Python version

[tapastro]

How will this interact with the github settings that specify which CI are required and which aren't? If they are generated programmatically, are we no longer allowed to say which one must complete successfully?

[pllim]

are we no longer allowed to say which one must complete successfully?

You are allowed but you have to react fast with the branch protection rules when the values change to unblock PRs.

[zacharyburnett]

How will this interact with the github settings that specify which CI are required and which aren't? If they are generated programmatically, are we no longer allowed to say which one must complete successfully?

Currently, when we want to change the versions of Python that a package supports, we go through the following process:

1. update requires-python in pyproject.toml
2. update the tested versions of Python in test.yml and test_extra.yml (sometimes we forget to do this)
3. update the "required status checks" in the ruleset (which is simply a list of exact job names)

This PR simply automates step 2.

I've been thinking for a while about how we can also remove the necessity for step 3. The problem is that the repository ruleset of required status checks takes a list of exact job names, so if a job name includes the version of Python in it (i.e. py312-xdist) then it'll eventually be outdated. As a possible solution I've made an action https://github.com/marketplace/actions/supported-pythons that does something very similar to the fill: parameter (reuses the same script in fact) , but adds oldest and latest outputs that we can use to make dedicated py-xdist (latest Python) and corresponding py-oldestdeps (oldest Python) jobs that have static names but always use the latest or oldest supported Python versions.

The question is whether we want to keep requiring ALL python versions, in which case we'll need to keep updating the ruleset manually when new python versions come out, or if we want to only require the oldest supported and latest supported versions of Python for a job to be merge-able (though we should still strive for all python versions to be green before merging) #10579

[tapastro]

update requires-python in pyproject.toml
update the tested versions of Python in test.yml and test_extra.yml (sometimes we forget to do this)
update the "required status checks" in the ruleset (which is simply a list of exact job names)

This PR simply automates step 2.

It wasn't clear to me that automating step 2 would not cause downstream harm to step 3, where the ruleset names are exact. What are the job names that fill will generate here?

[zacharyburnett]

What are the job names that fill will generate here?

in this case, the jobs appended by fill: true, fill_factors: xdist, fill_platforms: linux have these names:

• py314-xdist (ubuntu-latest)
• py313-xdist (ubuntu-latest)
• py312-xdist (ubuntu-latest)
• py311-xdist (ubuntu-latest)

So the current result of this PR on the list of jobs is to keep the list of jobs the same, and add py314-xdist (ubuntu-latest) and py313-xdist (ubuntu-latest) (which aren't currently explicitly defined), so the ruleset will be unchanged since the jobs it looks for are still there

test jobs from last night's scheduled run without fill::

test jobs with this PR (fill: appends jobs to the end of the matrix):

[zacharyburnett]

I've refactored this PR to do the following:

• make a dynamic list of Python versions that this package supports, that we can manually update at our own pace to increase the latest version (so that when Python 3.15 comes out the CI doesn't immediately fail)
• set static job names run tests (oldest supported Python, ubuntu-latest) and run tests with coverage (latest supported Python, ubuntu-latest), that we can then set as "required" in the branch protection rules but which will always align with our oldest and latest supported Python versions

[zacharyburnett]

(this PR basically copies the upcoming package template as closely as possible, so that rollout of that template will have minimal changes when it happens)

[tapastro]

I'm on board with getting this merged, but one remaining question - it looks like we have three pending checks, which look like required jobs that this PR no longer runs. What's up with the check-dependencies action?

[zacharyburnett]

I'm on board with getting this merged, but one remaining question - it looks like we have three pending checks, which look like required jobs that this PR no longer runs. What's up with the check-dependencies action?

oh good catch! I accidentally left out that check job

[zacharyburnett]

as for the required status checks in the repository settings, before or after merging this PR we can change the current required checks:

check / check-dependencies
test / py311-oldestdeps-xdist-cov (ubuntu-latest)
test / py313-cov-xdist (ubuntu-latest)

to

test / check-dependencies
test / run tests (oldest supported Python, ubuntu-latest)
test / run tests with coverage (latest supported Python, ubuntu-latest)

and remove the other version-specific ones