Is your feature request related to a problem? Please describe.
As Marek points out, the current set up of environments for the system-level test suite installs different versions of anemoi-datasets for the dataset tests and the training tests. — In both cases installing the repo from main (or a branch) and the anemoi dependencies from pypi. (See below for more context)
Solution proposed:
Installing all anemoi dependencies from main (or from a branch if selected in the workflow dispatch). This will allow testing breaking changes across the pipeline before merging and releasing them.
That is,
- add all repos as options for the workflow dispatch and pass them to the suite as overrides
- for each test environment, install the anemoi dependencies from the branch specified (main by default)
Upsides:
- Allows testing of breaking changes in upstream repos (or across the pipeline).
- Enables indirect testing of repos like anemoi-transform as dependencies.
- Could be used to test which anemoi versions are compatible.
Considerations: we don’t notice when the user workflow breaks and would require a version bump. (Assumed user workflow: install anemoi-datasets to create a dataset, install anemoi-training to run the training.)
Additional context
It appears to me, that this action does not test compatibility across all main branches but that the dependencies for each test are rather taken from PyPI. E.g. one task tests anemoi-datasets==0.1.dev1+gfa0b7e809 while the core tests use the version anemoi-datasets==0.5.26 from PyPI. Is this intended?
From the anemoi-datasets tests:
From the anemoi-core tests:
+ anemoi-datasets==0.5.26
+ anemoi-graphs==0.6.5.post3 (from file:///lus/h1resw02/project/prepml/ecflow_server/workdirs/testing/anemoi_tests/nightly/local/build/training_env/anemoi_training/graphs)
+ anemoi-models==0.9.4.post3 (from file:///lus/h1resw02/project/prepml/ecflow_server/workdirs/testing/anemoi_tests/nightly/local/build/training_env/anemoi_training/models)
+ anemoi-training==0.6.4.post3 (from file:///lus/h1resw02/project/prepml/ecflow_server/workdirs/testing/anemoi_tests/nightly/local/build/training_env/anemoi_training/training)
+ anemoi-transform==0.1.16
+ anemoi-utils==0.4.35
Originally posted by @MeraX in #38 (comment)
Organisation
No response
Is your feature request related to a problem? Please describe.
As Marek points out, the current set up of environments for the system-level test suite installs different versions of anemoi-datasets for the dataset tests and the training tests. — In both cases installing the repo from main (or a branch) and the anemoi dependencies from pypi. (See below for more context)
Solution proposed:
Installing all anemoi dependencies from main (or from a branch if selected in the workflow dispatch). This will allow testing breaking changes across the pipeline before merging and releasing them.
That is,
Upsides:
Considerations: we don’t notice when the user workflow breaks and would require a version bump. (Assumed user workflow: install anemoi-datasets to create a dataset, install anemoi-training to run the training.)
Additional context
Originally posted by @MeraX in #38 (comment)
Organisation
No response