What is the recommended way to upgrade Slurm Clusters and Templates after upgrading CycleCloud OS Package?

[garymansellricardo]

Hi,

I have recently upgraded my CycleCloud OS Package from 8.6 to 8.7 and now in the CycleCloud UI - I see my custom Slurm Clusters showing with an "upgrade" link in the UI. I presume I need to do this, but what exactly does it do behind the scenes, and are there risks that I might break my cluster by pressing this (if so, how can I revert)?

Further (and the main focus of my query). I have a custom cluster template that I generated from the previous Slurm Cluster template (3.0.8) that I used to build my CC Slurm Clusters originally on a previous version of CycleCloud. I searched the new CycleCloud loadpoint and found reference to a new Slurm Cluster template (/opt/cycle_server/system/work/.plugins_expanded/.expanded/cloud-ce665d9d-3f40-4c5c-b053-3c0a63a67597/plugins/cloud/initial_data/templates/slurm/slurm_template_3.0.11.txt) - which looks quite different to my custom template.

What is the correct method of upgrading my custom Slurm cluster template to work with the new 8.7.1 version of CycleCloud?

• Do I take a copy of the above (3.0.11) file and merge my changes across to it and then import this and build new clusters from it before deleting my old one? Or, is there a better place to access/download a copy of the new template?
• I notice that there is a slurm.txt and slurm-cs.txt template here: https://github.com/Azure/cyclecloud-slurm/blob/master/templates but these seem to refer to a version 4.0 - is this safe/better to use with CycleCloud 8.7.1 or is it a pre-release version not suited for 8.7.1 yet?
• Should I then delete/re-create my CC Slurm Clusters from the new template version - or will pressing the upgrade button in the UI do exactly this anyway, so is not needed?
• Presumably, I will still need to delete my custom Slurm Cluster template from the system and install my new merged version based on the 3.0.11 template for building new Slurm Clusters?

Thanks for any advice you can offer, as I am running a business critical env - and want to make sure I am following the right process and will be on a supported configuration.

FWIW - I could not find any official MS doco on this anywhere, so maybe could be added?

Thanks and Regards to the CycleCloud Slurm team for your great software package...

Gary

[garymansellricardo]

I also notice that there are some slurm 3.0.11 templates in the templates folder of the cyclecloud-slurm 3.0.11 release source code from the repo, which are different again, so I am not sure which to use?

[aditigaur4]

Hi Gary:

The upgrade button upgrades our cyclecloud client (jetpack), it does not update the template a project uses at the moment. Template is designed for the user to define how their cluster is created, so upgrade does not change the template.

What is the correct method of upgrading my custom Slurm cluster template to work with the new 8.7.1 version of CycleCloud?
Do I take a copy of the above (3.0.11) file and merge my changes across to it and then import this and build new clusters from it before deleting my old one? Or, is there a better place to access/download a copy of the new template?

That is correct. You can look into our last tagged release, 3.0.11 and get templates from there. It does have some UI changes, so you would need to do a merge of your changes with the new template.

I notice that there is a slurm.txt and slurm-cs.txt template here: https://github.com/Azure/cyclecloud-slurm/blob/master/templates but these seem to refer to a version 4.0 - is this safe/better to use with CycleCloud 8.7.1 or is it a pre-release version not suited for 8.7.1 yet?

Our next release is 4.0 but we have not done a release on that branch yet, so it is not safe to use in production just yet.

Should I then delete/re-create my CC Slurm Clusters from the new template version - or will pressing the upgrade button in the UI do exactly this anyway, so is not needed?

If you are updating the template then you should create a new cluster, that will get all the template changes in. Upgrade button is useful to get new installation of cycle server/jetpack but keeping your cluster definition as is.

Presumably, I will still need to delete my custom Slurm Cluster template from the system and install my new merged version based on the 3.0.11 template for building new Slurm Clusters?

That is correct. Once you merge your cluster template with the new 3.0.11 template, you can re-import your new template.

[garymansellricardo]

Thanks for the speedy reply (as ever - really helpful) that all makes sense. Please close the issue and have a great day.