Optimize short runs on compute clusters

[PascalIversen]

Description of feature

Currently, we start a job for every model fitting procedure, which is optimal for compute-intensive model fitting procedures like neural networks. For simpler models, such as naive predictors and linear models, we should consider reverting to a loop or parallel execution on a single node to avoid allocation overhang

[JudithBernett]

e.g., for SingleDrugProteomicsElasticNet

[nictru]

Would task arrays partially solve this? It is supported by slurm and some cloud computing providers

I think that would be a quite elegant approach, as opposed to adding logic that allows actually sending multiple training jobs to a single nextflow task

[JudithBernett]

It could, but it ties closely together with the question if we want to keep doing Grid Search or not. Because if we switch to, e.g., optuna, it would be better to have 1 optimizing process with many cores instead of many processes running in parallel.

[nictru]

I understand the issue description more like this is about bundling smaller models even if hyperparameter optimization is disabled, which would be a bit complicated to orchestrate. But if it is more about hyperparameter optimization, then it should be relatively straightforward to scale up the compute and memory resources by a certain factor and then let optuna parallelize within the job

[JudithBernett]

Yes, it was really mostly an issue during hyperparameter optimization because e.g. the ElasticNet runs in like 1 minute