Running on a cluster using multiple compute nodes with SLURM/SLURMGraph and MultiProc

[lcebaman]

Summary

Currently running on a cluster inside a compute node however it is desired to run on more than 1 node at the same time.

Information

How can we use SLURMGraph plugin to run on multiple compute nodes and MultiProc to fully populate the compute nodes? From the documentation, all we can see is:

workflow.run(plugin='SLURMGraph', plugin_args = {'dont_resubmit_completed_jobs': True})

How can we specify the number of compute nodes?

[d-vogel]

Hey !
I'm currently running my pipelines on SLURM, here is my setup:

The pipeline is run with normalize_wf.run(plugin="SLURM", plugin_args={"template":"SLURM_template.sh"})

SLURM_template.sh:

#!/bin/bash
#SBATCH -J singularity_job
#SBATCH -o singularity_job.out
#SBATCH -e singularity_job.err
#SBATCH --nodes=1
#SBATCH --cpus-per-task=32
#SBATCH --mem=32G
singularity run -B $WORKDIR:$WORKDIR $SINGULARITY_IMG \ 

About the SLURM settings:
the nodes available on the HPC are 32 cores. It seems having several tasks on the same node is a special setting and does not seem enabled in my case, which is why I affected the max amount of cores for each node.
In my Python code I set the number of cores to 32 for each tool (I had that setup anyways to run on my workstation in order to balance CPU vs RAM vs file IO) with num_threads for itk-based nodes,

At run time, each nipype node will be converted into a python file in $NIPYPE_TMP/$WORKFLOW_NAME/batch which has an associated bash script based on the template you provide the latter is the file that will actually be executed by the Compute node. The bash script is just a concatenation of your template and the python call to the python node.

In the case of my template, I wanted the python code to be called within the singularity container (where ANTS,FSL & co are installed) which is why there's a \ at the end of the call. I however had to make the change in #3297. You can however do the change for yourself by editing the file in your conda env.

So this is the setup that allows me to run many nodes on many compute nodes with multiprocessing on each. To go back to your question: if you still want to restrict the number of compute nodes to feed the jobs to, I suspect that to be a SLURM option, nipype just submits the jobs to the queue.

EDIT: replace exec with run in the singularity call. It seems the first does not run the entrypoint script (/neurodocker/startup.sh) while the second apparently does.