Provide Support for Non-Containerized Slurm Executor

[sam-h-bean]

I am in a situation where I want to use NeMo-Run on Slurm but also have no way to run containerized sbatch commands. Does this mean that I should just build my sbatch commands around the core NeMo 2.0 API in the torch lightning style or should I try orchestrating the multi-node local executor from an sbatch command when it is ready? Curious to hear what you would consider the best practice here!

[hemildesai]

Hey @sam-h-bean, we can definitely look into adding support for Non-Containerized Slurm Executor. Is your use case to just remove the --container-* flags from srun?

[sam-h-bean]

@hemildesai I guess it would also mean removing the requirement to have Pyxis on the slurm cluster. What I'm doing now is just wrapping python nemo_training_script.py with srun and putting that in an sbatch file. Which is fine but I'd rather get the benefits of the best practices NeMo-Run brings to the table.

I'm also fussing with trying to get this running on a Ray cluster on Slurm since that was my flow previously building distributed training framework around Deepspeed. I have a lot of flows that depend on ray data and feeding DDP ranks through ray. It would be sick to be able to do that except with the megatron style distribution methods NeMo has unlocked for me. That would be my dream state. Lemme know if this makes sense.

[hemildesai]

Thanks for the reply @sam-h-bean, we just recently added a Ray launcher for Slurm here - #159.

We can look into adding non pyxis support. Can you share an example srun command (for your usecase) for reference?

[AudranBert]

Hi, it would be useful to have a Slurm executor that do not need a container to run (but instead it takes a bash command so we can set up a conda env for example). Either because we can't run docker or singularity or because options are not similar. Local executor can only run on one node, so there is a real need for a slurm executor without container.

[hemildesai]

Hi, it would be useful to have a Slurm executor that do not need a container to run (but instead it takes a bash command so we can set up a conda env for example). Either because we can't run docker or singularity or because options are not similar. Local executor can only run on one node, so there is a real need for a slurm executor without container.

We just merged #143 to support multi node jobs using local executor.

[armsp]

I would also like to know how to get at least the evaluation running in a slurm environment but without containers. Is there a way to use VLLM directly somehow? I have checked the documentation and I could not find anything. @hemildesai @sam-h-bean

[armsp]

I think its possible now - NVIDIA-NeMo/Skills#446

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale.