Describe the bug
The scripts/speechlm2/oomptimizer.py script is very cool but in order to be able to work with the produced batch_sizes I need to use --memory-fraction 0.5.
It seems that the Oomptimizer is right but only for the beginning of the training. After some time the VRAM will bump to a new value which is not anticipated by the Oomptimizer. I don't know if it is a real issue or maybe the issue is somewhere else. These bumps can happen after a few hours of training, and they can be multiples bumps (one after a few minutes, then one after one hour and one after 2 hours) which can lead to OOM. I understand that these bumps are probably caused by the data but shouldn't be the role of the oomptimizer to prevent OOM by simulating the worst case scenario for each batch? I still haven't found how to limit or prevent these bumps except starting from a verly low VRAM usage.
Steps/Code to reproduce bug
Use oomptimizer and then launch a training and track the VRAM usage.
Expected behavior
A clear and concise description of what you expected to happen.
Environment overview (please complete the following information)
- Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
- Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
- If method of install is [Docker], provide
docker pull & docker run commands used
Environment details
NeMo 2.8.0 rc0n
Additional context
H100
Describe the bug
The scripts/speechlm2/oomptimizer.py script is very cool but in order to be able to work with the produced batch_sizes I need to use
--memory-fraction 0.5.It seems that the Oomptimizer is right but only for the beginning of the training. After some time the VRAM will bump to a new value which is not anticipated by the Oomptimizer. I don't know if it is a real issue or maybe the issue is somewhere else. These bumps can happen after a few hours of training, and they can be multiples bumps (one after a few minutes, then one after one hour and one after 2 hours) which can lead to OOM. I understand that these bumps are probably caused by the data but shouldn't be the role of the oomptimizer to prevent OOM by simulating the worst case scenario for each batch? I still haven't found how to limit or prevent these bumps except starting from a verly low VRAM usage.
Steps/Code to reproduce bug
Use oomptimizer and then launch a training and track the VRAM usage.
Expected behavior
A clear and concise description of what you expected to happen.
Environment overview (please complete the following information)
docker pull&docker runcommands usedEnvironment details
NeMo 2.8.0 rc0n
Additional context
H100