increasing GPU memory and finally OOM

[whq-pku1]

I've noticed an interesting phenomenon: as training progresses, GPU memory usage shows a significant upward trend, eventually leading to Out-of-Memory (OOM) failure. What could be the cause of this? Have you observed a similar phenomenon?