Out of Memory Issues Segmenting Example MINC data [Question]

[dyt811]

Hi, I have managed to set everything up and run default training for the corpus callosum segmentation with the tensorflow GPU without issues. so I know the tensorflow GPU integration works. HOWEVER, when I tried to conduct the segmentation on the example MINC file provided, it failed for Out of RAM issue. Could you elaborate a bit on 1) the minimal hardwarew 2) aximal file dimensions requirement for the 2DConvNet segmentation that is provided? I was using a 8GB 1070GTX on Windows 64 with 32GB and there are not many 16GB RAM GPU out there currently...

[dyt811]

Attempted on Ubuntu 18 as well but no dice. How much VRAM did you guys train this 2DUnet on when even a single volume prediction I am running out of ram. 😆

/home/dyt811/Git/DeepDiscovery/venv/bin/python /home/dyt811/Git/DeepDiscovery/examples/brainSegmentation/segment.py
2018-10-02 15:23:18.014572: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-10-02 15:23:18.016181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties: 
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.835
pciBusID: 0000:05:00.0
totalMemory: 7.93GiB freeMemory: 7.48GiB
2018-10-02 15:23:18.075653: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-10-02 15:23:18.076804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 1 with properties: 
name: Quadro K2000 major: 3 minor: 0 memoryClockRate(GHz): 0.954
pciBusID: 0000:04:00.0
totalMemory: 1.95GiB freeMemory: 1.93GiB
2018-10-02 15:23:18.077469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1461] Ignoring visible gpu device (device: 1, name: Quadro K2000, pci bus id: 0000:04:00.0, compute capability: 3.0) with Cuda compute capability 3.0. The minimum required Cuda capability is 3.5.
2018-10-02 15:23:18.077495: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0
2018-10-02 15:23:23.734511: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-10-02 15:23:23.734545: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977]      0 1 
2018-10-02 15:23:23.734555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0:   N N 
2018-10-02 15:23:23.734562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1:   N N 
2018-10-02 15:23:23.734829: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7218 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:05:00.0, compute capability: 6.1)
/home/dyt811/Git/DeepDiscovery/DeepDiscovery/utility.py:58: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  newShape,dims1,dims2 = computePad(numpy.array(img.shape)[[spatialAxes]],depth,shape)
/home/dyt811/Git/DeepDiscovery/DeepDiscovery/Data/Data.py:360: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  centre = x[slices];
2018-10-02 15:23:34.317911: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.87GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-10-02 15:23:34.963288: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.42GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
/home/dyt811/Git/DeepDiscovery/DeepDiscovery/utility.py:76: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  ret = img[slices]
/home/dyt811/Git/DeepDiscovery/examples/brainSegmentation/segment.py:21: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  pylab.imshow(image.get_data()[slc],cmap=pylab.cm.gray,origin='lower')
/home/dyt811/Git/DeepDiscovery/examples/brainSegmentation/segment.py:22: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  pylab.contour(brainProb[slc],[0.5],colors='g')

Process finished with exit code 0

[robb-brown]

Can you confirm that this is actually a failure and not just tensor flow complaining that it would like more memory please, but running anyway?

The CC segmentation runs fine on my 4 GB 970, and is lightweight enough that it can be trained quite successfully in a few minutes on a mobile CPU.