Cannot reproduce 0.3s inference time

[EasternJournalist]

Hi, thank you for this amazing work!

I am benchmarking the inference speed but noticed that the per-image inference time on an A100 GPU is around 940ms, at float32 precision, which is more than 2× the number reported in the paper (341ms on V100).

I am wondering if there are any specific optimizations or settings I might have missed that could explain this discrepancy.

inference_time = []
for i in range(20):
    torch.cuda.synchronize()
    start_time = time.time()
    prediction = model.infer(image)
    torch.cuda.synchronize()
    end_time = time.time()
    inference_time.append(end_time - start_time)
average_time = sum(inference_time) / len(inference_time)
print(f"Inference time: {average_time:.4f}s")

[REDDITARUN]

Use torch.compile

[bekhzod-olimov]

@EasternJournalist, did you ensure that no other processes were running on your GPU during inference? I tested on an RTX 4090 the following script:

import sys, time, torch
import depth_pro

# Load model and preprocessing transform
model_time = time.time()
device = "cuda" if torch.cuda.is_available else "cpu"
model, transform = depth_pro.create_model_and_transforms(device = device)
model.eval()
print(f"Model is loaded in {(time.time() - model_time):.3f} secs")

# Load and preprocess an image.
image_path = "test.jpg"
image, _, f_px = depth_pro.load_rgb(image_path)
image = transform(image)

inference_time = []
for i in range(20):    
    start_time = time.time()    
    prediction = model.infer(image, f_px=f_px)    
    end_time = time.time()
    print(f"Inference time for a single image -> {(end_time - start_time):.3f} secs!")
    inference_time.append(end_time - start_time)
average_time = sum(inference_time) / len(inference_time)
print(f"Inference time: {average_time:.4f}s")

and when the GPU was idle, the average inference time was approximately 0.4 seconds. However, when another model was being trained simultaneously (i.e., the GPU was not entirely free), the inference time increased by nearly threefold.

Additionally, model loading time can be quite significant. While your code does not appear to be loading the model for each image, I would suggest double-checking to ensure that the model is loaded only once before inference, as this could otherwise contribute to increased inference time.

[EasternJournalist]

@EasternJournalist, did you ensure that no other processes were running on your GPU during inference? I tested on an RTX 4090 the following script:

import sys, time, torch
import depth_pro

# Load model and preprocessing transform
model_time = time.time()
device = "cuda" if torch.cuda.is_available else "cpu"
model, transform = depth_pro.create_model_and_transforms(device = device)
model.eval()
print(f"Model is loaded in {(time.time() - model_time):.3f} secs")

# Load and preprocess an image.
image_path = "test.jpg"
image, _, f_px = depth_pro.load_rgb(image_path)
image = transform(image)

inference_time = []
for i in range(20):    
    start_time = time.time()    
    prediction = model.infer(image, f_px=f_px)    
    end_time = time.time()
    print(f"Inference time for a single image -> {(end_time - start_time):.3f} secs!")
    inference_time.append(end_time - start_time)
average_time = sum(inference_time) / len(inference_time)
print(f"Inference time: {average_time:.4f}s")

and when the GPU was idle, the average inference time was approximately 0.4 seconds. However, when another model was being trained simultaneously (i.e., the GPU was not entirely free), the inference time increased by nearly threefold.

Additionally, model loading time can be quite significant. While your code does not appear to be loading the model for each image, I would suggest double-checking to ensure that the model is loaded only once before inference, as this could otherwise contribute to increased inference time.

Hi, thanks for your kind reply. It was tested with no other running GPU process. The 0.4s latency on RTX4090 actually confirms the issue. A100 is generally ~2x slower than RTX4090, consistent with the 0.9s latency I tested.

However, in the paper, the 0.3s latency reported was tested on V100, which is ~4x slower than RTX4090.

[UygarUsta]

Have you tried fp16 precision when inferencing ?

[gojira5]

Wondering where I went wrong, spending 21 seconds per image at fp32, on 4070ti.

model, transform = depth_pro.create_model_and_transforms(device=torch.device("cuda"))
model = torch.compile(model, mode="reduce-overhead")
def infer_depth_to_numpy(im_path: str) -> np.ndarray:
    """Return the depth map as a H×W float32 numpy array (metres)."""
    image, _, f_px = depth_pro.load_rgb(im_path)
    image = transform(image).to(device)

    with torch.inference_mode():
        pred = model.infer(image, f_px=f_px)

    depth = pred["depth"]           # torch.Tensor on device
    return depth.squeeze().cpu().numpy()  # H×W np.float32

[gojira5]

Dang! I'm getting 0.42 seconds/image with a few modifications:

torch.backends.cudnn.benchmark = True      # Pick fastest kernels
torch.set_float32_matmul_precision('high') 

model, transform = depth_pro.create_model_and_transforms(device=torch.device("cuda"))
model = torch.compile(model, mode="reduce-overhead")


@torch.no_grad()
def infer_depth_to_numpy(im_path: str) -> np.ndarray:
    from torch.cuda.amp import autocast
    image, _, f_px = depth_pro.load_rgb(im_path)
    image = transform(image).unsqueeze(0).cuda()

    with autocast():
        pred = model.infer(image, f_px=f_px)

    return pred["depth"].squeeze().cpu().numpy()