Calculation of inference time

[karthi0804]

Hi . I want to calculate the inference time of my model. I am not sure where to the code for measuring the time. I thought its better to do it inside predict_step of the LightningModule. Something like this

class LitModule(pl.LightningModule):
        def __init__(self, hparams):
               ....
               self.starter, self.ender = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
               .....
        ......
        ......
         def predict_step(self, batch, batch_idx, dataloader_idx=None):
                 torch.cuda.empty_cache()
                 minibatch_model_in, _ = batch
                 self.starter.record()
                 _ = self(minibatch_model_in)
                 self.ender.record()
                 # wait for gpu sync
                 torch.cuda.synchronize()
                 inference_time = self.starter.elapsed_time(self.ender)
                 return inference_time*1e-3

And in the main funtion,

inference_metrics = trainer.predict(model=pl_model, datamodule=pl_data)

After removing the initial measurements (considering GPU warm-up) and taking mean of 200 samples, I get 0.0196 seconds.

If I do the measurement outside the LightningModule then I get a different value. This is how I measured

# Inferencing
pl_data.setup()
# Initializing cuda event loggers
starter, ender = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
metrics = []
pl_model.eval()
with torch.no_grad():
for batch in pl_data.train_dataloader():
      torch.cuda.empty_cache()
      minibatch_model_in, _, _, _, _, _ = batch
      starter.record()
        _, _, _ = pl_model(minibatch_model_in)
      # wait for gpu sync
       torch.cuda.synchronize()
       curr_time = starter.elapsed_time(ender)
       metrics.append(curr_time)

Using the above code, I get 0.027 seconds. Please someone tell me if I am doing something wrong. Which method is correct? Why is there a decrease in the inference time when measured within predict_step?

[justusschock]

Hi, the first snippet measures raw prediction time, whereas the second one doesn't seem to be on gpu at all (e.g. your data likely isn't on gpu) or (if it is) can already include the data preparation and host to device transfers.

[karthi0804]

Hi, the first snippet measures raw prediction time, whereas the second one doesn't seem to be on gpu at all (e.g. your data likely isn't on gpu) or (if it is) can already include the data preparation and host to device transfers.

Thanks for your reply. You are right! data was on cpu. I realized now that the trainer has no influence since I am using the model directly. I manually loaded the model and the data on gpu and measured the time. It is similar to the method 1 value.

[tchaton]

Dear @karthi0804,

You could do get the profiling automatically.

trainer = Trainer(profiler="simple")
trainer.predict(model=pl_model, datamodule=pl_data)

From pytorch_lightning/loops/epoch/prediction_epoch_loop.py

        with self.trainer.profiler.profile("predict_batch_to_device"):
            batch = self.trainer.accelerator.batch_to_device(batch, dataloader_idx=dataloader_idx)

        self.batch_progress.increment_ready()

        with self.trainer.profiler.profile("predict_step"):
            self._predict_step(batch, batch_idx, dataloader_idx)