Show GC execution time

As we now call GC everytime when checkpointing, GC execution time can start to affect the async checkpointing and zero-overhead checkpointing overall efficiency. Log the GC execution for better understanding the effect.

ghstack-source-id: 6b741b6
Pull Request resolved: #870

Author: fegin

torchtitan/tools/utils.py

@@ -9,6 +9,7 @@
 import os
 import subprocess
 import sys
+import time
 from dataclasses import dataclass
 from typing import Optional
 
@@ -43,8 +44,9 @@ def run(self, step_count):
 
     @staticmethod
     def collect(reason: str):
-        logger.info(reason)
+        begin = time.monotonic()
         gc.collect(1)
+        logger.info("[GC] %s %.2f seconds.", reason, time.monotonic() - begin)
 
 
 def get_num_params(model: torch.nn.Module, exclude_embedding: bool = False) -> int: