How to have per GPU logging?

[RuABraun]

Is there a way to have logs per GPU? I want to understand why training fails and at the moment the log just shows a time out error. I know from previous experience that this likely means a nonzero rank GPU process had an error, and I want to validate that (and see what the error is).

[miladfa7]

you can configure per-GPU logging by setting up a separate logger for each GPU process.

import logging

def setup_logging(rank):
    # Create a logger instance with a unique name for each process
    logger = logging.getLogger(f"process_{rank}_logger")
    logger.setLevel(logging.INFO)

    # Set up a console handler to log to the console
    console_handler = logging.StreamHandler()
    console_handler.setLevel(logging.INFO)
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    console_handler.setFormatter(formatter)
    logger.addHandler(console_handler)

    # Set up a file handler to log to a file
    file_handler = logging.FileHandler(f"process_{rank}_log.txt")
    file_handler.setLevel(logging.INFO)
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    file_handler.setFormatter(formatter)
    logger.addHandler(file_handler)

    return logger

You can then call this function at the beginning of each GPU process and use the returned logger instance to log messages. For example:

import torch

# Initialize each GPU process with a unique rank
rank = torch.distributed.get_rank()

# Set up logging for this process
logger = setup_logging(rank)

# Log a message
logger.info("Hello from process %d", rank)

[RuABraun]

nice thank you!

[RuABraun]

realized this doesn't really answer the question. any errors that happen will not be redirected to the file. only logged statements will be

[Leenaa1255]

Hi guys
To achieve per GPU logging, ensure each GPU has a unique identifier. Then, customize your logging mechanism to include the GPU identifier alongside relevant information.