What you would like to be added?
Add parameter options for streaming logs line-by-line and limiting returned log lines to TrainerClient.get_job_logs().
Specifically:
1. follow: bool = False: When set to True, returns an Iterator[str] / generator that streams job logs in real time rather than returning a static list.
2. tail_lines: Optional[int] = None: Allows fetching only the last $N$ lines of log output.
Proposed Function Signature Update:
def get_job_logs(
self,
name: str,
follow: bool = False,
tail_lines: Optional[int] = None,
container_name: Optional[str] = None
) -> Union[List[str], Iterator[str]]:
"""Fetches or streams training job logs."""
...
Why is this needed?
Currently, TrainerClient.get_job_logs() fetches all logs into memory at once as a single List[str]. This creates two main issues for long-running PyTorch, JAX, or Spark jobs:
1. Memory Exhaustion (OOM): Long training runs generate hundreds of megabytes or gigabytes of logs, causing client-side memory spikes and MemoryError crashes in interactive environments like Jupyter Notebooks.
2. Lack of Live Monitoring: Blocking execution until all logs are fetched prevents users from monitoring training loss and progress in real time as the job executes.
Love this feature?
Give it a 👍 We prioritize the features with most 👍
What you would like to be added?
Add parameter options for streaming logs line-by-line and limiting returned log lines to TrainerClient.get_job_logs().$N$ lines of log output.
Specifically:
1. follow: bool = False: When set to True, returns an Iterator[str] / generator that streams job logs in real time rather than returning a static list.
2. tail_lines: Optional[int] = None: Allows fetching only the last
Proposed Function Signature Update:
def get_job_logs(
self,
name: str,
follow: bool = False,
tail_lines: Optional[int] = None,
container_name: Optional[str] = None
) -> Union[List[str], Iterator[str]]:
"""Fetches or streams training job logs."""
...
Why is this needed?
Currently, TrainerClient.get_job_logs() fetches all logs into memory at once as a single List[str]. This creates two main issues for long-running PyTorch, JAX, or Spark jobs:
Love this feature?
Give it a 👍 We prioritize the features with most 👍