Skip to content

fix(client): Add missing timeout support for job status polling in TrainerClient and SparkClient #696

Description

@Sumitha007

What happened?

When waiting for a job execution to complete using client-side status polling methods (such as wait_for_job_status() or wait_for_job()), there is no timeout mechanism or limit on how long the loop runs.

If a Kubernetes job gets stuck indefinitely in a Pending, ContainerCreating, or CrashLoopBackOff state due to missing cluster resources, image pull errors, or invalid node selectors, the client script blocks permanently.

In CI/CD automation pipelines or automated Python scripts, this causes job runners to hang indefinitely without throwing an exception or returning control to the caller.

What did you expect to happen?

  1. Polling and waiting methods (wait_for_job_status(), wait_for_job()) should accept an optional timeout parameter (e.g., timeout: Optional[int] = None in seconds).

  2. If timeout is provided, the client should track elapsed polling time.

  3. If the job fails to reach a terminal status (SUCCEEDED, FAILED, CANCELLED) before the timeout expires, the SDK should raise a TimeoutError (or a dedicated KubeflowSDKTimeoutError) providing details such as the Job ID and current status.

Environment

Kubernetes version: v1.30.0
Kubeflow Trainer version: v1.8.0
Kubeflow Python SDK version: 0.5.0a0
Python version: 3.11.8
OS: Ubuntu 22.04 LTS



### Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions