What happened?
When waiting for a job execution to complete using client-side status polling methods (such as wait_for_job_status() or wait_for_job()), there is no timeout mechanism or limit on how long the loop runs.
If a Kubernetes job gets stuck indefinitely in a Pending, ContainerCreating, or CrashLoopBackOff state due to missing cluster resources, image pull errors, or invalid node selectors, the client script blocks permanently.
In CI/CD automation pipelines or automated Python scripts, this causes job runners to hang indefinitely without throwing an exception or returning control to the caller.
What did you expect to happen?
-
Polling and waiting methods (wait_for_job_status(), wait_for_job()) should accept an optional timeout parameter (e.g., timeout: Optional[int] = None in seconds).
-
If timeout is provided, the client should track elapsed polling time.
-
If the job fails to reach a terminal status (SUCCEEDED, FAILED, CANCELLED) before the timeout expires, the SDK should raise a TimeoutError (or a dedicated KubeflowSDKTimeoutError) providing details such as the Job ID and current status.
Environment
Kubernetes version: v1.30.0
Kubeflow Trainer version: v1.8.0
Kubeflow Python SDK version: 0.5.0a0
Python version: 3.11.8
OS: Ubuntu 22.04 LTS
### Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍
What happened?
When waiting for a job execution to complete using client-side status polling methods (such as wait_for_job_status() or wait_for_job()), there is no timeout mechanism or limit on how long the loop runs.
If a Kubernetes job gets stuck indefinitely in a Pending, ContainerCreating, or CrashLoopBackOff state due to missing cluster resources, image pull errors, or invalid node selectors, the client script blocks permanently.
In CI/CD automation pipelines or automated Python scripts, this causes job runners to hang indefinitely without throwing an exception or returning control to the caller.
What did you expect to happen?
Polling and waiting methods (wait_for_job_status(), wait_for_job()) should accept an optional timeout parameter (e.g., timeout: Optional[int] = None in seconds).
If timeout is provided, the client should track elapsed polling time.
If the job fails to reach a terminal status (SUCCEEDED, FAILED, CANCELLED) before the timeout expires, the SDK should raise a TimeoutError (or a dedicated KubeflowSDKTimeoutError) providing details such as the Job ID and current status.
Environment
Kubernetes version: v1.30.0
Kubeflow Trainer version: v1.8.0
Kubeflow Python SDK version: 0.5.0a0
Python version: 3.11.8
OS: Ubuntu 22.04 LTS