Bug: SparkClient treats retry states as terminal failures

[vivek-gite]

What happened?

Summary

SparkClient.wait_for_job_status() can fail too early when a SparkApplication is being retried by Spark Operator.

The SDK currently maps several non-terminal Spark Operator states to SparkJobStatus.FAILED. As a result, callers may receive a failure before the operator has completed its configured retry flow.

Current behavior

SparkJobStatus.from_operator_state() maps these operator states to FAILED:

• SUBMISSION_FAILED
• FAILING
• PENDING_RERUN
• INVALIDATING
• UNKNOWN

wait_for_job_status() raises as soon as it sees SparkJobStatus.FAILED.

For example, with a retry-enabled RestartPolicy, a job may validly transition through:

SUBMISSION_FAILED
  → PENDING_RERUN
  → SUBMITTED
  → RUNNING
  → COMPLETED

Today, the SDK can raise during SUBMISSION_FAILED or PENDING_RERUN, preventing the caller from waiting for the retry to complete.

Proposed solution

• Add SDK status values for non-terminal states, such as RETRYING, SUSPENDED, and UNKNOWN.
• Map retry/transition states such as PENDING_RERUN, INVALIDATING, and retryable submission failures to a non-terminal SDK status.
• Preserve the raw Spark Operator state and errorMessage on SparkJob so callers can diagnose the current lifecycle state.
• Update wait_for_job_status() to raise only for a terminal operator FAILED state.
• Treat previously unknown operator states as UNKNOWN rather than immediately converting them to FAILED.

Acceptance criteria

• Every Spark Operator ApplicationStateType has a mapping test.
• A job transitioning from PENDING_RERUN to RUNNING and then COMPLETED does not raise from wait_for_job_status().
• A terminal FAILED job still raises when FAILED is not one of the requested target states.
• SparkJob exposes the raw operator state and error message when available.
• Existing public status values remain backward compatible.
• Documentation explains how retrying and terminal job states are represented.

References

• Spark Operator lifecycle design: https://kubeflow.github.io/spark-operator/docs/design.html
• Spark Operator ApplicationStateType: https://github.com/kubeflow/spark-operator/blob/master/api/v1beta2/sparkapplication_types.go
• Related LFX SparkClient observability/status work: [Epic] SparkClient – Production-Ready Distributed Data Processing & ML Workflows #655

What did you expect to happen?

SparkClient should distinguish terminal failures from retry and transition states.

Spark Operator documents COMPLETED and FAILED as terminal states. Other states may still transition or be retried depending on the configured restart policy.

wait_for_job_status() should continue polling while a job is retrying or transitioning, and raise only when the operator reaches a true terminal failure.

Environment

Kubernetes version:

$ kubectl version
Client Version: v1.31.0

Kubeflow Trainer version:

$ kubectl get pods -n kubeflow -l app.kubernetes.io/name=trainer -o jsonpath="{.items[*].spec.containers[*].image}"

Kubeflow Python SDK version:

$ pip show kubeflow

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first issue! We're happy to have you as part of our community 🚀

Here's what happens next:

• Our team will review your issue soon! cc @kubeflow/kubeflow-sdk-team
• If you'd like to contribute to this issue, check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[vivek-gite]

/assign