fix(trainer): raise RuntimeError for failed TrainJobs

[kingabdulsalamsas111-lgtm]

PR Description
What this PR does / why we need it:

This PR updates LocalProcessBackend.wait_for_job_status() to immediately raise a RuntimeError when a TrainJob enters the Failed state, unless TRAINJOB_FAILED is explicitly included in the expected status set.

Previously, LocalProcessBackend.wait_for_job_status() continued polling until the timeout even after the TrainJob had already failed. This behavior was inconsistent with the Container and Kubernetes backends, which already raise a RuntimeError as soon as an unexpected failed status is observed.

This change aligns the LocalProcessBackend behavior with the other backends and avoids unnecessary waiting after a job has already failed.

In addition to the implementation change, this PR includes a regression test to verify that wait_for_job_status() raises a RuntimeError immediately when a TrainJob reaches the Failed state unexpectedly.

Which issue(s) this PR fixes (optional, in Fixes #<662> format, will close the issue(s) when PR gets merged):

Fixes #<Fixes #662>

Checklist:

• Docs included if any changes are user facing

Testing

python -m pytest kubeflow/trainer/backends/localprocess/backend_test.py

Output:

============================= 28 passed =============================

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andreyvelich for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment