[RayJob] Don't fail a running Ray job when the submitter exits

[1fanwang]

Why are these changes needed?

KubeRay treats the submitter finishing as evidence the Ray job is terminal. It isn't. ray job logs --follow, the submitter's last command in K8sJobMode and SidecarMode, returns 0 on any non-abnormal WebSocket close (sdk.py):

elif msg.type == aiohttp.WSMsgType.CLOSED:
    if ws.close_code == aiohttp.WSCloseCode.ABNORMAL_CLOSURE:
        raise RuntimeError(...)
    break                                  # normal close: no error
...
return _log_job_status(client, job_id)     # only logs; never exits non-zero

checkSubmitterFinishedTimeoutAndUpdateStatusIfNeeded then waits 30s, overwrites RUNNING with FAILED, and under shutdownAfterJobFinishes deletes the cluster from under the driver. It also produces the inconsistent status pair in #2314.

Ask the cluster instead of guessing

The timeout can't just be removed: #4091 added it for a status that never resolves after a head restart. Both cases arrive with JobStatus: RUNNING.

A job status is only current while the node running its driver is alive. Ray already reports driver_node_id on the job; this deserializes it and adds an IsNodeAlive check against the dashboard's node list:

if rayJobIsActive(rayJob) {
    alive, err := rayJobDriverIsAlive(ctx, dashboardClient, jobInfo)
    ...
    if alive { return false, nil }   // job is genuinely running; leave it
}

The check moves after the job-info poll, where jobInfo and the client are already available. It compares no timestamps and reads no Pod lifecycle, so it is unaffected by readiness probes flapping, by a sidecar submitter dying, and by whether the head restarted before or after the submitter exited, earlier revisions got each of those wrong, and Bugbot caught them.

Scoped to RUNNING on purpose: Ray assigns driver_node_id when it schedules the job's supervisor, so a PENDING job has none to check against. Those keep today's behavior rather than being guessed at. An unreachable dashboard returns an error and requeues instead of counting as a dead driver, and a jobInfo that has already gone terminal is left to the normal handling.

The e2e test RayJob has exceed SubmitterFinishedTimeout asserted the old behaviour; it is rewritten to assert the job survives, and renamed.

Related issue number

Addresses #4921. Related: #2314.

Labels

• doc-updates-required
• breaking-change

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests

Manual test instructions

kind, rayproject/ray:2.55.1, shutdownAfterJobFinishes: false so the cluster survives for inspection.

Repro: SIGTERM the head's dashboard JobHead module mid-job. That ends the submitter's ray job logs --follow stream without touching the raylet, so the node stays up and the job keeps running.

On unpatched master — a healthy job is failed.

[07:03:09Z] SIGTERM JobHead
[07:03:25Z] submitter=Succeeded/0 | js=RUNNING dep=Running
[07:03:55Z] submitter=Succeeded/0 | js=FAILED  dep=Failed
            reason=JobDeploymentStatusTransitionGracePeriodExceeded

Ray's own view at that moment, while the CR said FAILED:

node 01c291fbf6981517 state=ALIVE
ray job status = RUNNING   driver_node = 01c291fbf6981517
entrypoint process: still running

It then ran to completion, four minutes after KubeRay gave up on it:

[07:04:37Z] ray=RUNNING     CR=FAILED
[07:08:02Z] ray=SUCCEEDED   CR=FAILED

With shutdownAfterJobFinishes on, the cluster is deleted at 07:03:55 and that work is lost.

On this branch — same repro, job survives.

[06:55:11Z] submitter finished
[06:55:41Z] "The RayJob submitter finished but the Ray job is still active on
             a live node; not transitioning to terminal."
             JobStatus=RUNNING DriverNodeID=1fdaa65c52efe92a...
[07:00:02Z] js=SUCCEEDED dep=Complete

The #4091 case still fails, as it must. Killing the head's DashboardAgent takes the node down: the entrypoint process dies and Ray's status freezes at RUNNING. The driver's node stops reporting ALIVE, so the timeout fires exactly as before.

[06:42:58Z] killed DashboardAgent   → node state=DEAD, entrypoint gone
[06:43:31Z] js=FAILED dep=Failed
            reason=JobDeploymentStatusTransitionGracePeriodExceeded

Force-deleting the head is also still terminal:

[06:39:09Z] dep=Failed reason=AppFailed
            "Submitter completed but Ray job not found in RayCluster."

Unit tests

=== RUN   TestCheckSubmitterFinishedTimeoutAndUpdateStatusIfNeeded
    --- PASS: .../submitter_still_running
    --- PASS: .../grace_period_not_yet_elapsed
    --- PASS: .../job_RUNNING_on_a_live_driver_node_is_left_alone
    --- PASS: .../job_PENDING_has_no_driver_node_yet_and_times_out
    --- PASS: .../job_RUNNING_but_the_driver_node_is_gone_times_out
    --- PASS: .../job_RUNNING_with_no_driver_node_reported_times_out
    --- PASS: .../job_never_observed_times_out
    --- PASS: .../fresh_job_info_is_terminal,_CR_is_stale
    --- PASS: .../node_liveness_errors_leave_the_job_alone
--- PASS: TestCheckSubmitterFinishedTimeoutAndUpdateStatusIfNeeded (0.00s)

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit f01abb6. Configure here.}