[Feature] Support recreate pods for RayCluster using RayClusterSpec.upgradeStrategy

[win5923]

Why are these changes needed?

Currently, when users update a RayCluster spec (e.g., update the image), users must re-create the cluster or set spec.suspend to true and after all Pods are deleted and then set it back to false.

Ref: #2534 (comment)

Design doc: https://docs.google.com/document/d/1xQLm0-WQWD-FkufxBJYklOJGvVn4RLk0_vPjLD5ax7o/edit?usp=sharing

Changes

• Add spec.upgradeStrategy field to RayCluster CRD
• Supports two values:
  • Recreate: During upgrade, Recreate strategy will delete all existing pods before creating new ones.
  • None: No new pod will be created while the strategy is set to None

Implementation

• Store hash of HeadGroupSpec.Template to head pod and workerGroup.Template to worker pod annotations during creation with ray.io/pod-template-hash
• Compare stored hash with current head pod or worker pod template hash to detect changes and recreate all pods

I only compare the HeadGroupSpec.Template and workerGroup.Template because these define the pod-related configurations. The RayCluster.Spec contains many dynamic and component-specific settings (e.g., autoscaler configs, rayStartParams).

Example:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-kuberay
spec:
  upgradeStrategy:
    type: Recreate
  rayVersion: '2.48.0'

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[win5923]

Hi @andrewsykim, I followed you previous comments to adding a spec.upgradeStrategy API to RayCluster. But for now. I'm concerned this approach may introduce some issues:

1. Confusion with existing API: We already have upgradeStrategy for RayService. Adding another upgradeStrategy to RayCluster could be confusing for users and creates unclear separation of concerns.
2. Breaking RayJob workflows: For RayJob, setting upgradeStrategy=Recreate on the RayCluster would cause pod recreation during job execution, leading to job interruption and loss of running jobs.

Maybe we can just add a feature gate instead of using spec.upgradeStrategy.type field in RayCluster to enable the recreate behavior. WDYT?

[andrewsykim]

Maybe we can just add a feature gate instead of using spec.upgradeStrategy.type field in RayCluster to enable the recreate behavior. WDYT?

Feature gates are used to gate features that are in early development and not ready for wider adoption, it shouldn't be used to change the behavior of RayCluster because it will eventually be on by default (and forced on).

[andrewsykim]

I think both of those concerns are valid, but I don't think this is a problem with separation of concerns as RayCluster is a building block for both RayService and RayJob. For those cases you mentioned, we should have validation to ensure RayCluster upgrade strategy cannot be set when used w/ RayJob and RayService

[win5923]

Hi @andrewsykim, PTAL, thanks.

[400Ping]

Consider logging the error and returning it up the call stack so that reconciliation can be retried.

[400Ping]

Consider adding a unit test that also verifies that invalid strategy values.