Add top-level Labels and Resources Structed fields to HeadGroupSpec and WorkerGroupSpec (#4106)

* Add top-level Labels and Resources fields

Signed-off-by: Ryan O'Leary <[email protected]>

* Update API comment

Signed-off-by: Ryan O'Leary <[email protected]>

* Fix comments

Signed-off-by: Ryan O'Leary <[email protected]>

* Add validation logic

Signed-off-by: Ryan O'Leary <[email protected]>

* Update ray-operator/controllers/ray/common/pod.go

Co-authored-by: Kai-Hsun Chen <[email protected]>
Signed-off-by: Ryan O'Leary <[email protected]>

* Add k8s syntax validation

Signed-off-by: Ryan O'Leary <[email protected]>

---------

Signed-off-by: Ryan O'Leary <[email protected]>
Signed-off-by: Ryan O'Leary <[email protected]>
Co-authored-by: Kai-Hsun Chen <[email protected]>

Author: ryanaoleary

docs/reference/api.md

@@ -193,6 +193,8 @@ _Appears in:_
 | `template` _[PodTemplateSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#podtemplatespec-v1-core)_ | Template is the exact pod template used in K8s deployments, statefulsets, etc. |  |  |
 | `headService` _[Service](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#service-v1-core)_ | HeadService is the Kubernetes service of the head pod. |  |  |
 | `enableIngress` _boolean_ | EnableIngress indicates whether operator should create ingress object for head service or not. |  |  |
+| `resources` _object (keys:string, values:string)_ | Resources specifies the resource quantities for the head group.<br />These values override the resources passed to `rayStartParams` for the group, but<br />have no effect on the resources set at the K8s Pod container level. |  |  |
+| `labels` _object (keys:string, values:string)_ | Labels specifies the Ray node labels for the head group.<br />These labels will also be added to the Pods of this head group and override the `--labels`<br />argument passed to `rayStartParams`. |  |  |
 | `rayStartParams` _object (keys:string, values:string)_ | RayStartParams are the params of the start command: node-manager-port, object-store-memory, ... |  |  |
 | `serviceType` _[ServiceType](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#servicetype-v1-core)_ | ServiceType is Kubernetes service type of the head service. it will be used by the workers to connect to the head pod |  |  |
 
@@ -472,6 +474,8 @@ _Appears in:_
 | `minReplicas` _integer_ | MinReplicas denotes the minimum number of desired Pods for this worker group. | 0 |  |
 | `maxReplicas` _integer_ | MaxReplicas denotes the maximum number of desired Pods for this worker group, and the default value is maxInt32. | 2147483647 |  |
 | `idleTimeoutSeconds` _integer_ | IdleTimeoutSeconds denotes the number of seconds to wait before the v2 autoscaler terminates an idle worker pod of this type.<br />This value is only used with the Ray Autoscaler enabled and defaults to the value set by the AutoscalingConfig if not specified for this worker group. |  |  |
+| `resources` _object (keys:string, values:string)_ | Resources specifies the resource quantities for this worker group.<br />These values override the resources passed to `rayStartParams` for the group, but<br />have no effect on the resources set at the K8s Pod container level. |  |  |
+| `labels` _object (keys:string, values:string)_ | Labels specifies the Ray node labels for this worker group.<br />These labels will also be added to the Pods of this worker group and override the `--labels`<br />argument passed to `rayStartParams`. |  |  |
 | `rayStartParams` _object (keys:string, values:string)_ | RayStartParams are the params of the start command: address, object-store-memory, ... |  |  |
 | `template` _[PodTemplateSpec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#podtemplatespec-v1-core)_ | Template is a pod template for the worker |  |  |
 | `scaleStrategy` _[ScaleStrategy](#scalestrategy)_ | ScaleStrategy defines which pods to remove |  |  |

helm-chart/kuberay-operator/crds/ray.io_rayclusters.yaml

@@ -75,6 +75,16 @@ type HeadGroupSpec struct {
 	// EnableIngress indicates whether operator should create ingress object for head service or not.
 	// +optional
 	EnableIngress *bool `json:"enableIngress,omitempty"`
+	// Resources specifies the resource quantities for the head group.
+	// These values override the resources passed to `rayStartParams` for the group, but
+	// have no effect on the resources set at the K8s Pod container level.
+	// +optional
+	Resources map[string]string `json:"resources,omitempty"`
+	// Labels specifies the Ray node labels for the head group.
+	// These labels will also be added to the Pods of this head group and override the `--labels`
+	// argument passed to `rayStartParams`.
+	// +optional
+	Labels map[string]string `json:"labels,omitempty"`
 	// RayStartParams are the params of the start command: node-manager-port, object-store-memory, ...
 	// +optional
 	RayStartParams map[string]string `json:"rayStartParams"`
@@ -106,6 +116,16 @@ type WorkerGroupSpec struct {
 	// This value is only used with the Ray Autoscaler enabled and defaults to the value set by the AutoscalingConfig if not specified for this worker group.
 	// +optional
 	IdleTimeoutSeconds *int32 `json:"idleTimeoutSeconds,omitempty"`
+	// Resources specifies the resource quantities for this worker group.
+	// These values override the resources passed to `rayStartParams` for the group, but
+	// have no effect on the resources set at the K8s Pod container level.
+	// +optional
+	Resources map[string]string `json:"resources,omitempty"`
+	// Labels specifies the Ray node labels for this worker group.
+	// These labels will also be added to the Pods of this worker group and override the `--labels`
+	// argument passed to `rayStartParams`.
+	// +optional
+	Labels map[string]string `json:"labels,omitempty"`
 	// RayStartParams are the params of the start command: address, object-store-memory, ...
 	// +optional
 	RayStartParams map[string]string `json:"rayStartParams"`