[RayService] Validate ray.io/num-worker-groups annotation to prevent operator panic - #5104
Conversation
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
|
The |
No I didn't hit it in practice, this issue got identified as a possible vulnerability in the Ray operator through vulnerability scanning of our managed component (Ray on GKE which installs the operator). KubeRay does always sets this to a non-negative value under normal operation, but since CR annotations are exposed via the Kubernetes API unexpected modifications of this field could trigger the panic and crash the operator pod. I'd agree the issue isn't a super high priority since it just causes a panic and would be due erroneous user modification of the CR, but I think it's still worth guarding against. This PR also cleans up the error branches where parse/hash failures previously returned |
Why are these changes needed?
In
RayServicecontroller'sisClusterSpecHashEqual, theray.io/num-worker-groupsannotation on aRayClusteris parsed viastrconv.Atoiwithout validating that the value is non-negative.If the annotation contains a negative integer (e.g.
"-1"):goalNumWorkerGroups >= clusterNumWorkerGroupsevaluates totrue(sincelen()is non-negative).goalClusterSpec.WorkerGroupSpecs[:clusterNumWorkerGroups]with a negative index triggers a runtime panic (slice bounds out of range), crashing the operator during reconciliation.Additionally:
strconv.Atoi) or hash generation failed, the function previously returnedtrue, incorrectly treating the specs as matching."" == "").Changes in this PR:
clusterNumWorkerGroupsis parsed successfully and isWorkerGroupSpecs.falseifclusterHashis empty, if annotation parsing fails, or if hash generation fails.TestIsClusterSpecHashEqualcovering negative values, invalid non-integer strings, empty annotations, and missing cluster hashes.Related issue number
Labels
doc-updates-requiredlabel.breaking-changelabel.Checks