KEP 5554: In place update pod resources alongside static cpu manager policy KEP creation #5555

esotsal · 2025-09-21T18:21:24Z

One-line PR description: Create new KEP 5554: In place update pod resources alongside static cpu manager policy

Issue link: Support In place update pod resources alongside static cpu manager policy #5554

Other comments:

k8s-ci-robot · 2025-09-21T18:21:32Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: esotsal
Once this PR has been reviewed and has the lgtm label, please assign johnbelamaric for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/prod-readiness/OWNERS
keps/sig-node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

esotsal · 2025-09-21T18:50:14Z

/cc @natasha41575 @tallclair @pravk03 @Chunxia202410 @ffromani

k8s-ci-robot · 2025-09-21T18:50:20Z

@esotsal: GitHub didn't allow me to request PR reviews from the following users: Chunxia202410.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @natasha41575 @tallclair @pravk03 @Chunxia202410 @ffromani

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Chunxia202410 · 2025-09-28T08:11:38Z

keps/sig-node/5554-in-place-update-pod-resources-alongside-static-cpu-manager-policy/README.md

+* The pod's current NUMA Affinity (single or multiple NUMA nodes) can accommodate the resized CPU request. If yes, the current affinity is maintained.
+* If the current affinity is insufficient and expansion is needed the new NUMA affinity must always include all NUMA nodes from the current affinity.


Hi, @esotsal , do you think it is necessary to generate more fine-grained NUMA hints here?
I consider an example for scale down.
The server has 2 NUMA nodes:
NUMA 0: CPUs {0,1,2,3,4,5,6,7}
NUMA 1: CPUs {8,9,10,11,12,13,14,15}
CPU 0 is reserved.

There is a Pod with 10 CPUs, and the CPUset is {4,5,8-15}, where CPUs 10 and 11 are promised CPUs.
When the POD scales down from 10 to 8 CPUs (actually requiring only 1 NUMA node).

According to the KEP design.
The CPU NUMA hints would be {0,1} with preferred = false (if I understand correctly).
Assuming the Memory and Device both have two NUMA hints: {1} with preferred = true, and {0,1} with preferred = false.
After merging, the best NUMA hint would be {0,1} with preferred = false (according to my understanding).
If the topology manager policy is restricted, since preferred = false for best NUMA hint, the POD resizing would be rejected.

Consider more fine-grained solution for scale down: NUMA hints should be the combination of nodes containing the original CPUs, and must include the promised CPUs.
The CPU NUMA hints would be {1} with preferred = true, and {0,1} with preferred = false
Assuming the Memory and Device both have two NUMA hints: {1} with preferred = true, and {0,1} with preferred = false.
After merging, the best NUMA hints would be {1} with preferred = true. If the topology manager policy is restricted, the POD would be resized.

If my understanding is incorrect, please feel free to point it out. Thank you.

I'm having a hard time picturing a downsize which can cause a topology manager policy failure. I can think of examples for upsize, but none for downsize. Point in case, a downsize is a transition which at worst makes the container shape as restrictive than the original allocation, can't be worse (e.g we can't downsize in a way which violate the single-numa-node policy: if the allocation spans across 2 nodes, at worse the downsized one will still span across 2 node).

While writing this comment, I realized that we should probably just make explicit that on downsize the end result must be a subset of the original result. This will help.

Yes, option 2 is what i was thinking as well @Chunxia202410 for the use case you describe. @ffromani has described it very well, will add this sentence in the KEP for clarity.

I have confirmed this behaviour, in this KEPs PR, against a VM with a same CPU topology as the example you have provided with restricted topology manager policy option, creating a pod with one container 2cpus, then scaling it up to 10, afterwards scaling down to 2. Will push the added tests ( as well with tests against other topology manager policies single-numa-mode, best-effort ) in the PR, need to do some clean up, will try to push the commit today or tomorrow.

ffromani

Gave an initial pass. Designwise mostly questions and some areas to be clarified, I didn't see anything unexpected or concerning.

ffromani · 2025-09-29T12:50:02Z

keps/sig-node/5554-in-place-update-pod-resources-alongside-static-cpu-manager-policy/README.md

+* prefer-closest-numa-modes ( GA, visible by default ) (1.32 or higher) [KEP-3545](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3545-improved-multi-numa-alignment)
+* max-allowable-numa-modes ( beta, visible by default ) (1.31 or higher)  [KEP-4622](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/4622-topologymanager-max-allowable-numa-nodes) 


these are policy options, not topology manager policies. The options alter the behavior of the policies (where applicable)

ffromani · 2025-09-29T12:50:52Z

keps/sig-node/5554-in-place-update-pod-resources-alongside-static-cpu-manager-policy/README.md

+* strict-cpu-reservation, 
+* prefer-align-cpus-by-uncorecache 
+
+The Kubelet requires the total CPU reservation from `--kube-reserved` and `--system-reserved` to be greater than zero when CPU management static policy is enabled. This KEP should ensure those reserved CPUs are kept during resize.


and/or --reserved-cpus

ffromani · 2025-09-29T12:59:03Z

keps/sig-node/5554-in-place-update-pod-resources-alongside-static-cpu-manager-policy/README.md

+* must keep “promised” CPUs of a running container of a Guaranteed Pod CPUs.
+  + With the term “promised” we mean the CPUs allocated upon creation of the Guaranteed Pod checkpointed in CPU Managers checkpoint file, please refer to [Promised CPUs checkpoint](#promised-cpus-checkpoint) for more details of the proposed implementation using local storage.
+* attempt of allocating additional CPUs should adhere to the combination of Topology/Memory/CPU/Device/kubelet reservation policies
+  + All 40 possible combinations should work


this reads scary but I'm fairly confident that we can qucikly rule out some combinations. For example, strict-cpu-reservation should never be a factor. OTOH ensuring continued correctness (= testing) may turn nontrivial.

ffromani · 2025-09-29T13:00:27Z

keps/sig-node/5554-in-place-update-pod-resources-alongside-static-cpu-manager-policy/README.md

+When the CPU Manager, under a static policy, generates a NUMA Topology hint for a Guaranteed pod undergoing an in place CPU pod-resize,  it follows these rules to determine the new affinity:
+
+* The pod's current NUMA Affinity (single or multiple NUMA nodes) can accommodate the resized CPU request. If yes, the current affinity is maintained.
+* If the current affinity is insufficient and expansion is needed the new NUMA affinity must always include all NUMA nodes from the current affinity.


I'm worried that extending the hint may violate some topology manager policies. But this should be handled at topology manager level. It makes sense for the cpumanager to generate that hint.

ffromani · 2025-09-29T13:26:00Z

keps/sig-node/5554-in-place-update-pod-resources-alongside-static-cpu-manager-policy/README.md

+* The pod's current NUMA Affinity (single or multiple NUMA nodes) can accommodate the resized CPU request. If yes, the current affinity is maintained.
+* If the current affinity is insufficient and expansion is needed the new NUMA affinity must always include all NUMA nodes from the current affinity.


I'm having a hard time picturing a downsize which can cause a topology manager policy failure. I can think of examples for upsize, but none for downsize. Point in case, a downsize is a transition which at worst makes the container shape as restrictive than the original allocation, can't be worse (e.g we can't downsize in a way which violate the single-numa-node policy: if the allocation spans across 2 nodes, at worse the downsized one will still span across 2 node).

While writing this comment, I realized that we should probably just make explicit that on downsize the end result must be a subset of the original result. This will help.

ffromani · 2025-09-29T13:58:44Z

keps/sig-node/5554-in-place-update-pod-resources-alongside-static-cpu-manager-policy/README.md

+* Under pkg/kubelet/types
+  + constants.go
+    + ErrorIncosistentCPUAllocation
+    + ErrorProhibitedCPUAllocation
+    + ErrorGetPromisedCPUSet


If these errors can bubble up to the user, and I think they can, we should be consistent in naming with TopologyAffinityError SMTAlignmentError and UnexpectedAdmissionError

ffromani · 2025-09-29T14:01:04Z

keps/sig-node/5554-in-place-update-pod-resources-alongside-static-cpu-manager-policy/README.md

+#### GA
+
+* User feedback (ideally from at least two distinct users) is green,
+* No major bugs reported for three months.
+* 2 examples of real-world usage
+* 2 installs
+* Allowing time for feedback


this may likely needed to moved to beta because the new graduation rules

ffromani · 2025-09-29T14:02:11Z

keps/sig-node/5554-in-place-update-pod-resources-alongside-static-cpu-manager-policy/README.md

+// CPUManagerCheckpoint struct is used to store cpu/pod assignments in a checkpoint in v3 format
+type CPUManagerCheckpoint struct {
+    PolicyName    string                       `json:"policyName"`
+    DefaultCPUSet string                       `json:"defaultCpuSet"`
+    Entries       map[string]map[string]string `json:"entries,omitempty"`
+    Promised      map[string]map[string]string `json:"promised,omitempty"`
+    Checksum      checksum.Checksum            `json:"checksum"`
+}


the "v2" format is now the default and the "v1" format is largely unsupported, this makes things easier. Note that "checkpoint version" is largely an internal construct, is never mentioned or set explicitly anywhere

ffromani · 2025-09-29T14:05:03Z

keps/sig-node/5554-in-place-update-pod-resources-alongside-static-cpu-manager-policy/README.md

+
+Inspect the kubelet configuration of the nodes: check feature gate and usage of the new option.
+
+Checking /var/lib/kubelet/cpu_manager_state, and look for `promised` field.


my only true nack so far. We should stop directing users to peek to this file, which is already largely abused. Is not, never was and won't be part of the API, yet is consumed and considered source of truth by quite few users, and this already makes things hard when we need to change it, like in this very KEP.

So we would need a different mechanism, be it a metric or a new field in the podresources API or in any other official API.

ffromani · 2025-09-29T14:05:55Z

keps/sig-node/5554-in-place-update-pod-resources-alongside-static-cpu-manager-policy/README.md

+
+In order to verify this feature is working, one should first inspect the kubelet configuration and ensure that all required feature gates are enabled as described in [Summary](#summary). 
+
+Then user should create a guaranteed QoS Pod with integer CPU requests, inspect the `/var/lib/kubelet/cpu_manager_state` and check promised CPU set, for the created PoD. Upon creation promised should be equal to the assigned CPU. If it exists it means the feature is enabled and working. 


ffromani · 2025-09-29T14:46:32Z

keps/prod-readiness/sig-node/5554.yaml

@@ -0,0 +1,3 @@
+kep-number: 5554
+alpha:
+  approver: "TBD"


not sure you need to put a name and let the PRR team rebalance. Let's check if this is sufficient to make sure the PRR team is aware of this KEP; if so, we can surely keep TBD

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Sep 21, 2025

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/node Categorizes an issue or PR as relevant to SIG Node. labels Sep 21, 2025

k8s-ci-robot requested review from dchen1107 and jeremyrickard September 21, 2025 18:21

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Sep 21, 2025

esotsal force-pushed the ippvs-alognside-static-cpu-policy-KEP branch from 4c5c393 to 1240d58 Compare September 21, 2025 18:24

esotsal changed the title ~~KEP 5554: In place update pod resources alongside static cpu manager policy~~ KEP 5554: In place update pod resources alongside static cpu manager policy KEP creation Sep 21, 2025

esotsal mentioned this pull request Sep 21, 2025

Support In place update pod resources alongside static cpu manager policy #5554

Open

7 tasks

esotsal changed the title ~~KEP 5554: In place update pod resources alongside static cpu manager policy KEP creation~~ [WIP] KEP 5554: In place update pod resources alongside static cpu manager policy KEP creation Sep 21, 2025

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 21, 2025

k8s-ci-robot requested review from tallclair, pravk03, ffromani and natasha41575 September 21, 2025 18:50

esotsal force-pushed the ippvs-alognside-static-cpu-policy-KEP branch from 1240d58 to 8973b16 Compare September 26, 2025 10:59

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 26, 2025

esotsal force-pushed the ippvs-alognside-static-cpu-policy-KEP branch 9 times, most recently from c651aa2 to 24bfb5c Compare September 26, 2025 17:11

esotsal force-pushed the ippvs-alognside-static-cpu-policy-KEP branch 9 times, most recently from a0b72b1 to abc61a0 Compare September 26, 2025 18:30

esotsal changed the title ~~[WIP] KEP 5554: In place update pod resources alongside static cpu manager policy KEP creation~~ KEP 5554: In place update pod resources alongside static cpu manager policy KEP creation Sep 26, 2025

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 26, 2025

Chunxia202410 reviewed Sep 28, 2025

View reviewed changes

kep/5554: Initial KEP creation

37fa4b5

esotsal force-pushed the ippvs-alognside-static-cpu-policy-KEP branch from abc61a0 to 37fa4b5 Compare September 29, 2025 09:03

ffromani reviewed Sep 29, 2025

View reviewed changes

		* The pod's current NUMA Affinity (single or multiple NUMA nodes) can accommodate the resized CPU request. If yes, the current affinity is maintained.
		* If the current affinity is insufficient and expansion is needed the new NUMA affinity must always include all NUMA nodes from the current affinity.

		* prefer-closest-numa-modes ( GA, visible by default ) (1.32 or higher) [KEP-3545](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3545-improved-multi-numa-alignment)
		* max-allowable-numa-modes ( beta, visible by default ) (1.31 or higher) [KEP-4622](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/4622-topologymanager-max-allowable-numa-nodes)


		Inspect the kubelet configuration of the nodes: check feature gate and usage of the new option.

		Checking /var/lib/kubelet/cpu_manager_state, and look for `promised` field.


		In order to verify this feature is working, one should first inspect the kubelet configuration and ensure that all required feature gates are enabled as described in [Summary](#summary).

		Then user should create a guaranteed QoS Pod with integer CPU requests, inspect the `/var/lib/kubelet/cpu_manager_state` and check promised CPU set, for the created PoD. Upon creation promised should be equal to the assigned CPU. If it exists it means the feature is enabled and working.

KEP 5554: In place update pod resources alongside static cpu manager policy KEP creation #5555

Are you sure you want to change the base?

KEP 5554: In place update pod resources alongside static cpu manager policy KEP creation #5555

Uh oh!

Conversation

esotsal commented Sep 21, 2025

Uh oh!

k8s-ci-robot commented Sep 21, 2025

Uh oh!

esotsal commented Sep 21, 2025

Uh oh!

k8s-ci-robot commented Sep 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

esotsal Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ffromani left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

esotsal Sep 30, 2025 •

edited

Loading