Understanding why nvidia.com/gpu.shared is not decremented under a node's status.allocatable upon scheduling

[chipzoller]

I'm curious to understand why a Node's status.allocatable value for nvidia.com/gpu.shared is not decremented when Pods are scheduled on a node using a GPU sharing strategy. For example, when enabling time-slicing on a T4 in an EKS cluster and using v0.15.1 of the device plugin, with renameByDefault: true there are a number of replicas made available. These show correctly in a Node's status.capacity and the Node label nvidia.com/gpu.replicas reflects this properly as does the nvidia.com/gpu.sharing-strategy. But when Pods are scheduled which use one of these GPU slices, the value is never adjusted in the status.allocatable field. That said, scheduling of additional Pods beyond the replica limit is not successful (as expected). The logs for device plugin reveal nothing helpful, so I'm wondering if this is by design or possibly a bug. The documentation does not make mention of this and a cursory search of the issues in this repository does not bring up anything similar. Is this behavior expected or unexpected, @elezar or @klueska? Thanks for any commentary on this discussion.

[klueska]

status.allocatable is never decremented as resources get allocated for any extended resource type. This is independent of gpu.shared resources, as it is true for every extended resource.

This is a property of Kubernetes itself and not something under the control of individual device plugins.

The intent of status.allocatable is not to show the "real time" number of resources that can be Allocatable, but rather the total jumper of resources that could be allocated (independent of if they currently are or not).

This is different than status.capacity which shows the total number of devices that have ever been available for a given resource type (including those that may have gone unhealthy over time). Whereas status.allocatable only shows those devices that are healthy now and could be allocated at some point now or in the future (at least until they possibly go unhealthy at some point).

[chipzoller]

Thanks for the information. Although I could have sworn I saw this value decrement when using exclusive GPUs, on another test in a node with two GPUs I do not see this behavior.