Enhance NVIDIA DRA driver and change the two pods to use one GPU with timeslice

[ronaldosaheki]

What does this PR do?

• Fixed DRA timeslicing example
• Updated DRA timeslicing and MPS documentation
• Updated nvidia-dra-driver.yaml to enable TimeSlicingSettings and MPSSupport.
• Introduced timeslicing-claim.yaml to define a shared ResourceClaim for GPU time-slicing.
• Modified timeslicing-pod.yaml to reference the new shared ResourceClaim.
• Added timeslicing-podgroup.yaml to support PodGroup scheduling with time-slicing.
• Expanded documentation on dynamic resource allocation to include time-slicing use cases and configurations.

Motivation

1. Missing helm configuration to enable DRA timeslicing and MPS.
2. Current example doesn't share gpu with two pods, it actually needs two gpus and setup with timeslice with one pod on each, so updated to make more sense using one gpu and two pods with timeslice.

More

• Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
• Mandatory for new blueprints. Yes, I have added a example to support my blueprint PR
• Mandatory for new blueprints. Yes, I have updated the website/docs or website/blog section for this feature
• Yes, I ran pre-commit run -a with this PR. Link for installing pre-commit locally

For Moderators

• E2E Test successfully complete before merge?

Additional Notes

resources created after apply:

kubectl get resourceclaim,pods -n timeslicing-gpu
NAME                                                   STATE                AGE
resourceclaim.resource.k8s.io/shared-timeslicing-gpu   allocated,reserved   112m

NAME                  READY   STATUS    RESTARTS   AGE
pod/inference-pod-1   1/1     Running   0          46s
pod/training-pod-2    1/1     Running   0          45s

checking resourceclaim used by both pods:

$ kubectl get resourceclaim shared-timeslicing-gpu -n timeslicing-gpu -o json | jq -r '
  "device=\(.status.allocation.devices.results[0].device // "n/a")",
  "node=\(.status.allocation.nodeSelector.nodeSelectorTerms[0].matchFields[0].values[0] // "n/a")",
  "reservedFor=\([.status.reservedFor[]?.name] | join(" "))"
'
device=gpu-0
node=ip-100-64-109-171.us-west-2.compute.internal
reservedFor=inference-pod-1 training-pod-2

checking both pods using the same gpu:

$ kubectl get pods -n timeslicing-gpu -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName
NAME              NODE
inference-pod-1   ....us-west-2.compute.internal
training-pod-2    .....us-west-2.compute.internal

$ kubectl exec -n timeslicing-gpu inference-pod-1 -- nvidia-smi -L
GPU 0: NVIDIA L4 (UUID: GPU-0072...5e89)

$ kubectl exec -n timeslicing-gpu training-pod-2 -- nvidia-smi -L
GPU 0: NVIDIA L4 (UUID: GPU-0072...5e89)

pre-commit run:

trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check for merge conflicts................................................Passed
detect private key.......................................................Passed
detect aws credentials...................................................Passed
Terraform fmt............................................................Passed
Terraform docs...........................................................Passed
Terraform validate with tflint...........................................Passed
Terraform validate.......................................................Passed
TruffleHog...............................................................Passed

MPS check:

$ kubectl get pods,resourceclaim -n mps-gpu 2>&1; echo "===CLAIM==="; kubectl get resourceclaim -n mps-gpu -o json 2>/dev/null | jq -r '"device=\(.items[0].status.allocation.devices.results[0].device // "n/a")", "sharing=\(.items[0].status.allocation.devices.config[0].opaque.parameters.sharing.strategy // "n/a")"'; echo "===MPS DAEMON==="; kubectl get pods -n nvidia-dra-driver-gpu 2>/dev/null | grep -E 'mps|NAME'; echo "===CUDA INF==="; kubectl exec -n mps-gpu mps-multi-container-pod -c inference-container -- python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))" 2>&1; echo "===CUDA TRAIN==="; kubectl exec -n mps-gpu mps-multi-container-pod -c training-container -- python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))" 2>&1; echo "===PS==="; kubectl exec -n mps-gpu mps-multi-container-pod -c inference-container -- ps aux 2>&1 | grep -E 'python|USER'; kubectl exec -n mps-gpu mps-multi-container-pod -c training-container -- ps aux 2>&1 | grep -E 'python|USER'; echo "===SMI==="; kubectl exec -n mps-gpu mps-multi-container-pod -c training-container -- nvidia-smi 2>&1
NAME                          READY   STATUS    RESTARTS   AGE
pod/mps-multi-container-pod   2/2     Running   0          7m31s

NAME                                                                           STATE                AGE
resourceclaim.resource.k8s.io/mps-multi-container-pod-shared-gpu-claim-6sm2k   allocated,reserved   7m32s
===CLAIM===
device=gpu-0
sharing=MPS
===MPS DAEMON===
NAME                                                              READY   STATUS    RESTARTS   AGE
mps-control-daemon-445ccb5d-e0c0-471f-b88a-ff21c8cdf269-c3t854h   1/1     Running   0          7m33s
===CUDA INF===
True NVIDIA L4
===CUDA TRAIN===
True NVIDIA L4
===PS===
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.7  1.1 10800632 736548 ?     Ssl  12:28   0:03 python /scripts/inference-script.py
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  1.0  1.6 11327968 1026472 ?    Ssl  12:28   0:04 python /scripts/training-script.py
===SMI===
Tue May 26 12:36:35 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03             Driver Version: 580.159.03     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                      On  |   00000000:35:00.0 Off |                    0 |
| N/A   49C    P0             34W /   72W |     603MiB /  23034MiB |      0%   E. Process |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A               1    M+C   python                                  316MiB |
+-----------------------------------------------------------------------------------------+

[erezzarum]

can you validate if this is still needed when using the nvidia dra driver from k8s sig?