You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@shengnuo as requested in #2552, here is the vGPU side of the same problem.
I would like the vGPU Manager images to ship Fabric Manager and start it on NVSwitch hosts, with the mode set through the mechanism NVIDIA/gpu-driver-container#893 already added, so that the mode comes from NVFM_CONFIG_FABRIC_MODE=2 rather than from a second knob invented for vGPU.
The Fabric Manager User Guide says the vGPU host image must have the FM package and the FM SDK package installed, and "To support vGPU virtualization, start the FM service in vGPU virtualization mode by setting the FABRIC_MODE=2 FM config item". Partition activation is always required before starting a vGPU VM, even for VMs that use only one vGPU. On an 8-GPU HGX H200 host, with FM left in the default bare metal mode, a whole-card vGPU guest could not initialize CUDA: cuInit returned 802 and the guest reported fabric status "Insufficient Permissions". Mode 2 plus per-VM partition activation fixed it. The activation itself lives in NVIDIA/kubevirt-gpu-device-plugin#193.
In gpu-driver-container the nvidia-driver entrypoint collects NVFM_CONFIG_* from the environment and writes those keys into fabricmanager.cfg before starting FM. None of that reaches the vGPU Manager images. I find no Fabric Manager handling under vgpu-manager/, so those images neither install nor start it. In gpu-operator neither FABRIC_MODE nor NVFM_CONFIG_ appears anywhere, though vgpuManager.env already reaches the container and the vGPU host manager daemonset rendered from an NVIDIADriver CR already mounts /run/nvidia-fabricmanager from the host.
So on an operator-managed NVSwitch vGPU node, FM still has to be installed and configured out of band.
Two questions before any code:
Is this wanted on the vGPU Manager path at all, given that the fabric work is heading into the DRA driver? If vGPU on NVSwitch is meant to be served there instead, I would rather hear it now.
I have two drafts from before #893 that use their own FABRIC_MODE variable, #2602 and NVIDIA/gpu-driver-container#854. If the answer to the first question is yes, I will rebase both onto NVFM_CONFIG_FABRIC_MODE and open them for review.
@shengnuo as requested in #2552, here is the vGPU side of the same problem.
I would like the vGPU Manager images to ship Fabric Manager and start it on NVSwitch hosts, with the mode set through the mechanism NVIDIA/gpu-driver-container#893 already added, so that the mode comes from
NVFM_CONFIG_FABRIC_MODE=2rather than from a second knob invented for vGPU.The Fabric Manager User Guide says the vGPU host image must have the FM package and the FM SDK package installed, and "To support vGPU virtualization, start the FM service in vGPU virtualization mode by setting the FABRIC_MODE=2 FM config item". Partition activation is always required before starting a vGPU VM, even for VMs that use only one vGPU. On an 8-GPU HGX H200 host, with FM left in the default bare metal mode, a whole-card vGPU guest could not initialize CUDA:
cuInitreturned 802 and the guest reported fabric status "Insufficient Permissions". Mode 2 plus per-VM partition activation fixed it. The activation itself lives in NVIDIA/kubevirt-gpu-device-plugin#193.In gpu-driver-container the
nvidia-driverentrypoint collectsNVFM_CONFIG_*from the environment and writes those keys intofabricmanager.cfgbefore starting FM. None of that reaches the vGPU Manager images. I find no Fabric Manager handling undervgpu-manager/, so those images neither install nor start it. In gpu-operator neitherFABRIC_MODEnorNVFM_CONFIG_appears anywhere, thoughvgpuManager.envalready reaches the container and the vGPU host manager daemonset rendered from an NVIDIADriver CR already mounts/run/nvidia-fabricmanagerfrom the host.So on an operator-managed NVSwitch vGPU node, FM still has to be installed and configured out of band.
Two questions before any code:
I have two drafts from before #893 that use their own
FABRIC_MODEvariable, #2602 and NVIDIA/gpu-driver-container#854. If the answer to the first question is yes, I will rebase both ontoNVFM_CONFIG_FABRIC_MODEand open them for review.