Skip to content

[Feature]: Start Fabric Manager in vGPU multitenancy mode (NVFM_CONFIG_FABRIC_MODE=2) on the vGPU Manager path #2923

Description

@lexfrei

@shengnuo as requested in #2552, here is the vGPU side of the same problem.

I would like the vGPU Manager images to ship Fabric Manager and start it on NVSwitch hosts, with the mode set through the mechanism NVIDIA/gpu-driver-container#893 already added, so that the mode comes from NVFM_CONFIG_FABRIC_MODE=2 rather than from a second knob invented for vGPU.

The Fabric Manager User Guide says the vGPU host image must have the FM package and the FM SDK package installed, and "To support vGPU virtualization, start the FM service in vGPU virtualization mode by setting the FABRIC_MODE=2 FM config item". Partition activation is always required before starting a vGPU VM, even for VMs that use only one vGPU. On an 8-GPU HGX H200 host, with FM left in the default bare metal mode, a whole-card vGPU guest could not initialize CUDA: cuInit returned 802 and the guest reported fabric status "Insufficient Permissions". Mode 2 plus per-VM partition activation fixed it. The activation itself lives in NVIDIA/kubevirt-gpu-device-plugin#193.

In gpu-driver-container the nvidia-driver entrypoint collects NVFM_CONFIG_* from the environment and writes those keys into fabricmanager.cfg before starting FM. None of that reaches the vGPU Manager images. I find no Fabric Manager handling under vgpu-manager/, so those images neither install nor start it. In gpu-operator neither FABRIC_MODE nor NVFM_CONFIG_ appears anywhere, though vgpuManager.env already reaches the container and the vGPU host manager daemonset rendered from an NVIDIADriver CR already mounts /run/nvidia-fabricmanager from the host.

So on an operator-managed NVSwitch vGPU node, FM still has to be installed and configured out of band.

Two questions before any code:

  1. Is this wanted on the vGPU Manager path at all, given that the fabric work is heading into the DRA driver? If vGPU on NVSwitch is meant to be served there instead, I would rather hear it now.
  2. If it is wanted, does the GPU reset added in [B200+][NVLink5+] reset GPUs after FABRIC_MODE=1 is set gpu-driver-container#909 for NVLink5 systems apply to mode 2? Today it triggers on the literal value 1.

I have two drafts from before #893 that use their own FABRIC_MODE variable, #2602 and NVIDIA/gpu-driver-container#854. If the answer to the first question is yes, I will rebase both onto NVFM_CONFIG_FABRIC_MODE and open them for review.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions