openvmm - PCIe attached NVMe emulator and virtio vpci conflict #2257
Description
When using the new PCIe roots for nvme storage (#2194 ), the storage controller fails to work in the guest OS if vmbus is also enabled. This appears to be because a virtio vpci serial device is also automatically added, regardless of whether it was requested on the command line (openvmm/hvlite_core/src/worker/dispatch.rs line 2045). Commenting out this line allows the nvme controller to work (as long as no other vpci device is explicitly added?).
if partition.supports_virtual_devices() {
if vmbus_server.is_some() {
let serial = VirtioSerialDevice::new(1, &gm);
vpci_serial = Some(serial.io());
add_virtio_vpci(
&driver_source,
&partition,
&vmbus_server,
&mapper,
"virtio-serial-vpci",
&mut chipset_builder,
Box::new(LegacyWrapper::new(&driver_source, serial, &gm)),
)
.await?;
}Command line:
openvmm.exe -m 16GB --uefi --disk memdiff:file:D:\VM\Fedora.vhdx --net consomme --nvme mem:4GB,pcie_port=rp1 --pcie-root-complex rc0 --pcie-root-port rc0:rp0 --pcie-root-port rc0:rp1 --uefi-firmware "MSVM.fd" --com1=term
It doesn't require two pcie-root-complexes to repro the issue; creating two was just one of the steps I did along the way of debugging this.
Without code changes
- lspci shows that the nvme controller is visible, but the kernel driver is not in use for the device.
- dmesg shows that the nvme driver is failing because the address range requested by the nvme controller BAR 0 is already in use
- /proc/iomem shows that the address range in question is reserved by "0d01af27-7a73-4c24-b7cf-055467d0ea99"
- hwinfo shows that device 0d01af27 is the virtio-console
- This is not the requested address range for the device though. I don't know why it is taking this address range as well as the one it actually wants.
- My extremely naive theory is that the system is mixing up the vpci and pcie roots?
- I didn't want a virtio vpci serial console in the first place.
- This is being automatically added via the code linked above.
[root@fedora ~]# lspci -knn
0000:00:00.0 PCI bridge [0604]: Microsoft Corporation Device [1414:c030]
0000:00:01.0 PCI bridge [0604]: Microsoft Corporation Device [1414:c030]
0000:02:00.0 Non-Volatile memory controller [0108]: Microsoft Corporation Device [1414:00a9]
Kernel modules: nvme
7a73:00:00.0 System peripheral [0880]: Red Hat, Inc. Virtio 1.0 console [1af4:1043] (rev 01)
Subsystem: Red Hat, Inc. Device [1af4:0040]
Kernel driver in use: virtio-pci
[root@fedora ~]# dmesg | grep nvme
[ 5.861193] nvme 0000:02:00.0: BAR 0: can't reserve [mem 0xf8000000-0xf800ffff 64bit]
[root@fedora ~]# cat /proc/iomem
00000000-00000fff : Reserved
00001000-0009ffff : System RAM
000c0000-000fffff : Reserved
000f0000-000fffff : System ROM
00100000-044fdfff : System RAM
044fe000-048fdfff : Reserved
048fe000-f5826fff : System RAM
62c00000-6418378e : Kernel code
64200000-6538efff : Kernel rodata
65400000-6586f9bf : Kernel data
66400000-669fffff : Kernel bss
f5827000-f582efff : Reserved
f582f000-f7f40fff : System RAM
f7f41000-f7fc4fff : Reserved
f7fc5000-f7fd0fff : System RAM
f7fd1000-f7ffafff : ACPI Tables
f7ffb000-f7ffefff : ACPI Non-volatile Storage
f7fff000-f7ffffff : System RAM
f8000000-f80fffff : PCI Bus 0000:02
f8000000-f800ffff : 0000:02:00.0
f8000000-f8001fff : 0d01af27-7a73-4c24-b7cf-055467d0ea99
f8010000-f8010fff : 0000:02:00.0
fec00000-fec00fff : PNP0003:00
fec00000-fec003ff : IOAPIC 0
fee00000-fee00fff : PNP0003:00
100000000-407ffffff : System RAM
800000000-80fffffff : PCI ECAM 0000 [bus 00-ff]
800000000-80fffffff : pnp 00:03
fe0000000-fe0001fff : 0d01af27-7a73-4c24-b7cf-055467d0ea99
fe0000000-fe0000fff : 7a73:00:00.0
fe0000000-fe0000fff : virtio-pci-modern
fe0001000-fe0001fff : 7a73:00:00.0
[root@fedora ~]# hwinfo
<snip>
>> pci.10: virtio
virtio device: name = virtio0
path = /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0004:00/MSFT1000:00/0d01af27-7a73-4c24-b7cf-055467d0ea99/pci7a73:00/7a73:00:00.0/virtio0
driver = "virtio_console"
modalias = "virtio:d00000003v00001AF4"
device = 3
<snip>
09: PCI 200.0: 0108 Non-Volatile memory controller (NVM Express)
[Created at pci.386]
Unique ID: B35A.UDhxi8+dlA3
Parent ID: vSkL.4DvVMVTTG8A
SysFS ID: /devices/pci0000:00/0000:00:01.0/0000:02:00.0
SysFS BusID: 0000:02:00.0
Hardware Class: storage
Model: "Microsoft Non-Volatile memory controller"
Vendor: pci 0x1414 "Microsoft Corporation"
Device: pci 0x00a9
Memory Range: 0xf8000000-0xf800ffff (rw,non-prefetchable,disabled)
Memory Range: 0xf8010000-0xf8010fff (rw,non-prefetchable,disabled)
Module Alias: "pci:v00001414d000000A9sv00000000sd00000000bc01sc08i02"
Driver Info #0:
Driver Status: nvme is active
Driver Activation Cmd: "modprobe nvme"
Config Status: cfg=new, avail=yes, need=no, active=unknown
Attached to: #10 (PCI bridge)
<snip>
12: PCI 7a730000.0: 0880 System peripheral
[Created at pci.386]
Unique ID: G9sP.S9mwuZjekqB
SysFS ID: /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0004:00/MSFT1000:00/0d01af27-7a73-4c24-b7cf-055467d0ea99/pci7a73:00/7a73:00:00.0
SysFS BusID: 7a73:00:00.0
Hardware Class: unknown
Model: "Red Hat Virtio console"
Vendor: pci 0x1af4 "Red Hat, Inc."
Device: pci 0x1043 "Virtio console"
SubVendor: pci 0x1af4 "Red Hat, Inc."
SubDevice: pci 0x0040
Revision: 0x01
Driver: "virtio-pci"
Driver Modules: "virtio_pci"
Memory Range: 0xfe0000000-0xfe0000fff (rw,non-prefetchable)
Memory Range: 0xfe0001000-0xfe0001fff (rw,non-prefetchable)
Module Alias: "pci:v00001AF4d00001043sv00001AF4sd00000040bc08sc80i00"
Config Status: cfg=new, avail=yes, need=no, active=unknown
<snip>
15: None 00.0: 0000 Unclassified device
[Created at pci.1723]
Unique ID: oLkU.V9DMoCY1vr5
Parent ID: G9sP.S9mwuZjekqB
SysFS ID: /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0004:00/MSFT1000:00/0d01af27-7a73-4c24-b7cf-055467d0ea99/pci7a73:00/7a73:00:00.0/virtio0
SysFS BusID: virtio0
Hardware Class: unknown
Model: "Virtio Unclassified device"
Vendor: int 0x6014 "Virtio"
Device: int 0x0003
Driver: "virtio_console"
Module Alias: "virtio:d00000003v00001AF4"
Config Status: cfg=new, avail=yes, need=no, active=unknown
Attached to: #12 (System peripheral)
After commenting line
- Kernel driver is in use for the nvme controller
- No errors in dmesg
- /proc/iomem shows BAR 0 and BAR 4 are both mapped properly by the nvme kernel driver
[root@fedora ~]# lspci -knn
00:00.0 PCI bridge [0604]: Microsoft Corporation Device [1414:c030]
00:01.0 PCI bridge [0604]: Microsoft Corporation Device [1414:c030]
02:00.0 Non-Volatile memory controller [0108]: Microsoft Corporation Device [1414:00a9]
Kernel driver in use: nvme
Kernel modules: nvme
[root@fedora ~]# cat /proc/iomem
00000000-00000fff : Reserved
00001000-0009ffff : System RAM
000c0000-000fffff : Reserved
000f0000-000fffff : System ROM
00100000-044fdfff : System RAM
044fe000-048fdfff : Reserved
048fe000-f5826fff : System RAM
12400000-1398378e : Kernel code
13a00000-14b8efff : Kernel rodata
14c00000-1506f9bf : Kernel data
15c00000-161fffff : Kernel bss
f5827000-f582efff : Reserved
f582f000-f7f40fff : System RAM
f7f41000-f7fc4fff : Reserved
f7fc5000-f7fd0fff : System RAM
f7fd1000-f7ffafff : ACPI Tables
f7ffb000-f7ffefff : ACPI Non-volatile Storage
f7fff000-f7ffffff : System RAM
f8000000-f80fffff : PCI Bus 0000:02
f8000000-f800ffff : 0000:02:00.0
f8000000-f800ffff : nvme
f8010000-f8010fff : 0000:02:00.0
f8010000-f8010fff : nvme
fec00000-fec00fff : PNP0003:00
fec00000-fec003ff : IOAPIC 0
fee00000-fee00fff : PNP0003:00
100000000-407ffffff : System RAM
800000000-80fffffff : PCI ECAM 0000 [bus 00-ff]
800000000-80fffffff : pnp 00:03
[root@fedora ~]# nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1 /dev/ng0n1 SN: 000001 MSFT NVMe Accelerator v1.0 0x1 4.29 GB / 4.29 GB 512 B + 0 B v1.00000
Metadata
Metadata
Assignees
Labels
No labels