Skip to content

Conversation

@marmarek
Copy link
Member

@marmarek marmarek commented Mar 16, 2025

PCI SBDF may change if another device is added/removed from the system.
Use PCI device path to avoid this issue. The PCI device path is more or
less sysfs path, but with bus number behind bridges replaced with relative
bus number to bridge's secondary bus number. Each component of the path is
separated with '-', as that is allowed in qrexec arguments.

For example device 0000:c1:00.0 behind bridge at 0000:00:08.1 will get path
0000_00_08.1-00_00.0 (assuming bridge 0000:00:08.1 has secondary bus at
0xc1). Resolving device path requires access to the system's sysfs.

PCIDevice object will automatically resolve old device identifiers, which
partially covers migration path.

TODO:

  • finalize migration path - adjust device assignments too
  • consider still allowing attaching via SBDF, or otherwise expose function
    to translate SBDF to PCI path (an Admin API call?)
  • fix salt listing/attaching devices to sys-net/sys-usb - depending on
    the above
  • support path-based PCI in the dracut module too

Fixes QubesOS/qubes-issues#8681

This goes further in the direction of d2d2ad8 "pci: get device class
out of libvirt xml".
PCI device function is just one hex digit.
The issue was about device with segment part exceeding 16 bits, which
technically is not spec compliant. Adjust the test to be about a device
that was problematic initially.

QubesOS/qubes-issues#6932
@codecov
Copy link

codecov bot commented Mar 16, 2025

Codecov Report

Attention: Patch coverage is 88.23529% with 10 lines in your changes missing coverage. Please review.

Project coverage is 70.31%. Comparing base (a16975e) to head (20e3c20).
Report is 25 commits behind head on main.

Files with missing lines Patch % Lines
qubes/ext/pci.py 60.00% 6 Missing ⚠️
qubes/utils.py 94.02% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #667      +/-   ##
==========================================
+ Coverage   70.14%   70.31%   +0.16%     
==========================================
  Files          59       59              
  Lines       12726    12819      +93     
==========================================
+ Hits         8927     9014      +87     
- Misses       3799     3805       +6     
Flag Coverage Δ
unittests 70.31% <88.23%> (+0.16%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@qubesos-bot
Copy link

qubesos-bot commented Mar 17, 2025

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025041403-4.3&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025031804-4.3&flavor=update

  • system_tests_whonix

    • whonixcheck: wait_serial (wait serial expected)
      # wait_serial expected: qr/_oatY-\d+-/...

    • whonixcheck: Failed (test died + timed out)
      # Test died: command 'qvm-run -ap anon-whonix 'LC_ALL=C whonixchec...

  • system_tests_kde_gui_interactive

    • gui_keyboard_layout: wait_serial (wait serial expected)
      # wait_serial expected: "echo -e '[Layout]\nLayoutList=us,de' | sud...

    • gui_keyboard_layout: Failed (test died)
      # Test died: command 'test "$(cd ~user;ls e1*)" = "$(qvm-run -p wor...

  • system_tests_qwt_win10_seamless@hw13

    • windows_clipboard_and_filecopy: unnamed test (unknown)
    • windows_clipboard_and_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'windows-Explorer-empt...
  • system_tests_qwt_win11@hw13

    • windows_install: wait_serial (wait serial expected)
      # wait_serial expected: qr/dcWzE-\d+-/...

    • windows_install: Failed (test died + timed out)
      # Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...

  • system_tests_basic_vm_qrexec_gui_ext4

    • TC_30_Gui_daemon: test_000_clipboard (error)
      FileNotFoundError: [Errno 2] No such file or directory: '/var/run/q...
  • system_tests_basic_vm_qrexec_gui_zfs

    • startup: wait_serial (wait serial expected)
      # wait_serial expected: qr/PS1="root# "/u...
  • system_tests_suspend@hw1

    • suspend: unnamed test (unknown)
    • suspend: Failed (test died)
      # Test died: no candidate needle with tag(s) 'SUSPEND-FAILED' match...

Failed tests

16 failures
  • system_tests_whonix

    • whonixcheck: wait_serial (wait serial expected)
      # wait_serial expected: qr/_oatY-\d+-/...

    • whonixcheck: Failed (test died + timed out)
      # Test died: command 'qvm-run -ap anon-whonix 'LC_ALL=C whonixchec...

  • system_tests_suspend

    • suspend: unnamed test (unknown)
    • suspend: Failed (test died)
      # Test died: no candidate needle with tag(s) 'SUSPEND-FAILED' match...
  • system_tests_kde_gui_interactive

    • gui_keyboard_layout: wait_serial (wait serial expected)
      # wait_serial expected: "echo -e '[Layout]\nLayoutList=us,de' | sud...

    • gui_keyboard_layout: Failed (test died)
      # Test died: command 'test "$(cd ~user;ls e1*)" = "$(qvm-run -p wor...

  • system_tests_whonix@hw7

    • whonixcheck: fail (unknown)
      Whonixcheck for sys-whonix failed...

    • whonixcheck: unnamed test (unknown)

  • system_tests_qwt_win10_seamless@hw13

    • windows_clipboard_and_filecopy: unnamed test (unknown)
    • windows_clipboard_and_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'windows-Explorer-empt...
  • system_tests_qwt_win11@hw13

    • windows_install: wait_serial (wait serial expected)
      # wait_serial expected: qr/dcWzE-\d+-/...

    • windows_install: Failed (test died + timed out)
      # Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...

  • system_tests_basic_vm_qrexec_gui_ext4

    • TC_30_Gui_daemon: test_000_clipboard (error)
      FileNotFoundError: [Errno 2] No such file or directory: '/var/run/q...
  • system_tests_basic_vm_qrexec_gui_zfs

    • startup: wait_serial (wait serial expected)
      # wait_serial expected: qr/PS1="root# "/u...
  • system_tests_suspend@hw1

    • suspend: unnamed test (unknown)
    • suspend: Failed (test died)
      # Test died: no candidate needle with tag(s) 'SUSPEND-FAILED' match...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/132953#dependencies

10 fixed
  • system_tests_whonix

    • whonixcheck: fail (unknown)
      Whonixcheck for sys-whonix failed...

    • whonixcheck: unnamed test (unknown)

  • system_tests_basic_vm_qrexec_gui

  • system_tests_qrexec

  • system_tests_kde_gui_interactive

    • clipboard_and_web: unnamed test (unknown)

    • clipboard_and_web: Failed (test died)
      # Test died: no candidate needle with tag(s) 'qubes-website' matche...

    • clipboard_and_web: wait_serial (wait serial expected)
      # wait_serial expected: "lspci; echo 2E8vz-\$?-"...

  • system_tests_guivm_vnc_gui_interactive

    • gui_filecopy: unnamed test (unknown)
    • gui_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'files-work' matched...
  • system_tests_audio

Unstable tests

Details

Performance Tests

Performance degradation:

16 performance degradations
  • dom0_root_seq1m_q8t1_read 3:read_bandwidth_kb: 197883.00 :small_red_triangle_up: ( previous job: 446963.00, degradation: 44.27%)
  • dom0_root_seq1m_q1t1_read 3:read_bandwidth_kb: 81084.00 :small_red_triangle_up: ( previous job: 294295.00, degradation: 27.55%)
  • dom0_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 35814.00 :small_red_triangle_up: ( previous job: 79803.00, degradation: 44.88%)
  • dom0_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 3944.00 :small_red_triangle_up: ( previous job: 4826.00, degradation: 81.72%)
  • dom0_varlibqubes_seq1m_q8t1_write 3:write_bandwidth_kb: 182791.00 :small_red_triangle_up: ( previous job: 250795.00, degradation: 72.88%)
  • fedora-41-xfce_root_seq1m_q1t1_write 3:write_bandwidth_kb: 43999.00 :small_red_triangle_up: ( previous job: 87940.00, degradation: 50.03%)
  • fedora-41-xfce_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 2197.00 :small_red_triangle_up: ( previous job: 3599.00, degradation: 61.04%)
  • fedora-41-xfce_private_rnd4k_q1t1_write 3:write_bandwidth_kb: 651.00 :small_red_triangle_up: ( previous job: 1130.00, degradation: 57.61%)
  • fedora-41-xfce_volatile_seq1m_q8t1_write 3:write_bandwidth_kb: 158116.00 :small_red_triangle_up: ( previous job: 179949.00, degradation: 87.87%)
  • fedora-41-xfce_volatile_rnd4k_q32t1_write 3:write_bandwidth_kb: 2999.00 :small_red_triangle_up: ( previous job: 5672.00, degradation: 52.87%)
  • fedora-41-xfce_volatile_rnd4k_q1t1_write 3:write_bandwidth_kb: 1036.00 :small_red_triangle_up: ( previous job: 1953.00, degradation: 53.05%)
  • debian-12-xfce_exec: 8.57 :small_red_triangle_up: ( previous job: 7.12, degradation: 120.38%)
  • whonix-gateway-17_socket: 9.31 :small_red_triangle_up: ( previous job: 7.24, degradation: 128.64%)
  • whonix-gateway-17_exec-data-duplex-root: 100.06 :small_red_triangle_up: ( previous job: 88.52, degradation: 113.04%)
  • whonix-workstation-17_exec: 8.75 :small_red_triangle_up: ( previous job: 7.67, degradation: 114.14%)
  • whonix-workstation-17_socket: 9.13 :small_red_triangle_up: ( previous job: 8.19, degradation: 111.35%)

Remaining performance tests:

56 tests
  • dom0_root_seq1m_q8t1_write 3:write_bandwidth_kb: 188899.00 :green_circle: ( previous job: 129298.00, improvement: 146.10%)
  • dom0_root_seq1m_q1t1_write 3:write_bandwidth_kb: 180270.00 :green_circle: ( previous job: 95454.00, improvement: 188.86%)
  • dom0_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 10794.00 :green_circle: ( previous job: 6149.00, improvement: 175.54%)
  • dom0_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 11821.00 :green_circle: ( previous job: 10795.00, improvement: 109.50%)
  • dom0_varlibqubes_seq1m_q8t1_read 3:read_bandwidth_kb: 496484.00 :green_circle: ( previous job: 382273.00, improvement: 129.88%)
  • dom0_varlibqubes_seq1m_q1t1_read 3:read_bandwidth_kb: 420945.00 :small_red_triangle_up: ( previous job: 437636.00, degradation: 96.19%)
  • dom0_varlibqubes_seq1m_q1t1_write 3:write_bandwidth_kb: 178433.00 :small_red_triangle_up: ( previous job: 184752.00, degradation: 96.58%)
  • dom0_varlibqubes_rnd4k_q32t1_read 3:read_bandwidth_kb: 99347.00 :green_circle: ( previous job: 62195.00, improvement: 159.73%)
  • dom0_varlibqubes_rnd4k_q32t1_write 3:write_bandwidth_kb: 8299.00 :green_circle: ( previous job: 6479.00, improvement: 128.09%)
  • dom0_varlibqubes_rnd4k_q1t1_read 3:read_bandwidth_kb: 8269.00 :green_circle: ( previous job: 7669.00, improvement: 107.82%)
  • dom0_varlibqubes_rnd4k_q1t1_write 3:write_bandwidth_kb: 4748.00 :small_red_triangle_up: ( previous job: 4903.00, degradation: 96.84%)
  • fedora-41-xfce_root_seq1m_q8t1_read 3:read_bandwidth_kb: 384657.00 :green_circle: ( previous job: 368309.00, improvement: 104.44%)
  • fedora-41-xfce_root_seq1m_q8t1_write 3:write_bandwidth_kb: 213211.00 :green_circle: ( previous job: 162081.00, improvement: 131.55%)
  • fedora-41-xfce_root_seq1m_q1t1_read 3:read_bandwidth_kb: 302444.00 :small_red_triangle_up: ( previous job: 318716.00, degradation: 94.89%)
  • fedora-41-xfce_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 88569.00 :green_circle: ( previous job: 82694.00, improvement: 107.10%)
  • fedora-41-xfce_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 8841.00 :green_circle: ( previous job: 8485.00, improvement: 104.20%)
  • fedora-41-xfce_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 1504.00 :green_circle: ( previous job: 542.00, improvement: 277.49%)
  • fedora-41-xfce_private_seq1m_q8t1_read 3:read_bandwidth_kb: 356658.00 :small_red_triangle_up: ( previous job: 373957.00, degradation: 95.37%)
  • fedora-41-xfce_private_seq1m_q8t1_write 3:write_bandwidth_kb: 166697.00 :small_red_triangle_up: ( previous job: 170062.00, degradation: 98.02%)
  • fedora-41-xfce_private_seq1m_q1t1_read 3:read_bandwidth_kb: 341444.00 :green_circle: ( previous job: 334687.00, improvement: 102.02%)
  • fedora-41-xfce_private_seq1m_q1t1_write 3:write_bandwidth_kb: 84430.00 :green_circle: ( previous job: 61534.00, improvement: 137.21%)
  • fedora-41-xfce_private_rnd4k_q32t1_read 3:read_bandwidth_kb: 86624.00 :green_circle: ( previous job: 80283.00, improvement: 107.90%)
  • fedora-41-xfce_private_rnd4k_q32t1_write 3:write_bandwidth_kb: 2219.00 :green_circle: ( previous job: 2215.00, improvement: 100.18%)
  • fedora-41-xfce_private_rnd4k_q1t1_read 3:read_bandwidth_kb: 7814.00 :green_circle: ( previous job: 7540.00, improvement: 103.63%)
  • fedora-41-xfce_volatile_seq1m_q8t1_read 3:read_bandwidth_kb: 353412.00 :small_red_triangle_up: ( previous job: 369868.00, degradation: 95.55%)
  • fedora-41-xfce_volatile_seq1m_q1t1_read 3:read_bandwidth_kb: 316790.00 :small_red_triangle_up: ( previous job: 324737.00, degradation: 97.55%)
  • fedora-41-xfce_volatile_seq1m_q1t1_write 3:write_bandwidth_kb: 94466.00 :green_circle: ( previous job: 17567.00, improvement: 537.75%)
  • fedora-41-xfce_volatile_rnd4k_q32t1_read 3:read_bandwidth_kb: 80609.00 :green_circle: ( previous job: 79021.00, improvement: 102.01%)
  • fedora-41-xfce_volatile_rnd4k_q1t1_read 3:read_bandwidth_kb: 8400.00 :green_circle: ( previous job: 7867.00, improvement: 106.78%)
  • debian-12-xfce_exec-root: 28.18 🟢 ( previous job: 28.65, improvement: 98.35%)
  • debian-12-xfce_socket: 8.87 :small_red_triangle_up: ( previous job: 8.60, degradation: 103.15%)
  • debian-12-xfce_socket-root: 8.52 🟢 ( previous job: 8.52, improvement: 99.93%)
  • debian-12-xfce_exec-data-simplex: 62.25 🟢 ( previous job: 71.62, improvement: 86.91%)
  • debian-12-xfce_exec-data-duplex: 72.13 :small_red_triangle_up: ( previous job: 70.34, degradation: 102.54%)
  • debian-12-xfce_exec-data-duplex-root: 88.59 :small_red_triangle_up: ( previous job: 82.72, degradation: 107.10%)
  • debian-12-xfce_socket-data-duplex: 145.97 🟢 ( previous job: 156.96, improvement: 93.00%)
  • fedora-41-xfce_exec: 9.33 :small_red_triangle_up: ( previous job: 9.27, degradation: 100.69%)
  • fedora-41-xfce_exec-root: 65.28 :small_red_triangle_up: ( previous job: 61.51, degradation: 106.14%)
  • fedora-41-xfce_socket: 8.80 :small_red_triangle_up: ( previous job: 8.63, degradation: 102.04%)
  • fedora-41-xfce_socket-root: 7.89 🟢 ( previous job: 8.71, improvement: 90.65%)
  • fedora-41-xfce_exec-data-simplex: 72.45 🟢 ( previous job: 75.53, improvement: 95.92%)
  • fedora-41-xfce_exec-data-duplex: 62.89 🟢 ( previous job: 71.56, improvement: 87.88%)
  • fedora-41-xfce_exec-data-duplex-root: 96.70 🟢 ( previous job: 109.13, improvement: 88.62%)
  • fedora-41-xfce_socket-data-duplex: 154.61 :small_red_triangle_up: ( previous job: 150.61, degradation: 102.66%)
  • whonix-gateway-17_exec: 6.78 🟢 ( previous job: 6.82, improvement: 99.49%)
  • whonix-gateway-17_exec-root: 38.55 🟢 ( previous job: 40.43, improvement: 95.35%)
  • whonix-gateway-17_socket-root: 7.65 :small_red_triangle_up: ( previous job: 7.65, degradation: 100.00%)
  • whonix-gateway-17_exec-data-simplex: 77.81 🟢 ( previous job: 78.32, improvement: 99.35%)
  • whonix-gateway-17_exec-data-duplex: 76.82 :small_red_triangle_up: ( previous job: 76.65, degradation: 100.23%)
  • whonix-gateway-17_socket-data-duplex: 167.65 🟢 ( previous job: 171.76, improvement: 97.61%)
  • whonix-workstation-17_exec-root: 56.58 🟢 ( previous job: 58.26, improvement: 97.11%)
  • whonix-workstation-17_socket-root: 8.43 :small_red_triangle_up: ( previous job: 8.13, degradation: 103.69%)
  • whonix-workstation-17_exec-data-simplex: 73.44 🟢 ( previous job: 74.99, improvement: 97.93%)
  • whonix-workstation-17_exec-data-duplex: 73.50 :small_red_triangle_up: ( previous job: 72.71, degradation: 101.08%)
  • whonix-workstation-17_exec-data-duplex-root: 103.23 :small_red_triangle_up: ( previous job: 99.82, degradation: 103.42%)
  • whonix-workstation-17_socket-data-duplex: 169.39 🟢 ( previous job: 169.50, improvement: 99.93%)

@DemiMarie
Copy link
Contributor

Has this been tested on real hardware? I like the idea but I think it would be best to make sure that it actually works.

0000_00_08.1-00_00.0

I read this as:

  • Segment 0000.
  • Bus 00 on the root complex.
  • Slot 08.
  • Function 01, which is a bridge.
  • Something 00.
  • Slot 00 on the previous bridge.
  • Function 00 of the device on that bridge.

What is the something?

@marmarek
Copy link
Member Author

What is the something?

See issue description, commit message, or simply this PR description. But yes, this needs to be also added to a documentation.

Has this been tested on real hardware? I like the idea but I think it would be best to make sure that it actually works.

Yes.

Copy link
Contributor

@DemiMarie DemiMarie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you checked that the PCI path stays stable when other devices behind the same bridge or other bridges are added or removed?

segment_match = segment_re.match(path_part)
if segment_match:
segment = segment_match["segment"]
path_part = segment_match["rest"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for a regex match here. Just use [5:].

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be very easy to miss if the regex changes in the future.

qubes/utils.py Outdated
segment = dev_match["segment"]
else:
segment = "0000"
if int(dev_match["bus"], 16) == 0:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if int(dev_match["bus"], 16) == 0:
if dev_match["bus"] == "00":

qubes/utils.py Outdated
Comment on lines 395 to 396
# example: ../../../devices/pci0000:00/0000:00:1a.0/0000:02:00.0
path = sysfs_path.partition(f"/pci{segment}:")[2]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest following each .. component to make sure that one does reach /sys/devices/pci{segment}.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added an assert

Comment on lines +407 to +413
bus_num = int(bridge_match["bus"], 16) - bus_offset
bridge_str = (
f"{bus_num:02x}_{bridge_match['device']}."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
bus_num = int(bridge_match["bus"], 16) - bus_offset
bridge_str = (
f"{bus_num:02x}_{bridge_match['device']}."
bridge_str = (
f"{bridge_match["bus"]}_{bridge_match['device']}."

No need to go from string to int and back.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, you missed the key part: - bus offset

@marmarek marmarek marked this pull request as ready for review April 14, 2025 17:25
PCI SBDF may change if _another_ device is added/removed from the
system. Use PCI device path to avoid this issue. The PCI device path is
more or less sysfs path, but with bus number behind bridges replaced
with relative bus number to bridge's secondary bus number. Each
component of the path is separated with '-', as that is allowed in
qrexec arguments.

For example device 0000:c1:00.0 behind bridge at 0000:00:08.1 will get
path 0000_00_08.1-00_00.0 (assuming bridge 0000:00:08.1 has secondary
bus at 0xc1). Resolving device path requires access to the system's
sysfs.

As a side effect, this adds also support for multi-segment systems, PCI
segment is no longer hardcoded to 0000.

PCIDevice object will automatically resolve old device identifiers,
which partially covers migration path. The other part is converting BDF
to device path when loading device assignments.

Tests include subset of sysfs from a real system that has multiple
bridges.

TODO:
- consider still allowing attaching via SBDF, or otherwise expose function
  to translate SBDF to PCI path (an Admin API call?)
- fix salt listing/attaching devices to sys-net/sys-usb - depending on
  the above

Fixes QubesOS/qubes-issues#8681
@marmarek marmarek merged commit 3fe1e6c into QubesOS:main Apr 16, 2025
4 of 5 checks passed
@marmarek marmarek deleted the pci-path branch July 19, 2025 21:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace bus/slot/function with full PCI paths

3 participants