Skip to content

Conversation

@ben-grande
Copy link
Contributor

@ben-grande ben-grande commented Jul 24, 2025

@codecov
Copy link

codecov bot commented Jul 24, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.42%. Comparing base (c2bd753) to head (b14f6f8).
⚠️ Report is 10 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #708      +/-   ##
==========================================
- Coverage   70.53%   70.42%   -0.11%     
==========================================
  Files          61       61              
  Lines       13546    13513      -33     
==========================================
- Hits         9554     9517      -37     
- Misses       3992     3996       +4     
Flag Coverage Δ
unittests 70.42% <ø> (-0.11%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@marmarek
Copy link
Member

Using similar structure is fine, but please move it to a separate file at least in qubes.tests, so it can be scheduled separately. And likely the same for the file in tests/ too - given you added "if dispvm test do this else do the other" more or less everywhere.
And BTW, the TestConfig class is done this way to not rely on test name for its behavior...

@ben-grande ben-grande force-pushed the preload-perf branch 9 times, most recently from fbcbe67 to affd60b Compare July 25, 2025 13:40
ben-grande added a commit to ben-grande/openqa-tests-qubesos that referenced this pull request Jul 25, 2025
@ben-grande ben-grande force-pushed the preload-perf branch 3 times, most recently from b49c778 to 9f939b7 Compare July 25, 2025 16:51
@ben-grande

This comment was marked as outdated.

@ben-grande
Copy link
Contributor Author

ben-grande commented Jul 25, 2025 via email

@ben-grande ben-grande force-pushed the preload-perf branch 7 times, most recently from 2fb5fc3 to 9688a00 Compare July 28, 2025 13:29
@ben-grande
Copy link
Contributor Author

16 iterations, max preload of 2, but on concurrency it is 4.

name average total
vm-dispvm calls 7.263 116.214
vm-dispvm GUI calls 8.903 142.448
vm-dispvm concurrent calls 3.267 52.270
vm-dispvm concurrent GUI calls 3.711 59.372
dom0-dispvm calls 8.356 133.703
dom0-dispvm GUI calls 9.745 155.922
dom0-dispvm concurrent calls 3.526 56.415
dom0-dispvm concurrent GUI calls 3.901 62.409
vm-dispvm (preload) calls 3.706 59.300
vm-dispvm (preload) GUI calls 3.681 58.900
vm-dispvm (preload) concurrent calls 2.476 39.609
vm-dispvm (preload) concurrent GUI calls 2.820 45.116
dom0-dispvm (preload) calls 3.787 60.594
dom0-dispvm (preload) GUI calls 4.205 67.272
dom0-dispvm (preload) concurrent calls 2.826 45.223
dom0-dispvm (preload) concurrent GUI calls 3.078 49.241

@marmarek
Copy link
Member

Those numbers look much more plausible :)

ben-grande added a commit to ben-grande/openqa-tests-qubesos that referenced this pull request Aug 4, 2025
@ben-grande ben-grande force-pushed the preload-perf branch 3 times, most recently from 726e75f to 87085f9 Compare August 4, 2025 18:33
@ben-grande
Copy link
Contributor Author

  • Remove Whonix Gateway from tests
  • Cool down to get a little bit more independent results
  • Port the Bash script to POSIX cause why not
  • Add more vm tests

Please add #712 to be OpenQAd together with this PR.

@ben-grande
Copy link
Contributor Author

Still ironing out the tests to avoid expiring the timeout. Testing with 16GB on my testbench.

@ben-grande
Copy link
Contributor Author

Concurrent tests (4 qubes at a time) on a 16GB system can lead to qmemman logging: Xen free = N too small to satisfy assignments!. The good news is that the tests are not failing though, hoping OpenQA likes it.

@ben-grande
Copy link
Contributor Author

Concurrent tests (4 qubes at a time) on a 16GB system can lead to qmemman logging: Xen free = N too small to satisfy assignments!. The good news is that the tests are not failing though, hoping OpenQA likes it.

Besides this, full test of fedora-42-xfce with 16GB RAM was completed in 58 minutes (time increased due to cool down/delay/sleep added),

marmarek added a commit that referenced this pull request Aug 6, 2025
* origin/pr/712:
  Skip memory assignment on domain still starting

Pull request description:

For: QubesOS/qubes-issues#1512

---

This does not fix all cases of *Failed to set memory* (but lowers it a lot) and does fix the TypeError exception being raised when `mem_used` is `None`: #708 (comment)
@marmarek marmarek merged commit 540024c into QubesOS:main Aug 6, 2025
3 of 5 checks passed
@ben-grande
Copy link
Contributor Author

  • system_tests_dispvm_perf@hw7

    * TC_00_DispVMPerf_debian-12-xfce: [test_407_dispvm_preload_gui_concurrent_api](https://openqa.qubes-os.org/tests/148784#step/TC_00_DispVMPerf_debian-12-xfce/26) (failure)
      `AssertionError: '/usr/lib/qubes/tests/dispvm_perf.py --dvm=test-ins...`
    

The timeout is 60 seconds for a single API call, I don't understand yet why it happened. I added the logs of the previous run also for comparison, no iteration is reaching over 60 seconds. The test 407 is heavy (concurrent and GUI), but it failed on the most lightweight qube (comparing the results to the 406 test), Debian. It looks like a temporary failure, although I'd prefer to not see it on the qubesos-bot messages every time.

Adding the sleep/delay already helped no happen many times. Sleep is already a considerable amount of the test run, but I see no other alternative to reduce load.

qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_debian-12-xfce/test_406_dispvm_preload_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_debian-12-xfce.test_406_dispvm_preload_concurrent_api:starting
WARNING:vm.disp9411:Failed to set memory
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
ERROR:appmenus:Failed to update appmenus for 'disp5696' in 'dom0': Traceback
ERROR:appmenus:Failed to update appmenus for 'disp8810' in 'dom0': Traceback

iter   dom  disp   exec clean  total dom% disp% exec% clean%
   1 0.023 0.000  0.800 2.590  3.413   1%    0%   23%    76%
   2 0.064 0.000  1.107 2.296  3.467   2%    0%   32%    66%
   3 0.063 0.000 16.475 2.661 19.200   0%    0%   86%    14%
   4 0.063 0.000  9.338 4.925 14.326   0%    0%   65%    34%
   5 0.017 0.000  7.564 2.953 10.533   0%    0%   72%    28%
   6 0.012 0.000 10.551 3.176 13.739   0%    0%   77%    23%
   7 0.090 0.000  4.686 3.769  8.544   1%    0%   55%    44%
   8 0.033 0.000  6.934 2.924  9.891   0%    0%   70%    30%
   9 0.059 0.000 12.521 3.143 15.722   0%    0%   80%    20%
  10 1.407 0.000  6.222 2.889 10.518  13%    0%   59%    27%
  11 0.065 0.000 10.463 3.684 14.213   0%    0%   74%    26%
  12 0.041 0.000 11.537 3.548 15.125   0%    0%   76%    23%
  13 0.059 0.000  9.616 4.399 14.074   0%    0%   68%    31%
  14 0.148 0.000  8.636 2.714 11.498   1%    0%   75%    24%
  15 0.071 0.000  7.073 3.038 10.182   1%    0%   69%    30%
  16 0.097 0.000  7.518 2.499 10.114   1%    0%   74%    25%
Run time (iterations=16 average=3.091): 49.463s
WARNING:vm.disp7168:Failed to set memory
ok

qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_debian-12-xfce/test_407_dispvm_preload_gui_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_debian-12-xfce.test_407_dispvm_preload_gui_concurrent_api:starting
WARNING:vm.disp9783:Failed to set memory
WARNING:vm.disp8679:Failed to set memory
WARNING:vm.disp8617:Failed to set memory
Traceback (most recent call last):
  File "/usr/lib/qubes/tests/dispvm_perf.py", line 347, in call_api
    target_qube.run_service_for_stdio(service, timeout=60)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/qubesadmin/vm/__init__.py", line 331, in run_service_for_stdio
    stdouterr = p.communicate(input=input, timeout=timeout)
  File "/usr/lib64/python3.13/subprocess.py", line 1222, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/subprocess.py", line 2129, in _communicate
    self._check_timeout(endtime, orig_timeout, stdout, stderr)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/subprocess.py", line 1269, in _check_timeout
    raise TimeoutExpired(
    ...<2 lines>...
            stderr=b''.join(stderr_seq) if stderr_seq else None)
subprocess.TimeoutExpired: Command '['/usr/lib/qubes/qrexec-client', '-d', 'disp9783', '-T', '-w', '60', 'DEFAULT:QUBESRPC qubes.WaitForSession+ dom0']' timed out after 60 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/qubes/tests/dispvm_perf.py", line 575, in <module>
    main()
    ~~~~^^
  File "/usr/lib/qubes/tests/dispvm_perf.py", line 571, in main
    run.run_test(test)
    ~~~~~~~~~~~~^^^^^^
  File "/usr/lib/qubes/tests/dispvm_perf.py", line 518, in run_test
    result = self.run_latency_api_calls(test)
  File "/usr/lib/qubes/tests/dispvm_perf.py", line 401, in run_latency_api_calls
    all_results = asyncio.run(self.api_thread(test, service, qube))
  File "/usr/lib64/python3.13/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "/usr/lib64/python3.13/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib64/python3.13/asyncio/base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/usr/lib/qubes/tests/dispvm_perf.py", line 385, in api_thread
    all_results = await asyncio.gather(*tasks)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/qubes/tests/dispvm_perf.py", line 355, in call_api
    raise Exception(f"service '{service}' failed: timeout expired")
Exception: service 'qubes.WaitForSession' failed: timeout expired
FAIL
Fedora

qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_fedora-42-xfce/test_406_dispvm_preload_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_fedora-42-xfce.test_406_dispvm_preload_concurrent_api:starting
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
ERROR:appmenus:Failed to update appmenus for 'disp4014' in 'dom0': Traceback 
ERROR:appmenus:Failed to update appmenus for 'disp5806' in 'dom0': Traceback 
ERROR:appmenus:Failed to update appmenus for 'disp344' in 'dom0': Traceback 
ERROR:appmenus:Failed to update appmenus for 'disp9837' in 'dom0': Traceback 

iter   dom  disp   exec clean  total dom% disp% exec% clean%
   1 0.014 0.000  1.373 3.346  4.732   0%    0%   29%    71%
   2 0.055 0.000  1.056 2.444  3.555   2%    0%   30%    69%
   3 0.055 0.000 10.905 3.997 14.958   0%    0%   73%    27%
   4 0.055 0.000 17.128 4.277 21.460   0%    0%   80%    20%
   5 0.014 0.000 10.272 5.372 15.658   0%    0%   66%    34%
   6 0.017 0.000 10.376 4.837 15.230   0%    0%   68%    32%
   7 0.065 0.000  6.037 2.625  8.728   1%    0%   69%    30%
   8 0.122 0.000  6.722 4.096 10.939   1%    0%   61%    37%
   9 0.686 0.000 10.647 3.124 14.457   5%    0%   74%    22%
  10 0.087 0.000 12.985 5.142 18.213   0%    0%   71%    28%
  11 0.042 0.000 17.388 3.377 20.806   0%    0%   84%    16%
  12 0.070 0.000  7.463 3.381 10.914   1%    0%   68%    31%
  13 0.053 0.000 10.591 2.980 13.623   0%    0%   78%    22%
  14 0.119 0.000  8.432 3.004 11.555   1%    0%   73%    26%
  15 0.129 0.000 10.346 2.484 12.959   1%    0%   80%    19%
  16 0.071 0.000 10.126 1.700 11.898   1%    0%   85%    14%
Run time (iterations=16 average=3.525): 56.393s
WARNING:vm.disp9084:Failed to set memory
ok

qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_fedora-42-xfce/test_407_dispvm_preload_gui_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_fedora-42-xfce.test_407_dispvm_preload_gui_concurrent_api:starting
WARNING:vm.disp1791:Failed to set memory
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached

iter   dom  disp   exec clean  total dom% disp% exec% clean%
   1 0.020 0.000  3.453 4.598  8.070   0%    0%   43%    57%
   2 0.065 0.000  3.903 4.130  8.098   1%    0%   48%    51%
   3 0.064 0.000 15.426 6.935 22.425   0%    0%   69%    31%
   4 0.064 0.000 16.346 5.871 22.281   0%    0%   73%    26%
   5 0.072 0.000 18.769 3.118 21.960   0%    0%   85%    14%
   6 0.101 0.000 11.916 4.934 16.952   1%    0%   70%    29%
   7 0.083 0.000 10.631 3.205 13.919   1%    0%   76%    23%
   8 0.076 0.000  6.412 4.423 10.911   1%    0%   59%    41%
   9 0.091 0.000 18.597 5.466 24.154   0%    0%   77%    23%
  10 0.040 0.000 14.117 6.272 20.429   0%    0%   69%    31%
  11 0.056 0.000 14.380 4.445 18.881   0%    0%   76%    24%
  12 0.094 0.000 18.648 6.210 24.952   0%    0%   75%    25%
  13 0.116 0.000 10.686 5.547 16.349   1%    0%   65%    34%
  14 0.373 0.000  6.203 4.628 11.204   3%    0%   55%    41%
  15 0.023 0.000 16.141 2.457 18.620   0%    0%   87%    13%
  16 0.051 0.000 10.429 2.190 12.670   0%    0%   82%    17%
Run time (iterations=16 average=4.614): 73.828s
ok

Whonix-Workstation

qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_whonix-workstation-17/test_406_dispvm_preload_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_whonix-workstation-17.test_406_dispvm_preload_concurrent_api:starting
ERROR:appmenus:Failed to update appmenus for 'disp2840' in 'dom0': Traceback 
ERROR:appmenus:Failed to update appmenus for 'disp7929' in 'dom0': Traceback 

iter   dom  disp   exec clean  total dom% disp% exec% clean%
   1 0.016 0.000  1.655 2.779  4.451   0%    0%   37%    62%
   2 0.025 0.000  1.747 4.055  5.827   0%    0%   30%    70%
   3 0.047 0.000 16.768 7.214 24.030   0%    0%   70%    30%
   4 0.047 0.000 11.803 4.451 16.301   0%    0%   72%    27%
   5 0.014 0.000 15.058 5.069 20.142   0%    0%   75%    25%
   6 0.018 0.000 16.295 6.438 22.750   0%    0%   72%    28%
   7 0.063 0.000  9.997 4.802 14.863   0%    0%   67%    32%
   8 0.191 0.000  9.151 3.237 12.579   2%    0%   73%    26%
   9 0.046 0.000 11.509 3.876 15.431   0%    0%   75%    25%
  10 0.038 0.000 16.261 5.643 21.942   0%    0%   74%    26%
  11 0.079 0.000 10.522 3.750 14.351   1%    0%   73%    26%
  12 0.056 0.000 11.093 4.775 15.924   0%    0%   70%    30%
  13 0.073 0.000 11.317 3.063 14.454   1%    0%   78%    21%
  14 0.068 0.000 12.650 4.146 16.864   0%    0%   75%    25%
  15 0.033 0.000 11.376 3.998 15.407   0%    0%   74%    26%
  16 0.021 0.000 12.843 2.381 15.245   0%    0%   84%    16%
Run time (iterations=16 average=4.236): 67.782s
ok

qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_whonix-workstation-17/test_407_dispvm_preload_gui_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_whonix-workstation-17.test_407_dispvm_preload_gui_concurrent_api:starting
WARNING:vm.disp3153:Failed to set memory

iter   dom  disp   exec clean  total dom% disp% exec% clean%
   1 0.019 0.000  4.108 3.652  7.778   0%    0%   53%    47%
   2 0.041 0.000  4.570 3.424  8.035   1%    0%   57%    43%
   3 0.041 0.000 27.668 5.829 33.538   0%    0%   82%    17%
   4 0.053 0.000 19.230 6.941 26.224   0%    0%   73%    26%
   5 0.043 0.000 10.327 7.314 17.684   0%    0%   58%    41%
   6 0.034 0.000 16.031 5.359 21.425   0%    0%   75%    25%
   7 0.150 0.000 12.879 4.751 17.781   1%    0%   72%    27%
   8 0.095 0.000  5.860 6.112 12.067   1%    0%   49%    51%
   9 0.132 0.000 20.925 7.305 28.362   0%    0%   74%    26%
  10 0.052 0.000 12.197 6.264 18.513   0%    0%   66%    34%
  11 0.099 0.000 22.760 6.863 29.722   0%    0%   77%    23%
  12 0.093 0.000 11.509 4.199 15.802   1%    0%   73%    27%
  13 0.063 0.000 14.292 6.674 21.030   0%    0%   68%    32%
  14 0.101 0.000 14.519 4.781 19.401   1%    0%   75%    25%
  15 0.062 0.000 16.909 3.194 20.165   0%    0%   84%    16%
  16 0.104 0.000 14.017 2.021 16.143   1%    0%   87%    13%
Run time (iterations=16 average=5.260): 84.159s
WARNING:vm.disp7226:Failed to set memory
ok

@marmarek
Copy link
Member

marmarek commented Aug 7, 2025

I'm not sure what happened there, the console log at https://openqa.qubes-os.org/tests/148784/file/system_tests-qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_debian-12-xfce.test_407.guest-disp9783.log doesn't show any failures... Can you include stderr of the qubes.WaitForSession service also in case of timeout?

ben-grande added a commit to ben-grande/qubes-core-admin that referenced this pull request Aug 7, 2025
ben-grande added a commit to ben-grande/qubes-core-admin that referenced this pull request Aug 7, 2025
ben-grande added a commit to ben-grande/qubes-core-admin that referenced this pull request Aug 7, 2025
ben-grande added a commit to ben-grande/qubes-core-admin that referenced this pull request Aug 7, 2025
ben-grande added a commit to ben-grande/qubes-core-admin that referenced this pull request Aug 7, 2025
ben-grande added a commit to ben-grande/qubes-core-admin that referenced this pull request Aug 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants