-
-
Notifications
You must be signed in to change notification settings - Fork 117
Add disposable performance benchmark #708
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #708 +/- ##
==========================================
- Coverage 70.53% 70.42% -0.11%
==========================================
Files 61 61
Lines 13546 13513 -33
==========================================
- Hits 9554 9517 -37
- Misses 3992 3996 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Using similar structure is fine, but please move it to a separate file at least in qubes.tests, so it can be scheduled separately. And likely the same for the file in tests/ too - given you added "if dispvm test do this else do the other" more or less everywhere. |
fbcbe67 to
affd60b
Compare
b49c778 to
9f939b7
Compare
This comment was marked as outdated.
This comment was marked as outdated.
|
It was worse because it was not using a preloaded....
…On Fri, Jul 25, 2025, 10:33 PM ben-grande ***@***.***> wrote:
*ben-grande* left a comment (QubesOS/qubes-core-admin#708)
<#708 (comment)>
Did a table manually with the results. 16 iterations, preload normally 2,
except on concurrent that it preloads double (4).
test run time (s) average (s)
fedora-42-xfce_dispvm 136.79583648600965 8.549739780375603
fedora-42-xfce_dispvm-gui 166.28912552101247 10.393070345063279
fedora-42-xfce_dispvm-concurrent 72.48606694900081 4.530379184312551
fedora-42-xfce_dispvm-concurrent-gui 77.78402063000249 4.861501289375155
fedora-42-xfce_dispvm-dom0 159.93910823899205 9.996194264937003
fedora-42-xfce_dispvm-dom0-gui 184.57327080400137 11.535829425250085
fedora-42-xfce_dispvm-dom0-concurrent 78.68325804101187 4.917703627563242
fedora-42-xfce_dispvm-dom0-concurrent-gui 82.98177742300322
5.186361088937701
fedora-42-xfce_dispvm-preload 77.30045937601244 4.831278711000777
fedora-42-xfce_dispvm-preload-gui 73.71754707499349 4.607346692187093
fedora-42-xfce_dispvm-preload-concurrent 57.57939739200810
3.598712337000506
fedora-42-xfce_dispvm-preload-concurrent-gui 58.38620778400218
3.649137986500136
fedora-42-xfce_dispvm-preload-dom0 165.32496509299380 10.332810318312113
fedora-42-xfce_dispvm-preload-dom0-gui 183.59772987499310
11.474858117187068
fedora-42-xfce_dispvm-preload-dom0-concurrent 82.48804083499999
5.155502552187499
fedora-42-xfce_dispvm-preload-dom0-concurrent-gui 90.14072562199726
5.633795351374829
fedora-42-xfce_vm 9.68348605500068 0.605217878437542
Interesting that the dom0 discrepancy was not as big, it was actually
worse...
—
Reply to this email directly, view it on GitHub
<#708 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BCE2O4PDT4XMP2IEDRNW5X33KKIARAVCNFSM6AAAAACCJ3DJJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCMRQGI3DGMZRG4>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
2fb5fc3 to
9688a00
Compare
|
16 iterations, max preload of 2, but on concurrency it is 4.
|
|
Those numbers look much more plausible :) |
726e75f to
87085f9
Compare
Please add #712 to be OpenQAd together with this PR. |
|
Still ironing out the tests to avoid expiring the timeout. Testing with 16GB on my testbench. |
87085f9 to
b14f6f8
Compare
|
Concurrent tests (4 qubes at a time) on a 16GB system can lead to qmemman logging: Xen free = N too small to satisfy assignments!. The good news is that the tests are not failing though, hoping OpenQA likes it. |
Besides this, full test of fedora-42-xfce with 16GB RAM was completed in 58 minutes (time increased due to cool down/delay/sleep added), |
* origin/pr/712: Skip memory assignment on domain still starting Pull request description: For: QubesOS/qubes-issues#1512 --- This does not fix all cases of *Failed to set memory* (but lowers it a lot) and does fix the TypeError exception being raised when `mem_used` is `None`: #708 (comment)
The timeout is 60 seconds for a single API call, I don't understand yet why it happened. I added the logs of the previous run also for comparison, no iteration is reaching over 60 seconds. The test Adding the sleep/delay already helped no happen many times. Sleep is already a considerable amount of the test run, but I see no other alternative to reduce load. qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_debian-12-xfce/test_406_dispvm_preload_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_debian-12-xfce.test_406_dispvm_preload_concurrent_api:starting
WARNING:vm.disp9411:Failed to set memory
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
ERROR:appmenus:Failed to update appmenus for 'disp5696' in 'dom0': Traceback
ERROR:appmenus:Failed to update appmenus for 'disp8810' in 'dom0': Traceback
iter dom disp exec clean total dom% disp% exec% clean%
1 0.023 0.000 0.800 2.590 3.413 1% 0% 23% 76%
2 0.064 0.000 1.107 2.296 3.467 2% 0% 32% 66%
3 0.063 0.000 16.475 2.661 19.200 0% 0% 86% 14%
4 0.063 0.000 9.338 4.925 14.326 0% 0% 65% 34%
5 0.017 0.000 7.564 2.953 10.533 0% 0% 72% 28%
6 0.012 0.000 10.551 3.176 13.739 0% 0% 77% 23%
7 0.090 0.000 4.686 3.769 8.544 1% 0% 55% 44%
8 0.033 0.000 6.934 2.924 9.891 0% 0% 70% 30%
9 0.059 0.000 12.521 3.143 15.722 0% 0% 80% 20%
10 1.407 0.000 6.222 2.889 10.518 13% 0% 59% 27%
11 0.065 0.000 10.463 3.684 14.213 0% 0% 74% 26%
12 0.041 0.000 11.537 3.548 15.125 0% 0% 76% 23%
13 0.059 0.000 9.616 4.399 14.074 0% 0% 68% 31%
14 0.148 0.000 8.636 2.714 11.498 1% 0% 75% 24%
15 0.071 0.000 7.073 3.038 10.182 1% 0% 69% 30%
16 0.097 0.000 7.518 2.499 10.114 1% 0% 74% 25%
Run time (iterations=16 average=3.091): 49.463s
WARNING:vm.disp7168:Failed to set memory
ok
qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_debian-12-xfce/test_407_dispvm_preload_gui_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_debian-12-xfce.test_407_dispvm_preload_gui_concurrent_api:starting
WARNING:vm.disp9783:Failed to set memory
WARNING:vm.disp8679:Failed to set memory
WARNING:vm.disp8617:Failed to set memory
Traceback (most recent call last):
File "/usr/lib/qubes/tests/dispvm_perf.py", line 347, in call_api
target_qube.run_service_for_stdio(service, timeout=60)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/site-packages/qubesadmin/vm/__init__.py", line 331, in run_service_for_stdio
stdouterr = p.communicate(input=input, timeout=timeout)
File "/usr/lib64/python3.13/subprocess.py", line 1222, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.13/subprocess.py", line 2129, in _communicate
self._check_timeout(endtime, orig_timeout, stdout, stderr)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.13/subprocess.py", line 1269, in _check_timeout
raise TimeoutExpired(
...<2 lines>...
stderr=b''.join(stderr_seq) if stderr_seq else None)
subprocess.TimeoutExpired: Command '['/usr/lib/qubes/qrexec-client', '-d', 'disp9783', '-T', '-w', '60', 'DEFAULT:QUBESRPC qubes.WaitForSession+ dom0']' timed out after 60 seconds
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/qubes/tests/dispvm_perf.py", line 575, in <module>
main()
~~~~^^
File "/usr/lib/qubes/tests/dispvm_perf.py", line 571, in main
run.run_test(test)
~~~~~~~~~~~~^^^^^^
File "/usr/lib/qubes/tests/dispvm_perf.py", line 518, in run_test
result = self.run_latency_api_calls(test)
File "/usr/lib/qubes/tests/dispvm_perf.py", line 401, in run_latency_api_calls
all_results = asyncio.run(self.api_thread(test, service, qube))
File "/usr/lib64/python3.13/asyncio/runners.py", line 195, in run
return runner.run(main)
~~~~~~~~~~^^^^^^
File "/usr/lib64/python3.13/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/usr/lib64/python3.13/asyncio/base_events.py", line 725, in run_until_complete
return future.result()
~~~~~~~~~~~~~^^
File "/usr/lib/qubes/tests/dispvm_perf.py", line 385, in api_thread
all_results = await asyncio.gather(*tasks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.13/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/qubes/tests/dispvm_perf.py", line 355, in call_api
raise Exception(f"service '{service}' failed: timeout expired")
Exception: service 'qubes.WaitForSession' failed: timeout expired
FAILFedora
qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_fedora-42-xfce/test_406_dispvm_preload_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_fedora-42-xfce.test_406_dispvm_preload_concurrent_api:starting
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
ERROR:appmenus:Failed to update appmenus for 'disp4014' in 'dom0': Traceback
ERROR:appmenus:Failed to update appmenus for 'disp5806' in 'dom0': Traceback
ERROR:appmenus:Failed to update appmenus for 'disp344' in 'dom0': Traceback
ERROR:appmenus:Failed to update appmenus for 'disp9837' in 'dom0': Traceback
iter dom disp exec clean total dom% disp% exec% clean%
1 0.014 0.000 1.373 3.346 4.732 0% 0% 29% 71%
2 0.055 0.000 1.056 2.444 3.555 2% 0% 30% 69%
3 0.055 0.000 10.905 3.997 14.958 0% 0% 73% 27%
4 0.055 0.000 17.128 4.277 21.460 0% 0% 80% 20%
5 0.014 0.000 10.272 5.372 15.658 0% 0% 66% 34%
6 0.017 0.000 10.376 4.837 15.230 0% 0% 68% 32%
7 0.065 0.000 6.037 2.625 8.728 1% 0% 69% 30%
8 0.122 0.000 6.722 4.096 10.939 1% 0% 61% 37%
9 0.686 0.000 10.647 3.124 14.457 5% 0% 74% 22%
10 0.087 0.000 12.985 5.142 18.213 0% 0% 71% 28%
11 0.042 0.000 17.388 3.377 20.806 0% 0% 84% 16%
12 0.070 0.000 7.463 3.381 10.914 1% 0% 68% 31%
13 0.053 0.000 10.591 2.980 13.623 0% 0% 78% 22%
14 0.119 0.000 8.432 3.004 11.555 1% 0% 73% 26%
15 0.129 0.000 10.346 2.484 12.959 1% 0% 80% 19%
16 0.071 0.000 10.126 1.700 11.898 1% 0% 85% 14%
Run time (iterations=16 average=3.525): 56.393s
WARNING:vm.disp9084:Failed to set memory
ok
qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_fedora-42-xfce/test_407_dispvm_preload_gui_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_fedora-42-xfce.test_407_dispvm_preload_gui_concurrent_api:starting
WARNING:vm.disp1791:Failed to set memory
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
WARNING:vm.test-inst-dvm:Failed to create preloaded disposable, limit reached
iter dom disp exec clean total dom% disp% exec% clean%
1 0.020 0.000 3.453 4.598 8.070 0% 0% 43% 57%
2 0.065 0.000 3.903 4.130 8.098 1% 0% 48% 51%
3 0.064 0.000 15.426 6.935 22.425 0% 0% 69% 31%
4 0.064 0.000 16.346 5.871 22.281 0% 0% 73% 26%
5 0.072 0.000 18.769 3.118 21.960 0% 0% 85% 14%
6 0.101 0.000 11.916 4.934 16.952 1% 0% 70% 29%
7 0.083 0.000 10.631 3.205 13.919 1% 0% 76% 23%
8 0.076 0.000 6.412 4.423 10.911 1% 0% 59% 41%
9 0.091 0.000 18.597 5.466 24.154 0% 0% 77% 23%
10 0.040 0.000 14.117 6.272 20.429 0% 0% 69% 31%
11 0.056 0.000 14.380 4.445 18.881 0% 0% 76% 24%
12 0.094 0.000 18.648 6.210 24.952 0% 0% 75% 25%
13 0.116 0.000 10.686 5.547 16.349 1% 0% 65% 34%
14 0.373 0.000 6.203 4.628 11.204 3% 0% 55% 41%
15 0.023 0.000 16.141 2.457 18.620 0% 0% 87% 13%
16 0.051 0.000 10.429 2.190 12.670 0% 0% 82% 17%
Run time (iterations=16 average=4.614): 73.828s
okWhonix-Workstation
qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_whonix-workstation-17/test_406_dispvm_preload_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_whonix-workstation-17.test_406_dispvm_preload_concurrent_api:starting
ERROR:appmenus:Failed to update appmenus for 'disp2840' in 'dom0': Traceback
ERROR:appmenus:Failed to update appmenus for 'disp7929' in 'dom0': Traceback
iter dom disp exec clean total dom% disp% exec% clean%
1 0.016 0.000 1.655 2.779 4.451 0% 0% 37% 62%
2 0.025 0.000 1.747 4.055 5.827 0% 0% 30% 70%
3 0.047 0.000 16.768 7.214 24.030 0% 0% 70% 30%
4 0.047 0.000 11.803 4.451 16.301 0% 0% 72% 27%
5 0.014 0.000 15.058 5.069 20.142 0% 0% 75% 25%
6 0.018 0.000 16.295 6.438 22.750 0% 0% 72% 28%
7 0.063 0.000 9.997 4.802 14.863 0% 0% 67% 32%
8 0.191 0.000 9.151 3.237 12.579 2% 0% 73% 26%
9 0.046 0.000 11.509 3.876 15.431 0% 0% 75% 25%
10 0.038 0.000 16.261 5.643 21.942 0% 0% 74% 26%
11 0.079 0.000 10.522 3.750 14.351 1% 0% 73% 26%
12 0.056 0.000 11.093 4.775 15.924 0% 0% 70% 30%
13 0.073 0.000 11.317 3.063 14.454 1% 0% 78% 21%
14 0.068 0.000 12.650 4.146 16.864 0% 0% 75% 25%
15 0.033 0.000 11.376 3.998 15.407 0% 0% 74% 26%
16 0.021 0.000 12.843 2.381 15.245 0% 0% 84% 16%
Run time (iterations=16 average=4.236): 67.782s
ok
qubes.tests.integ.dispvm_perf/TC_00_DispVMPerf_whonix-workstation-17/test_407_dispvm_preload_gui_concurrent_api
Latency of dom0-dispvm (preload) concurrent GUI API calls ... CRITICAL:qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_whonix-workstation-17.test_407_dispvm_preload_gui_concurrent_api:starting
WARNING:vm.disp3153:Failed to set memory
iter dom disp exec clean total dom% disp% exec% clean%
1 0.019 0.000 4.108 3.652 7.778 0% 0% 53% 47%
2 0.041 0.000 4.570 3.424 8.035 1% 0% 57% 43%
3 0.041 0.000 27.668 5.829 33.538 0% 0% 82% 17%
4 0.053 0.000 19.230 6.941 26.224 0% 0% 73% 26%
5 0.043 0.000 10.327 7.314 17.684 0% 0% 58% 41%
6 0.034 0.000 16.031 5.359 21.425 0% 0% 75% 25%
7 0.150 0.000 12.879 4.751 17.781 1% 0% 72% 27%
8 0.095 0.000 5.860 6.112 12.067 1% 0% 49% 51%
9 0.132 0.000 20.925 7.305 28.362 0% 0% 74% 26%
10 0.052 0.000 12.197 6.264 18.513 0% 0% 66% 34%
11 0.099 0.000 22.760 6.863 29.722 0% 0% 77% 23%
12 0.093 0.000 11.509 4.199 15.802 1% 0% 73% 27%
13 0.063 0.000 14.292 6.674 21.030 0% 0% 68% 32%
14 0.101 0.000 14.519 4.781 19.401 1% 0% 75% 25%
15 0.062 0.000 16.909 3.194 20.165 0% 0% 84% 16%
16 0.104 0.000 14.017 2.021 16.143 1% 0% 87% 13%
Run time (iterations=16 average=5.260): 84.159s
WARNING:vm.disp7226:Failed to set memory
ok |
|
I'm not sure what happened there, the console log at https://openqa.qubes-os.org/tests/148784/file/system_tests-qubes.tests.integ.dispvm_perf.TC_00_DispVMPerf_debian-12-xfce.test_407.guest-disp9783.log doesn't show any failures... Can you include stderr of the qubes.WaitForSession service also in case of timeout? |
For: QubesOS/qubes-issues#1512
Requires: QubesOS/qubes-core-admin-client#376