Repository navigation
perf(libkernel): end short sleeps when the GPU or a flip completes - #6658
Conversation
A sleep of 1 ms or less returns once the AGC driver sees a GPU batch complete (a thread waits on the recorder's timeline semaphore), runs a batch's completion actions, or VideoOut completes a flip. A thread returns early at most 8 times in a run of back-to-back sleeps; the run restarts when the thread submits AGC work or did not sleep for the requested time. APS5_NO_SLEEP_WAKE=1 keeps every sleep in full. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
… other PRs change Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Moved the 🤖 Addressed by Claude Code |
jotitadev
left a comment
jotitadev
left a comment
There was a problem hiding this comment.
Linux run of the condition-variable path in TimedWait::SleepNanosUnlessChanged, which the description lists as not run. Validated at 82dd28619 fast-forwarded onto main 5d9dcc42d; the current head 6ce90a728 only moves the SleepWake.hpp include in Recorder.cpp and the debt entry, no code change.
Machine: Ubuntu 26.04, Intel i7-8700 (6C/12T), GCC 15.2, full build, Release, BUILD_TESTING=ON.
- Builds clean, including the new
guest_sleep_waketarget. guest_sleep_wakepasses on Linux: 100/100 runs alone, 120/120 with 12 copies in parallel and 120/120 with 24 copies in parallel (2x the logical CPUs).- The test does catch a missing wake: with
APS5_NO_SLEEP_WAKE=1it aborts. - Non-GPU suite (
ctest -E "^agc|gpu|vulkan|video_out" -j12): 649/651; the only failures arefile_writer_binaryandfile_writer_text, which also fail onmainon this machine (648/650).guest_nanosleeppasses.
Not covered here: the GPU progress watcher in Recorder.cpp and the VideoOut flip wake (no AGC/VideoOut tests or titles were run).
Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.
|
Thanks @jotitadev for the Linux run. I added it to the description, with what it doesn't cover (GPU progress watcher, flip wake, macOS). |
|
Data point: RTX 3060 12 GB (NVIDIA 617.42), i5-12400F, Windows 10 19045, MinGW GCC Release. Minecraft Dungeons II (PPSA16064 01.100.100), title screen, no input, 200 s rows,
Neutral on this scene (mean +0.4 %, within the row spread); no regression. 0 fatal. AI-assisted: yes (Claude Code) |
|
Additional Windows test on Ryzen 7 5800H, 64 GiB RAM, RTX 3070 Laptop 8 GiB (NVIDIA 617.42), Windows 11 Pro 26H2 (build 26300.9550; NT 10.0). I repeated the SuperTuxKart PPSA69421 comparison without the other local optimizations: clean Startup/tutorial prompt over the main menu, 1920×1080 internal rendering. Same executable and shared warm shader cache, music/SFX disabled on both sides, normal flip pacing, no host-import-disabling switches. Both measured windows stayed foreground in the same maximized window, client area 2560×1369. No builds or GPU tests ran during the captures. Each title run was limited to 35 seconds; the table uses the late
The separately sampled last ~8 seconds also agreed in direction: 9.81 versus 15.67 FPS. Loaded library SHA256s were verified; both traces report All 7 directly related sleep/import checks passed, including AI-assisted: yes. The builds, tests and measurements above were executed locally. |
# Conflicts: # docs/dev/debt/libkernel.md
github-actions
Bot
added
perf
|
This pull request is labeled |
github-actions
Bot
removed
perf numbers missing
|
@nicolasabosch thanks, this reads well, and the menu gain reproducing on three machines (one AMD) is convincing. One Windows fix before merge: the per-thread auto-reset Two smaller points, no change needed unless you prefer: on Windows the new path skips |
… coarse and sub-50 us sleeps as before A wake set after the previous wait returned left the per-thread auto-reset event signalled, so the next short sleep returned from its wait at once and spun to its deadline. With APS5_COARSE_TIMED_WAITS or a request under 50 us the sleep now takes TimedWait::SleepNanos as on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@oneandonlydean thanks, you're right. Fixed in 09093bf: Checked: the 12 sleep and timer tests ( 🤖 Addressed by Claude Code |
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
@oneandonlydean follow-up with what I measured, since my earlier reply fixed this without reproducing it first. The stale event (Windows). Reproduced before keeping the fix. Loop of 1000 rounds: a wake lands 150 us into a 300 us sleep (which spins, so nothing consumes the event), then a 1 ms sleep is timed in thread cycles ( (a) (b) Sleeps under 50 us. Also changed: they take Tests: 🤖 Addressed by Claude Code |
With APS5_COARSE_TIMED_WAITS set a sleep is never ended early. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Correction to my previous comment: its point (a) was garbled. The accurate version: (a) The rest of the previous comment stands (stale-event measurement, (b), test and benchmark numbers). The branch is now at 🤖 Addressed by Claude Code |
|
Thanks for the The new The code behaves correctly here. On Linux, |
APS5_COARSE_TIMED_WAITS is read under _WIN32 only; on Linux the condition variable path always ends a sleep when a completion arrives. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
@oneandonlydean you're right, and thanks for the diagnosis: I registered 🤖 Addressed by Claude Code |
|
Ran this on GTA V (PPSA04263 01.010.002), Linux 7.2.8-zen1-2-zen, NVIDIA GeForce RTX 5060 Laptop GPU, NVIDIA 615.71.09, Graphics Mode Performance, the Story prologue after Resume Story, with no input. Summary lines (main, then the PR): |
|
@oneandonlydean do you need something more from me? |
|
Race-scene data on AMD, as a follow-up to my menu numbers: this helps there too, by about 5%. SuperTuxKart (PPSA69421 00.001.000), tutorial race, kart idle on the start line, Both trees include #8538 (the point-light fix) and #6718 (the texture-array upload saving):
The waiting this PR targets is most of the frame:
The gain here is smaller than in the menu (6.7 → 9.7 FPS there), so in the race most of each wait seems to be the submission round trip itself rather than the rest of the 1 ms sleep. AI-assisted: yes (Claude Code) |
# Conflicts: # core/libs/prx/libSceAgcDriver/Execution/src/Driver/EntryPoints.cpp # core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
7 participants
What
A
sceKernelUsleep,usleep,nanosleeporsceKernelNanosleepof 1 ms or less now returns as soon as one of these happens:Titles that poll a GPU label or the flip status between short sleeps no longer sleep out the rest of each 1 ms. SuperTuxKart's GL layer is one: it spins 200 us, then calls
sceKernelUsleep(1000)until its marker lands, about 35 times per frame in the menu.Limits that keep the console behaviour for everything else:
APS5_COARSE_TIMED_WAITSset, takes the same path as on main.APS5_NO_SLEEP_WAKE=1keeps every sleep in full.The early return is recorded in technical debt (libkernel, Functional).
Tested
Windows 11 Pro 10.0.26300, Intel i9-14900HX, 32 GB, NVIDIA GeForce RTX 4070 Laptop GPU (driver 616.92), 1920x1080 window.
Captured as the Performance report form describes:
APS5_TRACE_SECONDSonly, no redirected log, warm caches.43a2abaf7and this branch876ad540f(main merged in) built from clean checkouts and alternated in one session, same relinked executable.31cbb55d7)31cbb55d7)Other kart-selection runs of this branch gave 150.5 ms p50 (one more run) and, in an earlier session on main
31cbb55d7, 187.5 against 324.6 ms. One further run of the branch did not reach that scene and is not counted. DOOM was measured on the earlier base and not repeated; it paces itself at 60 Hz and its GPU time per present did not change.In the STK menu the guest no longer waits out its 1 ms sleeps: in the trace, a batch's cycle drops from about 1.7 ms to about 0.9 ms.
ctest -j 16(Windows, mingw) on this branch: 1689 of 1691 pass.agc_driver_pass_input_hazardfails the same way on main43a2abaf7without this change (3 of 3 runs).conversion_diagnosticstimed out under the parallel run and passes alone (3 of 3).agc_driver_preparation_scope_settleandagc_driver_bda_device, which failed in earlier runs, are explained by main failing or timing out the same way (7 of 144 and 3 of 3 runs).New test:
guest_sleep_wake.Validation layers (this change only, not compared with main): this branch with main
dbac9fa15b61merged in, built Release on the same machine and GPU,ctest -V -R 'sleep_wake|agc_driver_flip'withVK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation,ANYPS5_NO_SHADER_CACHE=1andANYPS5_REQUIRE_VULKAN=1. 5 of 5 pass (agc_driver_flip,agc_driver_flip_stores_small_results,agc_driver_flip_compute,guest_sleep_wake,guest_sleep_wake_coarse); the log has noVUID-line and nowas not found but was requested by env var VK_INSTANCE_LAYERS. The same run on main was not done, so the no-new-VUID comparison with main is not made.Review of the Windows wake event: a wake that lands after a sleep stopped waiting left the per-thread event signalled, and the next sleep on that thread could spin its whole length. Measured on this machine with a 1000-run loop (a wake during a 300 us sleep, then a 1 ms sleep timed in thread cycles): without
ResetEvent65, 102 and 173 of 1000 of those second sleeps burned over 800 us of CPU; with the fix 3, 2 and 2. The same loop with no earlier wake gave 0 of 1000 in both. The fix is inSleepNanosUnlessChanged.Linux and macOS take the condition-variable path in
TimedWait::SleepNanosUnlessChanged. @jotitadev validated it on Linux in this review (Ubuntu 26.04, i7-8700, GCC 15.2):guest_sleep_wakepasses 100/100 alone and 120/120 with 12 and 24 parallel copies, and aborts withAPS5_NO_SLEEP_WAKE=1. That run was before the Windows fix above, which does not touch the Linux path. The GPU progress watcher and the flip wake were not run on Linux. macOS was not run.Summary lines, traces and
trace_digest.pyoutput for every capture: https://github.com/nicolasabosch/AnyPS5/releases/tag/perf-evidence-gpu-wake-sleeps (filesfinal-*are the captures in the table; the earlier session's are unprefixed).Summary lines
STK main menu, main
43a2abaf7:{"commit":"43a2abaf7753","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"40:10"},"gpu":"NVIDIA GeForce RTX 4070 Laptop GPU","driver":"NVIDIA 616.92","os":"Windows 10.0.26300","resolution":"1920x1080","window":{"spec":"seconds 40:10","first_present":401,"ms":10013.889},"presents":184,"frame_ms":{"p50":53.331,"p90":57.606,"p99":77.508,"max":79.033},"gpu_busy_ms_per_present":1.516,"worker_busy_pct":{"queue worker 0x0":2.686},"top_phases_ms_per_frame":{"Graphics.Draw":4.919,"Graphics.Draw.draw_and_resource_release":4.756,"Vulkan.DrawFromRecipe":1.597,"Graphics.DrawWithRecipe":1.590,"Graphics.DrawWithRecipe.draw_and_resource_release":1.586,"Driver.Packet":1.438,"Driver.Packet.draw":1.092,"Driver.Draw":1.064},"trace_dropped":false,"trace":"aps5-trace-43a2abaf7753-4132.json.gz"}STK main menu, this branch
876ad540f:{"commit":"876ad540ff5c","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"40:10"},"gpu":"NVIDIA GeForce RTX 4070 Laptop GPU","driver":"NVIDIA 616.92","os":"Windows 10.0.26300","resolution":"1920x1080","window":{"spec":"seconds 40:10","first_present":851,"ms":10029.038},"presents":310,"frame_ms":{"p50":32.141,"p90":33.898,"p99":39.566,"max":40.269},"gpu_busy_ms_per_present":1.124,"worker_busy_pct":{"queue worker 0x0":3.924},"top_phases_ms_per_frame":{"Graphics.Draw":4.743,"Graphics.Draw.draw_and_resource_release":4.587,"Vulkan.DrawFromRecipe":1.515,"Graphics.DrawWithRecipe":1.508,"Graphics.DrawWithRecipe.draw_and_resource_release":1.505,"Driver.Packet":1.248,"Driver.Packet.draw":0.951,"Driver.Draw":0.925},"trace_dropped":false,"trace":"aps5-trace-876ad540ff5c-40272.json.gz"}STK kart selection, main:
{"commit":"43a2abaf7753","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"70:10"},"gpu":"NVIDIA GeForce RTX 4070 Laptop GPU","driver":"NVIDIA 616.92","os":"Windows 10.0.26300","resolution":"1920x1080","window":{"spec":"seconds 70:10","first_present":411,"ms":10253.988},"presents":32,"frame_ms":{"p50":319.174,"p90":328.618,"p99":343.988,"max":343.988},"gpu_busy_ms_per_present":18.050,"worker_busy_pct":{"queue worker 0x0":2.420},"top_phases_ms_per_frame":{"Vulkan.DrawFromRecipe":20.167,"Graphics.DrawWithRecipe":20.066,"Graphics.DrawWithRecipe.draw_and_resource_release":19.708,"Graphics.Draw":17.411,"Graphics.Draw.draw_and_resource_release":14.140,"Driver.Packet":7.386,"Recorder.Submit":5.801,"Driver.Packet.draw":5.363},"trace_dropped":false,"trace":"aps5-trace-43a2abaf7753-5624.json.gz"}STK kart selection, this branch:
{"commit":"876ad540ff5c","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"70:10"},"gpu":"NVIDIA GeForce RTX 4070 Laptop GPU","driver":"NVIDIA 616.92","os":"Windows 10.0.26300","resolution":"1920x1080","window":{"spec":"seconds 70:10","first_present":708,"ms":10030.452},"presents":62,"frame_ms":{"p50":163.165,"p90":172.619,"p99":191.444,"max":191.444},"gpu_busy_ms_per_present":12.149,"worker_busy_pct":{"queue worker 0x0":4.686},"top_phases_ms_per_frame":{"Vulkan.DrawFromRecipe":20.087,"Graphics.DrawWithRecipe":19.992,"Graphics.DrawWithRecipe.draw_and_resource_release":19.643,"Graphics.Draw":15.843,"Graphics.Draw.draw_and_resource_release":12.864,"Driver.Packet":7.202,"Recorder.Submit":5.692,"Driver.Packet.draw":5.256},"trace_dropped":false,"trace":"aps5-trace-876ad540ff5c-39480.json.gz"}tools/trace_digest.py, STK main menu, mainseconds 40:10 from present 401: 184 presents, frame p50 53.3 / p90 57.6 / p99 77.5 / max 79.0 ms, GPU busy 1.5 ms per present
NVIDIA GeForce RTX 4070 Laptop GPU, NVIDIA 616.92, Windows 10.0.26300, 1920x1080, commit 43a2aba, dropped: no
{"commit":"43a2abaf7753","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"40:10"},"gpu":"NVIDIA GeForce RTX 4070 Laptop GPU","driver":"NVIDIA 616.92","os":"Windows 10.0.26300","resolution":"1920x1080","window":{"spec":"seconds 40:10","first_present":401,"ms":10013.889},"presents":184,"frame_ms":{"p50":53.331,"p90":57.606,"p99":77.508,"max":79.033},"gpu_busy_ms_per_present":1.516,"worker_busy_pct":{"queue worker 0x0":2.686},"top_phases_ms_per_frame":{"Graphics.Draw":4.919,"Graphics.Draw.draw_and_resource_release":4.756,"Vulkan.DrawFromRecipe":1.597,"Graphics.DrawWithRecipe":1.59,"Graphics.DrawWithRecipe.draw_and_resource_release":1.586,"Driver.Packet":1.438,"Driver.Packet.draw":1.092,"Driver.Draw":1.064},"trace_dropped":false,"trace":"aps5-trace-43a2abaf7753-4132.json.gz"}Slowest frames (largest slices inside each, inclusive)
Threads (ms per frame)
Largest slices (ms per frame, inclusive, all threads): Driver.WorkerWait 52.8, Driver.WorkerWait.idle_and_completion_poll 52.7, Graphics.Draw 4.9, Graphics.Draw.draw_and_resource_release 4.8, Vulkan.DrawFromRecipe 1.6, Graphics.DrawWithRecipe 1.6, Graphics.DrawWithRecipe.draw_and_resource_release 1.6, batch 1.5, Driver.Packet 1.0, Driver.Packet.draw 0.9
Longest waits
Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)
tools/trace_digest.py, STK main menu, this branchseconds 40:10 from present 851: 310 presents, frame p50 32.1 / p90 33.9 / p99 39.6 / max 40.3 ms, GPU busy 1.1 ms per present
NVIDIA GeForce RTX 4070 Laptop GPU, NVIDIA 616.92, Windows 10.0.26300, 1920x1080, commit 876ad54, dropped: no
{"commit":"876ad540ff5c","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"40:10"},"gpu":"NVIDIA GeForce RTX 4070 Laptop GPU","driver":"NVIDIA 616.92","os":"Windows 10.0.26300","resolution":"1920x1080","window":{"spec":"seconds 40:10","first_present":851,"ms":10029.038},"presents":310,"frame_ms":{"p50":32.141,"p90":33.898,"p99":39.566,"max":40.269},"gpu_busy_ms_per_present":1.124,"worker_busy_pct":{"queue worker 0x0":3.924},"top_phases_ms_per_frame":{"Graphics.Draw":4.743,"Graphics.Draw.draw_and_resource_release":4.587,"Vulkan.DrawFromRecipe":1.515,"Graphics.DrawWithRecipe":1.508,"Graphics.DrawWithRecipe.draw_and_resource_release":1.505,"Driver.Packet":1.248,"Driver.Packet.draw":0.951,"Driver.Draw":0.925},"trace_dropped":false,"trace":"aps5-trace-876ad540ff5c-40272.json.gz"}Slowest frames (largest slices inside each, inclusive)
Threads (ms per frame)
Largest slices (ms per frame, inclusive, all threads): Driver.WorkerWait 30.9, Driver.WorkerWait.idle_and_completion_poll 30.8, Graphics.Draw 4.7, Graphics.Draw.draw_and_resource_release 4.6, Vulkan.DrawFromRecipe 1.5, Graphics.DrawWithRecipe 1.5, Graphics.DrawWithRecipe.draw_and_resource_release 1.5, batch 1.1, Driver.Packet 0.8, Driver.Packet.draw 0.8
Longest waits
Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)
Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)🤖 Generated with Claude Code