Skip to content

perf(agc): keep unimported label stores on the CPU - #7464

Open
Dripl1x wants to merge 8 commits into
boykopovar:mainfrom
Dripl1x:codex/cpu-unimported-labels
Open

Dripl1x wants to merge 8 commits into
boykopovar:mainfrom
Dripl1x:codex/cpu-unimported-labels

Conversation

@Dripl1x

@Dripl1x Dripl1x commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

What

Store a queue-0 label on the CPU after a recorder sync when no host import covers its destination, instead of importing the destination's allocation for the label. VulkanDevice::WriteLabelOnGpu takes that path only for queue 0 and only while no submit is held (Recorder::SubmitHeld()); it returns a new code 7, which every caller (commitLabel in PacketMemory.cpp, recordDeferredLabels in DeferredLabels.cpp) handles like code 1: a CPU store without a device drain. Labels on compute queues, labels behind a held submit and labels whose memory an import already covers keep main's path (import on demand, store recorded on the GPU). The [sync] profile line counts the new path on its own: no covering import (recorder synced) among the fallbacks and on the CPU with no covering import (recorder synced) among the stores (SynchronizationStatistics).

A label the driver stores from the CPU (commitLabel, recordDeferredLabels) also ends short guest sleeps (#6658), as a completed GPU batch or a flip already does.

Why it depends on #7465. On main a small block is imported when the title maps it, so its labels find a covering import and this path does not run. #7465 leaves mappings of 64 KiB or less unimported until their first GPU use. In SuperTuxKart (PPSA69421) that first use is a label, so with #7465 alone each of the blocks the menu maps every frame is imported by its first label instead, and the frame rate stays at main's. With this PR on top, those labels are stored on the CPU after a recorder sync and the blocks are not imported at all:

Build Commit FPS, run 1 / run 2 frame p50 ms
main 94fbb8cea8 9.91 / 10.24 99.8 / 94.3
main + #7465 a1cee90f52 10.29 / 10.19 94.7 / 95.6
main + #7465 + this PR 223f2469f5 25.87 / 25.99 37.7 / 37.9

The cost of the path is visible in the same traces: Recorder.Sync takes 12.0 ms per frame on the draw committer, which #7465 alone does not show, and the frame still drops from 94.7 to 37.7 ms (p50).

Tested

This head is 223f2469f5: current main 94fbb8cea8, #7465's head a1cee90f52, and this PR's five commits on top, picked without conflicts and unchanged from the previous head b31bc845fb. Windows 11 Pro 26300, Ryzen 7 5800H, WinLibs GCC 15.2.0 r7, Release.

  • NVIDIA GeForce RTX 3070 Laptop GPU, NVIDIA 617.42, ANYPS5_NO_SHADER_CACHE=1, three runs each: the 74 tests built from tests/Recorder.cpp, and agc_driver_indirect_arguments_pending_producer, agc_driver_unimported_label_order, agc_driver_cpu_label_wake, agc_driver_same_queue_label_wait, agc_driver_alias_copy_label, agc_driver_pipelined_dma, agc_driver_graphics, agc_driver_host_import_windows, agc_driver_host_import_at_map, agc_driver_host_import_teardown, guest_sleep_wake, guest_sleep_wake_coarse, guest_nanosleep, precise_wait. 88/88 on this head; 87/87 on main and on perf(agc): defer small host imports until first GPU use #7465 (agc_driver_cpu_label_wake is added here).
  • Validation layer, the same set once on main 94fbb8cea8 and on this head (VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation, ANYPS5_NO_SHADER_CACHE=1, ANYPS5_REQUIRE_VULKAN=1, ctest -V, RTX 3070 Laptop GPU, NVIDIA 617.42; DISABLE_LAYER=1 turns off a third-party implicit layer installed here, FPS Monitor, whose vkDestroyInstance hook crashes later instance lookups): 87/87 and 88/88 pass, neither log reports was not found but was requested by env var VK_INSTANCE_LAYERS, and both report the same VUIDs the same number of times (VUID-VkImageViewCreateInfo-usage-02275 4, VUID-vkCmdCopyBuffer-srcBuffer-00118 2). The change adds none.
  • AMD Radeon(TM) Graphics (the Ryzen 7 5800H's integrated GPU, driver 31.0.21925.1001, selected with ANYPS5_GPU=AMD), the same set once and once under the validation layer: 85/88 on this head, 84/87 on main and on perf(agc): defer small host imports until first GPU use #7465. The same three tests fail on all three trees (agc_sample_dumps, agc_driver_vertex_target_range_copied, agc_driver_vertex_target_range_imported), so they are main's; the validation logs report the same VUIDs as on NVIDIA and none new.
  • Linux (WSL Ubuntu 26.04, GCC, lavapipe llvmpipe (LLVM 21.1.8)): libSceAgcDriver builds; agc_driver_unimported_label_order, agc_driver_cpu_label_wake, agc_driver_same_queue_label_wait, agc_driver_indirect_arguments_pending_producer, guest_sleep_wake and agc_driver_host_import_windows pass.
  • Negative controls, on the previous head b31bc845fb with the same source: without the queue-0 gate the compute-queue case fails with reason 7; with SubmitHeld() ignoring a held batch still in flight the held-batch case fails after its 5 s watchdog; without the two wake calls agc_driver_cpu_label_wake fails.
  • tools/check_conventions.py passes for this range.
  • Not run: the whole CTest suite, and titles other than SuperTuxKart.

SuperTuxKart captures. RTX 3070 Laptop GPU, NVIDIA 617.42, same machine and build as above; SuperTuxKart (PPSA69421) tutorial prompt over the main menu, 1920x1080 internal rendering in a maximized 2560x1369 window, warm shader caches, audio muted, no input; APS5_TRACE_SECONDS=20:8 with APS5_TRACE_EVENTS=1048576, console output not redirected, no other APS5_* switch; 2026-10-22, one session, order main, #7465, this PR, this PR, #7465, main; every run kept the window in the foreground, and every summary line reports dirty: false and trace_dropped: false.

Summary lines (main, #7465, this PR, this PR, #7465, main)
{"commit":"94fbb8cea80a","dirty":false,"env":{"APS5_TRACE_EVENTS":"1048576","APS5_TRACE_SECONDS":"20:8"},"gpu":"NVIDIA GeForce RTX 3070 Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 5800H with Radeon Graphics","cpu_threads":16,"memory_mib":64863,"title_id":"PPSA69421","title":"SuperTuxKart","content_version":"00.001.000","resolution":"1920x1080","window":{"spec":"seconds 20:8","first_present":38,"ms":8076.426},"presents":80,"frame_ms":{"p50":99.757,"p90":110.517,"p99":128.310,"max":128.310},"gpu_busy_ms_per_present":5.610,"worker_busy_pct":{"queue worker 0x0":2.846},"top_phases_ms_per_frame":{"Graphics.Draw":7.873,"Graphics.Draw.draw_and_resource_release":7.574,"Vulkan.DrawFromRecipe":6.254,"Graphics.DrawWithRecipe":6.229,"Graphics.DrawWithRecipe.draw_and_resource_release":6.216,"Recorder.Submit":2.908,"Driver.Packet":2.834,"Driver.Packet.draw":2.260},"trace_dropped":false,"trace":"aps5-trace-94fbb8cea80a-424.json.gz"}
{"commit":"a1cee90f52cc","dirty":false,"env":{"APS5_TRACE_EVENTS":"1048576","APS5_TRACE_SECONDS":"20:8"},"gpu":"NVIDIA GeForce RTX 3070 Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 5800H with Radeon Graphics","cpu_threads":16,"memory_mib":64863,"title_id":"PPSA69421","title":"SuperTuxKart","content_version":"00.001.000","resolution":"1920x1080","window":{"spec":"seconds 20:8","first_present":55,"ms":8065.429},"presents":83,"frame_ms":{"p50":94.730,"p90":106.217,"p99":129.954,"max":129.954},"gpu_busy_ms_per_present":5.650,"worker_busy_pct":{"queue worker 0x0":2.889},"top_phases_ms_per_frame":{"Graphics.Draw":8.166,"Graphics.Draw.draw_and_resource_release":7.877,"Vulkan.DrawFromRecipe":6.616,"Graphics.DrawWithRecipe":6.589,"Graphics.DrawWithRecipe.draw_and_resource_release":6.576,"Recorder.Submit":2.785,"Driver.Packet":2.769,"Driver.Packet.draw":2.207},"trace_dropped":false,"trace":"aps5-trace-a1cee90f52cc-21452.json.gz"}
{"commit":"223f2469f5b7","dirty":false,"env":{"APS5_TRACE_EVENTS":"1048576","APS5_TRACE_SECONDS":"20:8"},"gpu":"NVIDIA GeForce RTX 3070 Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 5800H with Radeon Graphics","cpu_threads":16,"memory_mib":64863,"title_id":"PPSA69421","title":"SuperTuxKart","content_version":"00.001.000","resolution":"1920x1080","window":{"spec":"seconds 20:8","first_present":131,"ms":8002.835},"presents":207,"frame_ms":{"p50":37.675,"p90":42.179,"p99":54.788,"max":57.409},"gpu_busy_ms_per_present":4.721,"worker_busy_pct":{"queue worker 0x0":5.801},"top_phases_ms_per_frame":{"Recorder.Sync":11.955,"Graphics.Draw":6.348,"Graphics.Draw.draw_and_resource_release":6.104,"Vulkan.DrawFromRecipe":5.275,"Graphics.DrawWithRecipe":5.255,"Graphics.DrawWithRecipe.draw_and_resource_release":5.243,"Recorder.Submit":2.463,"Driver.Packet":2.212},"trace_dropped":false,"trace":"aps5-trace-223f2469f5b7-7524.json.gz"}
{"commit":"223f2469f5b7","dirty":false,"env":{"APS5_TRACE_EVENTS":"1048576","APS5_TRACE_SECONDS":"20:8"},"gpu":"NVIDIA GeForce RTX 3070 Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 5800H with Radeon Graphics","cpu_threads":16,"memory_mib":64863,"title_id":"PPSA69421","title":"SuperTuxKart","content_version":"00.001.000","resolution":"1920x1080","window":{"spec":"seconds 20:8","first_present":136,"ms":8002.542},"presents":208,"frame_ms":{"p50":37.925,"p90":41.354,"p99":46.729,"max":49.249},"gpu_busy_ms_per_present":4.706,"worker_busy_pct":{"queue worker 0x0":5.815},"top_phases_ms_per_frame":{"Recorder.Sync":11.984,"Graphics.Draw":6.381,"Graphics.Draw.draw_and_resource_release":6.138,"Vulkan.DrawFromRecipe":5.259,"Graphics.DrawWithRecipe":5.238,"Graphics.DrawWithRecipe.draw_and_resource_release":5.227,"Recorder.Submit":2.455,"Driver.Packet":2.205},"trace_dropped":false,"trace":"aps5-trace-223f2469f5b7-36088.json.gz"}
{"commit":"a1cee90f52cc","dirty":false,"env":{"APS5_TRACE_EVENTS":"1048576","APS5_TRACE_SECONDS":"20:8"},"gpu":"NVIDIA GeForce RTX 3070 Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 5800H with Radeon Graphics","cpu_threads":16,"memory_mib":64863,"title_id":"PPSA69421","title":"SuperTuxKart","content_version":"00.001.000","resolution":"1920x1080","window":{"spec":"seconds 20:8","first_present":60,"ms":8047.206},"presents":82,"frame_ms":{"p50":95.592,"p90":109.603,"p99":121.282,"max":121.282},"gpu_busy_ms_per_present":5.638,"worker_busy_pct":{"queue worker 0x0":2.804},"top_phases_ms_per_frame":{"Graphics.Draw":8.185,"Graphics.Draw.draw_and_resource_release":7.900,"Vulkan.DrawFromRecipe":6.988,"Graphics.DrawWithRecipe":6.962,"Graphics.DrawWithRecipe.draw_and_resource_release":6.949,"Recorder.Submit":2.777,"Driver.Packet":2.710,"Driver.Packet.draw":2.157},"trace_dropped":false,"trace":"aps5-trace-a1cee90f52cc-30808.json.gz"}
{"commit":"94fbb8cea80a","dirty":false,"env":{"APS5_TRACE_EVENTS":"1048576","APS5_TRACE_SECONDS":"20:8"},"gpu":"NVIDIA GeForce RTX 3070 Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 5800H with Radeon Graphics","cpu_threads":16,"memory_mib":64863,"title_id":"PPSA69421","title":"SuperTuxKart","content_version":"00.001.000","resolution":"1920x1080","window":{"spec":"seconds 20:8","first_present":60,"ms":8004.322},"presents":82,"frame_ms":{"p50":94.350,"p90":108.262,"p99":142.124,"max":142.124},"gpu_busy_ms_per_present":5.714,"worker_busy_pct":{"queue worker 0x0":2.913},"top_phases_ms_per_frame":{"Graphics.Draw":7.571,"Graphics.Draw.draw_and_resource_release":7.276,"Vulkan.DrawFromRecipe":6.235,"Graphics.DrawWithRecipe":6.207,"Graphics.DrawWithRecipe.draw_and_resource_release":6.194,"Recorder.Submit":2.886,"Driver.Packet":2.803,"Driver.Packet.draw":2.235},"trace_dropped":false,"trace":"aps5-trace-94fbb8cea80a-36604.json.gz"}
tools/performance_compare.py (first capture of #7465 against the first capture of this PR)

**Performance comparison
**

Capture Commit Trace Presents Window (ms)
baseline a1cee90 aps5-trace-a1cee90f52cc-21452.json.gz 83 8065.429
candidate 223f246 aps5-trace-223f2469f5b7-7524.json.gz 207 8002.835
  • Compare captures from the same machine, title, content version and scene; matching recorded metadata cannot verify the physical machine or scene.
  • Changes are candidate minus baseline. A negative frame-time change means a shorter frame interval.
  • A missing top phase is unreported, not zero; its change is unavailable.
  • Recorded configuration differs; these measurements do not isolate the code change.

**Configuration differences
**

Setting Baseline Candidate
window.first_present 55 131

**Frame and GPU times (ms)
**

Metric Baseline Candidate Change Change (%)
frame_ms.p50 94.730 37.675 -57.055 -60.229
frame_ms.p90 106.217 42.179 -64.038 -60.290
frame_ms.p99 129.954 54.788 -75.166 -57.840
frame_ms.max 129.954 57.409 -72.545 -55.824
mean_frame_ms 97.174 38.661 -58.513 -60.215
gpu_busy_ms_per_present 5.650 4.721 -0.929 -16.442

**Top phases (ms per frame)
**

Metric Baseline Candidate Change Change (%)
Driver.Packet 2.769 2.212 -0.557 -20.116
Driver.Packet.draw 2.207 n/a n/a n/a
Graphics.Draw 8.166 6.348 -1.818 -22.263
Graphics.Draw.draw_and_resource_release 7.877 6.104 -1.773 -22.509
Graphics.DrawWithRecipe 6.589 5.255 -1.334 -20.246
Graphics.DrawWithRecipe.draw_and_resource_release 6.576 5.243 -1.333 -20.271
Recorder.Submit 2.785 2.463 -0.322 -11.562
Recorder.Sync n/a 11.955 n/a n/a
Vulkan.DrawFromRecipe 6.616 5.275 -1.341 -20.269
tools/trace_digest.py (first capture of #7465, then of this PR; every digest is in the evidence archive)

v23-s02-7465

seconds 20:8 from present 55: 83 presents, frame p50 94.7 / p90 106.2 / p99 130.0 / max 130.0 ms, GPU busy 5.7 ms per present
NVIDIA GeForce RTX 3070 Laptop GPU, NVIDIA 617.42, Windows 10.0.26300, 1920x1080, commit a1cee90, dropped: no

{"commit":"a1cee90f52cc","dirty":false,"env":{"APS5_TRACE_EVENTS":"1048576","APS5_TRACE_SECONDS":"20:8"},"gpu":"NVIDIA GeForce RTX 3070 Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 5800H with Radeon Graphics","cpu_threads":16,"memory_mib":64863,"title_id":"PPSA69421","title":"SuperTuxKart","content_version":"00.001.000","resolution":"1920x1080","window":{"spec":"seconds 20:8","first_present":55,"ms":8065.429},"presents":83,"frame_ms":{"p50":94.73,"p90":106.217,"p99":129.954,"max":129.954},"gpu_busy_ms_per_present":5.65,"worker_busy_pct":{"queue worker 0x0":2.889},"top_phases_ms_per_frame":{"Graphics.Draw":8.166,"Graphics.Draw.draw_and_resource_release":7.877,"Vulkan.DrawFromRecipe":6.616,"Graphics.DrawWithRecipe":6.589,"Graphics.DrawWithRecipe.draw_and_resource_release":6.576,"Recorder.Submit":2.785,"Driver.Packet":2.769,"Driver.Packet.draw":2.207},"trace_dropped":false,"trace":"aps5-trace-a1cee90f52cc-21452.json.gz"}

Slowest frames (largest slices inside each, inclusive)

  • present 128: 130.0 ms at 7006 ms: Driver.WorkerWait (queue worker 0x0) 125.1, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x0) 125.0, Graphics.Draw (draw committer) 13.4, Graphics.Draw.draw_and_resource_release (draw committer) 13.0
  • present 127: 124.1 ms at 6882 ms: Driver.WorkerWait (queue worker 0x0) 119.4, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x0) 119.3, Graphics.Draw (draw committer) 12.5, Graphics.Draw.draw_and_resource_release (draw committer) 12.1
  • present 129: 122.4 ms at 7136 ms: Driver.WorkerWait (queue worker 0x0) 117.9, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x0) 117.7, Graphics.Draw (draw committer) 12.4, Graphics.Draw.draw_and_resource_release (draw committer) 12.0

Threads (ms per frame)

thread busy wait untraced in slices < 20 us largest
title 0.2 (0 %) 0.0 (0 %) 97.0 0.4 Recorder.Sync 0.2
queue worker 0x0 2.0 (2 %) 94.0 (97 %) 1.1 4.0 Driver.Packet 2.0
draw committer 17.6 (18 %) 0.6 (1 %) 79.0 0.2 Graphics.Draw 8.2, Vulkan.DrawFromRecipe 6.6
presenter 0.9 (1 %) 0.0 (0 %) 96.2 0.1 VideoOut.Flip 0.9, Driver.Present 0.0
gpu 5.7 (6 %) 0.0 (0 %) 91.5 0.0 batch 5.7

Largest slices (ms per frame, inclusive, all threads): Driver.WorkerWait 94.0, Driver.WorkerWait.idle_and_completion_poll 94.0, Graphics.Draw 8.2, Graphics.Draw.draw_and_resource_release 7.9, Vulkan.DrawFromRecipe 6.6, Graphics.DrawWithRecipe 6.6

Longest waits

  • Driver.WorkerWait (queue worker 0x0) 18.2 ms at 7659 ms, present 133
  • Driver.WorkerWait (queue worker 0x0) 8.3 ms at 1529 ms, present 70
  • Driver.WorkerWait (queue worker 0x0) 6.3 ms at 5616 ms, present 114
  • Driver.WorkerWait (queue worker 0x0) 6.1 ms at 5047 ms, present 108
  • Driver.WorkerWait (queue worker 0x0) 5.7 ms at 7120 ms, present 128
  • Driver.WorkerWait (queue worker 0x0) 5.7 ms at 989 ms, present 65

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

  • batch 1637: 2.5 ms at 2655 ms, present 82: GPU no ranges; recorded by Vulkan.DrawFromRecipe (draw committer) 0.2, Graphics.DrawWithRecipe (draw committer) 0.2, Graphics.DrawWithRecipe.draw_and_resource_release (draw committer) 0.2, Recorder.Submit (draw committer) 0.1
  • batch 4174: 1.7 ms at 6737 ms, present 125: GPU no ranges; recorded by Vulkan.DrawFromRecipe (draw committer) 0.2, Graphics.DrawWithRecipe (draw committer) 0.2, Graphics.DrawWithRecipe.draw_and_resource_release (draw committer) 0.2, Recorder.Submit (draw committer) 0.1
  • batch 1991: 1.7 ms at 3218 ms, present 88: GPU no ranges; recorded by Vulkan.DrawFromRecipe (draw committer) 0.2, Graphics.DrawWithRecipe (draw committer) 0.2, Graphics.DrawWithRecipe.draw_and_resource_release (draw committer) 0.2, Recorder.Submit (draw committer) 0.0

v23-s03-pair

seconds 20:8 from present 131: 207 presents, frame p50 37.7 / p90 42.2 / p99 54.8 / max 57.4 ms, GPU busy 4.7 ms per present
NVIDIA GeForce RTX 3070 Laptop GPU, NVIDIA 617.42, Windows 10.0.26300, 1920x1080, commit 223f246, dropped: no

{"commit":"223f2469f5b7","dirty":false,"env":{"APS5_TRACE_EVENTS":"1048576","APS5_TRACE_SECONDS":"20:8"},"gpu":"NVIDIA GeForce RTX 3070 Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","cpu":"AMD Ryzen 7 5800H with Radeon Graphics","cpu_threads":16,"memory_mib":64863,"title_id":"PPSA69421","title":"SuperTuxKart","content_version":"00.001.000","resolution":"1920x1080","window":{"spec":"seconds 20:8","first_present":131,"ms":8002.835},"presents":207,"frame_ms":{"p50":37.675,"p90":42.179,"p99":54.788,"max":57.409},"gpu_busy_ms_per_present":4.721,"worker_busy_pct":{"queue worker 0x0":5.801},"top_phases_ms_per_frame":{"Recorder.Sync":11.955,"Graphics.Draw":6.348,"Graphics.Draw.draw_and_resource_release":6.104,"Vulkan.DrawFromRecipe":5.275,"Graphics.DrawWithRecipe":5.255,"Graphics.DrawWithRecipe.draw_and_resource_release":5.243,"Recorder.Submit":2.463,"Driver.Packet":2.212},"trace_dropped":false,"trace":"aps5-trace-223f2469f5b7-7524.json.gz"}

Slowest frames (largest slices inside each, inclusive)

  • present 139: 57.4 ms at 399 ms: Driver.WorkerWait (queue worker 0x0) 53.4, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x0) 53.3, Recorder.Sync (draw committer) 16.0, Graphics.Draw (draw committer) 11.6
  • present 131: 57.2 ms at 0 ms: Driver.WorkerWait (queue worker 0x0) 53.5, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x0) 53.4, Recorder.Sync (draw committer) 14.2, Vulkan.DrawFromRecipe (draw committer) 12.0
  • present 136: 54.8 ms at 254 ms: Driver.WorkerWait (queue worker 0x0) 51.2, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x0) 51.1, Recorder.Sync (draw committer) 15.9, Vulkan.DrawFromRecipe (draw committer) 9.8

Threads (ms per frame)

thread busy wait untraced in slices < 20 us largest
queue worker 0x0 1.4 (4 %) 36.1 (93 %) 1.1 3.9 Driver.Packet 1.4, Shader.DrawStage 0.0
draw committer 23.6 (61 %) 0.7 (2 %) 14.4 0.1 Recorder.Sync 12.0, Graphics.Draw 6.3
presenter 0.9 (2 %) 0.0 (0 %) 37.7 0.1 VideoOut.Flip 0.9, Driver.Present 0.0
gpu 4.7 (12 %) 0.0 (0 %) 33.9 0.0 batch 4.7

Largest slices (ms per frame, inclusive, all threads): Driver.WorkerWait 36.1, Driver.WorkerWait.idle_and_completion_poll 36.1, Recorder.Sync 12.0, Graphics.Draw 6.3, Graphics.Draw.draw_and_resource_release 6.1, Vulkan.DrawFromRecipe 5.3

Longest waits

  • Driver.WorkerWait (queue worker 0x0) 3.5 ms at 145 ms, present 133
  • Driver.WorkerWait (queue worker 0x0) 2.9 ms at 897 ms, present 150
  • Driver.WorkerWait (queue worker 0x0) 2.9 ms at 233 ms, present 135
  • Driver.WorkerWait (queue worker 0x0) 2.9 ms at 196 ms, present 134
  • Driver.WorkerWait (queue worker 0x0) 2.8 ms at 4073 ms, present 233
  • Driver.WorkerWait (queue worker 0x0) 2.8 ms at 774 ms, present 147

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

  • batch 1155: 2.5 ms at 897 ms, present 150: GPU no ranges; recorded by Recorder.Sync (draw committer) 2.6, Vulkan.DrawFromRecipe (draw committer) 0.0, Graphics.DrawWithRecipe (draw committer) 0.0, Graphics.DrawWithRecipe.draw_and_resource_release (draw committer) 0.0
  • batch 270: 2.4 ms at 233 ms, present 135: GPU no ranges; recorded by Recorder.Sync (draw committer) 2.5, Vulkan.DrawFromRecipe (draw committer) 0.0, Graphics.DrawWithRecipe (draw committer) 0.0, Graphics.DrawWithRecipe.draw_and_resource_release (draw committer) 0.0
  • batch 978: 2.4 ms at 774 ms, present 147: GPU no ranges; recorded by Recorder.Sync (draw committer) 2.5, Vulkan.DrawFromRecipe (draw committer) 0.0, Graphics.DrawWithRecipe (draw committer) 0.0, Graphics.DrawWithRecipe.draw_and_resource_release (draw committer) 0.0

Evidence: 7464-evidence-20261022.zip (the six traces, summary lines, digests and comparisons, test and validation-layer logs).

Where it helps, and what it costs

Probe counters from 2026-10-21, on main 2185dbf92e with a local counter build that is not part of this PR (APS5_PROFILE_DRAW=1, medians of two runs per flip over the same 8-s window; they carry profiling overhead and are not used for FPS), in the scene above: the menu maps about 48 blocks of 64 KiB per frame and writes about 59 labels into them. main imports each block when it is mapped (48.5 imports per frame, ImportMappedRanges 784 µs per call). In a build of main with the eleven PRs of that series, which include #7465 and this one, no block is imported (0 per frame) and the labels take this path instead: 58.9 recorder syncs per frame, WriteLabelOnGpu 194 µs per call. With this PR reduced to its wake calls in the same build, 47.8 imports per frame come back, every one made by a label (WriteLabelOnGpu 696 µs per call).

@LeandroLP23 measured the same path in a SuperTuxKart race on a Radeon 860M (comment above): it is the largest cost left there (Recorder.Sync 41 ms of a ~96 ms frame), and the two alternatives he tried, importing the label pages and deferring the store to the batch reap, were slower (5.4 and 0.7 FPS against 10.5).

Known limitations.

Dependencies and overlap

Checklist

  • Based on current main 94fbb8cea8 through perf(agc): defer small host imports until first GPU use #7465 a1cee90f52; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source (none added)
  • Generic behaviour, not specific to one title
  • Depends on: perf(agc): defer small host imports until first GPU use #7465
  • Fixes: none
  • Unblocks: none
  • GPUs tested: NVIDIA GeForce RTX 3070 Laptop GPU, NVIDIA 617.42, Windows 11 Pro 26300; AMD Radeon(TM) Graphics (Ryzen 7 5800H), 31.0.21925.1001, Windows 11 Pro 26300; lavapipe (llvmpipe, LLVM 21.1.8) on Linux
  • AI-assisted: yes

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 7963 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp L164
PR 8622 core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp L981-982
PR 9011 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp L3444, L3447-3448
PR 9495 core/libs/prx/libSceAgcDriver/CMakeLists.txt L2025

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 1191 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 1368 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 1395 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 2434 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 4040 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 4072 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 4448 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 4449 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 4472 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 5021 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
docs/dev/debt/libkernel.md
PR 5371 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 5372 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 5431 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
PR 5441 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 5609 core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
PR 5925 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 6044 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6149 docs/dev/debt/libkernel.md
PR 6169 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6187 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 6188 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 6192 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 6197 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 6202 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6209 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 6280 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6292 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Synchronization/DeferredLabels.cpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
PR 6300 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6412 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 6469 docs/dev/debt/libkernel.md
PR 6481 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 6501 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/include/Driver/Synchronization/SynchronizationStatistics.hpp
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Synchronization/SynchronizationStatistics.cpp
PR 6539 core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
PR 6548 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 6615 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Synchronization/DeferredLabels.cpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
PR 6642 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6740 docs/dev/debt/libkernel.md
PR 6788 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6808 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6830 docs/dev/debt/libkernel.md
PR 6868 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 6879 docs/dev/debt/libkernel.md
PR 6946 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 6960 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6971 core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 6973 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 6987 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 6989 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 6995 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 7002 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 7088 docs/dev/debt/libkernel.md
PR 7105 docs/dev/debt/libkernel.md
PR 7149 docs/dev/debt/libkernel.md
PR 7201 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
PR 7275 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7447 docs/dev/debt/libkernel.md
PR 7457 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 7509 docs/dev/debt/libkernel.md
PR 7565 docs/dev/debt/libkernel.md
PR 7604 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 7622 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
PR 7642 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 7653 docs/dev/debt/libkernel.md
PR 7705 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7803 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 8014 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 8105 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 8126 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 8246 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 8278 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 8484 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 8520 docs/dev/debt/libkernel.md
PR 8527 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 8547 docs/dev/debt/libkernel.md
PR 8552 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 8553 docs/dev/debt/libkernel.md
PR 8592 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 8658 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
docs/dev/debt/libkernel.md
PR 8672 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 9020 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
PR 9239 docs/dev/debt/libkernel.md
PR 9335 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
PR 9356 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
PR 9387 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
PR 9508 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
core/libs/prx/libSceAgcDriver/tests/BdaResources.cpp
core/libs/prx/libSceAgcDriver/tests/Recorder.cpp
PR 9533 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp
PR 9717 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9742 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9748 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp

@github-actions github-actions Bot added libkernel build docs prx agc libSceAgc* needs rebase Conflicts with main, rebase needed labels Oct 18, 2026
@blawesom

Copy link
Copy Markdown
Contributor

Rebase handoff — rebased onto current main (both commits, authorship unchanged). One docs-debt conflict, unioned.

Verified on Linux (CachyOS, kernel 7.2.9, GCC 16.2.1, Ninja, Release, BUILD_TESTING=ON, AMD RX 9070 XT / RADV): builds; conventions pass; full suite green apart from the two known main regressions (agc_driver_tessellation — Mesa, #5444; agc_driver_bda_device — #7584), neither from this PR.

To pick it up:

git fetch https://github.com/blawesom/AnyPS5 rebase/7464
git checkout <your branch>
git merge --ff-only FETCH_HEAD

Head is fdf2b016d.

@blawesom

Copy link
Copy Markdown
Contributor

Correction: the first push missed a required ENVIRONMENT.md row for APS5_NO_SLEEP_WAKE (the conventions check reads the commit, not the working tree — the row is now amended in). Head is now f7e0a81c2; conventions pass.

@Dripl1x
Dripl1x force-pushed the codex/cpu-unimported-labels branch from ca7b45b to aa92e2c Compare October 18, 2026 18:28
@Dripl1x

Dripl1x commented Oct 18, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the rebase handoff, Linux validation, and the follow-up ENVIRONMENT.md row. Picked up your corrected handoff and rebased it onto main e1dae23; published head is aa92e2c. Both sides of the debt-doc conflict are retained, and I clarified that APS5_NO_SLEEP_WAKE disables early wakeups. The six focused Windows Release checks pass, including unimported-label ordering, CPU wake, same-queue waits, and the sleep checks. A matched SuperTuxKart A-B-B-A repeat with only libSceAgcDriver.prx differing gives 11.945 FPS for the exact #6658 control and 11.900 FPS for this topic (-0.045 FPS); the description now reports the individual captures and their limits instead of using the combined series to justify this topic. Logs, manifests and raw traces: https://github.com/Dripl1x/AnyPS5/releases/download/performance-series-20261018/review-rebase-20261018.zip

@github-actions github-actions Bot added perf Performance change perf numbers missing perf pull request without a before and after summary line and removed needs rebase Conflicts with main, rebase needed perf numbers missing perf pull request without a before and after summary line labels Oct 18, 2026
@oneandonlydean oneandonlydean self-assigned this Oct 19, 2026
@oneandonlydean

Copy link
Copy Markdown
Collaborator

@Dripl1x thanks. The ordering is right. Three things before this can merge:

  1. Main imports an unimported label store once and then frees later GPU label stores; this PR syncs the recorder on every label in unimported memory instead, and its own A/B is -0.38%. Could you post the [sync] profile line from main and from this PR in the measured scene, with the new stall counted separately: how many labels per frame take the Sync path, and how many imports does it avoid?
  2. Please give "unimported, synced" its own return code or counter, so it isn't counted as idle in that line.
  3. The first commit carries perf(libkernel): end short sleeps when the GPU or a flip completes #6658. Once perf(libkernel): end short sleeps when the GPU or a flip completes #6658 merges, please rebase and drop the duplicate APS5_NO_SLEEP_WAKE row.

@screamm

screamm commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Windows NVIDIA run on the rebase, with the [sync] lines and the per-label counts you asked for. Two blockers came up on current main.

Host: Windows 11 Pro 26220, NVIDIA GeForce RTX 4070 driver 596.21, Ryzen 7 5700X, WinLibs MinGW-w64 GCC 15.2 r7, Release. Head aa92e2c47, which is already newer than blawesom's rebase/7464 (f7e0a81c2), merged onto main 1d4164552. Two include/field unions: EntryPoints.cpp (main's Screenshot.hpp include and yours) and Recorder.hpp (main's submitHold/submitHoldValue and your progressWatcher).

1. agc_driver_indirect_arguments_pending_producer hangs with this PR. It times out at 300 s in the GPU pass and 3/3 at 60 s on main+this PR alone, also with APS5_NO_SLEEP_WAKE=1. main's binary passes 3/3 in about 1 s. The hang is in the last variant, group counts read on the CPU behind an argument label of an earlier graphics batch, which loops on [gpu] recorded batch still running on the GPU after N s. If I restore only this PR's VulkanDevice.cpp hunk (the HostImportCovers / recorder.Sync() path) to main, it passes 3/3. That case records a label into slot.arguments while the fixture's timeline-semaphore gate holds the batch until the producer is submitted (16683be01), and it requires WriteLabelOnGpu(...) == 0. main imports the label memory and records the store. Here the unimported path syncs the held batch instead. The test arrived on main on 10-19 (f6c8e8474, 16683be01), after this PR's base e1dae23e0.

2. The new eight-byte case in agc_driver_unimported_label_order is layout-flaky. The first x3 passed, but 4 of 12 further runs failed with an eight-byte label did not select its unimported CPU path, and it also failed 1 of 1 on the stack build. With a printf it fails exactly when VirtualAlloc places temporary right after a 128 KiB-aligned labels block. The FillBuffer import of labels is then 131072 bytes and covers temporary, so the label correctly goes to the GPU (reason 0):

reason labels temporary import live bytes
0 0x1f138b60000 0x1f138b70000 131072
1 0x2829f8b0000 0x2829f8c0000 65536
1 0x29515640000 0x29515830000 65536

A gap between the two blocks, or a HostImportCovers check on temporary first, would make the case deterministic.

Tests and control. agc_driver_cpu_label_wake, agc_driver_same_queue_label_wait, agc_driver_unimported_label_order and guest_sleep_wake pass 4/4 in the first until-fail:3 run. With APS5_NO_SLEEP_WAKE=1, cpu_label_wake fails (a completed CPU label did not wake the guest short sleep) and guest_sleep_wake aborts (0xc0000409), 4/4 each. same_queue_label_wait still passes.

[sync] line, vkQuake PPSA99010. Shareware pak0 demo loop at 3840x2160, 90 s, APS5_PROFILE_DRAW=1 APS5_PROFILE_GPU=1 APS5_TRACE_SUMMARY_EVERY=10, interleaved with main in one session. The values are cumulative at the drain where the line printed.

main (8223 presents at the next [flip]):
[sync] 5 device drains by packet: DMA_DATA=5 (4 waited without the GPU mutex); 344724 labels written on the GPU, 0 deferred behind completions, 0 no-op, fallbacks: idle 5679, completions 0, not imported 0, undecodable 0; 0 labels queued per worker, recorded in 0 groups, 350403 recorded at once; stores: 12969 recorded on the GPU, 0 behind completions, 3052 on the CPU (idle), 5 synced (drained), 0 DMA fills stored as buffer fills; 0 copies into host memory synced on their source; 0 copies of up to 64 KiB recorded on the GPU behind their source's pending stores
this PR (8419 presents):
[sync] 5 device drains by packet: DMA_DATA=5 (4 waited without the GPU mutex); 352091 labels written on the GPU, 0 deferred behind completions, 0 no-op, fallbacks: idle 6738, completions 0, not imported 0, undecodable 0; 0 labels queued per worker, recorded in 0 groups, 358829 recorded at once; stores: 13653 recorded on the GPU, 0 behind completions, 2742 on the CPU (idle), 5 synced (drained), 0 DMA fills stored as buffer fills; 0 copies into host memory synced on their source; 0 copies of up to 64 KiB recorded on the GPU behind their source's pending stores

Labels on the Sync path and imports avoided. Your point 2 stands: the Sync path returns 1, so in [sync] it shows up as idle. To separate it I added local counters (not proposed for merge) in WriteLabelOnGpu after the Idle() return, on main and on this PR: busy-recorder labels with no covering import (HostImportCovers false; on this PR, exactly the labels that take recorder.Sync()), and Access imports made inside a label's HostImportFor. They printed next to [flip], because Ping Pong and Showcase print no [sync] line in 60 s.

Title (60 s) main: labels without import / imports made this PR: Sync-path labels (per present) / imports made GPU labels per present, main / PR
Middle Ages Ping Pong PPSA28319 2 / 2 2990 (1.00) / 0 13.0 / 9.0
Showcase PPSA99005 3 / 3 2 (0.004) / 0 37.0 / 37.0
vkQuake PPSA99010 0 / 0 0 / 0 42.6 / 42.6

vkQuake gave 0 / 0 in a 120 s run as well. So in Ping Pong this PR avoids 2 imports and syncs the recorder about once per frame instead, as you described. Ping Pong is vsync-bound and stays at 59.94 FPS with p50 16.683 ms on both sides. vkQuake takes the path 0 times, so its FPS difference against main in the same session (+6.8/+5.4% in a loaded round, +2.8/+2.5% in a quiet one: 105.44/105.68 vs 108.36/108.36 FPS) cannot come from the label change. #6658's short-sleep wake is the other change in this PR, but I did not build #6658 alone to confirm that.

GPU pass on main + #7473 + this PR: 599 passed, 2 skipped as on main, 2 failed (pass_input_hazard also fails on main; indirect_arguments_pending_producer as above). Conventions: 0 errors, 1 warning (docs/dev/debt/libkernel.md:228 debt entry 1022 characters). git diff --check is clean.

Not checked: #6658 on its own, the PR head on its own base, Linux, the full non-GPU suite.

Written with Claude Code assistance (validation by a subagent, reviewed in this session).

@Dripl1x

Dripl1x commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased onto main 2185dbf92e. The #6658 copy is gone (merged as b7596be595), and so are the duplicate APS5_NO_SLEEP_WAKE row and the docs commit, which only reworded text main now has.

@screamm, thank you, both blockers are confirmed and fixed. (1) The unimported path called recorder.Sync() under the GPU mutex, which submits the open batch and waits for every batch in flight. In the argument-label case the open batch is held on the fixture's semaphore until a later producer submit, so Sync never returned; the rebased old topic still times out here. The CPU path now runs only for queue 0 and never while a submit is held (Recorder::SubmitHeld()), so that label is imported and recorded on the GPU as on main; compute queues, which must never wait, keep main's path too. (2) The temporary block sat right after a 128-KiB-aligned labels block and was covered by its import; it is now registered between two unregistered guard blocks, and an adjacent aligned placement that failed the old case passes.

@oneandonlydean, on your three points:

  1. The [sync] line does not print in the SuperTuxKart scene (it prints only at a device drain by packet or with deferred label groups, as @screamm saw in Ping Pong), so I counted the same events with probe counters in main and in the series, A-B-B-A. Per frame the title maps about 48.5 blocks of 64 KiB. Main imports each at map time. perf(agc): defer small host imports until first GPU use #7465 alone defers those imports, and without this PR the next label imports the block anyway (47.8 label imports per frame). With this PR and perf(agc): defer small host imports until first GPU use #7465 in the same build, the label path takes about 59 labels per frame, about 194 µs each, and no import is created. Alone this PR changes nothing measurable here (9.21 / 9.33 FPS against main 9.24 / 9.30); together: main 9.18 / 9.03 FPS, main + perf(agc): defer small host imports until first GPU use #7465 8.66 / 8.91, main + perf(agc): defer small host imports until first GPU use #7465 + perf(agc): keep unimported label stores on the CPU #7464 20.85 / 22.02, the whole series 29.01 / 28.70 (SuperTuxKart menu scene, RTX 3070 Laptop, two runs each in A-B-C-D-D-C-B-A order).
  2. Done: the synced path returns its own code 7, handled by every caller like 1, and is counted as no covering import (recorder synced) among the fallbacks and on the CPU with no covering import (recorder synced) among the stores.
  3. Done after perf(libkernel): end short sleeps when the GPU or a flip completes #6658 merged.

Tests (Windows 11, RTX 3070 Laptop, Release): indirect_arguments_pending_producer 3/3, unimported_label_order 15/15 twice, cpu_label_wake, same_queue_label_wait, guest_sleep_wake, guest_sleep_wake_coarse, guest_nanosleep, precise_wait, alias_copy_label, pipelined_dma 3/3 each, negative controls for the queue gate, the held batch and the wake. The whole ^agc_driver_ set on this head: 612 tests, 601 passed, 3 skipped, 8 failed. I then ran the eight (depth_clear_hle, depth_fast_clear_submit, multisample_fill_host_only, native_multisample_depth, native_depth_clears, prepared_shaders_indirect, prepared_shaders_debug, prepared_shaders_debug_failure) three times each on main 2185dbf92e and on this head: 13 of 24 failed on main and 11 of 24 here, the same tests on both sides (access violations and timeouts), so I do not attribute them to this change. The description has the counters, the remaining limitation (recorded guest GPU work that waits for memory a later CPU write produces would block the sync), summary lines and evidence.

AI-assisted: yes (Claude Code)

@LeandroLP23

Copy link
Copy Markdown
Contributor

SuperTuxKart race data on the unimported-label path, in case it helps with this PR and #8858. It doesn't call for a change; the two alternatives I tried were worse.

Setup.

Where the remaining time goes.

  • In a trace_digest, the draw committer is busy 71.6 ms of a ~96 ms frame, and Recorder.Sync takes 41.1 ms of that at its top level.
  • The [recorder] line with APS5_PROFILE_DRAW=1 lists 68256 syncs waiting 47.0 s over the run. 68115 of them, 46.9 s, come from one site: +0x131156, the return from Recorder::SyncForCpuLabel in VulkanDevice::WriteLabelOnGpu. That is this PR's syncUnimported path.
  • The title's thread, ps5-opengl's fence wait, spins briefly, then calls sceKernelUsleep(1000) until the label matches, for up to 2000 tries. That is about 64 % of its samples.
  • So every queue-0 label into memory without an import (about 32 per frame here) drains the recorder.

Two alternatives, both worse. Measured locally, two runs each, same scene:

unimported queue-0 label FPS GPU busy per present
SyncForCpuLabel, then a CPU store (this PR) 10.46 / 10.68 33.9 / 31.0 ms
no HostImportCovers early return, so HostImportFor imports the range and the store goes on the GPU 5.41 / 5.35 73.9 / 78.1 ms
AfterCompletions(..., storedOnGpu = false) and reason 5, so the label lands at the batch's reap with no drain 0.74 / 0.72 28.2 / 77.5 ms
  • Importing the label pages doubles the GPU time per frame.
  • Deferring the store to the reap leaves the label unwritten for about 1.35 s a frame: nothing reaps that batch promptly, and the title polls in 1 ms sleeps until then.

The drain looks like the right choice among these. Unless a completion could be reaped as soon as the progress watcher sees the batch finish (it already wakes the sleepers), the label has to land when the title wakes. I haven't tried that.

Dripl1x and others added 8 commits October 22, 2026 09:35
…PU use

The import-requester test now counts the map requester on a 128 KiB range
and asserts that a 64 KiB range is not imported at map time and is imported,
for the access requester, at its first use. The over-budget map test maps
128 KiB so it still reaches the budget check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…time

importMappedFailureGenerationTests (boykopovar#8802) maps a single 64 KiB range and
expects ImportMappedRanges to try its import. Ranges of 64 KiB are now left
to their first GPU use, so the probe saw no allocation. The fixture maps
128 KiB as one window, which is still imported at map time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orts

An import covers a run of adjacent registered allocations. When
VirtualAlloc placed the temporary block right after the 128 KiB-aligned
labels block, the labels import also covered the temporary block, the
eight-byte label correctly went to the GPU and the case failed (4 of 12
runs in review). The temporary block is now registered between two
unregistered 64 KiB guards of its own allocation, so no import made for
another block can cover it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The unimported label path synced the recorder: it submitted the open
batch and waited for every batch in flight under the GPU mutex, running
their completion actions. A compute worker must never run them (labels
from other queues only submit), and the sync waited forever on a batch
whose submit was held on a semaphore that only a later submission
releases: agc_driver_indirect_arguments_pending_producer timed out in
its argument-label case, which needs the label imported and recorded on
the GPU.

The path is now taken only for queue 0 while no submit is held (a hold
armed for the next submit, or a held batch the GPU has not finished).
Other labels import their memory and are recorded on the GPU, as on
main. UnimportedLabelOrder covers a compute-queue label and a label
behind a held batch, each of which must be recorded without a submit or
a wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eir own

The unimported path returned 1, the code of a label the CPU stores
because nothing is recorded, so the [sync] line counted its labels as
idle fallbacks and its stores as idle CPU stores. It now returns 7,
which every caller handles as it handles 1 (the CPU writes the bytes
without a device wait, and the label is noted and wakes short sleeps
where 1 does), and the [sync] line counts it separately as "no covering
import (recorder synced)" among the fallbacks and "on the CPU with no
covering import (recorder synced)" among the stores. PipelinedDma counts
these stores with the other committed ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rces

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Dripl1x

Dripl1x commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Updated on 2026-10-22; the description is rewritten to match.

@JJenkx

JJenkx commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Lodoss, which the description lists as not run: neutral on AMD/Linux, with no measurable cost from the new path.

Setup

Run presents / FPS frame p50 / p99 ms Recorder.Sync ms per frame host imports made (access) map made / refused / failed
#7465 1799 / 59.94 16.68 / 16.70 0.2 132 (20) 112 / 60 / 75
PR 1799 / 59.94 16.68 / 16.70 0.2 128 (18) 110 / 61 / 75
PR 1799 / 59.94 16.68 / 16.70 0.3 129 (18) 111 / 60 / 75
#7465 1799 / 59.94 16.68 / 16.70 0.2 132 (20) 112 / 60 / 75
  • Both sides hold the title's 60 FPS cap, so this scene cannot show a gain.
  • The PR removes about two first-use imports per run (access 20 → 18) and adds no measurable Recorder.Sync time, unlike the 12 ms per frame in the SuperTuxKart capture. In this scene, Lodoss's small blocks are rarely first touched by a label.
  • The 75 failed map imports appear identically on both sides, so they are not from this PR.

AI-assisted: yes (Claude Code); results above were run on this machine.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants