Skip to content

perf(agc): store a resident heap mirror's write-back from its readback buffer - #12827

Open
JJenkx wants to merge 1 commit into
boykopovar:mainfrom
JJenkx:perf/agc-heap-writeback-readback
Open

JJenkx wants to merge 1 commit into
boykopovar:mainfrom
JJenkx:perf/agc-heap-writeback-readback

Conversation

@JJenkx

@JJenkx JJenkx commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Additional title measurements supplied by @mrfelek, 2026-11-04: full route, source pins, summaries and trace digests. On PPSA22408 (Hot Wheels Infinite Rush), RTX3080Ti/NVIDIA615.71.09, Ryzen5950X/Linux7.3, 3840×2160, warm shader caches and identical import/staging/shadow settings, their alternating A/B/A/B route compares plain main b11c7bf84f with that main plus this PR's c12a20f45d topic (5fcf8be8a4). The post-load flyover improves from 0.30/0.30 to 0.62/0.63 presents/s, with median frame p50 moving from3.94/4.10s to1.82/1.86s. This remains very slow gameplay.

The race-load window has no established improvement; the attached 330:10s trace windows land in loading and do not measure that post-load gain. The comment separately provides the post-load summary records. Later magenta/green artifacts also occur on their longer main control, and no exceptions occurred. These are the contributor's measurements, not locally reproduced title results. Their PPSA28416 multi-PR-tree switch pairs are attribution-only, not plain-main performance evidence. All measurements retain these historical source/configuration pins and do not qualify later heads.

Previous implementation report and contribution template:

What

GuestBufferMemory::WriteBack stores a resident heap mirror's write-back from its recorded readback buffer. It used to copy the readback into a new std::vector<std::byte> of the written range and store from that copy. ResidentBuffer::Store and storeChanged/storeSparse take a std::span<const std::byte>, and the readback buffer stays alive in heapReadbacks for the whole write-back, so the copy is dropped. A write-back without a recorded readback (ResidentBuffer::ReadDevice) still reads into a vector, as before. The bytes stored in the resident shadow and in guest memory are the same.

Why

In Echoes of the End (PPSA25636), a compute dispatch with atomic writes into a resident sparse heap mirror completes about once per second. Each completion writes back the dispatch's whole written range, about 540 MiB. A later build that reads the same heap waits for that write-back in CompletePendingCpuWrites (UploadFinish). Allocating, zeroing and filling the 540 MiB vector took most of each write-back.

The attribution below comes from a private integration tree (older main plus open PRs) with private counters (APS5_PROFILE_DRAW plus diagnostics). It is attribution, not a speed claim: 4 interleaved 900 s runs of the same build, with a private switch restoring the copy.

Per write-back (mean) with the copy without (this change)
copy into the vector 342 / 332 ms 0 / 0 ms
ResidentBuffer::Store 42 / 47 ms 78 / 81 ms
guest store (storeChanged) 35 / 36 ms 36 / 38 ms
CompletePendingCpuWrites sync per copied-writer overlap 511 / 503 ms 168 / 172 ms
total in those syncs per 900 s run 368 / 349 s 135 / 141 s

ResidentBuffer::Store reads the host-visible readback mapping directly now, so it got slower. The total per write-back still falls from about 419 ms to about 115 ms.

Tested

Arch Linux 7.2.9-arch1-1, AMD Radeon RX 6650 XT, RADV Mesa 26.2.4, Ryzen 9 7950X.

Focused tests on main and on the change (ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ctest -V -j1 -R 'agc_driver_resident_heap_mirror|heap_mirror'; 9 tests):

  • main 4c7d275: 9/9 passed, 0 VUID-.
  • Change (same code on 4c7d275): 9/9 passed, 0 VUID-.
  • Neither log reports was not found but was requested by env var VK_INSTANCE_LAYERS.
  • agc_driver_resident_heap_mirror ran its resident tests on this device (the device has sparse residency; the log prints resident heap mirror tests passed, not the skip message). They dispatch stores into a resident heap mirror and check guest memory after the write-back.

Echoes of the End: same-scene main/change pairs on one machine (also posted with screenshots in #12827 (comment)). Echoes does not boot on main alone:

Both sides therefore use the same private base: main 899ae3d + #4449 + #8089 + #11873 + #6867, commit 27db71c581b. The change side is that base + this PR's commit, 3e83ee94776; its git patch-id equals c12a20f45da's. These runs therefore compare this PR on that private stack, not on bare main.

Runs:

  • 1800 s each, scripted pad input, the same warm seeded cache before every run (seed manifest 6a3bbca6eb39b28b), APS5_TRACE_SECONDS=1500:240, no other APS5_TRACE_*/APS5_PROFILE_* switch set.
  • Order: base, change, then six runs for other PRs, then change, base. Nothing crashed.
  • Machine: Arch Linux 7.2.9-arch1-1, Ryzen 9 7950X, RX 6650 XT, RADV Mesa 26.2.4, 3840x2160.
  • Scenes were checked with screenshots at 1440 s and 1795 s, which bracket the 1500–1740 s window.

Pair 1: in-game "Loading..." screen (cathedral with the blue crystal) in both windows. Base, then change:

{"commit":"27db71c581b4","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"1500:240"},"gpu":"AMD Radeon RX 6650 XT (RADV NAVI23)","driver":"radv Mesa 26.2.4-arch1.1","os":"Linux 7.2.9-arch1-1","cpu":"AMD Ryzen 9 7950X 16-Core Processor","cpu_threads":32,"memory_mib":127901,"title_id":"PPSA25636","title":"Echoes of the End","content_version":"01.012.000","resolution":"3840x2160","window":{"spec":"seconds 1500:240","first_present":803,"ms":240007.014},"presents":125,"frame_ms":{"p50":1621.105,"p90":2233.242,"p99":8046.195,"max":8428.698},"gpu_busy_ms_per_present":816.406,"worker_busy_pct":{"queue worker 0x0":99.503,"queue worker 0x20":98.191},"top_phases_ms_per_frame":{"Driver.Packet":3760.159,"Driver.Packet.dispatch":2070.479,"Driver.Dispatch":1796.986,"Driver.Packet.memory_wait":1211.455,"Vulkan.Dispatch":1111.888,"Resources.Complete":1068.127,"Vulkan.Dispatch.shader_resources":1033.539,"Graphics.Draw":286.697},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":64.295,"events_lost":646426,"totals_full":false},"queue worker 0x20":{"events_full_after_s":217.500,"events_lost":27431,"totals_full":false}},"trace":"aps5-trace-27db71c581b4-196709.json.gz"}
{"commit":"3e83ee94776d","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"1500:240"},"gpu":"AMD Radeon RX 6650 XT (RADV NAVI23)","driver":"radv Mesa 26.2.4-arch1.1","os":"Linux 7.2.9-arch1-1","cpu":"AMD Ryzen 9 7950X 16-Core Processor","cpu_threads":32,"memory_mib":127901,"title_id":"PPSA25636","title":"Echoes of the End","content_version":"01.012.000","resolution":"3840x2160","window":{"spec":"seconds 1500:240","first_present":1452,"ms":240019.614},"presents":341,"frame_ms":{"p50":704.873,"p90":877.547,"p99":1018.779,"max":1114.243},"gpu_busy_ms_per_present":335.082,"worker_busy_pct":{"queue worker 0x0":98.756,"queue worker 0x20":97.053},"top_phases_ms_per_frame":{"Driver.Packet":1357.357,"Driver.Packet.dispatch":671.973,"Driver.Dispatch":595.953,"Driver.Packet.memory_wait":456.245,"Vulkan.Dispatch":245.115,"Vulkan.Dispatch.shader_resources":233.316,"Resources.Complete":188.105,"Graphics.Draw":126.533},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":28.328,"events_lost":2033306,"totals_full":false},"queue worker 0x20":{"events_full_after_s":94.147,"events_lost":460849,"totals_full":false}},"trace":"aps5-trace-3e83ee94776d-824611.json.gz"}
  • frame p50 1621.1 → 704.9 ms, p90 2233.2 → 877.5 ms, GPU busy 816.4 → 335.1 ms per present.
  • Attribution from the first 25 s of each trace, the span both traces recorded fully: Resources.Complete on queue worker 0x20 (the copied-writer write-back) took 7.4 s on base and 3.5 s on the change. Recorder.Sync on that worker took 1.6 s and 0.8 s.

Pair 2: in-world scene (two characters at a ruined archway) in both windows. Base, then change:

{"commit":"27db71c581b4","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"1500:240"},"gpu":"AMD Radeon RX 6650 XT (RADV NAVI23)","driver":"radv Mesa 26.2.4-arch1.1","os":"Linux 7.2.9-arch1-1","cpu":"AMD Ryzen 9 7950X 16-Core Processor","cpu_threads":32,"memory_mib":127901,"title_id":"PPSA25636","title":"Echoes of the End","content_version":"01.012.000","resolution":"3840x2160","window":{"spec":"seconds 1500:240","first_present":1054,"ms":240025.444},"presents":220,"frame_ms":{"p50":1084.471,"p90":1276.934,"p99":1413.076,"max":1538.887},"gpu_busy_ms_per_present":563.648,"worker_busy_pct":{"queue worker 0x0":99.116,"queue worker 0x20":97.803},"top_phases_ms_per_frame":{"Driver.Packet":2123.795,"Driver.Packet.dispatch":1102.659,"Driver.Dispatch":997.477,"Driver.Packet.memory_wait":748.864,"Vulkan.Dispatch":540.299,"Vulkan.Dispatch.shader_resources":527.130,"Resources.Complete":476.271,"GpuMutex.wait.indirect":155.197},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":39.845,"events_lost":1294831,"totals_full":false},"queue worker 0x20":{"events_full_after_s":127.742,"events_lost":221256,"totals_full":false}},"trace":"aps5-trace-27db71c581b4-1422189.json.gz"}
{"commit":"3e83ee94776d","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"1500:240"},"gpu":"AMD Radeon RX 6650 XT (RADV NAVI23)","driver":"radv Mesa 26.2.4-arch1.1","os":"Linux 7.2.9-arch1-1","cpu":"AMD Ryzen 9 7950X 16-Core Processor","cpu_threads":32,"memory_mib":127901,"title_id":"PPSA25636","title":"Echoes of the End","content_version":"01.012.000","resolution":"3840x2160","window":{"spec":"seconds 1500:240","first_present":1305,"ms":240957.123},"presents":239,"frame_ms":{"p50":1018.347,"p90":1212.909,"p99":1379.782,"max":1666.358},"gpu_busy_ms_per_present":515.481,"worker_busy_pct":{"queue worker 0x0":99.162,"queue worker 0x20":97.691},"top_phases_ms_per_frame":{"Driver.Packet":1961.533,"Driver.Packet.dispatch":1018.941,"Driver.Dispatch":918.054,"Driver.Packet.memory_wait":701.249,"Vulkan.Dispatch":484.835,"Vulkan.Dispatch.shader_resources":472.343,"Resources.Complete":425.887,"Graphics.Draw":145.052},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":40.275,"events_lost":1403009,"totals_full":false},"queue worker 0x20":{"events_full_after_s":125.406,"events_lost":258029,"totals_full":false}},"trace":"aps5-trace-3e83ee94776d-787378.json.gz"}
  • frame p50 1084.5 vs 1018.3 ms, p90 1276.9 vs 1212.9 ms.
  • I don't read that as a difference with one run per side. The write-back this change shortens is mostly loading-phase work.
  • The change run reached this scene between 1300 s and 1440 s; the base run had reached it by 1300 s.

Caveats:

  • One run per side per scene. Pair 1 shows a drop in the loading scene; it doesn't establish a size for it.
  • All four traces filled their per-thread event buffers during the window (trace_dropped: true; the digests list the times). The frame times come from presents and are complete; only the per-phase totals are partial.

An earlier pair in this description (900 s runs, APS5_TRACE_SECONDS=600:240) landed on different scenes, so it is removed and is not evidence.

tools/trace_digest.py output for the four traces:

Pair 1 base (27db71c581b4)

seconds 1500:240 from present 803: 125 presents, frame p50 1621.1 / p90 2233.2 / p99 8046.2 / max 8428.7 ms, GPU busy 816.4 ms per present
AMD Radeon RX 6650 XT (RADV NAVI23), radv Mesa 26.2.4-arch1.1, Linux 7.2.9-arch1-1, 3840x2160, commit 27db71c581b4, dropped: yes

{"commit":"27db71c581b4","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"1500:240"},"gpu":"AMD Radeon RX 6650 XT (RADV NAVI23)","driver":"radv Mesa 26.2.4-arch1.1","os":"Linux 7.2.9-arch1-1","cpu":"AMD Ryzen 9 7950X 16-Core Processor","cpu_threads":32,"memory_mib":127901,"title_id":"PPSA25636","title":"Echoes of the End","content_version":"01.012.000","resolution":"3840x2160","window":{"spec":"seconds 1500:240","first_present":803,"ms":240007.014},"presents":125,"frame_ms":{"p50":1621.105,"p90":2233.242,"p99":8046.195,"max":8428.698},"gpu_busy_ms_per_present":816.406,"worker_busy_pct":{"queue worker 0x0":99.503,"queue worker 0x20":98.191},"top_phases_ms_per_frame":{"Driver.Packet":3760.159,"Driver.Packet.dispatch":2070.479,"Driver.Dispatch":1796.986,"Driver.Packet.memory_wait":1211.455,"Vulkan.Dispatch":1111.888,"Resources.Complete":1068.127,"Vulkan.Dispatch.shader_resources":1033.539,"Graphics.Draw":286.697},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":64.295,"events_lost":646426,"totals_full":false},"queue worker 0x20":{"events_full_after_s":217.5,"events_lost":27431,"totals_full":false}},"trace":"aps5-trace-27db71c581b4-196709.json.gz"}

Slowest frames (largest slices inside each, inclusive)

  • present 891: 8428.7 ms at 158334 ms: Driver.Packet (queue worker 0x20) 8360.6, Driver.Packet.dispatch (queue worker 0x20) 5757.9, Driver.Dispatch (queue worker 0x20) 5523.9, Vulkan.Dispatch (queue worker 0x20) 5329.4
  • present 804: 8046.2 ms at 1538 ms: Driver.Packet (queue worker 0x20) 8000.2, Driver.Packet (queue worker 0x0) 7996.0, Driver.Packet.dispatch (queue worker 0x20) 5418.2, Driver.Dispatch (queue worker 0x20) 5363.3
  • present 871: 7934.5 ms at 115262 ms: Driver.Packet (queue worker 0x20) 7864.6, Driver.Packet.dispatch (queue worker 0x20) 6465.5, Driver.Dispatch (queue worker 0x20) 6382.0, Vulkan.Dispatch (queue worker 0x20) 6218.2
  • present 880: 7158.8 ms at 135237 ms: Driver.Packet (queue worker 0x20) 7103.9, Driver.Packet.dispatch (queue worker 0x20) 4626.4, Driver.Dispatch (queue worker 0x20) 4602.4, Vulkan.Dispatch (queue worker 0x20) 4348.6
  • present 840: 5908.3 ms at 65391 ms: Driver.Packet (queue worker 0x20) 5859.2, Driver.Packet.dispatch (queue worker 0x20) 3127.9, Driver.Dispatch (queue worker 0x20) 3056.1, Vulkan.Dispatch (queue worker 0x20) 2725.0

Threads (ms per frame)

thread busy wait untraced in slices < 20 us largest
shader prepare 0.4 (0 %) 0.0 (0 %) 1919.6 0.0 Shader.PrepareArtifact 0.4
shader prepare 0.1 (0 %) 0.0 (0 %) 1919.9 0.0 Shader.PrepareArtifact 0.1
shader prepare 0.4 (0 %) 0.0 (0 %) 1919.6 0.0 Shader.PrepareArtifact 0.4
shader prepare 0.2 (0 %) 0.0 (0 %) 1919.8 0.0 Shader.PrepareArtifact 0.2
shader prepare 0.1 (0 %) 0.0 (0 %) 1919.9 0.0 Shader.PrepareArtifact 0.1
shader prepare 0.2 (0 %) 0.0 (0 %) 1919.9 0.0 Shader.PrepareArtifact 0.2
shader prepare 0.4 (0 %) 0.0 (0 %) 1919.6 0.0 Shader.PrepareArtifact 0.4
shader prepare 0.1 (0 %) 0.0 (0 %) 1919.9 0.0 Shader.PrepareArtifact 0.1
shader prepare 0.3 (0 %) 0.0 (0 %) 1919.8 0.0 Shader.PrepareArtifact 0.3
shader prepare 0.6 (0 %) 0.0 (0 %) 1919.5 0.0 Shader.PrepareArtifact 0.6
shader prepare 0.1 (0 %) 0.0 (0 %) 1919.9 0.0 Shader.PrepareArtifact 0.1
shader prepare 16.9 (1 %) 0.0 (0 %) 1903.2 0.0 Shader.PrepareArtifact 16.9
shader prepare 0.4 (0 %) 0.0 (0 %) 1919.7 0.0 Shader.PrepareArtifact 0.4
shader prepare 0.6 (0 %) 0.0 (0 %) 1919.4 0.0 Shader.PrepareArtifact 0.6
shader prepare 0.1 (0 %) 0.0 (0 %) 1920.0 0.0 Shader.PrepareArtifact 0.1
shader prepare 0.1 (0 %) 0.0 (0 %) 1920.0 0.0 Shader.PrepareArtifact 0.1
shader prepare 2.4 (0 %) 0.0 (0 %) 1917.6 0.0 Vulkan.GraphicsPipelinePrebuild 2.4
shader prepare 2.6 (0 %) 0.0 (0 %) 1917.5 0.0 Vulkan.GraphicsPipelinePrebuild 2.6
shader prepare 1.7 (0 %) 0.0 (0 %) 1918.3 0.0 Vulkan.GraphicsPipelinePrebuild 1.7
shader prepare 0.8 (0 %) 0.0 (0 %) 1919.3 0.0 Vulkan.GraphicsPipelinePrebuild 0.8
shader prepare 0.2 (0 %) 0.0 (0 %) 1919.9 0.0 Vulkan.GraphicsPipelinePrebuild 0.2
shader prepare 0.1 (0 %) 0.0 (0 %) 1920.0 0.0 Vulkan.GraphicsPipelinePrebuild 0.1
shader prepare 2.1 (0 %) 0.0 (0 %) 1918.0 0.0 Vulkan.GraphicsPipelinePrebuild 2.1
shader prepare 0.8 (0 %) 0.0 (0 %) 1919.2 0.0 Vulkan.GraphicsPipelinePrebuild 0.8
shader prepare 0.6 (0 %) 0.0 (0 %) 1919.5 0.0 Vulkan.GraphicsPipelinePrebuild 0.6
shader prepare 1.8 (0 %) 0.0 (0 %) 1918.3 0.0 Vulkan.GraphicsPipelinePrebuild 1.8
shader prepare 0.4 (0 %) 0.0 (0 %) 1919.7 0.0 Vulkan.GraphicsPipelinePrebuild 0.4
shader prepare 1.8 (0 %) 0.0 (0 %) 1918.3 0.0 Vulkan.GraphicsPipelinePrebuild 1.8
shader prepare 1.1 (0 %) 0.0 (0 %) 1918.9 0.0 Vulkan.GraphicsPipelinePrebuild 1.1
shader prepare 2.2 (0 %) 0.0 (0 %) 1917.8 0.0 Vulkan.GraphicsPipelinePrebuild 2.2
shader prepare 1.8 (0 %) 0.0 (0 %) 1918.3 0.0 Vulkan.GraphicsPipelinePrebuild 1.8
shader prepare 0.3 (0 %) 0.0 (0 %) 1919.8 0.0 Vulkan.GraphicsPipelinePrebuild 0.3
queue worker 0x0 372.4 (19 %) 137.7 (7 %) 1410.0 21.7 Driver.Packet 503.7, Driver.EndSubmission 4.2
draw committer 386.0 (20 %) 103.1 (5 %) 1430.9 6.1 Graphics.Draw 265.8, Recorder.Submit 100.4
queue worker 0x20 553.3 (29 %) 1183.7 (62 %) 183.0 19.9 Driver.Packet 1687.9, Driver.EndSubmission 17.3
presenter 1.4 (0 %) 2.5 (0 %) 1916.1 0.1 VideoOut.Flip 3.9, Driver.Present 0.0
title 0.7 (0 %) 0.0 (0 %) 1919.3 0.0 Shader.ResolveAbi 0.5, Shader.ResolveGraphicsStagesAbi 0.1
title 0.9 (0 %) 0.0 (0 %) 1919.2 0.0 Shader.ResolveGraphicsStagesAbi 0.5, Shader.ResolveAbi 0.4
title 1.4 (0 %) 0.0 (0 %) 1918.6 0.0 Shader.ResolveAbi 0.9, Shader.ResolveGraphicsStagesAbi 0.5
title 1.2 (0 %) 0.0 (0 %) 1918.9 0.0 Shader.ResolveAbi 0.7, Shader.ResolveGraphicsAbi 0.5
title 2.0 (0 %) 0.0 (0 %) 1918.0 0.0 Shader.ResolveAbi 1.6, Shader.ResolveGraphicsStagesAbi 0.4
title 1.5 (0 %) 0.0 (0 %) 1918.5 0.0 Shader.ResolveAbi 0.9, Shader.ResolveGraphicsStagesAbi 0.6
title 0.9 (0 %) 0.0 (0 %) 1919.1 0.0 Shader.ResolveAbi 0.7, Shader.ResolveGraphicsStagesAbi 0.1
title 0.0 (0 %) 0.0 (0 %) 1920.0 0.0 Shader.Register 0.0, Shader.ResolveGraphicsStagesAbi 0.0
title 1.6 (0 %) 0.0 (0 %) 1918.4 0.0 Shader.ResolveAbi 1.0, Shader.ResolveGraphicsStagesAbi 0.5
title 1.8 (0 %) 0.0 (0 %) 1918.2 0.0 Shader.ResolveAbi 1.7, Shader.ResolveGraphicsStagesAbi 0.1
title 1.1 (0 %) 0.0 (0 %) 1919.0 0.0 Shader.ResolveAbi 1.0, Shader.ResolveGraphicsStagesAbi 0.1
title 1.9 (0 %) 0.0 (0 %) 1918.1 0.0 Shader.ResolveAbi 1.5, Shader.ResolveGraphicsStagesAbi 0.5
title 1.7 (0 %) 0.0 (0 %) 1918.3 0.0 Shader.ResolveAbi 0.9, Shader.ResolveGraphicsStagesAbi 0.6
title 1.0 (0 %) 0.0 (0 %) 1919.1 0.0 Shader.ResolveAbi 0.8, Shader.ResolveGraphicsStagesAbi 0.1
title 0.0 (0 %) 0.0 (0 %) 1920.1 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1920.1 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1920.1 0.0 Shader.ResolveGraphicsStagesAbi 0.0
gpu 817.7 (43 %) 0.0 (0 %) 1102.4 0.0 batch 817.7

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 2191.6, Driver.Packet.dispatch 1045.8, Driver.Packet.memory_wait 991.9, Driver.Dispatch 930.1, batch 817.7, Resources.Complete 639.5, Vulkan.Dispatch 621.0, Vulkan.Dispatch.shader_resources 552.2, Graphics.Draw 269.5, Graphics.Draw.draw_and_resource_release 259.3

Longest waits

  • GpuMutex.wait.label (draw committer) 5338.8 ms at 115912 ms, present 871
  • Driver.Packet.memory_wait (queue worker 0x0) 4861.7 ms at 1687 ms, present 804
  • Driver.Packet.memory_wait (queue worker 0x20) 1520.9 ms at 6550 ms, present 804
  • Driver.Packet.memory_wait (queue worker 0x20) 1486.4 ms at 67904 ms, present 840
  • Driver.Packet.memory_wait (queue worker 0x20) 1317.8 ms at 163408 ms, present 891
  • Driver.Packet.memory_wait (queue worker 0x20) 1290.9 ms at 139408 ms, present 880
  • Driver.Packet.memory_wait (queue worker 0x20) 1214.3 ms at 171794 ms, present 893
  • Driver.Packet.memory_wait (queue worker 0x20) 1175.6 ms at 153312 ms, present 887
  • Driver.Packet.memory_wait (queue worker 0x20) 1075.1 ms at 143089 ms, present 881
  • Driver.Packet.memory_wait (queue worker 0x20) 1038.7 ms at 44088 ms, present 825

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

  • batch 28342: 96.8 ms at 229615 ms, present 922: GPU no ranges; recorded by Graphics.Draw (draw committer) 17.3, Graphics.Draw.draw_and_resource_release (draw committer) 17.3, Resources.Complete (draw committer) 16.8, Recorder.Submit (draw committer) 1.3
  • batch 6271: 92.3 ms at 49316 ms, present 828: GPU no ranges; recorded by Graphics.Draw (draw committer) 18.2, Graphics.Draw.draw_and_resource_release (draw committer) 18.2, Resources.Complete (draw committer) 17.8, Recorder.Submit (draw committer) 1.3
  • batch 23373: 92.1 ms at 189292 ms, present 901: GPU no ranges; recorded by Graphics.Draw (draw committer) 4.4, Graphics.Draw.draw_and_resource_release (draw committer) 4.4, Resources.Complete (draw committer) 3.8, Recorder.Submit (draw committer) 1.0
  • batch 26484: 92.1 ms at 214467 ms, present 914: GPU no ranges; recorded by Graphics.Draw (draw committer) 19.5, Graphics.Draw.draw_and_resource_release (draw committer) 19.5, Resources.Complete (draw committer) 17.9, Recorder.Submit (draw committer) 1.7
  • batch 25528: 92.0 ms at 207050 ms, present 910: GPU no ranges; recorded by Graphics.Draw (draw committer) 17.8, Graphics.Draw.draw_and_resource_release (draw committer) 17.8, Resources.Complete (draw committer) 17.2, Recorder.Submit (draw committer) 1.6
Pair 1 change (3e83ee94776d)

seconds 1500:240 from present 1452: 341 presents, frame p50 704.9 / p90 877.5 / p99 1018.8 / max 1114.2 ms, GPU busy 335.1 ms per present
AMD Radeon RX 6650 XT (RADV NAVI23), radv Mesa 26.2.4-arch1.1, Linux 7.2.9-arch1-1, 3840x2160, commit 3e83ee94776d, dropped: yes

{"commit":"3e83ee94776d","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"1500:240"},"gpu":"AMD Radeon RX 6650 XT (RADV NAVI23)","driver":"radv Mesa 26.2.4-arch1.1","os":"Linux 7.2.9-arch1-1","cpu":"AMD Ryzen 9 7950X 16-Core Processor","cpu_threads":32,"memory_mib":127901,"title_id":"PPSA25636","title":"Echoes of the End","content_version":"01.012.000","resolution":"3840x2160","window":{"spec":"seconds 1500:240","first_present":1452,"ms":240019.614},"presents":341,"frame_ms":{"p50":704.873,"p90":877.547,"p99":1018.779,"max":1114.243},"gpu_busy_ms_per_present":335.082,"worker_busy_pct":{"queue worker 0x0":98.756,"queue worker 0x20":97.053},"top_phases_ms_per_frame":{"Driver.Packet":1357.357,"Driver.Packet.dispatch":671.973,"Driver.Dispatch":595.953,"Driver.Packet.memory_wait":456.245,"Vulkan.Dispatch":245.115,"Vulkan.Dispatch.shader_resources":233.316,"Resources.Complete":188.105,"Graphics.Draw":126.533},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":28.328,"events_lost":2033306,"totals_full":false},"queue worker 0x20":{"events_full_after_s":94.147,"events_lost":460849,"totals_full":false}},"trace":"aps5-trace-3e83ee94776d-824611.json.gz"}

Slowest frames (largest slices inside each, inclusive)

  • present 1537: 1114.2 ms at 64067 ms: Driver.Packet (queue worker 0x20) 1072.2, Driver.Packet.memory_wait (queue worker 0x20) 771.6, batch (gpu) 448.3, Graphics.Draw (draw committer) 346.5
  • present 1536: 1042.5 ms at 63025 ms: Driver.Packet (queue worker 0x20) 1003.4, Driver.Packet.memory_wait (queue worker 0x20) 606.2, batch (gpu) 434.7, Driver.Packet.dispatch (queue worker 0x20) 359.5
  • present 1582: 1040.9 ms at 102636 ms: batch (gpu) 445.1, Graphics.Draw (draw committer) 292.0, Graphics.Draw.draw_and_resource_release (draw committer) 286.1, Resources.Complete (draw committer) 177.8
  • present 1516: 1018.8 ms at 47246 ms: Driver.Packet (queue worker 0x20) 981.9, Driver.Packet.memory_wait (queue worker 0x20) 639.7, batch (gpu) 419.8, Driver.Packet.dispatch (queue worker 0x20) 302.5
  • present 1539: 995.8 ms at 66117 ms: Driver.Packet (queue worker 0x20) 951.9, Driver.Packet.memory_wait (queue worker 0x20) 653.9, batch (gpu) 429.6, Driver.Packet.dispatch (queue worker 0x20) 278.7

Threads (ms per frame)

thread busy wait untraced in slices < 20 us largest
queue worker 0x0 61.7 (9 %) 19.9 (3 %) 622.4 8.0 Driver.Packet 79.6, Driver.EndSubmission 1.1
draw committer 204.6 (29 %) 13.6 (2 %) 485.7 6.1 Graphics.Draw 118.1, Recorder.Submit 70.5
queue worker 0x20 78.6 (11 %) 196.4 (28 %) 428.9 7.6 Driver.Packet 262.9, Driver.EndSubmission 4.6
presenter 1.2 (0 %) 0.4 (0 %) 702.3 0.1 VideoOut.Flip 1.6, Driver.Present 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveAbi 0.0, Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveGraphicsAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 703.9 0.0 Shader.ResolveGraphicsStagesAbi 0.0
gpu 335.0 (48 %) 0.0 (0 %) 368.8 0.0 batch 335.0

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 342.5, batch 335.0, Driver.Packet.memory_wait 168.9, Driver.Packet.dispatch 142.4, Driver.Dispatch 130.2, Graphics.Draw 119.0, Graphics.Draw.draw_and_resource_release 114.0, Recorder.Submit 88.1, Resources.Complete 70.1, Vulkan.Dispatch 61.7

Longest waits

  • Driver.Packet.memory_wait (queue worker 0x20) 563.0 ms at 64595 ms, present 1537
  • Driver.Packet.memory_wait (queue worker 0x20) 528.3 ms at 47712 ms, present 1516
  • Driver.Packet.memory_wait (queue worker 0x20) 518.2 ms at 5077 ms, present 1458
  • Driver.Packet.memory_wait (queue worker 0x20) 483.0 ms at 77579 ms, present 1552
  • Driver.Packet.memory_wait (queue worker 0x20) 474.4 ms at 63573 ms, present 1536
  • Driver.Packet.memory_wait (queue worker 0x20) 468.9 ms at 60112 ms, present 1532
  • Driver.Packet.memory_wait (queue worker 0x20) 467.4 ms at 67594 ms, present 1540
  • Driver.Packet.memory_wait (queue worker 0x20) 462.3 ms at 93349 ms, present 1570
  • Driver.Packet.memory_wait (queue worker 0x20) 461.6 ms at 89623 ms, present 1566
  • Driver.Packet.memory_wait (queue worker 0x20) 459.2 ms at 72564 ms, present 1546

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

  • batch 1533: 63.0 ms at 6130 ms, present 1459: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 87.1, Driver.Packet.dispatch (queue worker 0x0) 67.2, Driver.Dispatch (queue worker 0x0) 66.7, Vulkan.Dispatch (queue worker 0x0) 66.3
  • batch 24304: 61.9 ms at 100732 ms, present 1579: GPU no ranges; recorded by no CPU slices
  • batch 47713: 58.2 ms at 190825 ms, present 1705: GPU no ranges; recorded by no CPU slices
  • batch 31496: 58.0 ms at 129247 ms, present 1617: GPU no ranges; recorded by no CPU slices
  • batch 54226: 55.5 ms at 211510 ms, present 1742: GPU no ranges; recorded by no CPU slices
Pair 2 base (27db71c581b4)

seconds 1500:240 from present 1054: 220 presents, frame p50 1084.5 / p90 1276.9 / p99 1413.1 / max 1538.9 ms, GPU busy 563.6 ms per present
AMD Radeon RX 6650 XT (RADV NAVI23), radv Mesa 26.2.4-arch1.1, Linux 7.2.9-arch1-1, 3840x2160, commit 27db71c581b4, dropped: yes

{"commit":"27db71c581b4","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"1500:240"},"gpu":"AMD Radeon RX 6650 XT (RADV NAVI23)","driver":"radv Mesa 26.2.4-arch1.1","os":"Linux 7.2.9-arch1-1","cpu":"AMD Ryzen 9 7950X 16-Core Processor","cpu_threads":32,"memory_mib":127901,"title_id":"PPSA25636","title":"Echoes of the End","content_version":"01.012.000","resolution":"3840x2160","window":{"spec":"seconds 1500:240","first_present":1054,"ms":240025.444},"presents":220,"frame_ms":{"p50":1084.471,"p90":1276.934,"p99":1413.076,"max":1538.887},"gpu_busy_ms_per_present":563.648,"worker_busy_pct":{"queue worker 0x0":99.116,"queue worker 0x20":97.803},"top_phases_ms_per_frame":{"Driver.Packet":2123.795,"Driver.Packet.dispatch":1102.659,"Driver.Dispatch":997.477,"Driver.Packet.memory_wait":748.864,"Vulkan.Dispatch":540.299,"Vulkan.Dispatch.shader_resources":527.13,"Resources.Complete":476.271,"GpuMutex.wait.indirect":155.197},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":39.845,"events_lost":1294831,"totals_full":false},"queue worker 0x20":{"events_full_after_s":127.742,"events_lost":221256,"totals_full":false}},"trace":"aps5-trace-27db71c581b4-1422189.json.gz"}

Slowest frames (largest slices inside each, inclusive)

  • present 1188: 1538.9 ms at 143780 ms: batch (gpu) 699.6, Graphics.Draw (draw committer) 261.7, Graphics.Draw.draw_and_resource_release (draw committer) 255.5, Resources.Complete (draw committer) 153.5
  • present 1075: 1467.7 ms at 23008 ms: Driver.Packet (queue worker 0x0) 1444.3, Driver.Packet (queue worker 0x20) 1431.0, Driver.Packet.dispatch (queue worker 0x0) 1163.0, Driver.Packet.memory_wait (queue worker 0x20) 1030.0
  • present 1131: 1413.1 ms at 79436 ms: Driver.Packet (queue worker 0x20) 1370.3, Driver.Packet.memory_wait (queue worker 0x20) 835.1, batch (gpu) 647.8, Driver.Packet.dispatch (queue worker 0x20) 391.8
  • present 1217: 1405.0 ms at 176843 ms: batch (gpu) 633.7, Graphics.Draw (draw committer) 282.5, Graphics.Draw.draw_and_resource_release (draw committer) 266.7, Resources.Complete (draw committer) 183.9
  • present 1211: 1405.0 ms at 169563 ms: batch (gpu) 641.1, Graphics.Draw (draw committer) 246.8, Graphics.Draw.draw_and_resource_release (draw committer) 241.0, Resources.Complete (draw committer) 139.4

Threads (ms per frame)

thread busy wait untraced in slices < 20 us largest
shader prepare 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Vulkan.GraphicsPipelinePrebuild 0.0
queue worker 0x0 142.6 (13 %) 36.1 (3 %) 912.3 12.6 Driver.Packet 176.1, Driver.EndSubmission 1.4
draw committer 237.7 (22 %) 15.9 (1 %) 837.4 6.2 Graphics.Draw 136.5, Recorder.Submit 81.0
queue worker 0x20 124.1 (11 %) 454.9 (42 %) 512.1 12.1 Driver.Packet 557.7, Driver.EndSubmission 8.3
presenter 1.2 (0 %) 0.4 (0 %) 1089.3 0.1 VideoOut.Flip 1.7, Driver.Present 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveGraphicsAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveGraphicsAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveGraphicsAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1091.0 0.0 Shader.ResolveGraphicsStagesAbi 0.0
gpu 563.6 (52 %) 0.0 (0 %) 527.5 0.0 batch 563.6

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 733.8, batch 563.6, Driver.Packet.memory_wait 392.9, Driver.Packet.dispatch 287.7, Driver.Dispatch 253.6, Resources.Complete 152.3, Graphics.Draw 139.3, Graphics.Draw.draw_and_resource_release 132.6, Vulkan.Dispatch 130.4, Vulkan.Dispatch.shader_resources 127.2

Longest waits

  • Driver.Packet.memory_wait (queue worker 0x20) 860.8 ms at 23593 ms, present 1075
  • Driver.Packet.memory_wait (queue worker 0x20) 803.9 ms at 93357 ms, present 1143
  • Driver.Packet.memory_wait (queue worker 0x20) 787.8 ms at 7836 ms, present 1060
  • Driver.Packet.memory_wait (queue worker 0x20) 763.5 ms at 86557 ms, present 1137
  • Driver.Packet.memory_wait (queue worker 0x20) 762.1 ms at 117553 ms, present 1165
  • Driver.Packet.memory_wait (queue worker 0x20) 752.7 ms at 26194 ms, present 1077
  • Driver.Packet.memory_wait (queue worker 0x20) 752.0 ms at 68716 ms, present 1121
  • Driver.Packet.memory_wait (queue worker 0x20) 751.3 ms at 29724 ms, present 1080
  • Driver.Packet.memory_wait (queue worker 0x20) 749.3 ms at 60117 ms, present 1113
  • Driver.Packet.memory_wait (queue worker 0x20) 736.6 ms at 31050 ms, present 1081

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

  • batch 20717: 92.8 ms at 104520 ms, present 1153: GPU no ranges; recorded by Graphics.Draw (draw committer) 1.8, Graphics.Draw.draw_and_resource_release (draw committer) 1.8, Resources.Complete (draw committer) 1.0, Recorder.Submit (draw committer) 1.0
  • batch 20082: 89.9 ms at 101161 ms, present 1150: GPU no ranges; recorded by no CPU slices
  • batch 16749: 89.4 ms at 83585 ms, present 1134: GPU no ranges; recorded by Graphics.Draw (draw committer) 1.3, Graphics.Draw.draw_and_resource_release (draw committer) 1.3, Recorder.Submit (draw committer) 1.1, Resources.Complete (draw committer) 0.8
  • batch 22993: 87.2 ms at 116465 ms, present 1164: GPU no ranges; recorded by Graphics.Draw (draw committer) 1.3, Graphics.Draw.draw_and_resource_release (draw committer) 1.3, Recorder.Submit (draw committer) 0.9, Resources.Complete (draw committer) 0.8
  • batch 11047: 84.3 ms at 54026 ms, present 1107: GPU no ranges; recorded by Graphics.Draw (draw committer) 1.8, Graphics.Draw.draw_and_resource_release (draw committer) 1.8, Resources.Complete (draw committer) 1.3, Recorder.Submit (draw committer) 1.2
Pair 2 change (3e83ee94776d)

seconds 1500:240 from present 1305: 239 presents, frame p50 1018.3 / p90 1212.9 / p99 1379.8 / max 1666.4 ms, GPU busy 515.5 ms per present
AMD Radeon RX 6650 XT (RADV NAVI23), radv Mesa 26.2.4-arch1.1, Linux 7.2.9-arch1-1, 3840x2160, commit 3e83ee94776d, dropped: yes

{"commit":"3e83ee94776d","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"1500:240"},"gpu":"AMD Radeon RX 6650 XT (RADV NAVI23)","driver":"radv Mesa 26.2.4-arch1.1","os":"Linux 7.2.9-arch1-1","cpu":"AMD Ryzen 9 7950X 16-Core Processor","cpu_threads":32,"memory_mib":127901,"title_id":"PPSA25636","title":"Echoes of the End","content_version":"01.012.000","resolution":"3840x2160","window":{"spec":"seconds 1500:240","first_present":1305,"ms":240957.123},"presents":239,"frame_ms":{"p50":1018.347,"p90":1212.909,"p99":1379.782,"max":1666.358},"gpu_busy_ms_per_present":515.481,"worker_busy_pct":{"queue worker 0x0":99.162,"queue worker 0x20":97.691},"top_phases_ms_per_frame":{"Driver.Packet":1961.533,"Driver.Packet.dispatch":1018.941,"Driver.Dispatch":918.054,"Driver.Packet.memory_wait":701.249,"Vulkan.Dispatch":484.835,"Vulkan.Dispatch.shader_resources":472.343,"Resources.Complete":425.887,"Graphics.Draw":145.052},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":40.275,"events_lost":1403009,"totals_full":false},"queue worker 0x20":{"events_full_after_s":125.406,"events_lost":258029,"totals_full":false}},"trace":"aps5-trace-3e83ee94776d-787378.json.gz"}

Slowest frames (largest slices inside each, inclusive)

  • present 1400: 1666.4 ms at 100003 ms: Driver.Packet (queue worker 0x20) 1609.5, Driver.Packet.memory_wait (queue worker 0x20) 1197.9, batch (gpu) 791.3, Driver.Packet.dispatch (queue worker 0x20) 392.3
  • present 1516: 1411.0 ms at 210954 ms: batch (gpu) 625.2, Graphics.Draw (draw committer) 212.6, Graphics.Draw.draw_and_resource_release (draw committer) 199.8, Resources.Complete (draw committer) 114.9
  • present 1526: 1379.8 ms at 222276 ms: batch (gpu) 625.2, Graphics.Draw (draw committer) 195.8, Graphics.Draw.draw_and_resource_release (draw committer) 182.9, Resources.Complete (draw committer) 101.6
  • present 1504: 1378.1 ms at 197928 ms: batch (gpu) 602.2, Graphics.Draw (draw committer) 248.9, Graphics.Draw.draw_and_resource_release (draw committer) 231.7, Resources.Complete (draw committer) 137.7
  • present 1329: 1376.8 ms at 26094 ms: Driver.Packet (queue worker 0x0) 1346.1, Driver.Packet (queue worker 0x20) 1329.8, Driver.Packet.dispatch (queue worker 0x0) 1107.1, Driver.Packet.memory_wait (queue worker 0x20) 934.6

Threads (ms per frame)

thread busy wait untraced in slices < 20 us largest
queue worker 0x0 133.0 (13 %) 33.3 (3 %) 841.9 11.3 Driver.Packet 164.0, Driver.EndSubmission 1.3
draw committer 222.2 (22 %) 13.3 (1 %) 772.7 6.2 Graphics.Draw 131.0, Recorder.Submit 69.2
queue worker 0x20 109.4 (11 %) 413.8 (41 %) 485.0 11.0 Driver.Packet 504.6, Driver.EndSubmission 7.0
presenter 1.1 (0 %) 0.4 (0 %) 1006.6 0.1 VideoOut.Flip 1.5, Driver.Present 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
title 0.0 (0 %) 0.0 (0 %) 1008.2 0.0 Shader.ResolveGraphicsStagesAbi 0.0
gpu 514.3 (51 %) 0.0 (0 %) 493.9 0.0 batch 514.3

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 668.7, batch 514.3, Driver.Packet.memory_wait 361.7, Driver.Packet.dispatch 261.5, Driver.Dispatch 238.7, Resources.Complete 150.2, Graphics.Draw 133.5, Graphics.Draw.draw_and_resource_release 127.1, Vulkan.Dispatch 127.1, Vulkan.Dispatch.shader_resources 124.1

Longest waits

  • Driver.Packet.memory_wait (queue worker 0x20) 1038.3 ms at 100613 ms, present 1400
  • Driver.Packet.memory_wait (queue worker 0x20) 841.4 ms at 1625 ms, present 1306
  • Driver.Packet.memory_wait (queue worker 0x20) 792.3 ms at 26652 ms, present 1329
  • Driver.Packet.memory_wait (queue worker 0x20) 788.0 ms at 4300 ms, present 1308
  • Driver.Packet.memory_wait (queue worker 0x20) 759.9 ms at 98126 ms, present 1398
  • Driver.Packet.memory_wait (queue worker 0x20) 754.4 ms at 38696 ms, present 1340
  • Driver.Packet.memory_wait (queue worker 0x20) 746.3 ms at 103174 ms, present 1402
  • Driver.Packet.memory_wait (queue worker 0x20) 728.5 ms at 17755 ms, present 1321
  • Driver.Packet.memory_wait (queue worker 0x20) 724.9 ms at 67461 ms, present 1368
  • Driver.Packet.memory_wait (queue worker 0x20) 709.3 ms at 32374 ms, present 1334

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

  • batch 1837: 82.9 ms at 9670 ms, present 1313: GPU no ranges; recorded by Graphics.Draw (draw committer) 1.2, Graphics.Draw.draw_and_resource_release (draw committer) 1.2, Recorder.Submit (draw committer) 1.0, Resources.Complete (draw committer) 0.7
  • batch 19740: 82.0 ms at 101010 ms, present 1400: GPU no ranges; recorded by Graphics.Draw (draw committer) 1.9, Graphics.Draw.draw_and_resource_release (draw committer) 1.9, Resources.Complete (draw committer) 1.4, Recorder.Submit (draw committer) 1.0
  • batch 35355: 81.1 ms at 176278 ms, present 1477: GPU no ranges; recorded by Graphics.Draw (draw committer) 1.1, Graphics.Draw.draw_and_resource_release (draw committer) 1.1, Recorder.Submit (draw committer) 0.7, Resources.Complete (draw committer) 0.7
  • batch 25257: 80.5 ms at 129168 ms, present 1427: GPU no ranges; recorded by Graphics.Draw (draw committer) 2.2, Graphics.Draw.draw_and_resource_release (draw committer) 2.2, Recorder.Submit (draw committer) 1.0, Resources.Complete (draw committer) 0.9
  • batch 44894: 80.4 ms at 221753 ms, present 1525: GPU no ranges; recorded by Graphics.Draw (draw committer) 1.2, Graphics.Draw.draw_and_resource_release (draw committer) 1.2, Recorder.Submit (draw committer) 0.8, Resources.Complete (draw committer) 0.7

Attached in the release https://github.com/JJenkx/AnyPS5/releases/tag/heap-writeback-readback-samescene-20261029: the four traces, the four summary lines, their digests, the 1440 s and 1795 s screenshots and the seed manifest. The earlier release https://github.com/JJenkx/AnyPS5/releases/tag/heap-writeback-readback-evidence holds the focused-test VVL logs.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none claimed
  • GPUs tested: AMD Radeon RX 6650 XT, RADV Mesa 26.2.4, Arch Linux 7.2.9-arch1-1
  • AI-assisted: yes

🤖 Generated with Claude Code

…k buffer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added perf Performance change perf numbers missing perf pull request without a before and after summary line labels Oct 29, 2026
@github-actions

Copy link
Copy Markdown

This pull request is labeled perf, so it needs the summary line of a capture on main and one with the change, from the same title, scene and machine, as the Performance report form asks. Please paste both aps5-summary.jsonl lines, attach the two traces and add the output of tools/trace_digest.py.

@XairRrR XairRrR left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — changes requested

The code change is correct, merges cleanly, builds, and the focused resident heap mirror tests pass with validation on. The block is on evidence only: this is a perf PR, and its description does not yet carry the same-scene main/change summary lines that CONTRIBUTING.md requires.

What I ran

  • Head: c12a20f45da55cd45da092e937ee196416cc2f72
  • main: ba5b5330b37724bca4b99f30faab504653635a04 (the head's parent, so the PR is on current main)
  • Machine: Arch Linux 7.2.11-zen, GCC 16.2.1, Intel Core i9-14900KF, NVIDIA GeForce RTX 4070 Ti SUPER (driver 615.78.08). Release build, Ninja, BUILD_TESTING=ON, in a clean worktree with main merged with the PR head (--no-ff).
git diff --check origin/main...refs/review/12827
python3 tools/check_conventions.py --base origin/main
cmake --build build -j10
VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1 VK_LOADER_DEBUG=error,warn,layer \
  ctest --test-dir build -R 'agc_driver_resident_heap_mirror|heap_mirror|resident' -V -j4 --output-on-failure
# the same build and ctest command on origin/main
# coverage check: gdb breakpoints with backtraces on ResidentBuffer::RecordReadback, ::ReadDevice and ::Store
gdb -batch -x count.gdb build/tests/agc_driver_resident_heap_mirror_tests
# perf label gate on the PR description
python3 -c "import pr_labels; print(pr_labels.has_numbers(body))"

Results

Check Result
Merge into main Clean
git diff --check Clean
check_conventions.py All conventions checks passed.
Build (merged tree and main) OK
Focused tests, merged tree 14/14 passed, 0 VUID- lines
Focused tests, main 14/14 passed, 0 VUID- lines
Validation layer loaded Yes: Inserted device layer "VK_LAYER_KHRONOS_validation" in 6 Vulkan tests in each log; no was not found but was requested by env var VK_INSTANCE_LAYERS
Changed branch exercised Yes: agc_driver_resident_heap_mirror_tests records 16 readbacks (RecordCopyBacks) and calls ResidentBuffer::Store 8 times from GuestBufferMemory::WriteBack; ReadDevice is never called
Test durations All under 1 s
tools/pr_labels.py has_numbers on the description False
ctest, merged tree
 1/14 Test #1304: agc_driver_partially_resident_snapshot ..........   Passed    0.40 sec
 2/14 Test #1548: agc_driver_partially_resident_pending_stores ....   Passed    0.14 sec
 3/14 Test #1583: agc_heap_mirror_unwatch .........................   Passed    0.01 sec
 4/14 Test #1584: agc_heap_mirror_unwatch_no_space_cache ..........   Passed    0.01 sec
 5/14 Test #1585: agc_sparse_heap_mirror_unwatch ..................   Passed    0.00 sec
 6/14 Test #1586: agc_sparse_heap_mirror_unwatch_no_space_cache ...   Passed    0.00 sec
 7/14 Test #1587: agc_sparse_heap_mirror_budget ...................   Passed    0.01 sec
 8/14 Test #1588: agc_heap_mirror_run_unwatch .....................   Passed    0.01 sec
 9/14 Test #1589: agc_heap_mirror_run_unwatch_no_space_cache ......   Passed    0.00 sec
10/14 Test #1590: agc_heap_mirror_unwatched_import ................   Passed    0.00 sec
11/14 Test  #829: agc_resident_present ............................   Passed    0.61 sec
12/14 Test #1257: agc_driver_linear_target_resident ...............   Passed    0.63 sec
13/14 Test #1368: agc_driver_resident_heap_mirror .................   Passed    0.74 sec
14/14 Test #1626: agc_driver_uint8x4_color_target_no_resident .....   Passed    0.37 sec
100% tests passed out of 14
1368: resident heap mirror tests passed
gdb call counts, agc_driver_resident_heap_mirror_tests (merged tree)
     16 HIT RecordReadback      (callers: ShaderResources::MarkGpuWrites -> GuestBufferMemory::RecordCopyBacks)
      8 HIT Store               (caller: GuestBufferMemory::WriteBack)
      0 HIT ReadDevice
resident heap mirror tests passed

Negative control

None. The PR adds or changes no tests, and the change is meant to store the same bytes as before, so no existing test should fail when it is reverted. The gdb run above shows that the existing agc_driver_resident_heap_mirror test does reach the changed recorded-readback branch and checks guest memory after it.

Findings

Blocking

  1. The perf evidence required by CONTRIBUTING.md is not in the description. CONTRIBUTING.md:71 requires a perf(...) PR to show the summary lines of main and of the change, from the same machine and scene. It keeps perf numbers missing on the PR until the description holds two valid summary lines with different commits. Two things are missing:

    • The four lines in the description are shortened with ..., so they are not JSON. tools/pr_labels.py has_numbers returns False on the current description, which is why the label is still set. The full lines are only in the external release.
    • By the description's own account, the base and change windows cover different scenes: logo and fade frames on the base side, the loading screen on the change side. So the lines cannot support or refute a speed change. "Reached the loading scene earlier in 2 of 2 runs" is suggestive, but it is not the same-scene comparison the rule asks for.

    Using main + #4449/#8089/#11873/#6867 as the base, because the title does not boot on main alone, is well explained. Whether that is an acceptable stand-in for main is for the maintainers to decide.

Non-blocking

None about the code. What I checked:

  • Lifetime. current now points into the readback Buffer mapping. That buffer is owned by heapReadbacks (and by recorder.Keep) from GuestBufferMemory.cpp:5876-5877 until heapReadbacks.clear() after the loop. Nothing in ResidentBuffer::Store, storeSparse or storeChanged modifies heapReadbacks, so the span stays valid across GuestBufferMemory.cpp:6291-6303.
  • Same bytes, same visibility. The old memcpy read the same mapping at the same point. ResidentBuffer::Store's GuestBytesZero check scans host pointers, so passing the mapping instead of a vector does not change its meaning.
  • Memory type. The readback Buffer uses the default HOST_VISIBLE | HOST_COHERENT flags, and Context::BufferMemoryType prefers a HOST_CACHED type for those flags. So the extra reads of the mapping (by Store, then by storeChanged/storeSparse) are cached reads wherever the driver exposes such a type. The description's ResidentBuffer::Store 42 → 78 ms is consistent with that.

Not verified

  • Any frame-time or per-write-back timing. I did not launch a title, and I did not measure the copy removal on this NVIDIA machine.
  • The ReadDevice fallback branch (no recorded readback). No focused test reaches it. Its behaviour is unchanged apart from moving the vector into read.
  • The private attribution counters in the description's table.

To get this to a good state

  1. Paste the full, unshortened aps5-summary.jsonl lines (one base, one change) into the description, so that the perf numbers missing gate can parse them.
  2. Provide a pair whose trace windows cover the same scene on both sides. For example, use a window that starts after both builds have reached the in-world loading screen, with runs long enough for the base to get there too. Attach the tools/trace_digest.py output for that pair as before.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 5021 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 6186 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 6197 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 6481 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 6918 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 7457 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 7514 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 7642 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 9012 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 9289 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 9508 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 9852 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 10193 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 10442 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 12157 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 13148 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 13486 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 13863 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 13920 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 13928 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 13929 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 15302 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
PR 15570 core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp

@TechLevelZero

TechLevelZero commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

✅ NVIDIA GPU test: passed

PR head c12a20f merged onto main b11c7bf · tested 2026-11-04 18:58 UTC · 30 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,767 passed · 6 failed (0 new, 3 also fail on main, 3 unstable on main) · 49 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,822 of 3,894 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (974 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.1 s
Unstable on main (3, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 3 of the last 10 main runs
agc_driver_submit_progress: failed in 3 of the last 10 main runs
agc_submit_progress: failed in 4 of the last 10 main runs
Failing on main too (3, not counted against this PR)
agc_driver_image_table
guest_mmap_executable
guest_socket_options_multicast
Validation IDs already present on main (13 IDs in 549 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (549 tests): agc_driver, agc_driver_address_draw_space_reuse, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, and 541 more
VUID-VkShaderModuleCreateInfo-pCode-08737 (8 tests): agc_driver_dpp16_operands, agc_driver_dpp8_pk_fmac, agc_driver_dx10_clamp, agc_driver_float16_fma_rounding, agc_driver_float16_unused_nan_payloads, agc_driver_packed_alu, agc_driver_packed_f16_denormal_modes, agc_driver_packed_float16_native
VUID-VkShaderModuleCreateInfo-pCode-08739 (8 tests): agc_driver_dpp16_operands, agc_driver_dpp8_pk_fmac, agc_driver_dx10_clamp, agc_driver_float16_fma_rounding, agc_driver_float16_unused_nan_payloads, agc_driver_packed_alu, agc_driver_packed_f16_denormal_modes, agc_driver_packed_float16_native
VUID-VkShaderModuleCreateInfo-pCode-08741 (8 tests): agc_driver_dpp16_operands, agc_driver_dpp8_pk_fmac, agc_driver_dx10_clamp, agc_driver_float16_fma_rounding, agc_driver_float16_unused_nan_payloads, agc_driver_packed_alu, agc_driver_packed_f16_denormal_modes, agc_driver_packed_float16_native
VUID-vkCmdCopyImage-srcImage-01548 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_srgb8_color_target
VUID-VkPipelineShaderStageCreateInfo-module-parameter (1 test): agc_driver_float16_unused_nan_payloads
VUID-VkPipelineShaderStageCreateInfo-pSpecializationInfo-06849 (1 test): agc_driver_float16_unused_nan_payloads
VUID-VkSamplerCreateInfo-addressModeU-01079 (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeU-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeV-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeW-parameter (1 test): agc_driver_image_gather_volume
Skipped (49)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_gcn_display_present: skipped, no display: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_deferred: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not av
Reproduce
git fetch origin c12a20f45da55cd45da092e937ee196416cc2f72 b11c7bf84f5d2f636ff60166cc3371bd071e24d5
git checkout b11c7bf84f5d2f636ff60166cc3371bd071e24d5 && git merge --no-ff c12a20f45da55cd45da092e937ee196416cc2f72
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@JJenkx

JJenkx commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

@XairRrR thanks for the check and the NVIDIA run. Agreed: the pair in the description lands on different scenes, so it is not a qualifying same-scene main/change report.

I'm rerunning it now. Echoes needs four open PRs to boot on main (#4449, #8089, #11873, #6867), so both sides use the same base: current main 899ae3d1ad3 plus those four. The change side adds this PR's commit to that base.

  • Runs: 1800 s, scripted pad input, warm seeded cache, APS5_TRACE_SECONDS=1500:240 (gameplay in earlier runs), no other APS5_TRACE_*/APS5_PROFILE_* switch, two interleaved runs per side.
  • Scene check: screenshots at 1500 s confirm the scene before any line is used.

I'll post the summary lines, traces and digests here. If the scenes still differ, I'll say so rather than claim a speed change.

AI-assisted: yes (Claude Code).

@adriadam10 adriadam10 self-assigned this Oct 29, 2026

@adriadam10 adriadam10 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code change itself checks out: the readback span in GuestBufferMemory.cpp:6295 stays alive in heapReadbacks until it is cleared after the write-back loop, Store/storeSparse/storeChanged only read it, and the size is checked against the write-back length. The NVIDIA CI run is clean (0 new failures, 0 new validation IDs).

What blocks it is evidence, as the perf label bot and @XairRrR's review say:

  • Paste the aps5-summary.jsonl line of main and of the change from the same title, scene and machine, plus the tools/trace_digest.py output. The pair in the description lands on different scenes (the description says so itself), so it does not qualify.
  • If the same-scene rerun you announced still can't be matched, drop the speed framing and state only the per-write-back attribution, which is already clearly labelled as such.

Happy to merge once the report is in and the review request is cleared.

@github-actions github-actions Bot added the agc libSceAgc* label Oct 29, 2026
@JJenkx

JJenkx commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

@XairRrR @adriadam10 the same-scene rerun is done. One of its two pairs is on the same scene in both windows and shows a frame-time drop. The other pair is also on matching scenes but shows no clear difference. Both are below.

Setup

Pair 1: in-game loading screen in both windows. Screenshots at 1440 s and 1795 s show the same "Loading..." screen (cathedral with the blue crystal) in both runs.

{"commit":"27db71c581b4","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"1500:240"},"gpu":"AMD Radeon RX 6650 XT (RADV NAVI23)","driver":"radv Mesa 26.2.4-arch1.1","os":"Linux 7.2.9-arch1-1","cpu":"AMD Ryzen 9 7950X 16-Core Processor","cpu_threads":32,"memory_mib":127901,"title_id":"PPSA25636","title":"Echoes of the End","content_version":"01.012.000","resolution":"3840x2160","window":{"spec":"seconds 1500:240","first_present":803,"ms":240007.014},"presents":125,"frame_ms":{"p50":1621.105,"p90":2233.242,"p99":8046.195,"max":8428.698},"gpu_busy_ms_per_present":816.406,"worker_busy_pct":{"queue worker 0x0":99.503,"queue worker 0x20":98.191},"top_phases_ms_per_frame":{"Driver.Packet":3760.159,"Driver.Packet.dispatch":2070.479,"Driver.Dispatch":1796.986,"Driver.Packet.memory_wait":1211.455,"Vulkan.Dispatch":1111.888,"Resources.Complete":1068.127,"Vulkan.Dispatch.shader_resources":1033.539,"Graphics.Draw":286.697},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":64.295,"events_lost":646426,"totals_full":false},"queue worker 0x20":{"events_full_after_s":217.500,"events_lost":27431,"totals_full":false}},"trace":"aps5-trace-27db71c581b4-196709.json.gz"}
{"commit":"3e83ee94776d","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"1500:240"},"gpu":"AMD Radeon RX 6650 XT (RADV NAVI23)","driver":"radv Mesa 26.2.4-arch1.1","os":"Linux 7.2.9-arch1-1","cpu":"AMD Ryzen 9 7950X 16-Core Processor","cpu_threads":32,"memory_mib":127901,"title_id":"PPSA25636","title":"Echoes of the End","content_version":"01.012.000","resolution":"3840x2160","window":{"spec":"seconds 1500:240","first_present":1452,"ms":240019.614},"presents":341,"frame_ms":{"p50":704.873,"p90":877.547,"p99":1018.779,"max":1114.243},"gpu_busy_ms_per_present":335.082,"worker_busy_pct":{"queue worker 0x0":98.756,"queue worker 0x20":97.053},"top_phases_ms_per_frame":{"Driver.Packet":1357.357,"Driver.Packet.dispatch":671.973,"Driver.Dispatch":595.953,"Driver.Packet.memory_wait":456.245,"Vulkan.Dispatch":245.115,"Vulkan.Dispatch.shader_resources":233.316,"Resources.Complete":188.105,"Graphics.Draw":126.533},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":28.328,"events_lost":2033306,"totals_full":false},"queue worker 0x20":{"events_full_after_s":94.147,"events_lost":460849,"totals_full":false}},"trace":"aps5-trace-3e83ee94776d-824611.json.gz"}
  • frame p50 1621.1 → 704.9 ms, p90 2233.2 → 877.5 ms, GPU busy 816.4 → 335.1 ms per present.
  • Attribution from the first 25 s of each trace, the span both traces recorded fully: Resources.Complete on queue worker 0x20 (the copied-writer write-back) took 7.4 s on base and 3.5 s on the change. Recorder.Sync on that worker took 1.6 s and 0.8 s.

Pair 2: in-world scene in both windows. At 1440 s and 1795 s both runs show the same in-world scene: two characters at a ruined archway.

{"commit":"27db71c581b4","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"1500:240"},"gpu":"AMD Radeon RX 6650 XT (RADV NAVI23)","driver":"radv Mesa 26.2.4-arch1.1","os":"Linux 7.2.9-arch1-1","cpu":"AMD Ryzen 9 7950X 16-Core Processor","cpu_threads":32,"memory_mib":127901,"title_id":"PPSA25636","title":"Echoes of the End","content_version":"01.012.000","resolution":"3840x2160","window":{"spec":"seconds 1500:240","first_present":1054,"ms":240025.444},"presents":220,"frame_ms":{"p50":1084.471,"p90":1276.934,"p99":1413.076,"max":1538.887},"gpu_busy_ms_per_present":563.648,"worker_busy_pct":{"queue worker 0x0":99.116,"queue worker 0x20":97.803},"top_phases_ms_per_frame":{"Driver.Packet":2123.795,"Driver.Packet.dispatch":1102.659,"Driver.Dispatch":997.477,"Driver.Packet.memory_wait":748.864,"Vulkan.Dispatch":540.299,"Vulkan.Dispatch.shader_resources":527.130,"Resources.Complete":476.271,"GpuMutex.wait.indirect":155.197},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":39.845,"events_lost":1294831,"totals_full":false},"queue worker 0x20":{"events_full_after_s":127.742,"events_lost":221256,"totals_full":false}},"trace":"aps5-trace-27db71c581b4-1422189.json.gz"}
{"commit":"3e83ee94776d","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_DIR":"<path>","APS5_TRACE_SECONDS":"1500:240"},"gpu":"AMD Radeon RX 6650 XT (RADV NAVI23)","driver":"radv Mesa 26.2.4-arch1.1","os":"Linux 7.2.9-arch1-1","cpu":"AMD Ryzen 9 7950X 16-Core Processor","cpu_threads":32,"memory_mib":127901,"title_id":"PPSA25636","title":"Echoes of the End","content_version":"01.012.000","resolution":"3840x2160","window":{"spec":"seconds 1500:240","first_present":1305,"ms":240957.123},"presents":239,"frame_ms":{"p50":1018.347,"p90":1212.909,"p99":1379.782,"max":1666.358},"gpu_busy_ms_per_present":515.481,"worker_busy_pct":{"queue worker 0x0":99.162,"queue worker 0x20":97.691},"top_phases_ms_per_frame":{"Driver.Packet":1961.533,"Driver.Packet.dispatch":1018.941,"Driver.Dispatch":918.054,"Driver.Packet.memory_wait":701.249,"Vulkan.Dispatch":484.835,"Vulkan.Dispatch.shader_resources":472.343,"Resources.Complete":425.887,"Graphics.Draw":145.052},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":40.275,"events_lost":1403009,"totals_full":false},"queue worker 0x20":{"events_full_after_s":125.406,"events_lost":258029,"totals_full":false}},"trace":"aps5-trace-3e83ee94776d-787378.json.gz"}
  • frame p50 1084.5 vs 1018.3 ms, p90 1276.9 vs 1212.9 ms.
  • I don't read that as a difference with one run per side. The write-back this PR shortens is mostly loading-phase work.
  • The change run reached this scene between 1300 s and 1440 s; the base run had reached it by 1300 s.

Caveats

  • One run per side per scene. Pair 1 shows a drop in the loading scene; it doesn't establish a size for it.
  • All four traces filled their per-thread event buffers during the window (trace_dropped: true; the digests list the times). The summary frame times come from presents and are complete; only the per-phase totals are partial.

Release https://github.com/JJenkx/AnyPS5/releases/tag/heap-writeback-readback-samescene-20261029 has the four aps5-summary.jsonl lines, the four traces, their tools/trace_digest.py outputs, the 1440 s and 1795 s screenshots and the seed manifest.

🤖 Generated with Claude Code

@github-actions github-actions Bot removed the perf numbers missing perf pull request without a before and after summary line label Oct 29, 2026
@JJenkx

JJenkx commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

@XairRrR @adriadam10 please re-check the evidence follow-up at #12827 (comment) and the current description against the two remaining evidence requests. The description now contains the full valid summary JSON lines and the matching-scene trace/digest pairs; the earlier different-scene pair is not presented as qualifying evidence.

The limitations remain explicit: the base contains the same four open prerequisites on both sides; one matched pair improves frame time and the second is inconclusive. This is not a claim of a repeatable plain-main or general FPS gain. Please assess whether that scoped report satisfies the request, or whether to retain only the per-write-back attribution framing you suggested. The code is unchanged; this follow-up requests re-review of the completed report rather than claiming new measurements. AI-assisted: yes.

@XairRrR XairRrR left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked — approved

The code is the same commit (c12a20f). The description now holds the full aps5-summary.jsonl lines of two same-scene base/change pairs, with trace digests, so the one blocking finding of my last review is resolved.

What changed since my last review

  • No new commits. delta.patch shows c12a20f45d = c12a20f45d.
  • The description was rewritten:
    • the earlier different-scene pair is gone and is marked as not evidence;
    • two pairs of full, unshortened summary lines from 1800 s runs (window 1500:240) were added, with scene screenshots at 1440 s and 1795 s;
    • the tools/trace_digest.py output for all four traces is now in the description.
  • Comments from the author on 2026-10-29 and 2026-11-02 ask for a re-check of that report.

What I ran

Head c12a20f45da55cd45da092e937ee196416cc2f72, main 096c45570f516de76a43bbe1b97781381c84f55f, merged tree deea710d7d1c33842a4cb6ce865061c3f49a9daa. Debian 13 VM (Xeon Gold 6136), GCC 14.2.0, lavapipe (Mesa 26.1.6). Release, Ninja, BUILD_TESTING=ON.

git diff --check 096c45570f516de76a43bbe1b97781381c84f55f...c12a20f45da55cd45da092e937ee196416cc2f72
python3 tools/check_conventions.py --base 096c45570f516de76a43bbe1b97781381c84f55f
# perf gate on the pair-1 base and change lines, copied verbatim from the description
python3 -I -c "import sys; sys.path.insert(0,'tools'); import pr_labels; print(pr_labels.has_numbers(sys.argv[1]+chr(10)+sys.argv[2]))" '<base line>' '<change line>'
VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1 VK_LOADER_DEBUG=error,warn,layer \
  ctest --test-dir build -j8 --test-load 32 -V -R "heap_mirror|resident"

The script also built the merged tree and ran the selected tests and the validation comparison against main.

Check Result
Merge into main Clean
git diff --check Clean
check_conventions.py All conventions checks passed.
Build and libs OK
Selected tests (2031 run) 0 new failures, 0 timeouts. 7 known failures also fail on main.
Validation vs main (757 tests) Layer loaded in 756 and missing in none; 0 tests with new VUID-
Focused heap_mirror|resident tests (16), with validation 16/16 passed; 0 VUID- lines; no was not found line; the layer was inserted in 6 tests; resident heap mirror tests passed printed, longest test 2.08 s
pr_labels.has_numbers on the pair-1 lines True

Previous findings

  • Blocking: the perf evidence required by CONTRIBUTING.md:73 was missing. Resolved.
    • The lines now parse. The four summary lines are full JSON inside fenced blocks. On the pair-1 base (27db71c581b4) and change (3e83ee94776d) lines, has_numbers returns True, so the perf numbers missing gate is satisfied.
    • Same scene on both sides. Pair 1 has the in-game "Loading..." screen in both windows, confirmed by screenshots at 1440 s and 1795 s (author-reported). Same title, machine, resolution, seed cache and window. Frame p50 goes from 1621.1 to 704.9 ms.
    • The second pair is reported honestly. Pair 2 (in-world) is called inconclusive, and the caveats state one run per side and that per-phase totals are partial (trace_dropped).
    • The screenshots and traces are in the author's release, which I did not open.
    • The base is main + #4449/#8089/#11873/#6867 on both sides. As I said before, the maintainers decide whether that stands in for main. The description is clear that this is not a bare-main result.

New findings

Blocking

None.

Non-blocking

None. I re-checked the span lifetime on the merged tree:

  • heapReadbacks is filled in RecordCopyBacks.
  • It is cleared only in releaseHeapReferences (GuestBufferMemory.cpp:6652-6656). That function is called from takeHeapReferences before the build and at the end of WriteBack (GuestBufferMemory.cpp:6781), after the loop.
  • So the span assigned at GuestBufferMemory.cpp:6720 stays valid through Store/storeSparse/storeChanged (GuestBufferMemory.cpp:6726-6728).

This agrees with adriadam10's review.

Not verified

  • Any frame-time or per-write-back timing. I launched no title and opened neither the external release nor its screenshots and traces. The scene match is author-reported.
  • The ReadDevice fallback branch (no recorded readback). No focused test reaches it; it only moved into the read vector.
  • Negative control: none. No tests were added or changed, and the change stores the same bytes.

@JJenkx

JJenkx commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

@adriadam10 both conditions from your 10-29 review are now met on the same head c12a20f45d: the description holds the full aps5-summary.jsonl lines of two same-scene main/change pairs with tools/trace_digest.py output, and @XairRrR re-checked them and approved today (their review says the one blocking finding is resolved; they ran the perf gate on the lines as copied). No code has changed since your review.

@EagleBearerYS

Copy link
Copy Markdown
Contributor

On the order with #12840 (BDA stores into writable heap mirrors), which conflicts with this PR in WriteBack's resident path: this PR can go first. #12840 copies the readback of each written piece into a vector, the copy this PR removes, so on rebase it takes a span over the readback's slice instead.

#12840's checklist will say Depends on: #12827 (merge order only).

@mrfelek

mrfelek commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Data: plain main A/B from PPSA22408 (Hot Wheels Infinite Rush, UE5), plus two pairs from PPSA28416 on another tree

PPSA22408, first race, main vs this PR. RTX 3080 Ti (NVIDIA 615.71.09), Ryzen 9 5950X, Ubuntu 26.10 (Linux 7.3), 3840x2160, the game's default settings.

Trees:

  • A: main b11c7bf84f.
  • B: A merged with this head c12a20f45d (5fcf8be8a4). It was built from the same build directory as A, with only the driver rebuilt, so only libSceAgcDriver.prx differs.

Run setup:

  • Route: the same scripted route every run: menus, then the garage, then Select Vehicle, which loads the first race. Each run then holds 420 s.
  • Order: A B A B in one session, with warm shader caches.
  • Env: no APS5_PROFILE_*.

What happens on main: loading that race brings resident heap-mirror write-backs of about 540 MiB each. That's measured with a temporary counter on a Radeon 780M on the same route: about 7 GB per 10 s during the load.

Presents per second A run 1 B run 1 A run 2 B run 2
Race load (Select Vehicle +20 to +240 s) 2.37 2.30 1.88 2.31
After the load (Select Vehicle +300 to +420 s) 0.30 0.62 0.30 0.63
Median frame p50 after the load 3.94 s 1.82 s 4.10 s 1.86 s
  • During the load: about the same on both trees.
  • After the load: the sequence before the race (the city flyover with the tutorial captions) runs about twice as fast with this PR in both pairs.
  • Errors: no exceptions in any run.
  • Rendering: the B runs reach later parts of that sequence within the window. Those parts show magenta and green artifacts, but a longer main run shows the same artifacts at the same point (Many activities to discover), so they aren't from this PR.

Trace:

  • Timing: one trace per tree (APS5_TRACE_SECONDS=330:10 APS5_TRACE_EVENTS=2097152, no other APS5_TRACE_* or APS5_PROFILE_*).
  • It landed in the load, not after it: 330 s from start is about 80 s after Select Vehicle, where the two trees are about equal. So it doesn't show the post-load difference, which is in the summary records below.
  • The traces themselves: I can attach them if useful.
trace_digest.py: A (main b11c7bf), during the load
**seconds 330:10** from present 4182: 128 presents, frame p50 73.7 / p90 84.5 / p99 144.4 / max 261.9 ms, GPU busy 42.4 ms per present
NVIDIA GeForce RTX 3080 Ti, NVIDIA 615.71.09, Linux 7.3.0-10-generic, 3840x2160, commit b11c7bf84f5d, dropped: no

```json
{"commit":"b11c7bf84f5d","dirty":false,"env":{"APS5_HOST_IMPORT_MIB":"16384","APS5_PAD_INPUT_FILE":"<path>","APS5_STAGING_POOL_MIB":"768","APS5_TRACE_EVENTS":"2097152","APS5_TRACE_SECONDS":"330:10","APS5_TRACE_SUMMARY_EVERY":"10","APS5_UNIT_SHADOW_MIB":"512"},"gpu":"NVIDIA GeForce RTX 3080 Ti","driver":"NVIDIA 615.71.09","os":"Linux 7.3.0-10-generic","cpu":"AMD Ryzen 9 5950X 16-Core Processor","cpu_threads":32,"memory_mib":32003,"title_id":"PPSA22408","title":"HOT WHEELS\u2122\ufe0f Infinite Rush","content_version":"01.000.000","resolution":"3840x2160","window":{"spec":"seconds 330:10","first_present":4182,"ms":10051.82},"presents":128,"frame_ms":{"p50":73.684,"p90":84.477,"p99":144.376,"max":261.859},"gpu_busy_ms_per_present":42.442,"worker_busy_pct":{"queue worker 0x0":99.527,"queue worker 0x20":97.903},"top_phases_ms_per_frame":{"Driver.Packet":154.985,"Driver.Packet.memory_wait":65.066,"Driver.Packet.dispatch":57.642,"Driver.Dispatch":57.106,"Vulkan.Dispatch":45.069,"Vulkan.Dispatch.shader_resources":42.845,"Recorder.Sync":24.206,"Resources.Complete":19.283},"trace_dropped":false,"trace":"aps5-trace-b11c7bf84f5d-1585635.json.gz"}
```

**Slowest frames** (largest slices inside each, inclusive)

- present 4204: 261.9 ms at 1913 ms: Driver.Packet (queue worker 0x0) 259.3, Driver.Packet (queue worker 0x20) 232.5, Driver.Packet.memory_wait (queue worker 0x20) 215.2, Driver.Packet.dispatch (queue worker 0x0) 134.4
- present 4203: 144.4 ms at 1768 ms: Driver.Packet (queue worker 0x20) 143.9, Driver.Packet (queue worker 0x0) 142.0, Driver.Packet.memory_wait (queue worker 0x20) 132.5, Driver.Packet.dispatch (queue worker 0x0) 73.1
- present 4200: 111.6 ms at 1491 ms: Driver.Packet (queue worker 0x20) 111.2, Driver.Packet (queue worker 0x0) 109.1, Driver.Packet.memory_wait (queue worker 0x20) 94.3, Driver.Packet.dispatch (queue worker 0x0) 65.3
- present 4192: 99.0 ms at 774 ms: Driver.Packet (queue worker 0x20) 98.5, Driver.Packet (queue worker 0x0) 96.3, Driver.Packet.memory_wait (queue worker 0x20) 86.9, Driver.Packet.dispatch (queue worker 0x0) 71.2
- present 4199: 98.8 ms at 1392 ms: Driver.Packet (queue worker 0x20) 98.3, Driver.Packet (queue worker 0x0) 96.2, Driver.Packet.memory_wait (queue worker 0x20) 87.2, Driver.Packet.dispatch (queue worker 0x0) 64.1

**Persistent spikes** (frame time above the fastest frame that separates it from a longer frame; one entry per stall, not per frame)

- present 4204: 261.9 ms at 1913 ms, persistence 190.3 ms
- present 4200: 111.6 ms at 1491 ms, persistence 29.9 ms
- present 4192: 99.0 ms at 774 ms, persistence 21.8 ms
- present 4196: 97.7 ms at 1126 ms, persistence 16.4 ms
- present 4186: 87.2 ms at 299 ms, persistence 14.5 ms
- present 4305: 85.1 ms at 9673 ms, persistence 13.5 ms
- present 4233: 84.1 ms at 4282 ms, persistence 11.8 ms
- present 4189: 85.2 ms at 540 ms, persistence 11.4 ms
- present 4246: 83.6 ms at 5324 ms, persistence 10.7 ms
- present 4290: 82.6 ms at 8568 ms, persistence 10.7 ms

**Threads** (ms per frame)

| thread | busy | wait | untraced | in slices < 20 us | largest |
|---|---|---|---|---|---|
| title | 0.1 (0 %) | 0.0 (0 %) | 78.3 | 0.0 | Recorder.Sync 0.1 |
| queue worker 0x0 | 68.5 (87 %) | 7.4 (9 %) | 2.7 | 11.5 | Driver.Packet 75.8, Driver.EndSubmission 0.0 |
| draw committer | 6.1 (8 %) | 7.4 (9 %) | 65.0 | 0.6 | Graphics.Draw 5.3, Recorder.Sync 0.8 |
| queue worker 0x20 | 10.4 (13 %) | 67.6 (86 %) | 0.6 | 2.2 | Driver.Packet 76.2, Driver.EndSubmission 0.0 |
| presenter | 0.8 (1 %) | 0.1 (0 %) | 77.6 | 0.1 | VideoOut.Flip 0.9 |
| title | 0.1 (0 %) | 0.1 (0 %) | 78.3 | 0.0 | Recorder.Sync 0.1 |
| title | 0.1 (0 %) | 0.0 (0 %) | 78.4 | 0.0 | Recorder.Sync 0.1 |
| title | 0.0 (0 %) | 0.0 (0 %) | 78.5 | 0.0 | Recorder.Sync 0.0 |
| title | 0.0 (0 %) | 0.0 (0 %) | 78.5 | 0.0 | Recorder.Sync 0.0 |
| title | 0.1 (0 %) | 0.4 (1 %) | 78.0 | 0.0 | Recorder.Sync 0.1 |
| title | 0.0 (0 %) | 0.0 (0 %) | 78.5 | 0.0 | Recorder.Sync 0.0 |
| title | 0.0 (0 %) | 0.0 (0 %) | 78.5 | 0.0 | Recorder.Sync 0.0 |
| title | 0.0 (0 %) | 0.0 (0 %) | 78.5 | 0.0 | Recorder.Sync 0.0 |
| title | 0.0 (0 %) | 0.0 (0 %) | 78.5 | 0.0 | Recorder.Sync 0.0 |
| title | 0.0 (0 %) | 0.0 (0 %) | 78.5 | 0.0 | Recorder.Sync 0.0 |
| title | 0.0 (0 %) | 0.0 (0 %) | 78.5 | 0.0 | Recorder.Sync 0.0 |
| title | 0.0 (0 %) | 0.0 (0 %) | 78.5 | 0.0 | Recorder.Sync 0.0 |
| gpu | 42.3 (54 %) | 0.0 (0 %) | 36.2 | 0.0 | batch 42.3 |

**Largest slices** (ms per frame, inclusive, all threads): Driver.Packet 152.1, Driver.Packet.memory_wait 64.8, Driver.Packet.dispatch 57.5, Driver.Dispatch 57.0, Vulkan.Dispatch 44.3, batch 42.3, Vulkan.Dispatch.shader_resources 41.9, Recorder.Sync 24.2, Resources.Complete 19.1, Driver.Packet.pipeline_drain 17.7

**Longest waits**

- Driver.Packet.memory_wait (queue worker 0x20) 242.2 ms at 1878 ms, present 4203
- Driver.Packet.memory_wait (queue worker 0x20) 86.9 ms at 1766 ms, present 4202
- Driver.Packet.memory_wait (queue worker 0x20) 86.5 ms at 1455 ms, present 4199
- Driver.Packet.memory_wait (queue worker 0x20) 76.5 ms at 746 ms, present 4191
- Driver.Packet.memory_wait (queue worker 0x20) 68.6 ms at 1093 ms, present 4195
- Driver.Packet.memory_wait (queue worker 0x20) 65.9 ms at 1186 ms, present 4196
- Driver.Packet.memory_wait (queue worker 0x20) 63.0 ms at 42 ms, present 4182
- Driver.Packet.memory_wait (queue worker 0x20) 60.3 ms at 6393 ms, present 4260
- Driver.Packet.memory_wait (queue worker 0x20) 60.1 ms at 4242 ms, present 4232
- Driver.Packet.memory_wait (queue worker 0x20) 59.8 ms at 839 ms, present 4192

**Largest GPU batches** (GPU ranges inside, and the CPU slices that recorded each, inclusive)

- batch 140962: 13.3 ms at 9068 ms, present 4296: GPU no ranges; recorded by Recorder.Sync (queue worker 0x0) 13.4, Driver.Packet (queue worker 0x0) 6.2, Driver.Packet.dispatch (queue worker 0x0) 6.0, Driver.Dispatch (queue worker 0x0) 6.0
- batch 136433: 13.2 ms at 2621 ms, present 4210: GPU no ranges; recorded by Recorder.Sync (queue worker 0x0) 13.2, Driver.Packet (queue worker 0x0) 4.6, Driver.Packet.dispatch (queue worker 0x0) 4.3, Driver.Dispatch (queue worker 0x0) 4.3
- batch 135923: 13.2 ms at 1743 ms, present 4202: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 6.4, Driver.Packet.dispatch (queue worker 0x0) 6.1, Driver.Dispatch (queue worker 0x0) 6.1, Vulkan.Dispatch (queue worker 0x0) 5.7
- batch 140106: 13.1 ms at 7814 ms, present 4279: GPU no ranges; recorded by Recorder.Sync (queue worker 0x0) 13.1, Driver.Packet (queue worker 0x0) 4.6, Driver.Packet.dispatch (queue worker 0x0) 4.3, Driver.Dispatch (queue worker 0x0) 4.3
- batch 135648: 13.1 ms at 1367 ms, present 4198: GPU no ranges; recorded by Recorder.Sync (queue worker 0x0) 13.2, Driver.Packet (queue worker 0x0) 4.5, Driver.Packet.dispatch (queue worker 0x0) 4.3, Driver.Dispatch (queue worker 0x0) 4.3
trace_digest.py: B (this head), during the load
**seconds 330:10** from present 4343: 64 presents, frame p50 97.2 / p90 159.2 / p99 1617.9 / max 1617.9 ms, GPU busy 55.6 ms per present
NVIDIA GeForce RTX 3080 Ti, NVIDIA 615.71.09, Linux 7.3.0-10-generic, 3840x2160, commit 5fcf8be8a405, dropped: no

```json
{"commit":"5fcf8be8a405","dirty":false,"env":{"APS5_HOST_IMPORT_MIB":"16384","APS5_PAD_INPUT_FILE":"<path>","APS5_STAGING_POOL_MIB":"768","APS5_TRACE_EVENTS":"2097152","APS5_TRACE_SECONDS":"330:10","APS5_TRACE_SUMMARY_EVERY":"10","APS5_UNIT_SHADOW_MIB":"512"},"gpu":"NVIDIA GeForce RTX 3080 Ti","driver":"NVIDIA 615.71.09","os":"Linux 7.3.0-10-generic","cpu":"AMD Ryzen 9 5950X 16-Core Processor","cpu_threads":32,"memory_mib":32003,"title_id":"PPSA22408","title":"HOT WHEELS\u2122\ufe0f Infinite Rush","content_version":"01.000.000","resolution":"3840x2160","window":{"spec":"seconds 330:10","first_present":4343,"ms":10019.697},"presents":64,"frame_ms":{"p50":97.157,"p90":159.207,"p99":1617.895,"max":1617.895},"gpu_busy_ms_per_present":55.568,"worker_busy_pct":{"queue worker 0x0":99.638,"queue worker 0x20":98.866},"top_phases_ms_per_frame":{"Driver.Packet":310.713,"Shader.PrepareArtifact":179.23,"Driver.Packet.memory_wait":120.991,"Driver.Packet.dispatch":111.186,"Driver.Dispatch":110.313,"Vulkan.Dispatch":77.461,"Vulkan.Dispatch.shader_resources":64.72,"Driver.Packet.pipeline_drain":49.853},"trace_dropped":false,"trace":"aps5-trace-5fcf8be8a405-1601561.json.gz"}
```

**Slowest frames** (largest slices inside each, inclusive)

- present 4348: 1617.9 ms at 1708 ms: Shader.PrepareArtifact (shader prepare) 4171.5, Driver.Packet (queue worker 0x20) 1617.3, Driver.Packet (queue worker 0x0) 1612.5, Driver.Packet.dispatch (queue worker 0x0) 855.5
- present 4347: 1180.9 ms at 527 ms: Driver.Packet (queue worker 0x20) 1180.4, Driver.Packet (queue worker 0x0) 1175.6, Shader.PrepareArtifact (shader prepare) 863.4, Driver.Packet.pipeline_drain (queue worker 0x0) 738.2
- present 4401: 791.6 ms at 8630 ms: Driver.Packet (queue worker 0x20) 791.1, Driver.Packet (queue worker 0x0) 786.5, Driver.Packet.memory_wait (queue worker 0x20) 781.8, Graphics.Draw (draw committer) 618.7
- present 4351: 190.2 ms at 3643 ms: Driver.Packet (queue worker 0x0) 185.4, Driver.Packet.dispatch (queue worker 0x0) 138.7, Driver.Dispatch (queue worker 0x0) 138.1, Driver.WorkerWait (queue worker 0x20) 105.5
- present 4405: 172.6 ms at 9738 ms: Shader.PrepareArtifact (shader prepare) 2015.9, Driver.Packet (queue worker 0x20) 172.0, Driver.Packet (queue worker 0x0) 167.2, Shader.Register (title) 154.9

**Persistent spikes** (frame time above the fastest frame that separates it from a longer frame; one entry per stall, not per frame)

- present 4348: 1617.9 ms at 1708 ms, persistence 1529.7 ms
- present 4401: 791.6 ms at 8630 ms, persistence 703.4 ms
- present 4405: 172.6 ms at 9738 ms, persistence 75.4 ms
- present 4343: 160.9 ms at 0 ms, persistence 61.3 ms
- present 4366: 125.8 ms at 5188 ms, persistence 37.2 ms
- present 4397: 129.7 ms at 8199 ms, persistence 36.0 ms
- present 4373: 124.1 ms at 5881 ms, persistence 32.7 ms
- present 4351: 190.2 ms at 3643 ms, persistence 32.1 ms
- present 4395: 122.1 ms at 7972 ms, persistence 17.9 ms
- present 4361: 108.1 ms at 4699 ms, persistence 16.3 ms

**Threads** (ms per frame)

| thread | busy | wait | untraced | in slices < 20 us | largest |
|---|---|---|---|---|---|
| title | 0.0 (0 %) | 0.0 (0 %) | 156.5 | 0.0 | Recorder.Sync 0.0 |
| shader prepare | 11.2 (7 %) | 0.0 (0 %) | 145.3 | 0.0 | Shader.PrepareArtifact 11.2 |
| shader prepare | 11.6 (7 %) | 0.0 (0 %) | 144.9 | 0.0 | Shader.PrepareArtifact 11.6 |
| shader prepare | 11.1 (7 %) | 0.0 (0 %) | 145.5 | 0.0 | Shader.PrepareArtifact 11.1 |
| shader prepare | 9.0 (6 %) | 0.0 (0 %) | 147.6 | 0.0 | Shader.PrepareArtifact 9.0 |
| shader prepare | 9.7 (6 %) | 0.0 (0 %) | 146.9 | 0.0 | Shader.PrepareArtifact 9.7 |
| shader prepare | 11.7 (7 %) | 0.0 (0 %) | 144.8 | 0.0 | Shader.PrepareArtifact 11.7 |
| shader prepare | 8.9 (6 %) | 0.0 (0 %) | 147.6 | 0.0 | Shader.PrepareArtifact 8.9 |
| shader prepare | 9.7 (6 %) | 0.0 (0 %) | 146.8 | 0.0 | Shader.PrepareArtifact 9.7 |
| shader prepare | 10.3 (7 %) | 0.0 (0 %) | 146.3 | 0.0 | Shader.PrepareArtifact 10.3 |
| shader prepare | 10.6 (7 %) | 0.0 (0 %) | 145.9 | 0.0 | Shader.PrepareArtifact 10.6 |
| shader prepare | 11.8 (8 %) | 0.0 (0 %) | 144.8 | 0.0 | Shader.PrepareArtifact 11.8 |
| shader prepare | 11.1 (7 %) | 0.0 (0 %) | 145.5 | 0.0 | Shader.PrepareArtifact 11.1 |
| shader prepare | 11.4 (7 %) | 0.0 (0 %) | 145.2 | 0.0 | Shader.PrepareArtifact 11.4 |
| shader prepare | 10.1 (6 %) | 0.0 (0 %) | 146.4 | 0.0 | Shader.PrepareArtifact 10.1 |
| shader prepare | 10.8 (7 %) | 0.0 (0 %) | 145.8 | 0.0 | Shader.PrepareArtifact 10.8 |
| shader prepare | 12.1 (8 %) | 0.0 (0 %) | 144.4 | 0.0 | Shader.PrepareArtifact 12.1 |
| shader prepare | 0.1 (0 %) | 0.0 (0 %) | 156.5 | 0.0 | Vulkan.GraphicsPipelinePrebuild 0.1 |
| shader prepare | 0.2 (0 %) | 0.0 (0 %) | 156.3 | 0.0 | Vulkan.GraphicsPipelinePrebuild 0.2 |
| shader prepare | 0.1 (0 %) | 0.0 (0 %) | 156.5 | 0.0 | Vulkan.GraphicsPipelinePrebuild 0.1 |
| queue worker 0x0 | 131.1 (84 %) | 20.8 (13 %) | 4.7 | 20.6 | Driver.Packet 151.9, Driver.EndSubmission 0.0 |
| draw committer | 23.2 (15 %) | 20.0 (13 %) | 113.4 | 0.9 | Graphics.Draw 23.3, Recorder.Sync 0.6 |
| queue worker 0x20 | 24.0 (15 %) | 131.4 (84 %) | 1.2 | 2.5 | Driver.Packet 151.9, Driver.EndSubmission 0.0 |
| presenter | 1.1 (1 %) | 0.1 (0 %) | 155.3 | 0.1 | VideoOut.Flip 1.2 |
| title | 3.8 (2 %) | 0.0 (0 %) | 152.7 | 0.0 | Shader.ResolveAbi 2.6, Shader.Register 1.1 |
| title | 1.6 (1 %) | 0.1 (0 %) | 154.9 | 0.0 | Shader.Register 0.7, Shader.ResolveAbi 0.4 |
| title | 2.2 (1 %) | 0.0 (0 %) | 154.4 | 0.0 | Shader.ResolveAbi 1.2, Shader.Register 0.9 |
| title | 7.8 (5 %) | 9.7 (6 %) | 139.0 | 0.0 | Recorder.Sync 7.2, Shader.Register 0.6 |
| title | 0.9 (1 %) | 0.0 (0 %) | 155.6 | 0.0 | Shader.Register 0.9, Recorder.Sync 0.0 |
| title | 1.1 (1 %) | 0.0 (0 %) | 155.4 | 0.0 | Shader.Register 1.1, Recorder.Sync 0.0 |
| title | 3.8 (2 %) | 0.1 (0 %) | 152.6 | 0.0 | Shader.ResolveAbi 2.7, Shader.Register 1.0 |
| title | 1.0 (1 %) | 0.0 (0 %) | 155.5 | 0.0 | Shader.Register 0.9, Recorder.Sync 0.1 |
| title | 0.9 (1 %) | 0.0 (0 %) | 155.6 | 0.0 | Shader.Register 0.8, Recorder.Sync 0.1 |
| title | 1.1 (1 %) | 0.0 (0 %) | 155.5 | 0.0 | Shader.Register 1.1, Recorder.Sync 0.0 |
| title | 0.9 (1 %) | 0.0 (0 %) | 155.6 | 0.0 | Shader.Register 0.9, Recorder.Sync 0.0 |
| title | 2.4 (2 %) | 0.0 (0 %) | 154.2 | 0.0 | Shader.ResolveAbi 1.1, Shader.Register 0.9 |
| title | 0.4 (0 %) | 0.0 (0 %) | 156.1 | 0.0 | Shader.Register 0.4 |
| gpu | 55.1 (35 %) | 0.0 (0 %) | 101.5 | 0.0 | batch 55.1 |

**Largest slices** (ms per frame, inclusive, all threads): Driver.Packet 303.8, Shader.PrepareArtifact 179.2, Driver.Packet.memory_wait 118.5, Driver.Packet.dispatch 111.0, Driver.Dispatch 110.2, Vulkan.Dispatch 76.6, Vulkan.Dispatch.shader_resources 62.9, batch 55.1, Driver.Packet.pipeline_drain 49.0, Resources.Complete 35.2

**Longest waits**

- Driver.Packet.memory_wait (queue worker 0x20) 773.0 ms at 8675 ms, present 4401
- Driver.Packet.memory_wait (queue worker 0x20) 495.4 ms at 1123 ms, present 4347
- Driver.Packet.memory_wait (queue worker 0x20) 451.3 ms at 2766 ms, present 4348
- GpuMutex.wait.label (draw committer) 346.9 ms at 536 ms, present 4347
- GpuMutex.wait.label (queue worker 0x0) 292.2 ms at 2119 ms, present 4348
- GpuMutex.wait.label (draw committer) 277.9 ms at 1841 ms, present 4348
- GpuMutex.wait.other (title) 271.2 ms at 1848 ms, present 4348
- GpuMutex.wait.label (draw committer) 236.5 ms at 887 ms, present 4347
- GpuMutex.wait.label (queue worker 0x20) 224.3 ms at 2411 ms, present 4348
- GpuMutex.wait.dispatch (queue worker 0x0) 220.2 ms at 2412 ms, present 4348

**Largest GPU batches** (GPU ranges inside, and the CPU slices that recorded each, inclusive)

- batch 146993: 85.2 ms at 2412 ms, present 4348: GPU no ranges; recorded by Driver.Packet (queue worker 0x20) 516.6, Driver.Packet.dispatch (queue worker 0x20) 292.2, Driver.Dispatch (queue worker 0x20) 292.2, Driver.Packet.prepare_memory (queue worker 0x20) 224.4
- batch 150939: 35.5 ms at 8203 ms, present 4397: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 37.8, Driver.Packet.pipeline_drain (queue worker 0x0) 36.6, Driver.Packet.dispatch (queue worker 0x0) 1.1, Driver.Dispatch (queue worker 0x0) 1.1
- batch 146903: 24.3 ms at 1153 ms, present 4347: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 266.7, Driver.Packet.pipeline_drain (queue worker 0x0) 236.5, Driver.Packet.prepare_memory (queue worker 0x0) 30.0, GpuMutex.wait.label (queue worker 0x0) 29.9
- batch 146991: 24.2 ms at 1841 ms, present 4348: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 130.9, Driver.Packet.dispatch (queue worker 0x0) 127.8, Driver.Dispatch (queue worker 0x0) 127.7, GpuMutex.wait.dispatch (queue worker 0x0) 126.0
- batch 146992: 23.9 ms at 2150 ms, present 4348: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 571.4, Driver.Packet.prepare_memory (queue worker 0x0) 293.2, GpuMutex.wait.label (queue worker 0x0) 292.2, Driver.Packet.pipeline_drain (queue worker 0x0) 278.0
Summary records: the window nearest Select Vehicle +400 s of each run (A1, B1, A2, B2)
{"commit":"b11c7bf84f5d","dirty":false,"env":{"APS5_HOST_IMPORT_MIB":"16384","APS5_PAD_INPUT_FILE":"<path>","APS5_STAGING_POOL_MIB":"768","APS5_TRACE_EVENTS":"2097152","APS5_TRACE_SECONDS":"330:10","APS5_TRACE_SUMMARY_EVERY":"10","APS5_UNIT_SHADOW_MIB":"512"},"gpu":"NVIDIA GeForce RTX 3080 Ti","driver":"NVIDIA 615.71.09","os":"Linux 7.3.0-10-generic","cpu":"AMD Ryzen 9 5950X 16-Core Processor","cpu_threads":32,"memory_mib":32003,"title_id":"PPSA22408","title":"HOT WHEELS™️ Infinite Rush","content_version":"01.000.000","resolution":"3840x2160","window":{"spec":"summary_every 10","first_present":4668,"ms":12346.241},"interval_start_s":638.478,"interval_end_s":650.825,"presents":3,"frame_ms":{"p50":3936.133,"p90":4613.252,"p99":4613.252,"max":4613.252},"gpu_busy_ms_per_present":558.598,"worker_busy_pct":{"queue worker 0x0":99.964,"queue worker 0x20":100.144},"top_phases_ms_per_frame":{"Driver.Packet":8235.157,"Driver.Packet.dispatch":3232.456,"Driver.Dispatch":3217.751,"Driver.Packet.pipeline_drain":2620.181,"Vulkan.Dispatch":2435.124,"Vulkan.Dispatch.shader_resources":2139.668,"GpuMutex.wait.label":2030.901,"Resources.Complete":1928.529},"trace_dropped":false,"trace":""}
{"commit":"5fcf8be8a405","dirty":false,"env":{"APS5_HOST_IMPORT_MIB":"16384","APS5_PAD_INPUT_FILE":"<path>","APS5_STAGING_POOL_MIB":"768","APS5_TRACE_EVENTS":"2097152","APS5_TRACE_SECONDS":"330:10","APS5_TRACE_SUMMARY_EVERY":"10","APS5_UNIT_SHADOW_MIB":"512"},"gpu":"NVIDIA GeForce RTX 3080 Ti","driver":"NVIDIA 615.71.09","os":"Linux 7.3.0-10-generic","cpu":"AMD Ryzen 9 5950X 16-Core Processor","cpu_threads":32,"memory_mib":32003,"title_id":"PPSA22408","title":"HOT WHEELS™️ Infinite Rush","content_version":"01.000.000","resolution":"3840x2160","window":{"spec":"summary_every 10","first_present":4681,"ms":10214.752},"interval_start_s":639.233,"interval_end_s":649.448,"presents":9,"frame_ms":{"p50":413.246,"p90":3546.532,"p99":3546.532,"max":3546.532},"gpu_busy_ms_per_present":280.645,"worker_busy_pct":{"queue worker 0x0":99.785,"queue worker 0x20":99.978},"top_phases_ms_per_frame":{"Driver.Packet":2267.152,"Driver.Packet.dispatch":912.713,"Driver.Dispatch":905.396,"Vulkan.Dispatch":719.170,"Driver.Packet.pipeline_drain":702.961,"Vulkan.Dispatch.shader_resources":585.713,"GpuMutex.wait.label":560.603,"Resources.Complete":497.619},"trace_dropped":false,"trace":""}
{"commit":"b11c7bf84f5d","dirty":false,"env":{"APS5_HOST_IMPORT_MIB":"16384","APS5_PAD_INPUT_FILE":"<path>","APS5_STAGING_POOL_MIB":"768","APS5_TRACE_SUMMARY_EVERY":"10","APS5_UNIT_SHADOW_MIB":"512"},"gpu":"NVIDIA GeForce RTX 3080 Ti","driver":"NVIDIA 615.71.09","os":"Linux 7.3.0-10-generic","cpu":"AMD Ryzen 9 5950X 16-Core Processor","cpu_threads":32,"memory_mib":32003,"title_id":"PPSA22408","title":"HOT WHEELS™️ Infinite Rush","content_version":"01.000.000","resolution":"3840x2160","window":{"spec":"summary_every 10","first_present":4631,"ms":14949.274},"interval_start_s":636.711,"interval_end_s":651.660,"presents":3,"frame_ms":{"p50":5342.536,"p90":6452.269,"p99":6452.269,"max":6452.269},"gpu_busy_ms_per_present":677.720,"worker_busy_pct":{"queue worker 0x0":99.965,"queue worker 0x20":99.919},"top_phases_ms_per_frame":{"Driver.Packet":9958.255,"Driver.Packet.dispatch":4457.849,"Driver.Dispatch":4440.460,"Driver.Packet.pipeline_drain":2603.026,"GpuMutex.wait.label":2422.954,"Vulkan.Dispatch":2246.948,"Recorder.Sync":2143.873,"GpuMutex.wait.dispatch":2102.271},"trace_dropped":false,"trace":""}
{"commit":"5fcf8be8a405","dirty":false,"env":{"APS5_HOST_IMPORT_MIB":"16384","APS5_PAD_INPUT_FILE":"<path>","APS5_STAGING_POOL_MIB":"768","APS5_TRACE_SUMMARY_EVERY":"10","APS5_UNIT_SHADOW_MIB":"512"},"gpu":"NVIDIA GeForce RTX 3080 Ti","driver":"NVIDIA 615.71.09","os":"Linux 7.3.0-10-generic","cpu":"AMD Ryzen 9 5950X 16-Core Processor","cpu_threads":32,"memory_mib":32003,"title_id":"PPSA22408","title":"HOT WHEELS™️ Infinite Rush","content_version":"01.000.000","resolution":"3840x2160","window":{"spec":"summary_every 10","first_present":4728,"ms":11600.235},"interval_start_s":631.819,"interval_end_s":643.419,"presents":7,"frame_ms":{"p50":1853.806,"p90":2623.291,"p99":2623.291,"max":2623.291},"gpu_busy_ms_per_present":308.046,"worker_busy_pct":{"queue worker 0x0":99.788,"queue worker 0x20":100.062},"top_phases_ms_per_frame":{"Driver.Packet":3311.687,"Driver.Packet.dispatch":1417.745,"Driver.Dispatch":1412.887,"Vulkan.Dispatch":847.593,"Driver.Packet.memory_wait":757.858,"Driver.Packet.pipeline_drain":680.552,"Vulkan.Dispatch.shader_resources":660.728,"GpuMutex.wait.label":636.610},"trace_dropped":false,"trace":""}

PPSA28416 (Star Wars Galactic Racer, UE5), first race of the campaign. This is attribution, not a main pair: the title needs the closed #15656 to boot.

The tree: main 1c8c0efc8c + #15656 + #11346 + #7963 + #11485, with this PR's change behind a local switch.

What happens there: resident heap-mirror write-backs of about 575 MiB each, about 220 GB per run.

Runs: two pairs, alternating, holding 180 s in the race. The second pair ran in reversed order.

copy (as on main) this change
Pair 1, presents per second 0.18 0.27
Pair 2, presents per second 0.16 0.32
Pair 2, frame p50 6.6–7.6 s 3.8–4.0 s

There were no exceptions.

AI-assisted.

@JJenkx

JJenkx commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Queue maintenance follow-up: the remaining older change request has no formal re-review request. The completed report and XairRrR's re-check/approval are already present, and the description now also links mrfelek's November 4 plain-main title measurements with their source/configuration limits.

I attempted to re-request the outstanding review through GitHub. REST returned 404; GraphQL explicitly returned FORBIDDEN: JJenkx does not have the correct permissions to execute RequestReviews. A reviewer or maintainer needs to perform that re-request or re-check. The source head is still c12a20f45da55cd45da092e937ee196416cc2f72.

@JJenkx

JJenkx commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

A second title where this helps: Mortal Shell II (PPSA34008), title key art. Linux 7.2.11, Ryzen 9 7950X, RX 6650 XT (RADV Mesa 26.2.6), Release. main 5444d9862f against main 5444d9862f + this PR (head c12a20f45d, a local merge, no other change), the same scripted input, 125 s per run, arms alternating A B B A, no compiler or other run on the host during the runs.

Why this title. Its key art runs compute dispatches that write a resident heap mirror of several hundred MiB. Each write-back reads the written range back, and on main it then copies the whole readback into a new vector before storing it. A perf profile of main at 90 s put 42 % of samples in the kernel (page allocation and clearing for these large buffers) and 37 % in glibc's copy and compare routines.

How I measured. The key art alternates between slow phases (about 0.1 to 0.6 FPS) and fast ones (about 5 to 10 FPS), so a mean over a fixed window mostly reflects where the fast phase falls. I report, per run, the time of the first fast phase (the first sample above 3 FPS after 20 s) and the median FPS of the slow-phase samples (below 2 FPS) from 20 to 120 s.

Run Build First fast phase Slow-phase median Fast-phase median
A1 main 70 s 0.40 5.11
B2 this PR 44 s 0.54 5.85
B3 this PR 44 s 0.68 5.64
A4 main 72 s 0.38 5.98

Earlier main runs on the same build and input today (four runs) reached the fast phase at 70 to 82 s, with slow-phase medians of 0.32 to 0.37. So with this PR the slow phases run at about 0.54 to 0.68 FPS instead of about 0.32 to 0.40, and the title reaches its fast phase roughly 30 s sooner.

Limits: one title, one machine, 2+2 runs.

Per-sample FPS, 20 to 120 s, every run
ms2pr2-A1-1: 0.2 0.2 0.6 1.3 1.3 1.3 0.2 0.2 0.4 0.4 0.4 0.4 0.4 0.4 0.3 0.3 0.4 0.4 0.4 0.4 0.5 0.5 0.5 0.3 0.5 4.9 5.3 5.3 5.4 5.1 5.1 5.1 5.1 4.7 4.7 0.9 0.9 0.9 0.9 0.9 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.2 0.2
ms2pr2-B2-1: 1.3 1.3 0.7 0.7 0.5 0.8 0.8 0.8 0.9 0.9 0.9 1.4 5.7 5.9 5.9 5.9 2.4 5.5 5.8 5.3 4.9 0.9 6.5 10.7 10.7 1.1 1.1 0.3 0.3 0.3 0.3 0.3 0.3 0.6 0.5 0.5 0.5 0.5 0.6 0.8 0.8 0.5 0.5 0.5 0.4 0.4 0.2 0.2 0.5 0.5
ms2pr2-B3-1: 1.4 0.7 0.7 0.7 0.6 0.6 0.9 0.7 1.1 1.1 1.5 2.4 5.0 5.7 5.2 5.2 1.7 5.6 5.8 5.6 0.9 5.7 5.7 5.7 5.7 5.0 1.1 1.1 0.3 0.3 0.3 0.3 0.5 0.5 0.4 0.4 0.5 0.7 0.7 0.6 0.5 0.5 0.3 0.6 0.6 0.8 0.8 0.8 0.6 0.7
ms2pr2-A4-1: 0.1 1.0 0.8 0.8 0.8 0.4 0.4 0.4 0.3 0.3 0.3 0.4 0.4 0.5 0.5 0.5 0.5 0.5 0.4 0.4 0.3 1.1 1.1 1.1 0.2 0.5 5.4 6.0 5.9 6.0 6.0 7.6 7.6 0.9 0.9 0.9 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.2 0.2 0.2 0.2 0.2 0.2

AI-assisted: yes

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* perf Performance change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants