Repository navigation
fix(agc): split oversized dispatches with workgroup offset specialization - #7445
Conversation
elliot1377
changed the title
|
Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
|
Thanks for this. We're going with this approach for #7375: plain Two things before it can merge:
We'll run the full CI and our game checks on the result. |
|
Thanks for the review!
The full |
|
Cross-validation of 1551e7a on NVIDIA. Scratch-merged on main a62c724 (clean), full Release build with
Not checked: X and Z splits on NVIDIA (the X limit is 2^31-1 and the test has no Z case), AMD, lavapipe, Linux, and any title that issues an oversized dispatch. Main a62c724, PR head 1551e7a. Written with Claude Code assistance (validation by a subagent, reviewed in this session). |
|
AMD on Windows: the tests this PR names pass on an RDNA 3.5 GPU. Radeon 860M (RDNA 3.5, driver 32.0.13058.8001), Windows 11 26300, WinLibs GCC 15.2.0 r7, Release.
AI-assisted |
|
Thanks a lot for the cross-check and the negative control, @screamm, and @LeandroLP23 for the AMD run. About the slower PPSA99005 start-up: I couldn't find a per-run cost in the change. Unsplit pipelines only gain the three offset values (all 0), and the pipeline journal stores and replays them with the rest of the result. What does change is the recompiler sources, so That would explain a slow first ~30 s on the first PR run, but not on runs 2 and 3 if they reused the same cache directory. Could you check two things?
If runs 2 and 3 replayed everything and were still slow, I'll dig further. |
|
Tested on AMD on Linux, which this PR had not covered.
Full
|
jotitadev
left a comment
jotitadev
left a comment
There was a problem hiding this comment.
A Linux run on NVIDIA and lavapipe, plus a standalone check of the NVIDIA behaviour that main's throw describes, for the open question in #7375.
Machine: Ubuntu 26.04, i7-8700, GCC 15.2, Release, BUILD_TESTING=ON, every execution test built. NVIDIA GeForce RTX 3070 (driver 595.99.02) and lavapipe (Mesa 26.0.3, LLVM 21.1.8) on the same host. Head 1551e7af0 merged onto main a373e85e5 (clean). ANYPS5_NO_SHADER_CACHE=1, -j1 --timeout 30. No validation layer on this host.
agc_driver_oversized_dispatch |
RTX 3070 | lavapipe |
|---|---|---|
head 1551e7af0 |
pass, 3 runs | pass |
the PR's test on main (production files from main) |
fails: workgroup count 1x69632x1 exceeds device limits 2147483647x65535x65535 and NVIDIA drops workgroups of a vkCmdDispatchBase split |
pass |
On lavapipe main passes the new Y-split check too, because the out-of-range base it uses is only caught by the validation layer, as #7375 says.
#7375 leaves open whether NVIDIA drops workgroups for a valid nonzero base as well. In a standalone Vulkan program (no AnyPS5 code), I recorded one vkCmdDispatchBase per case into a pipeline created with VK_PIPELINE_CREATE_DISPATCH_BASE_BIT. Each workgroup counted itself into a buffer slot indexed by gl_WorkGroupID, and every invocation compared gl_GlobalInvocationID with gl_WorkGroupID * gl_WorkGroupSize + gl_LocalInvocationID. I ran it with local size 1 and 64, 3 runs each on the RTX 3070.
| Case (base, count) | RTX 3070 | lavapipe |
|---|---|---|
| Y: 1000, 1000 / 61438, 4097 / 1, 64 (valid) | every group once, none outside the range | same |
| X: 1000, 1000 and Z: 1000, 1000 (valid) | every group once | same |
Y: 65535, 4097 and Z: 65535, 4097 (the base main uses for the second chunk, invalid) |
every group once | same |
| Y: 65535, 1 (invalid) | the group runs once | same |
None of the cases had a mismatched global ID. So on Ampere with this Linux driver, vkCmdDispatchBase drops no workgroups, even with the out-of-range base. The drop that 288ee89 measured on an RTX 5070 Ti does not reproduce here, and it may be specific to that GPU or driver. This does not change this PR, which stops using vkCmdDispatchBase and keeps every base valid. It only means the removed "NVIDIA drops workgroups" message was not true of every NVIDIA device.
Not run: Windows, AMD, Intel, the validation layer, any title.
GPUs tested: NVIDIA GeForce RTX 3070 (595.99.02) and lavapipe (Mesa 26.0.3), Ubuntu 26.04
Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.
…tion A dispatch over maxComputeWorkGroupCount was split into vkCmdDispatchBase chunks whose base reached the limit, which VUID-vkCmdDispatchBase-baseGroupX-00421 forbids, and NVIDIA refused the split outright. Compute modules now declare WorkgroupOffset specialization constants (default 0) that the recompiler adds to WorkgroupId and GlobalInvocationId and keeps unmaterialized. Each chunk is recorded with vkCmdDispatch from group 0, bound to a pipeline variant specialized with the chunk's base. The NVIDIA throw is gone; device-memory LDS splits still throw. Fixes boykopovar#7375
8237 and 8238 are taken by MeshLayerBase and MeshLayerCount in boykopovar#6959 and 8240 by DepthRemapBase in another open PR, so buildSpecializedModule would read a mesh module's IDs as workgroup offsets once both land.
1551e7a to
40d4cc0
Compare
|
Answers from the saved run records: the order was main, PR, main, PR, PR, main (then two runs of main Written with Claude Code assistance. |
What
A direct dispatch over
maxComputeWorkGroupCountwas split intovkCmdDispatchBasechunks starting at a multiple of the limit, whichVUID-vkCmdDispatchBase-baseGroupX-00421forbids. On NVIDIA the split was refused outright.WorkgroupOffsetspecialization constants (IDs 8241-8243, default 0), added toWorkgroupIdand toGlobalInvocationId(offset × local size). The spanned-wave path derives the global id from the offsetWorkgroupId.buildSpecializedModulekeeps these constants unmaterialized, and every compute result lists them with value 0, so unsplit dispatches run with offset 0.WorkgroupLocalSizeis now shared with theLocalSizeexecution mode, so the two cannot disagree.vkCmdDispatchfrom group 0. Chunks after the first bind a pipeline variant specialized with that chunk's base. Variants share the dispatch's pipeline layout, so bound descriptor sets and push constants stay valid, and they are cached per base onComputePipelineObjects. NovkCmdDispatchBasecall remains, and the NVIDIA throw is removed.NumWorkgroups. Recorded indocs/dev/debt/libSceAgcDriver.md, along with the record-time variant build.tests/execution/OversizedDispatch.cpp: the NVIDIA "must throw" check is replaced by a Y-split coverage check that runs on every vendor (limit + 4097 groups, each group writesgroup_y + 1at dwordgroup_y). The existing X-split coverage check is unchanged.Discussed in #7375.
#7390 fixes the same issue and was opened first. It writes each chunk's offset into the
ShaderDatabuffer withvkCmdUpdateBuffer, moves the affected shaders out of push data and bumpsRuntimeAbi::Version. This PR leaves unsplit dispatches, push data and the ABI unchanged, at the cost of a pipeline variant per chunk base; it also ran on hardware (below). Only one of the two should land; I'll close this one if #7390 is preferred.Tested
Windows 11, MinGW-w64 GCC 15.2.0 (WinLibs r7), full build with
BUILD_TESTING=ON, rebased on currentmain.agc_driver_oversized_dispatchpasses on both GPUs below, with the Khronos validation layer (vulkan-sdk-1.4.335.0) loaded: 0 validation errors. As a positive control, best-practices warnings were enabled and appeared in the same output.mainthis dispatch throws. The X split is skipped because the X limit is 2^31-1.cteston the RTX 4050: 1461/1462 passed.agc_driver_image_plane_address_on_linefails identically on the commit before this change (LOD blend 67.50 vs 60.00); it passes on the Intel GPU.main3ddbc4acc(conflicts with feat(agc): remap a viewport depth range the device cannot express into the vertex shader #4650'sDepthRemapBase = 8240uandreadsDepthRemapPush, resolved by keeping both; the offsets stay at 8241-8243): full build OK, andagc_driver_oversized_dispatchpasses on the RTX 4050 under the validation layer with 0 errors. The Intel UHD wasn't available for the rerun, so its result above is from before the rebase.Checklist
main; no other open PR implements the same functions (except fix(agc): split oversized dispatches with shader offsets #7390, an alternative fix for AGC: splitting a dispatch over maxComputeWorkGroupCount with vkCmdDispatchBase uses a base at or past the limit #7375, see What)NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)