Repository navigation
perf(shader): declare the read-only descriptor tables NonWritable - #6748
Conversation
jotitadev
left a comment
There was a problem hiding this comment.
Linux cross-check on a second driver (lavapipe) plus another NVIDIA card. Head b4d65c4fd, merged onto main 143f13055.
Machine: Ubuntu 26.04, i7-8700, GCC 15.2, Release, SPIRV-Tools off. RTX 3070, NVIDIA 595.99.02 (Vulkan 1.4.329), and lavapipe, Mesa 26.0.3 (LLVM 21.1.8, Vulkan 1.4.335). All runs through ctest.
- Red/green: the
ShaderMemory.cppchange applied on its own overmainfailsagc_shader_memorywiththe flattened SRT is not decorated NonWritable. With the fix,agc_shader_memory|^agc_driver_bdapasses 3/3, three runs, on the 3070. - 3070,
^agc|shader|recompiler(display/flip/lds_atomics64excluded,-j1): 614/615. The one failure isagc_driver_store_over_untracked(the result did not reach a block the CPU left alone). It fails 10/10 on plainmain143f13055here too, so it is not from this PR. - lavapipe, 80 shader/recompiler/image/BDA tests (
^agc_shader|^agc_spirv|^agc_driver_bda|^agc_driver_image|^agc_driver_graphics$|recompiler|shader_cache|...): 80/80 with the PR, same asmain. lavapipe compiles and runs the decorated modules without errors. Some cases skip themselves on the CPU device, the same on both sides: subgroup 8 per-wave image-table cases, EXEC-masked cases, and the BDA multi-dword write execution case.
Not run: pipeline-creation timing or memory, which is the point of this PR. No AMD/Intel, no Windows, no SPIR-V validation (Tools off in this build).
GPUs tested: NVIDIA GeForce RTX 3070 (595.99.02), lavapipe (Mesa 26.0.3), Ubuntu 26.04
Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.
|
Thanks @jotitadev, the lavapipe run is a useful second driver for the decorations, and the red/green on the 3070 matches mine. I added both to the description. AI-assisted: yes |
The flattened SRT, the BDA page table and the image table map are filled by the host and only loaded by the shader, but they were declared as writable storage buffers. A program that also stores through device addresses then gives the driver no way to know those loads are not aliased by its stores, so every descriptor read stays a separate load. Crimson Desert's largest compute program reads the flattened SRT 12,362 times; with the decoration NVIDIA 595.99 creates its pipeline in 90 s with a 4.3 GiB peak, against 200 s and 10.6 GiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b4d65c4 to
7e06103
Compare
|
AMD on Windows: the tests this PR names pass on an RDNA 3.5 GPU. Radeon 860M (RDNA 3.5, driver 32.0.13058.8001), Windows 11 26300, WinLibs GCC 15.2.0 r7, Release.
AI-assisted |
|
Thanks @LeandroLP23 for the AMD run on Windows. I added it to the description. AI-assisted: yes (Claude Code) |
|
Re-checked against current |
What
The recompiler declares three tables as plain storage buffers: the flattened SRT (
flattened_srt), the BDA page table (bda_pagetable) and the image table map (table_map). The host fills them and the shader only loads from them (SpirvMemoryInstructions.cpp,SpirvBdaFault.cpp,SpirvImageEmitter.cpp). Nothing marks them read-only, though. A program that also stores, tobuffersor through a device address, gives the driver no way to know those stores don't alias the tables. So every descriptor read stays a separate load after each store.Crimson Desert (PPSA25286) has compute programs where this adds up. Its largest one, 1,835,706 SPIR-V words, reads
flattened_srt12,362 times. Creating its pipeline on NVIDIA 595.99 takes 200 s and 10.6 GiB of memory, and in the title that compile is where my 16 GiB cap ended every run.The three variables are now decorated
NonWritable. The decoration goes on the variables, not on the block's member, because the block type is shared with the writtenbuffers.Tested
Linux (Ubuntu 24.04, GCC 13.3, Release), RTX 4070 Ti, NVIDIA 595.99.
agc_shader_memory: the recompiled test program'sflattened_srtmust carryNonWritable, and so mustbda_pagetableandtable_mapwhere a program has them. Withmain'sSpirvEmitterHelpers.cppit fails withthe flattened SRT is not decorated NonWritable. It passes 3 of 3 with this change.^agc_|recompiler|shader_cache|request_serializer|spirv, 609 tests, SPIR-V Tools on, so every recompiled module is validated): 608 pass.agc_driver_store_over_untrackedfails the same way onmain5d9dcc4 here; test(agc): keep waiting for the neighbour copy case's result #6385 addresses it.APS5_DUMP_COMPUTE_PIPELINES), as the title compiled them and with only the three decorations added, timed withagc_compute_pipeline_replayand the NVIDIA disk cache off (__GL_SHADER_DISK_CACHE=0). Peak is the process's maximum RSS:NonWritableThe gain is concentrated where a program reads the tables most. The largest program drops from 200 s and 10.6 GiB to 91 s and 4.3 GiB. Others gain a little (112k words: 2.3 s to 1.2 s) or stay within run-to-run variation, and none gets worse beyond that. One run each.
The title still needs more memory than my cap allows at that point, so it doesn't present a frame yet.
@jotitadev cross-checked it on Ubuntu 26.04 (GCC 15.2, Release, SPIR-V Tools off), with this head merged onto
main143f130:ShaderMemory.cppcheck alone failsagc_shader_memorywiththe flattened SRT is not decorated NonWritable, and with the fixagc_shader_memory|^agc_driver_bdapasses 3 of 3.^agc|shader|recompilergives 614 of 615; the one failure isagc_driver_store_over_untracked, which fails onmaintoo.main, so a second driver compiles and runs the decorated modules.Validated on Windows 11 with an AMD Radeon 860M (RDNA 3.5, driver 32.0.13058.8001) by @LeandroLP23 (comment), in one build of
main6efb726 with this head and 13 other pull requests:agc_driver_store_over_untrackedandagc_shader_memorypass 3 runs of 3. 713 of 716agc_*tests pass; the 3 that fail fail the same way onmainthere.Not checked on Intel or on a console.
Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)🤖 Generated with Claude Code