[milescoviello]

What

Since #6674, a direct FMASK image load reads the FMASK stored in guest memory, but only for Fmask32_S8_F8 (8 samples, 8 fragments). The other 12 FMASK formats throw FMASK format is not implemented; before #6674 they all read a synthetic identity. This PR reads every uncompressed FMASK format of up to 8 samples:

• DecodeTextureResource decodes such a T# as an integer Z_64KB_X surface of the format's storage width: R8_UINT, R16_UINT or R32_UINT. That is the same surface conversion fix(agc): read uncompressed FMASK images from guest memory #6674 uses for 32 bits, following PAL's FMASK-to-surface conversion.
• The shader expands the stored value to the 4 bits per sample an FMASK load returns. Mesa's nir_lower_ms_txf_to_fragment_fetch reads ubfe(fmask, sample * 4, 3) for every MSAA mode, and its comment notes that EQAA can return 8 for an unknown fragment. Each format stores 1, 2 or 4 bits per sample. The layouts come from PAL's packed expanded FMASK values: 0xE4 for S4_F4 (2 bits), 0xA4 for S4_F2 (2 bits, with 2 meaning unknown), 0x44443210 for S8_F4 (4 bits, with 4 meaning unknown), and so on. A stored index at or past the fragment count reads 8. Nibbles of samples past the sample count read 0, which is not measured on hardware.
• One runtime image mode covers all these formats (conversionFormat = FmaskExpansion). So a 32-bit image load gains one mode, the same count it had before fix(agc): read uncompressed FMASK images from guest memory #6674.
  • In registration modules, the expansion reads the format from the descriptor at run time.
  • Otherwise it reads it from a new specialization constant, FmaskFormatBase + image. The constant is supplied for every image that has the mode, and it is 0 unless the FMASK mode is the selected one, so other draws don't gain specialized modules.
• The expanded view is bound with identity selectors and the shader applies the T#'s selectors. So the materializer now checks them: X, zero or one, as fix(agc): read uncompressed FMASK images from guest memory #6674 requires.
• Fmask32_S8_F8 keeps fix(agc): read uncompressed FMASK images from guest memory #6674's path. The 16-sample formats, whose expanded value is 64 bits, still throw, as do packed accesses.
• CMASK compression follows feat(agc): expose expanded FMASK for tracked native owners #7948's rules on main: without readable metadata it throws, and with it only an 8-sample, 8-fragment FMASK goes to feat(agc): expose expanded FMASK for tracked native owners #7948's native owner path. A compressed FMASK in one of this PR's narrower formats now throws FMASK CMASK compression is implemented only for 8-sample, 8-fragment FMASK, both in DecodeTextureResource and in the materializer, since nothing decodes its compressed indices. The shader debt entry states the new limits.

Why

Ruffle (PPSA68091, 1.2.0) renders through its own RADV and resolves 4x MSAA in a fragment shader through FMASK8_S4_F4. On main a9c93a0 with #7897 (its start-up fix), the run stops 2 of 2 at 10 s, when the demo starts:

[gpu] driver failure: shader 0x201d95a00 stage 5: FMASK format is not implemented

A diagnostic print of the refused T#: format 161 (Fmask8_S4_F4), dword6 compression bits clear, 2D Z_64KB_X, 40043880 ca100000 810dc1df 91800924 00000000 00000000 00080000 00000000. Ruffle ran past this point on 10-18, before #6674.

With this PR, the run continues to the 120 s limit 2 of 2.

Tested

Rebased onto main f6075c2 as 4feda344e6, after #7948 merged. The conflicts were the two FMASK checks that #7948 changed:

• In GuestTextureResource.cpp and ResourceMaterializer.cpp, the format check is this PR's (any FMASK layout of up to 8 samples), and the CMASK checks are feat(agc): expose expanded FMASK for tracked native owners #7948's (readable, pipe-aligned metadata).
• A new check refuses CMASK compression of the narrower formats.
• agc_driver_graphics' decode test now asks that refusal of each of the eight narrow formats.

On that head (Ubuntu 26.04.1, GCC 15.2, RTX 4070 Ti, 615.71.09), the 81 tests matching ^agc_driver_(fmask|sample_multisample|msaa|multisample|graphics$|image_) pass 3 of 3. Under Khronos 1.4.363 the same set passes on main and on this head; both report the same VUID-VkImageViewCreateInfo-usage-02275 x2 as before, and nothing is new in head.

• @Amoenus (feat(agc): expose expanded FMASK for tracked native owners #7948's author) checked 4feda344e6 and found that the combined guards match feat(agc): expose expanded FMASK for tracked native owners #7948's support boundary. They also ran it, composed with main 9e568a3, on Windows 11 under the 1.4.363 layer, on an NVIDIA RTX 5080 (610.62) and an AMD Radeon iGPU (32.0.21036.18). agc_shader_memory, agc_driver_graphics, agc_driver_fmask_image_reads (3 of 3) and agc_driver_native_fmask pass on both, with no VUID- message, so feat(agc): expose expanded FMASK for tracked native owners #7948's eight-sample native-owner regression still passes.

Earlier: rebased onto main 8b3a6ba as 643bb61490. The one conflict was an include line in RdnaDescriptorFormat.hpp, where #9430 added <span> and this PR adds <optional>; both are kept, and the change is otherwise line for line the same. On that head, Ubuntu 26.04.1 (kernel 7.0), GCC 15.2, NVIDIA RTX 4070 Ti, driver 615.71.09, Release: agc_driver_fmask_image_reads, agc_driver_graphics and agc_shader_memory pass 3 of 3. The results below are from the pre-rebase head.

Ubuntu 24.04 (Linux 6.8), GCC 13.3, NVIDIA RTX 4070 Ti, driver 595.99.02, on main d50ec76.

• agc_driver_fmask_image_reads gains a pass over the eight new formats: S2_F1, S4_F1, S8_F1, S2_F2, S4_F2, S4_F4, S8_F2 (16-bit) and S8_F4 (32-bit).

  • Each format runs with and without registration modules, as 2D and 2D-array views. One round stores PAL's identity value in every pixel and expects its expanded form (for example 0xE4 → 0x3210, 0xA4 → 0x8810, 0x44443210 → 0x88883210). A second round stores varied values.
  • In total it runs 64 new dispatches, with 8,192 component checks.
  • With main's production code it fails 3 of 3 (FMASK format is not implemented). With this PR it passes 3 of 3.

• agc_shader_memory checks that the eight formats select the one expansion mode, that the 16-sample formats still throw, and that a Y selector is refused. agc_driver_graphics checks each format's surface width, that identity selectors are accepted on an expanded view and refused on Fmask32_S8_F8, and that the 16-sample formats are refused. Both pass 3 of 3.

• ctest -R '^(agc_|shader_cache)': 828 of 828 pass. The usual three skip themselves (agc_host_import_sparse, agc_driver_sparse_dma_buf_import, agc_driver_stencil_ref_export).

• Khronos validation layer (LunarG SDK 1.4.363), same machine: VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1 ctest -R '^agc_driver_(fmask|sample_multisample|msaa|multisample|graphics$|image_)' -V on main d50ec76 and on this PR (4660cdd). 73 of 73 pass on each. Neither log reports the layer missing. Both have the same VUID, VUID-VkImageViewCreateInfo-usage-02275 ×2, and none is new.

• PPSA68091 on main a9c93a0 + fix(libc): adopt the guest SDK allocator only when the title replaces the heap #7897 + this PR: the demo runs to the 120 s limit 2 of 2. The demo is an empty SWF, so I also ran Ruffle's own visual tests, the 36 that check frame 1, each injected as the only library game for 32 s.

  • 32 run to the limit. The other 4 stop on the same refusals as on 10-18 (main 385ed71 + fix(agc): take RADV's fill-memory kernel as a buffer fill #6543 + feat(agc): take a full DCC fill of a resident multisampled target as its fast clear #7250, when FMASK loads read the synthetic identity): a write to a resident multisampled DCC target, a compute storage view of a multisampled target, and an initial multisampled DCC key.
  • Against output.expected.png, 29 of the 32 match at a coarse (32×24 block) difference under 2, with the same per-test values as on 10-18.
  • Of the other three, define_bits_lossless2_rgb15 (26.5) and simple_shapes__gradients__repeat (2.5) score the same as on 10-18. filters__bevel varies between runs on these libs (6.7, 14.2 and 14.6); in the worse frames two of its rows haven't been drawn yet.

• SPIR-V size, from the shader replay corpus (466 dumped requests from MXGP 26 and Super Meat Boy 3D), agc_shader_replay --spv at main d50ec76 and at this PR: the same 462 modules come out; 197 of them grow, by at most 7.2 KB each (the expansion is emitted for each FMASK-capable 32-bit load), at most 10.3% on one 69 KB module, and 0.12% in total (150,120,600 to 150,298,672 bytes). Before fix(agc): read uncompressed FMASK images from guest memory #6674 the same loads carried a constant FMASK mode instead.

• tools/check_conventions.py --base upstream/main passes. Two open PRs edit the same lines:

  • feat(agc): expose expanded FMASK for tracked native owners #7948 changes the line after both FMASK format checks (GuestTextureResource.cpp, ResourceMaterializer.cpp).
  • fix(shader): read a 2D MSAA T# through DIM 2D as its sample 0 #7803 adds a debt entry beside the FMASK one.

  I moved two edits so they don't collide with fix(agc): support packed 16-bit storage images #6218 and fix(shader): round normalized storage image writes from the exact product #7666. Whichever of these lands second needs a short rebase.

• Windows 10 Pro 22H2, WinLibs GCC 15.2.0, Release, by @TheSovietWaffle on 4660cdd42: agc_driver_fmask_image_reads, agc_shader_memory and agc_driver_graphics pass 3 of 3 on an NVIDIA RTX 3080 (617.42) and on an AMD Radeon RDNA 2 iGPU (1002:164e, 32.0.21045.5002).

Not tested: the console itself.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none. PPSA68091 also needs fix(libc): adopt the guest SDK allocator only when the title replaces the heap #7897 to start.
• Fixes: none
• Unblocks: PPSA68091 (with fix(libc): adopt the guest SDK allocator only when the title replaces the heap #7897)
• GPUs tested: NVIDIA GeForce RTX 4070 Ti, driver 595.99.02, Ubuntu 24.04; NVIDIA RTX 3080 (617.42) and AMD Radeon RDNA 2 iGPU (32.0.21045.5002), Windows 10 (@TheSovietWaffle)
• AI-assisted: yes (Claude Code)

🤖 Generated with Claude Code

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 8792	docs/dev/debt/shader.md L49-50
PR 9183	core/shader/recompiler/PipelineSpecialization.hpp L44
PR 9889	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp L1866-1867

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 907	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 5021	core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp docs/dev/debt/shader.md
PR 5371	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/PipelineSpecialization.hpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp docs/dev/debt/shader.md
PR 5609	core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp
PR 5925	core/shader/recompiler/CMakeLists.txt
PR 6235	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp docs/dev/debt/shader.md
PR 6280	core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/PipelineSpecialization.hpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp docs/dev/debt/shader.md
PR 6412	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp docs/dev/debt/shader.md
PR 6488	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp docs/dev/debt/shader.md
PR 6539	core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp
PR 6879	core/shader/recompiler/CMakeLists.txt docs/dev/debt/shader.md
PR 6924	docs/dev/debt/shader.md
PR 6977	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp
PR 7201	core/shader/recompiler/CMakeLists.txt
PR 7410	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp docs/dev/debt/shader.md
PR 7550	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 7642	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/CMakeLists.txt core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/PipelineSpecialization.hpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 7684	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp docs/dev/debt/shader.md
PR 7689	core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp
PR 7803	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp docs/dev/debt/shader.md
PR 8004	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/CMakeLists.txt docs/dev/debt/shader.md
PR 8077	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/RdnaDecoder/include/RdnaDecoder/RdnaDescriptorFormat.hpp core/shader/recompiler/RdnaDecoder/src/RdnaDescriptorFormat.cpp docs/dev/debt/shader.md
PR 8281	docs/dev/debt/shader.md
PR 8406	docs/dev/debt/shader.md
PR 8612	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 8998	core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 9020	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp docs/dev/debt/shader.md
PR 9193	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp docs/dev/debt/shader.md
PR 9335	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp docs/dev/debt/shader.md
PR 9356	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/CMakeLists.txt
PR 9387	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/CMakeLists.txt
PR 9431	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp
PR 9432	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp
PR 9508	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/CMakeLists.txt core/shader/recompiler/Optimization/src/DescriptorBindingBuilder.cpp core/shader/recompiler/SpirvBackend/src/SpirvImageEmitter.cpp
PR 9672	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp
PR 9673	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp
PR 9674	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp
PR 9675	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp
PR 9676	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp
PR 9748	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Optimization/src/ResourceMaterializer.cpp docs/dev/debt/shader.md
PR 9891	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp core/libs/prx/libSceAgcDriver/tests/GuestTextureResource.cpp
PR 9892	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp
PR 9893	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp
PR 9895	core/libs/prx/libSceAgcDriver/Graphics/src/GuestTextureResource.cpp

[TheSovietWaffle]

Windows data point: RTX 3080 (NVIDIA 617.42) and AMD Radeon RDNA 2 iGPU (1002:164e, 32.0.21045.5002), Windows 10 Pro 22H2, WinLibs GCC 15.2.0, Release, head 4660cdd42: agc_driver_fmask_image_reads, agc_shader_memory and agc_driver_graphics pass 3 of 3 on both GPUs.

AI-assisted: yes (Claude Code).

[milescoviello]

Thanks @TheSovietWaffle. The AMD run is useful here, since the expansion now runs on both vendors' compilers. I added it to the description.

AI-assisted: yes (Claude Code)

[milescoviello]

Windows run of this head (4660cdd42b) and its merge base d50ec76da0: Windows 11 Enterprise, NVIDIA GeForce GT 1030 (driver 582.66), WinLibs GCC 15.2.0 r7 (UCRT), Release, BUILD_TESTING=ON.

test	merge base (3 runs)	this PR (3 runs)
agc_driver_fmask_image_reads	pass (0.9 to 4.0 s)	pass (1.6 to 3.6 s)

Both builds succeeded.

AI-assisted: yes (Claude Code)

[milescoviello]

Rebased onto main 8b3a6ba as 643bb61490, after #9430 merged. The only conflict was an include line in RdnaDescriptorFormat.hpp, and both includes are kept. The FMASK, graphics and shader memory tests pass 3 of 3 on the new head.

AI-assisted: yes (Claude Code)

[milescoviello]

Rebased over #7948, now on main f6075c2 as 4feda34. Its two FMASK checks conflicted with this PR's, and I combined them as agreed earlier. @Amoenus, please check that this matches your intent:

• The format check is this PR's: any FMASK layout of up to 8 samples.
• The CMASK checks are feat(agc): expose expanded FMASK for tracked native owners #7948's: readable, pipe-aligned metadata.
• New: CMASK compression of the narrower formats throws FMASK CMASK compression is implemented only for 8-sample, 8-fragment FMASK, in DecodeTextureResource and in the materializer. feat(agc): expose expanded FMASK for tracked native owners #7948's native owner path only handles 8 fragments, and nothing decodes the narrower formats' compressed indices.
• The decode test in agc_driver_graphics checks that refusal for each of the eight narrow formats.

The 81 FMASK, MSAA and image tests pass 3 of 3. Under the 1.4.363 layer they match main, with the same pre-existing 02275 x2 on both sides and nothing new. Details are in Tested.

AI-assisted: yes (Claude Code)

[Amoenus]

@milescoviello yes: I checked 4feda344e661078367b979327d38e2b24b107d70, and the combined guards match #7948's intended support boundary.

• The broader format gate enables the uncompressed FMASK layouts up to 8 samples. Fmask32_S8_F8 retains the existing raw-word/native-owner path.
• The decoder retains the readable, non-write-compressed, pipe-aligned CMASK requirements. Both the decoder and materializer explicitly reject CMASK compression for the eight narrower formats with the quoted diagnostic; this avoids routing unsupported compressed indices through the 8-fragment owner.
• The graphics decode regression covers those eight refusals. I found no unintended change to native-owner association/invalidation in this combination.

I also built and tested literal main 9e568a389b5fedc5b925d1a2c8777b48a16bd6e9 against your head composed with that same main. These candidate results are for that composition, not the unchanged PR head.

Selected checks, per source	Result
agc_shader_memory, agc_driver_graphics	Both pass
agc_driver_fmask_image_reads	3/3 on each GPU
agc_driver_native_fmask	1/1 on each GPU

20 test processes total: 4 CPU and 16 GPU. All pass, with no skips/timeouts or Vulkan validation warnings/errors/VUIDs. In particular, the existing eight-sample native-owner regression still passes on both vendors. Main does not contain the new narrow-format cases.

Environment: Windows 11 Enterprise build 26200; WinLibs GCC 15.2.0 r7/UCRT, Release; NVIDIA GeForce RTX 5080 driver 610.62 (32.0.16.1062), AMD Radeon(TM) Graphics driver 32.0.21036.18. Verbose serial CTest, 30-second limit; ANYPS5_REQUIRE_VULKAN=1, ANYPS5_NO_SHADER_CACHE=1, APS5_NO_PIPELINE_JOURNAL=1, VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation (SDK 1.4.363.0). Actual device and inserted validation layer verified per GPU fixture; driver caches were not controlled.

This is focused preservation/capability validation, not a repeat of your full 81-test matrix, a title run, or independent confirmation of console semantics. No further guard change requested from this review.

AI-assisted review and validation: yes (Codex).

[milescoviello]

@Amoenus thanks for checking the combined guards, and for the NVIDIA and AMD runs on Windows. I added them to Tested.

AI-assisted: yes (Claude Code)