[MichonGoddijn231849]

What

Draws whose PA_SC_AA_MASK was not all ones were refused with sample masks are unsupported. At one sample, each pixel of the 2x2 quad keeps a single coverage bit (bit 0 of its 16-bit field). So the mask chooses which pixels of every quad a draw may write. GTA V uses 0xffff0000/0x00000000 to render a target one quarter of its pixels at a time (pixel (1, 0) of each quad).

• Driver: DecodePixelStageInfo reduces the two registers to a 4-bit quadPixelMask (bit (x & 1) | (y & 1) << 1). The state rule and the draw precheck now refuse partial masks only on multisampled draws.
• Recompiler: ShaderPixelStageInfo::quadPixelMask. A fragment shader whose mask is not full:
  • starts by killing the invocation unless the bit for its gl_FragCoord quad position is set (the FragCoord input is added for it);
  • runs without early fragment tests, so a killed pixel writes no depth.
  • The mask is part of the recompile cache key and the serialized request (version 8).

Tested

Windows 11 Pro, AMD Ryzen 5 7600, NVIDIA GeForce RTX 5080 (driver 32.0.16.1074), 32 GB DDR5-5600, MinGW-w64 GCC 15.2.0.

• ctest: all tests pass. New cases:
  • agc_driver_quad_pixel_mask draws a full-screen triangle on the GPU with masks 0xf, 0x2 and 0x9, and checks every pixel;
  • the state decode and the precheck accept a single-sample mask, and PA_SC_AA_MASK decodes to the quad mask;
  • the mask survives request serialization and changes the recompile cache key.
• GTA V (PPSA04264), Prologue: about 1,800 draws per 10 s were refused with sample masks are unsupported. They now render, and the scene looks as before. Tested on my branch that combines this change with my other open PRs.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: feat(agc): vertex clip and cull distances #423 (request serialization version 7; this uses 8), which depends on fix(agc): fragment SampleId, Layer, FragDepth and SampleMask interfaces #375 and fix(shader): packed pixel ancillary as SampleId and Layer, gl_Layer via Geometry, Z export formats #295; their commits are included here until they merge
• AI-assisted: yes. Written with Claude (Anthropic) under my direction; I reviewed and tested it.

🤖 Generated with Claude Code

[oneandonlydean]

The quad mask decode matches PA_SC_AA_MASK at one sample, and the GPU test is nice. One exactness point: the mask is applied with an OpKill at the top of the shader. On the hardware, a pixel that AA_MASK leaves uncovered still runs as a helper lane of its quad, so derivatives in the covered pixels stay defined. After OpKill, Vulkan no longer keeps that invocation as a helper, so implicit-LOD samples and other derivatives in the remaining pixels are undefined, and with one covered pixel per quad that's every quad. OpDemoteToHelperInvocation (core in Vulkan 1.3, shaderDemoteToHelperInvocation) keeps the lane as a helper and would match the hardware. Without demote support the draw could be refused when the pixel program uses derivatives. Could the test also sample a texture with implicit LOD in the masked draw? Once #375 and #423 land, a rebase would leave just this commit.

[MichonGoddijn231849]

Good catch, thanks. Done in 7316e3a:

• When the device has shaderDemoteToHelperInvocation, the mask now emits OpDemoteToHelperInvocation. The device enables the feature and extension when available, and the validator accepts capability 5379 only then. A masked-off pixel's starting EXEC bit also reads as a helper, as on the hardware.
• Without demote it keeps OpKill only when the pixel program reads no neighbouring lanes: no implicit-LOD samples, LOD queries, DPP, swizzles, bpermute, permlane16 or readlane. Otherwise the draw is refused with a message naming the mask and the missing feature.
• The GPU test now samples a 9-level texture with implicit LOD at LOD 2 in the masked draw, for masks 0xf, 0x2 and 0x9. Covered pixels must match the level-2 colour exactly, and uncovered ones stay zero. On a GPU without demote it expects the refusal.

One honest limit: on my RTX 5080 the test passes even with OpKill forced, because this driver evidently keeps killed lanes as helpers. So it shows the demote path is correct, not that kill is wrong there. Treating readlane and DPP as derivative-like reads for the refusal is my own choice; the docs don't define it. I'll rebase once #375 and #423 land.

[oneandonlydean]

Thanks, that's what I meant. Demote is only emitted when the device enabled shaderDemoteToHelperInvocation, the validator only accepts it then, the uncovered pixels start with their EXEC bit clear like helpers do, and the test now samples at LOD 2 with one covered pixel per quad, which only works with the helpers alive.

One small thing for the path without demote, not blocking: a program that runs s_wqm_b64 exec, exec and then a ballot or readfirstlane can also see the helper lanes, which OpKill removes. Most such programs sample with implicit LOD and are refused already, but adding WqmU64 to ReadsNeighbourLanes would close it. I'll look again once #375 and #423 land and this is down to its own commits.

[MichonGoddijn231849]

Done in bed8cb0: WqmU64 now counts as a neighbour-lane read, so without demote a quad-masked program that enables whole-quad mode is refused like one that samples with implicit LOD. The refusal message names whole-quad mode too.

Also rebased onto current main together with the PRs it carries (#295, #375, #423, all rebased today). The only conflict was the include list in SpirvAnalysis.cpp. Build and ctest are clean apart from the failures main has locally.

[oneandonlydean]

Thanks, bed8cb0 does it: WqmU64 counts as a neighbour-lane read, so a quad-masked program that enables whole-quad mode is refused without demote, and the message says so. It's down to the stack now. #752 has landed since, and #375's ShaderValidation.cpp line conflicts with it (details on #375).

[MichonGoddijn231849]

Rebased onto current main and the rebased #375 and #423 (now eecdf09). Two conflicts:

• VulkanDevice.cpp: main's 64-bit buffer atomics feature and this PR's demote feature both chain into the device features at the same place. Both are kept, each with its own block.
• CacheKey.hpp, ShaderStageInputInfo.hpp and ShaderInputInfoBuilder.cpp: main's conservativeZExport and this PR's quadPixelMask sit side by side.

The request version also moved: #423 now uses version 8 (main took 7), so the quad pixel mask is read from version 9. Builds; the driver tests pass apart from agc_driver_scalar_sop1, which fails without this PR too.

[MichonGoddijn231849]

Parked as a draft to keep my rebase load sane after #639. I'll rebase it when someone wants to review it.

[fdfawef324]

I recovered this implementation on current main (18c438b), retaining your three original commits and authorship, and prepared a verified branch at fdfawef324:codex/samplemask-no-demote-review (e8c1057). It also covers uniform and varying multisample PA_SC_AA_MASK fields, with explicit refusals for unsupported ordering and centroid cases. I would like to offer that branch here for review/integration because the single-sample implementation overlaps this PR.

Adversarial verification found that the no-demote guard also needs ReadFirstLane, Ballot and PermuteU32: explicit EXEC can enable uncovered helpers, and a constant WQM instruction can fold away before analysis. The added CPU regression fails on the recovered implementation and passes with those three classifications. Unmasked programs, supported demote paths, and the constant-colour kill fallback are checked as controls.

The real DCB test fails at the unsupported-mask runtime path without the implementation. With it, non-skipped GPU readbacks pass for driver mask changes, 1x depth/stencil suppression and fragment stores, 4x uniform/varying coverage, per-sample stores, and implicit-LOD helper preservation. Current-main shader-memory and focused no-demote checks also pass; conventions are clean. GTA gameplay validation is still pending.

GPUs tested: NVIDIA GeForce RTX 3060 12 GB, driver 591.86 (32.0.15.9186), Vulkan 1.4.325, Windows Server 2025 10.0.26100.

AI-assisted: yes.

Co-Authored-By: Codex noreply@openai.com

[tibzejoker]

@fdfawef324 the GTA gameplay check for codex/samplemask-no-demote-review (e8c1057), the one you listed as pending. GTA V (PPSA04264), Windows 11, RTX 4090, MinGW-w64 GCC 15.2.0, Release.

Base: main 8d98e88 plus the open PRs GTA needs to reach Story: #5003, #4878, #1186, #1189, and a local port of #637 (without it, write watch crashes the title on current main). Your five commits went in as they are. The only conflict was one test registration line in tests/Graphics.cpp, where I kept both lines.

• Without these commits, Story stops with sample masks are unsupported. With them, the prologue runs for 150 s with no sample-mask error, and the scene, HUD and minimap render correctly. That is the same picture as with Michon's three commits on our older stack.
• It runs at about 1 FPS (frame counter delta), as before.
• The only dispatches the run still skips are three bindless table shaders (0x25b456800, 0x25b458e00, 0x25b45a200), which need feat(agc): implement image and sampler tables indexed at runtime offsets #757.

AI-assisted (Claude Code).

[MichonGoddijn231849]

@fdfawef324 thanks for recovering this on current main and extending it to the multisample fields, and @tibzejoker for the GTA V check. Your branch is further along than this one. Please open it as its own PR, since you can answer the review on the new parts. Link it here and I'll close this one in its favour. If you'd rather I carry it, say so and I'll push your branch here instead.

[oneandonlydean]

Opened #6015 with @fdfawef324's codex/samplemask-no-demote-review rebased onto current main. It includes @MichonGoddijn231849's three commits, and everyone keeps authorship. The request version is now 26/27 because main's dither flag took 25. On GTA V, Story Mode now gets to the prologue title card. The next stop there is a CB resolve into a non-resident target. Michon, you can close this one in favour of #6015, as you offered.