Skip to content

feat(shader): give reads the masked-select pass keeps the whole pixel ancillary word - #4031

Merged
oneandonlydean merged 3 commits into
boykopovar:mainfrom
AlxcBTW:carry/pixel-ancillary-word-v3
Oct 13, 2026
Merged

oneandonlydean merged 3 commits into
boykopovar:mainfrom
AlxcBTW:carry/pixel-ancillary-word-v3

Conversation

@AlxcBTW

@AlxcBTW AlxcBTW commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

What

Refs #2548 and #3504. With this change a select that the masked-lane walk keeps no longer stops the recompile, because every read the walk keeps gets the whole word. That lets #3550's walk budget stay low without breaking titles that need the select gone for the ancillary lowering (Stray's 0x2013010000, see #3504).

A select the masked-lane walk can't drop still stops the recompile on #3279. Palworld's 0x1b40e5c000 is one (Rohan03x on #2548: the kept select is reached through a ReadFirstLane, which reads other lanes), and so are Minecraft Dungeons II's 0x190095e900 and 0x190095fc00. With this commit every read the walk keeps gets the whole word:

  • prim type at bits 0-1: 0 point, 1 line, 2 triangle, 3 rectangle, from the draw (RasterPrimitiveOf);
  • sample index at 8-11 (SampleId, always 0 here);
  • render target index at 16-28 (Layer);
  • every other bit 0.

The layout is from the RDNA 3/3.5 ISA guides. The GCN register guides agree for layers below 2048, Mesa reads sample and layer from those bits, and AMD's LLPC reads the prim type from bits 1:0 with the same encoding. Nobody has checked it on gfx1013; TechnicalDebt.md says so.

Where the primitive class comes from (owner's call)

  1. This PR: a pipeline specialization constant, the way dual-source blending does it (feat(agc): implement dual-source blending from the MRT1 export #1492). The class is a request field. It is folded into the specialized module and is part of the result memo, the draw key and the serialized request (version 17 → 18). The class costs nothing per pixel: it is a constant in the module. Only shaders that read it get a module per class: ShaderInfo records the read (disk cache format 27 → 28), and a shader that reads only the sample or layer field keeps one module. The draw sets the class only when the pixel shader loads the ancillary VGPR, so other draws keep their draw-cache entries.
  2. Run-time per-draw data: needs a layout change, so not done. Nothing the fragment shader receives today carries the class. Its push data is its own compact shader-data block (user SGPRs, buffer offsets, image metadata: IrBindingLayout in DescriptorBinding.hpp), and no fragment builtin can tell a rect list (drawn as tessellated triangles) from a triangle. The smallest run-time variant adds one dword to that block in shaders that read the class, and patches it per draw from Graphics::State the way ShaderResources patches guest buffer offsets (pushByte for push constants, dataByte for the ShaderData buffer). The module key, result memo, draw key and request format could then stay as on main. But the shader-data layout changes (IrBindingLayout, disk cache format), the block spills to the ShaderData buffer one dword sooner, and every pixel wave loads the class instead of using a constant.
  3. Specialization when the pipeline is created (not built). The driver could pass constant 8233 through VkSpecializationInfo from Graphics::State, whose pipeline key already holds the topology and rect-list state. The request format, module key, result memo and draw key would then stay as on main. Prepared modules would have to leave that one constant open (buildSpecializedModule requires every SpecId today), and the GPL fragment-library key would need the class for shaders that read it.

Tested

main d592d54e1 is merged into the branch (no rebase). The head is e24413c44. main writes request version 17 with stencilRefExportEnable as its last pixel field, so rasterPrimitive is written after it and read from version 18: a version 17 request from main still reads, and ShaderMemory.cpp checks the version 18 prefix, rejects 19 and truncates over all 20 trailing pixel bytes. The disk cache format is 28. The conflict resolutions match @blawesom's rebase (771c00783) except for that version.

#4150, merged since, added agc_driver_relative_ancillary. Its last case expected a whole-word read through v_movrels to be refused; #4150 notes that this PR composes that word. The one commit on top of the merge turns that case into a check of the composed word: the shader converts the word to float and exports it, and a triangle draw's word is 2, so the pixel is 255, with and without the implicit position inputs in front of the ancillary VGPR. In TechnicalDebt, the relative addressing entry no longer says that whole-word uses throw.

  • Windows 11, RTX 5070 Laptop (NVIDIA; the AMD iGPU is hidden from the Vulkan loader), with test binaries built on a second Windows 11 machine (MinGW GCC 15.2.0, Release): agc_driver_pixel_ancillary_word, agc_driver_relative_ancillary, agc_driver_graphics, agc_driver_dual_source_blend and agc_driver_alpha_to_coverage pass. relative_ancillary failed on the bare merge (77f0a568b), before the test update, with "an unsupported ancillary word was accepted".
  • Fork CI, run 38278984147, with e24413c44 merged into the fork's main: Linux passes 968 of 968 (lavapipe; 43 skipped for features lavapipe lacks), including agc_driver_pixel_ancillary_word, agc_driver_relative_ancillary, pixel_ancillary and agc_shader_memory. Windows passes 974 of 974, with the GPU tests skipped for lack of a device.
  • The same build machine, non-GPU ctest -E '^agc_driver|^video_out|^guest_video': 618 of 619 pass, including pixel_ancillary, agc_shader_memory and agc_shader_disk_cache. log_macros fails there on every branch and passes in CI.
  • tools/check_conventions.py --base origin/main passes.

Earlier, rebased onto main a11c8f7f7 as one commit (db92ece3c):

Before the rebase (on main plus #3279):

Windows 11, RTX 5070 Laptop GPU (NVIDIA driver; the AMD iGPU is hidden from the Vulkan loader):

Checklist

🤖 Generated with Claude Code

@github-actions github-actions Bot added shader core/shader docs agc libSceAgc* labels Oct 11, 2026
@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 12, 2026
@blawesom

Copy link
Copy Markdown
Contributor

Rebase handoff — rebased onto current main, authorship unchanged.

Four files conflicted, all the additive-field shape: main landed stencilRefExportEnable where this PR threads rasterPrimitive through the pixel-stage info, serializer and tests. The resolution keeps both — both fields, both serialization entries, both round-trip checks, and both test families (main's StartingQuad series and this PR's whole-word series).

Verified on Linux (CachyOS, kernel 7.2.9, GCC 16.2.1, Ninja, Release, BUILD_TESTING=ON, AMD RX 9070 XT / RADV):

  • builds; tools/check_conventions.py --base origin/main passes
  • full suite: 955/955 passed

To pick it up:

git fetch https://github.com/blawesom/AnyPS5 rebase/4031
git checkout <your branch>
git merge --ff-only FETCH_HEAD

Head is 771c00783.

@AlxcBTW

AlxcBTW commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

@blawesom thanks for the rebase and for running the whole suite on RADV.

The branch now has main merged into it rather than rebased, so its history stays as it was. The conflicts are resolved the way you did it: both fields, both serializer entries, and both test families (the StartingQuad series and the whole-word series).

The one difference is the request serialization version. main already writes version 17 with stencilRefExportEnable as the last pixel field. If rasterPrimitive is also read from 17, a version 17 request written by main comes up one byte short: the reader expects the primitive class after the stencil flag and fails with "truncated data". This head writes version 18, reads the stencil flag from 17 and the primitive class from 18. The ShaderMemory.cpp checks follow: the NVNQQRIAAAA= prefix, rejection of version 19, and truncation over all 20 trailing pixel bytes, so the primitive class byte is covered too. The disk cache format stays 28.

AI-assisted: yes (Claude Code)

@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 12, 2026
@github-actions github-actions Bot mentioned this pull request Oct 12, 2026
9 tasks done
@blawesom

Copy link
Copy Markdown
Contributor

Verified your merged head (e24413c44): builds, conventions pass, and the full suite is 958/958 on RADV — including the serializer round-trips and the new version-18 coverage.

And credit where due: the version bump is a catch my rebase missed. I merged both fields but kept the primitive class read at version 17 — which is exactly the one-byte-short truncation you describe for main-written version-17 requests. Writing 18 and splitting the reads (stencil from 17, primitive from 18) is the right wire behavior, and the NVNQQRIAAAA= + version-19 rejection + 20-byte truncation coverage settles it. From my side this is merge-ready.

@LeandroLP23

LeandroLP23 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Ran this on AMD hardware, since the GPU tests in CI run only on lavapipe: Windows 11 (26300), Radeon 860M (RDNA 3.5, driver 32.0.13058.8001), WinLibs GCC 15.2.0, Release, BUILD_TESTING=ON. Head dd8403112 merged onto main 584cc4c57.

GPU tests (ctest -R '^agc_|pixel_ancillary' -j 1, test shader cache emptied first):

Build Result Failing
main 584cc4c57 445/446 agc_driver_relative_ancillary
main + this PR 445/447 agc_driver_relative_ancillary, agc_driver_pixel_ancillary_word

agc_driver_pixel_ancillary_word is this PR's new test, and it fails 5 runs of 5:

AGC graphics: ANCILLARY alone, triangle: pixel (0, 0) holds (nan, nan, nan, nan)

The target is filled with 0xff before the draw, so NaN means pixel (0, 0) was never written. The draw didn't land there, as opposed to writing a wrong word.

With APS5_NO_PIPELINE_LTO=1 it still fails 3 of 3, but not always on the same case: twice on "ANCILLARY alone, triangle" and once on "ANCILLARY after the perspective center, rectangle". Each time the pixel is unwritten. APS5_NO_GPL=1 and APS5_SPIRV_OPT=none give the triangle failure too.

agc_driver_relative_ancillary (from #4150) already fails on plain main on this GPU, 3 of 3: wave 32 packed read / relative ancillary draw produced an incorrect pixel: 64. That test fills its target with 0x40 (64) before drawing, so this is the same signature: the draw left the pixel unwritten. Both tests draw with a pixel shader that reads the ancillary VGPR, and on this GPU those draws leave pixels unwritten on main too. So it doesn't look specific to this PR. I'll report the main failure separately.

Other tests: ctest -E '^agc_|pixel_ancillary' -j 4: 572/572 with this PR.

Titles. Seven homebrew titles that render on main were run for 60 s each, with and without this PR: Stremio Plus, Capy Mobile, DriftCheck, EVO Player, Jelly5, Emby5 and OpenRCT2. All seven run the full 60 s either way, with no stop and no new error. None of them reaches the shaders this PR is for, so this only checks for regressions.

I haven't looked into why the draw leaves pixels unwritten on this GPU. I can run a diagnostic build or more cases if that helps.

AI-assisted: yes (Claude Code)

@AlxcBTW

AlxcBTW commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

@LeandroLP23 thanks for running both tests on the 860M, and for noticing that the main failure has the same signature. That pointed straight at the cause.

It is not the sample index and not invalid SPIR-V from this PR. It is AMD's shader compiler (LLPC) programming the position exports inconsistently when a pixel shader reads gl_Layer and the vertex shader writes gl_ClipDistance but not gl_Layer.

Both failing draws read the reconstructed ancillary word, so their pixel shaders read SampleId and Layer. On main, the packed read in relative_ancillary also builds the whole word from both. Since #3878 (the all-zero position clip), every vertex shader AnyPS5 emits that exports a position also writes gl_ClipDistance[0]. With that pair, the 2025 LLPC:

  • adds a position export for the missing layer (exp pos1 off, off, v0, off, only z) and moves the clip distance to pos2;
  • but derives SPI_SHADER_POS_FORMAT and PA_CL_VS_OUT_CNTL from the vertex shader's own outputs: two position exports, no misc vector, the clip distance vector at pos1, and CLIP_DIST_ENA_0 and CULL_DIST_ENA_0 set.

So the rasterizer takes clip distance 0 from pos1.x, which the shader never writes. It holds whatever was left in the position buffer, and when that value is negative the triangle is clipped or culled. That is why the target keeps its fill value, and why the failing case changes between runs.

How I checked it, all on the CPU:

  • I dumped the SPIR-V of both tests from lavapipe (MESA_SPIRV_DUMP_PATH). spirv-val --target-env vulkan1.3 passes on every module, and the validation layers on lavapipe report nothing for either test.
  • The built-ins are within the spec. SampleId turns on sample shading. Layer reads 0 when the pre-rasterization stage doesn't write it. The geometryShader and sampleRateShading features are enabled. The interface has no user varyings.
  • I compiled the exact vertex and pixel modules for gfx1152 with the amdllpc that ships in Radeon GPU Analyzer:
amdllpc from Position exports Registers
RGA 2.12 (March 2025), 2.13 (June 2025) pos0, pos1 (layer, z only), pos2 (clip) 2 exports, no misc vector, clip vector at pos1
RGA 2.14 (November 2025), 2.14.2 pos0, pos1 (clip) 2 exports, clip vector at pos1

With 2.13, the same clip distance with a pixel shader that reads only gl_SampleID, or no built-in at all, compiles consistently. Only the gl_Layer read adds the extra export. In the public LLPC source (April 2025), LowerInOut.cpp forces useLayer for the pixel shader's sake, while RegisterMetadataBuilder.cpp only looks at builtInUsage.vs.layer. Going by the version numbers, your driver (32.0.13058) comes from the same 24.30 branch as Adrenalin 25.3 and 25.4, which is before the change.

A minimal repro outside AnyPS5: the vertex shader writes gl_Position and gl_ClipDistance[0] = 0.0 but not gl_Layer, the fragment shader writes vec4(float(gl_Layer), 0.0, 0.0, 1.0), and a full-screen triangle draws into a target pre-filled with a sentinel. On an affected driver some or all pixels keep the sentinel. Dropping either the clip distance or the gl_Layer read makes it correct.

One check would confirm it on your machine. In core/shader/recompiler/SpirvBackend/src/SpirvMemory/SpirvEmitterHelpers.cpp, in DefineOutputs, change if (exportsPosition && (stage == IrShaderStage::Vertex || stage == IrShaderStage::TessellationEvaluation)) to if (false && exportsPosition && (stage == IrShaderStage::Vertex || stage == IrShaderStage::TessellationEvaluation)). That drops the zero-position clip distance. Rebuild and run ctest -R "relative_ancillary|pixel_ancillary_word" --output-on-failure a few times. If both pass on every run, this is the cause. agc_driver_zero_position_vertex will fail with that patch, which is expected. A current Adrenalin driver should pass without the patch, but I could only check the compiler from RGA, not the Windows driver itself.

If it confirms, AnyPS5 can keep clear of this on older drivers. When the pre-rasterization stage doesn't write Layer, the layer field is 0 by the spec, so the pixel shader doesn't need to read gl_Layer at all. That would be a separate PR.

AI-assisted: yes (Claude Code)

@LeandroLP23

Copy link
Copy Markdown
Contributor

@AlxcBTW the AMD failure above comes from reading the unwritten Layer builtin (#4800). #4842 replaces the render target index field with 0 in lowerPackedAncillary. With #4842 merged under this branch and the same 0 at your whole-word read (insertBuiltin(..., StageInputKind::Layer) shifted by 16), agc_driver_pixel_ancillary_word passes 3/3 on the Radeon 860M. Whichever of the two merges second needs that line and the pixel_ancillary cases reconciled; happy to do it on my side if this one goes first.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 12, 2026
@oneandonlydean

Copy link
Copy Markdown
Collaborator

Three things on dd84031 before the question of the unknown bits:

  • Request version. main reads target.vertexSubgroups at version 18 since fix(recompiler): run vertex programs as one-lane waves without vertex subgroups #4470 (RequestSerializer.cpp:793). This head reads rasterPrimitive at 18 as well (:387), so a version-18 request written by main is read one byte short. It needs 19, with the prefix and reject-version checks in ShaderMemory.cpp moved; it is the same collision you caught at 17.
  • Layer. ComposePackedAncillary still inserts GetBuiltin Layer << 16, so agc_driver_pixel_ancillary_word fails on the 860M as LeandroLP23 showed. fix(shader): read the ancillary render target index as 0 #4842 makes that field 0 on main; it should go in first and this take the same 0.
  • It conflicts with main in docs/dev/debt/shader.md.

On the word itself nothing has changed since #1179: bits 2-7, 12-15 and 29-31 are zero by argument, not by measurement, and that is the rule adriadam10 set there, so it is his or the owner's call and not mine.

Kena's remaining shader may not need the word at all. (used by SelectU32 1065353216 ?) is a select between 1.0f and the register, which reads like the VGPR reused for a float with the word left on the other arm, and that is a case for one more rule in the masked-select pass. @fma-rc could you attach the .req for 0x20b6440000, or the IR around that select? If a rule covers it, Kena compiles exactly and this PR is left with the shaders that really export the word, if there are any.

@oneandonlydean oneandonlydean self-assigned this Oct 12, 2026
AlxcBTW and others added 2 commits October 13, 2026 13:06
… ancillary word

A packed ancillary word that still reaches an export, or any other read
the masked-select pass cannot drop, no longer stops the recompile. After
that pass every remaining use gets the composed word: the primitive class
at bits 0-1, SampleId at bits 8-11 and Layer at bits 16-28, with every
other bit zero. Those lanes do export the ancillary VGPR on the console,
so the word has to be right, not just present.

The primitive class (0 point, 1 line, 2 triangle, 3 rectangle) comes from
the draw. RasterPrimitiveOf maps rect lists to 3, line lists on the vertex
path to 1 and everything else the driver accepts to 2, because its
geometry and tessellation paths only output triangles. It reaches the
shader as pipeline specialization constant 8233, the way dual-source
blending does: a request field that is folded into the specialized
module and is part of the result memo, the draw key and the serialized
request (version 17).

Only shaders that read the class pay for it. ShaderInfo records the read
(disk cache format 28), and only those shaders put the class in their
module key, so a shader that reads just the sample or layer field keeps
one module for every primitive class. The draw sets the class only when
the pixel shader loads the ancillary VGPR, so other draws keep the draw
cache entries they have today.

The VRS rates (bits 2-5) and the eye id (bit 7) are 0 because the driver
supports neither VRS nor multiview. SampleId is 0 because the driver
rejects MSAA and per-sample shading, and Layer is 0 while it ignores
vertex layer exports. TechnicalDebt.md records the layout sources and
these assumptions.

Tests:
- pixel_ancillary builds an export to target 0 whose two channels select
  between a packed value and a phi carrying the word. It checks the word
  for each primitive class, sample and layer, checks that the three
  shapes the masked-select pass keeps now compile with the composed word,
  and that a program whose select the pass drops reads no class.
- agc_driver_pixel_ancillary_word exports the word from a draw and reads
  back 2, 3 and 1 for triangle, rectangle and line. A shader that reads
  only the sample field shares one module across the classes, and the
  whole-word shader gets one per class. It also checks RasterPrimitiveOf.
- agc_shader_memory covers the version 17 round trip, and
  agc_shader_disk_cache the new flag.

Both ancillary tests fail with the old refusal when the composition is
disabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sed word

The relative-ancillary test expected the whole-word read through
v_movrels to be refused, which was main's behaviour before the word was
composed. It now converts that read to float and exports it: the
composed word of a triangle draw is 2 (primitive class 2, sample 0,
layer 0), so the pixel is 255, with and without the implicit position
inputs in front of the ancillary VGPR. TechnicalDebt's relative
addressing entry no longer says that whole-word uses throw.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fma-rc

fma-rc commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

@oneandonlydean here is the request for 0x20b6440000: https://github.com/fma-rc/AnyPS5/raw/evidence/kena-shader-20b6440000/shader_20b6440000.req

It is the file APS5_DUMP_SHADERS wrote, unchanged: 936 code words, 30 user data words, 2 memory regions, wave64. agc_shader_replay built from main d38d98b (with one unrelated local change in the driver's texture code) still stops on it with packed pixel ancillary input has an unsupported live use (used by SelectU32 1065353216 ?).

I have no IR dump at the failing point, the tool only prints the instruction listing for it. I can post that listing, or the other 16 Kena requests, if either helps with the select rule.

…se the ancillary word with render target index 0

main reads target.vertexSubgroups at request version 18 since boykopovar#4470, so
reading the pixel primitive class at 18 as well read a version 18
request written by main one byte short. The class is now written and
read at version 19. agc_shader_memory loads a version 18 request as
main writes it, and it fails with "truncated data" when the class is
read at 18.

The composed word no longer reads the Layer builtin for bits 16-28.
Draws render to a single layer and guest layer exports are ignored, so
no stage writes Layer, and AMD's Windows driver drops the writes of a
fragment shader that reads it unwritten (boykopovar#4800). The field is the
constant 0 and pixel_ancillary checks that the composed program has no
Layer read. The debt entry names the bits that have no source and are
assumed zero (2-7, 12-15, 29-31), and says that a change which makes
draws export a layer must restore the read.
@oneandonlydean
oneandonlydean force-pushed the carry/pixel-ancillary-word-v3 branch from dd84031 to c52321f Compare October 13, 2026 00:34
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
#3763 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L1811, L1816
#4451 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawDecode.cpp L117
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawLookup.cpp L72
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawRegisterKey.cpp L108
core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L1811, L1813-1814, L1816, L1905-1906
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp L328, L377, L739, L770
core/shader/recompiler/Recompiler.hpp L144
#4706 docs/dev/debt/shader.md L100-102

@oneandonlydean

Copy link
Copy Markdown
Collaborator

Pushed the rebase and one commit on top (c52321f), your two commits kept as they are:

  • Rebased onto main; the debt entry is now in docs/dev/debt/shader.md.
  • The primitive class is read at request version 19. Main already reads vertexSubgroups at 18, so a version-18 request from main was misread (RequestSerializer: truncated data); there is a test with one.
  • The whole word is composed as primitive | (SampleId << 8), without the Layer read, since draws render to one layer. The field-extract path still reads it; that is fix(shader): read the ancillary render target index as 0 #4842, which goes on top of this.
  • The debt entry now says which bits have a source (0-1, 8-11) and that the rest are assumed zero, not measured.

ctest -R "agc|shader" is 462/462 on Linux/NVIDIA, and @fma-rc's request for 0x20b6440000 fails on main with the SelectU32 refusal and recompiles with this (71528 words). Not run on AMD or Windows here; full build is running.

@oneandonlydean
oneandonlydean merged commit 510a218 into boykopovar:main Oct 13, 2026
8 checks passed
@AlxcBTW

AlxcBTW commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

@LeandroLP23 thanks for #4842, and for confirming agc_driver_pixel_ancillary_word 3/3 on the Radeon 860M with it merged under this branch. @oneandonlydean thanks for c52321f.

With that commit, #4031 reads the render target index field (bits 16+) of the whole composed word as the constant 0, with the same reasoning as #4842: no pre-rasterization stage writes Layer today, and reading it breaks the position exports on AMD's Windows driver (#4800). The pixel_ancillary whole-word cases expect 0 for the field.

The extract path in lowerPackedAncillary still reads Layer on this branch; that's the part #4842 changes. Whichever of the two merges second only has to reconcile that one line in ConstantFolder.cpp and the matching test cases. If #4842 goes first, we're happy to do that on our side.

AI-assisted: yes (Claude Code)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* docs shader core/shader

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants