[Swastik-51]

Depends on #1280 (which builds on #1265). Until they merge, this branch includes their commits; the last commit adds the 32-bit probe and fallback sharing.

32-bit BDA stores previously emitted both an aligned 4-byte store and four unaligned byte stores at every store site, each with a lookup and a page-write probe sequence. With note_bda_write inlined by drivers, programs like agc_driver_flat_store ballooned into dozens of lookups, hundreds of atomic CAS probes, and over 134K GEN instructions on Intel.

Probe the 4-byte span once with Write permission, storing directly when mapped and aligned. The unaligned and faulting fallback is shared across write sites via write_bda_dword_bytes, marked DontInline. note_bda_write is also marked DontInline.

Closes #1256 .

What

• Probe 32-bit stores: Defined probe_bda_write_pointer (recordFaults = false, BdaAbi::Write) to probe the 4-byte span once. When mapped and 4-aligned, EmitBdaStore emits a single 4-byte store directly and notes the write.
• Shared fallback function: Defined write_bda_dword_bytes (with a coherent variant) marked with spv::FunctionControlDontInlineMask to handle unaligned accesses, cross-range accesses, and faults without duplicating per-byte lookups at every store site.
• Prevent inlining of note_bda_write: Changed spv::FunctionControlMaskNone to spv::FunctionControlDontInlineMask in DefineBdaNoteWrite so drivers do not inline its atomic CAS probes into dozens of call sites.
• Added test: Added verifyBdaWriteFallbackFunctions to ShaderMemory.cpp to verify that note_bda_write and write_bda_dword_bytes have DontInlineMask, main has 0 inline get_bda_write_pointer calls, coherence is preserved, and SPIR-V code size delta per site is kept small.

Tested

• Checked conventions locally: python tools/check_conventions.py --base origin/main passed with 0 errors.
• Added and registered unit test verifyBdaWriteFallbackFunctions in ShaderMemory.cpp.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: perf(recompiler): load the written-page slot before the CAS #1280, perf(recompiler): loop the BDA byte stores and written-page probes #1265
• AI-assisted: yes

[hashaam1217]

On Linux this does not keep #1265's gain. Intel Iris Xe, Mesa 25.2.8 ANV, applied to 2cc5735, cold caches:

• flat_store's first pipeline compiles to 54,375 GEN instructions, close to main's 55,929; perf(recompiler): loop the BDA byte stores and written-page probes #1265 and perf(recompiler): load the written-page slot before the CAS #1280 give about 26,000–28,000.
• Compiling the second pipeline goes past a 7 GiB memory limit and is killed. With perf(recompiler): load the written-page slot before the CAS #1280 alone, the whole test peaks at 2.11 GiB.

The cause is the DontInline in e63cc30. Mesa inlines every function in Vulkan shaders and does not check that bit: ANV through vk_spirv_to_nir → nir_inline_functions, and RADV through the same pass. So write_bda_dword_bytes and note_bda_write are copied back into every store site, next to the new probe. The SPIR-V gets smaller while the compiled shader does not. Keeping the fallback in a loop, as #1265 does for the byte stores, is what kept it small here.

Measurements and Mesa source links: #1256 (comment)

[oneandonlydean]

DontInline doesn't hold on Mesa: ANV and RADV inline every function in vk_spirv_to_nir, so write_bda_dword_bytes and note_bda_write end up back at every store site and the second flat_store pipeline runs past 7 GiB (hashaam1217's numbers above). Please drop DontInline and keep the shared fallback in a loop, the way #1265 does for the byte stores, so the compiled shader stays small without relying on it.

Then please add numbers for this commit alone on top of #1280: GEN instruction count and compile memory for agc_driver_flat_store, plus a full ctest run.

[Swastik-51]

Updated commit fe567b52 with the requested changes:

• Dropped DontInline from note_bda_write: Reverted back to spv::FunctionControlMaskNone in DefineBdaNoteWrite.
• Removed write_bda_dword_bytes: Dropped DefineBdaDwordWriteFunctions and its unrolled stores.
• Routed fallback through looping write_bda_bytes: EmitBdaStore now routes unaligned and probe-rejected stores through storeBytes() (write_bda_bytes, in a loop) so that driver inlining does not duplicate byte store sequences across store sites.
• Updated test: verifyBdaWriteFallbackFunctions in ShaderMemory.cpp now verifies that fallback byte stores route through write_bda_bytes and keep access coherence with 1 store inside the loop.
• Passes python tools/check_conventions.py --base origin/main with 0 errors.

[LeandroLP23]

agc_shader_memory fails at fe567b5, both on its own (base c5e1018) and merged into main 1081602:

BDA write functions: the byte stores do not keep the access's coherence

It is the new verifyBdaWriteFallbackFunctions, in the non-coherent case: write_bda_bytes has 3 OpStores without Volatile, and the check at ShaderMemory.cpp:1450 expects exactly one. The coherent case doesn't run, because the loop stops at the first case. Windows 11, MinGW-w64 GCC 15.2.0, Release; the test doesn't use the GPU.

This PR has no CI checks yet, which is probably why it hasn't shown up. The rest of the suite is unchanged on a Radeon 860M (AMD numbers for the three PRs are in #1256).

Tested with AI assistance (Claude Code).

[Swastik-51]

Could you please run the Github CI please as i need maintainer approval to run it

[LeandroLP23]

I can't approve workflow runs here; GitHub only lets maintainers do that for first-time contributors. You don't need CI for this one, though: the test doesn't touch the GPU, so it reproduces on any build with ctest --test-dir build -R '^agc_shader_memory$' --output-on-failure. Once it passes locally, I can rerun the AMD measurements on the new head if that helps.

[Swastik-51]

Fixed the agc_shader_memory test issue in 515df443.

The test was counting the two function-local loop counter OpStores as non-volatile stores because length > 3u was part of the array-index expression. It now filters length > 3u before counting, so only the physical buffer store is checked.

python tools/check_conventions.py --base origin/main passes with 0 errors.

I unfortunately can't run the full ctest suite or the requested agc_driver_flat_store measurements locally because I don't currently have a C++ build toolchain or an Intel/Mesa ANV environment. In particular, I can't produce the requested GEN instruction count or peak compile-memory measurement.

Could someone with the Intel/Mesa setup rerun the agc_driver_flat_store benchmark on 515df443? If possible, I'd also appreciate a test-suite run on the updated head.

[LeandroLP23]

Ran it on 515df443 merged into main 9f4bd71: Radeon 860M (RDNA 3.5), Windows driver 32.0.13058.8001, Windows 11, MinGW-w64 GCC 15.2.0, Release.

• agc_shader_memory passes.
• Full ctest with ANYPS5_REQUIRE_VULKAN=1 --repeat until-fail:3: 455/457. The two failures also fail on main on this machine. agc_driver_vopc_compare_table is the AMD driver bug in agc_driver_vopc_compare_table fails in wave32 for LG compares in the e32 encoding on AMD RDNA 3.5 #825. agc_driver_shader_clock is flaky here (memtime: lane 0 went back from 1030876 to 1354, 11 of 60 runs on main, 25 of 60 with this PR). The value wraps at 2^20, the 20-bit subgroup clock that fix(shader): read s_memtime from the device clock where RADV's subgroup clock is 20 bits #1383 handles only for RADV, while this GPU runs AMD's own driver. I'll follow that up separately.

Cold AMD timings, measured as in #1256 (fresh executable name per run so the AMD driver cache misses, ANYPS5_NO_SHADER_CACHE=1, median of 3, whole test):

	flat_store	global_store_addtid	global_vcc_base
main 9f4bd71	3.9 s, 0.48 GiB, 128 KiB, 28 VGPRs	3.9 s, 0.43 GiB, 128 KiB	10.6 s, 0.53 GiB, 256 KiB
#1282 515df44	3.3 s, 0.48 GiB, 64 KiB, 25 VGPRs	3.5 s, 0.43 GiB, 128 KiB	8.8 s, 0.53 GiB, 256 KiB

The KiB column is the size of the AMD driver's pipeline cache file for the run. I can't produce the Intel GEN instruction count from here; that still needs someone with an Intel GPU.

Measured with AI assistance (Claude Code).

[Swastik-51]

Thanks! Great to see agc_shader_memory passing and the full suite results confirming the two failures are pre-existing on main.
The AMD numbers also look good, especially flat_store improving from 3.9s to 3.3s with 25 vs 28 VGPRs and the pipeline cache dropping from 128 KiB to 64 KiB.
That leaves only the Intel/Mesa ANV GEN instruction and compile-memory measurements requested in the review. I don't have access to that environment, so hopefully someone with Intel/Mesa can rerun the benchmark on 515df44.

[hashaam1217]

Intel numbers for 515df44, the commit alone on top of #1280, as requested above. Intel Iris Xe (Alder Lake GT2), Mesa 25.2.8 ANV, Linux. Each variant was merged onto this PR's base, c5e1018, because the head currently conflicts with main in SpirvBdaLookup.cpp and SpirvEmitterState.hpp. Cold runs with ANYPS5_NO_SHADER_CACHE=1 MESA_SHADER_CACHE_DISABLE=true APS5_PIPELINE_STATS=1, one test at a time in a cgroup limited to 7 GiB:

agc_driver_flat_store (passes in all three, no spills):

	main (c5e1018)	#1280 (1dca7b4)	this PR (515df44)
First pipeline: SPIR-V words → GEN instructions, compile time	10,897 → 55,929, 18.8 s	7,598 → 27,965, 7.2 s	8,024 → 21,601, 5.0 s
Second pipeline	17,450 → 111,860, 74.2 s	11,118 → 53,453, 27.0 s	11,229 → 41,175, 18.0 s
Wall time	93.3 s	34.4 s	23.1 s
Peak RSS	7.05 GiB	2.11 GiB	1.38 GiB

Other store-heavy tests. Both fail on this device for the subgroup reason fixed by #1185, which is not in this base; the compile numbers still apply:

	main	#1280	this PR
global_store_addtid GEN instructions	20,550	9,289	6,743
global_vcc_base GEN instructions, peak RSS	33,525 / 67,050, 2.73 GiB	22,326 / 44,652, 1.50 GiB	19,780 / 39,560, 1.36 GiB

Tests at the same base:

• agc_shader_memory passes.
• NVIDIA RTX 3060 (595.91.07) full suite: 400/400 with this PR, the same as perf(recompiler): load the written-page slot before the CAS #1280, none skipped.
• Intel full suite, run without flat_store and global_vcc_base: the same 33 failures as perf(recompiler): load the written-page slot before the CAS #1280, which are also exactly main's failures at 2cc5735.

So with DontInline gone and the fallback looping, this commit now makes the compiled shader smaller on Mesa ANV than #1280 alone, not larger as the first version did.

[Swastik-51]

Closing this PR for now because the branch has accumulated significant merge conflicts with the current main, particularly around SpirvBdaLookup.cpp and SpirvEmitterState.hpp.

The implementation and measurements are still useful, and the latest results look good after the requested changes. Rather than resolving the conflicts on top of the current dependency stack, I'll preserve the work and rebase/cherry-pick the relevant changes onto the latest main once the dependent PRs land, then open a clean follow-up PR.

Thanks everyone for the reviews, testing, and especially the Intel/Mesa measurements.