Skip to content

fix(shader): read s_memtime from the device clock where RADV's subgroup clock is 20 bits - #1383

Merged
adriadam10 merged 1 commit into
boykopovar:mainfrom
nisbenz:fix/shader-memtime-narrow-subgroup-clock
Oct 8, 2026
Merged

adriadam10 merged 1 commit into
boykopovar:mainfrom
nisbenz:fix/shader-memtime-narrow-subgroup-clock

Conversation

@nisbenz

@nisbenz nisbenz commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What

s_memtime went backwards on RADV with RDNA 2 and RDNA 3 GPUs (#985). It is emitted as OpReadClockKHR with Subgroup scope. On GFX10.3 to GFX11.7, ACO lowers that to s_getreg_b32 of SHADER_CYCLES, which is 20 bits wide and zero-extended to 64. The counter therefore wraps every 2^20 cycles.

This follows option 2 from the issue thread: on those GPUs only, s_memtime reads the Device-scope clock. Every other device, including RADV GFX12, NVIDIA and Intel, emits exactly what it did before.

  • Source for the 20-bit read: Mesa src/amd/compiler/instruction_selection/aco_select_nir_intrinsics.cpp, case nir_intrinsic_shader_clock.
    • The gfx_level >= GFX12 branch reads 64 bits (p_shader_cycles_hi_lo_hi).
    • The gfx_level >= GFX10_3 branch reads s_getreg_b32 ((20 - 1) << 11) | 29.
    • So GFX10.3, 11, 11.5 and 11.7 are affected.
  • Source for the family list: Mesa src/amd/common/amd_family.h, families CHIP_NAVI21 to CHIP_GFX1171. RADV reports the family in the device name as (RADV <family>), using the names from ac_get_family_name. I checked that these names appear in the installed libvulkan_radeon.so (Mesa 26.2.4).
  • Hardware data from the issue thread:
    • @fisherjoey measured on gfx1036 with the hw-oracle that s_memtime is 64-bit on the GPU itself, so the narrowing comes from ACO.
    • @EdouardSence confirmed that GFX12 on RADV reads 64 bits.

Changes:

  • NarrowSubgroupClock(driverID, deviceName) (new Execution/src/SubgroupClock.cpp) returns true only for VK_DRIVER_ID_MESA_RADV with one of the GFX10.3 to GFX11.7 family names. VulkanDevice queries VkPhysicalDeviceDriverProperties through VK_KHR_driver_properties, because the instance is Vulkan 1.1.
  • SpirvTarget::narrowSubgroupClock is passed to the emitter. EmitShaderClock uses ScopeDevice when it is set.
  • The field is part of the in-memory and disk cache keys. The request serializer goes to version 10, and older captures read as false.
  • TechnicalDebt.md (Functional) records the trade-off. The device clock is monotonic and 64-bit but counts the 100 MHz reference clock, so on those GPUs s_memtime deltas are about 20 times smaller than on hardware. A family added to Mesa later keeps the subgroup clock until it is listed.
  • agc_driver_shader_clock now runs its dispatch 128 times. On an affected GPU, a wrap then lands between two reads almost every run, so the test catches a regression instead of failing about 5% of the time.

Tests:

  • agc_subgroup_clock: family and driver matching, including malformed names, GFX10.1 and GFX12.
  • agc_shader_memory:
    • the OpReadClockKHR scope for s_memtime with the flag set and unset, and for s_memrealtime;
    • serializer round-trip, the legacy trailer, and the version-10 prefix.
  • agc_shader_disk_cache: the key changes with the flag.

Overlap with #1182 and #1190 (flagged below): both also take request serializer version 10, for scratchDwords, and append to the same trailer in RequestSerializer.cpp. This PR does not build on them and they do not build on it, so Depends on is none. Whichever merges second moves its field to version 11: write 11u, accept version > 11u, read the field under version >= 11u, and update the version pin in agc_shader_memory. I can rebase this one onto them if they land first.

Fixes #985

Tested

Linux (CachyOS, kernel 7.2.9), AMD Radeon RX 6750 XT (RADV NAVI22, gfx1031), Mesa 26.2.4, GCC 16.2.1, Release.

main (df16c4c) this PR
agc_driver_shader_clock, original single dispatch, 300 runs 14 failed (memtime: lane 0 went back from 1020093 to 17981 etc.) n/a
agc_driver_shader_clock, 128 dispatches per run 20 of 20 failed (emitter reverted locally) 0 of 300 failed
ctest -R agc_ -j6 (the repro from the issue), 10 runs n/a 201/201 each run, no went back
full ctest -j8 n/a 409/409
  • The device log reports Shader clock driver=3 narrowSubgroupClock=1.
  • tools/check_conventions.py --base upstream/main passes.
  • Not tested: Windows, other GPU vendors and RADV GFX11/GFX12. Their emitted SPIR-V is unchanged, and the unit tests cover the detection for those names.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • AI-assisted: yes

@EdouardSence

Copy link
Copy Markdown
Contributor

Tested on the two RADV devices in my machine. One is in the "not tested" list (GFX12); the other is one of the affected families. Same host as the PR: Linux 7.2.9 (CachyOS), Mesa 26.2.4, GCC, Release build of eb13861.

RX 9070 XT (RADV GFX1201, GFX12)

  • The device log reports Shader clock driver=3 narrowSubgroupClock=0, so the emitted SPIR-V is unchanged, as expected.
  • Full ctest -j8: 409/409 pass.
  • agc_driver_shader_clock_tests (128 dispatches): 30/30 runs pass.

Ryzen 7 7800X3D iGPU (RADV RAPHAEL_MENDOCINO, gfx1036, GFX10.3)

  • Selected with MESA_VK_DEVICE_SELECT=1002:164e MESA_VK_DEVICE_SELECT_FORCE_DEFAULT_DEVICE=1 (Physical device count=1).
  • The name matches the list: Shader clock driver=3 narrowSubgroupClock=1.
  • agc_driver_shader_clock_tests with this PR: 100/100 runs pass.
  • I could not reproduce the wrap on this iGPU:
    • main's single-dispatch test passed 100/100.
    • This PR's 128-dispatch test, with NarrowSubgroupClock forced to return false locally (log shows narrowSubgroupClock=0), passed 30/30, with no went back.
  • So on this chip the test does not catch a regression. The detection and the Device-scope path run cleanly, though.

AI-assisted: the runs and this write-up were done by Claude (Claude Code) on my machine.

…up clock is 20 bits

On GFX10.3 to GFX11.7, ACO lowers a subgroup-scope OpReadClockKHR to
s_getreg_b32 of SHADER_CYCLES, 20 bits zero-extended to 64, so s_memtime
wrapped every 2^20 cycles and went backwards. Detect those families from
the RADV device name and read the device clock there instead.

Fixes boykopovar#985

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@adriadam10

Copy link
Copy Markdown
Collaborator

Rebased onto current main: it conflicted in RequestSerializer.cpp and tests/ShaderMemory.cpp because main is already at serialization version 11 (#1190 line of work added scratchDwords at 10, orderedPixelShader at 11). As described in the PR, the field moved to version 12: written after scratchDwords, read under version >= 12u, limit > 12u, and the version pins in agc_shader_memory updated (NVNQQQwAAAA=, unsupported NVNQQQ0AAAA=).

Checked on the rebased head: build and full ctest green (453/453), the four PR tests pass, and they don't build without the change. On the hardware oracle (gfx1035) s_memtime advances by ~5.5M over a short loop with a zero high dword, so the counter is wider than 20 bits on HW and the narrowing is ACO's, as measured in #985. The 20x rate difference of the device clock is recorded in TechnicalDebt; fine as a trade-off. Merging once CI is green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* build docs needs rebase Conflicts with main, rebase needed shader core/shader

Projects

None yet

Development

Successfully merging this pull request may close these issues.

s_memtime goes backwards on RADV RDNA2/RDNA3: subgroup OpReadClockKHR is a 20-bit counter there

3 participants