Skip to content

fix(shader): LDS float atomics quiet signalling NaNs and flush denormals as the hardware does - #10

Merged
SleglJan merged 2 commits into
mainfrom
feat/shader-lds-float-atomics
Oct 10, 2026
Merged

SleglJan merged 2 commits into
mainfrom
feat/shader-lds-float-atomics

Conversation

@SleglJan

@SleglJan SleglJan commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

What

Follow-up to boykopovar#573 (the 64-bit LDS atomics), whose TechnicalDebt entry said the f64 forms were not measured on hardware. ds_min/max_f32, ds_cmpst_f32, ds_min/max_f64, ds_cmpst_f64 and their _rtn forms now follow the rule measured on RDNA2 (gfx1036, Ryzen 7950X3D iGPU) with tools/hw-oracle, over 14 × 14 operand sets per type (±0, ±1, ±inf, ±smallest denormal, quiet and signalling NaNs of both signs and with payloads) in IEEE 0 and 1 with the type's denormal field kept and flushed; every memory and returned result of the six f64 and nine f32 results per pair, in each of the four modes, 0 mismatches:

  • both operands are flushed to a signed zero when the type's denormal field flushes (FLOAT_MODE bits 5:4 for f32, bits 7:6 for f64; a shader without a known mode flushes f32 and keeps f64, as elsewhere);
  • a signalling-NaN data operand is stored quieted; else a signalling-NaN old value is stored quieted; else a quiet-NaN old value is replaced by the data; else a quiet-NaN data operand loses; else the ordered compare with −0 below +0 decides; the flushed value is what is written, also when the old value wins;
  • cmpst compares the flushed bits (both zeros equal, NaN never equal) and leaves memory unchanged when they differ;
  • the returned value is always the old value as it was in memory; IEEE_MODE has no effect.

This is the rule AtomicFloatBits::minMax already implements for the buffer float atomics, plus the flush. The shared f32 path used OpIsNan/OpFOrdLessThan/OpFOrdEqual on the bit-cast values, so its NaN and denormal results were the driver's (both RADV and NVIDIA quiet nothing and flush f32 denormals inside those compares in every mode); the shared f64 path (boykopovar#573) ordered the bits but had none of the signalling-NaN rules. Both now use AtomicFloatBits; the translation marks the shared float atomics with the mode's flush in MemoryInfo::flushDenormals. The 29 integer 64-bit forms, EXEC masking, immediate offsets, out-of-range addresses (0, no write) and contention (256 lanes, serial order) were measured as well and match the recompiler already; ds_add_f32 is not measured and is unchanged.

TechnicalDebt: the 64-bit entry loses its "not measured" sentence and gains the measured facts about mixed-width serialisation (hardware serialises 32-bit and 64-bit atomics on one address, the lock does not: in the measurement RADV lost 64 of 128 increments in wave32 and 32 in wave64, NVIDIA 48 in wave32, timing-dependent) and the misaligned fault; a new LDS float atomics entry records the rule; the f32 denormals entry points to it.

Measurement data, scripts and the host comparison before and after the change: https://github.com/SleglJan/AnyPS5/tree/audit/oracle-2026-10-08/oracle-audit/lds64 (report.md there is the summary).

Overlap: boykopovar#2025 and boykopovar#1290 add TechnicalDebt entries next to the edited ones; whichever merges second needs a one-line rebase. No open pull request changes the shared float atomics.

Tested

Linux (CachyOS, kernel 7.2.9, GCC 16.2.1), Release, ANYPS5_ENABLE_SPIRV_TOOLS=ON, RTX 3080 (615.71.09) and RADV (Mesa 26.2.4), on this branch (based on main d70b899).

  • New agc_driver_lds_float_atomic_modes: 32 operand pairs per type (the classes above, both orders), each lane runs the six f32 and six f64 forms on its LDS slot; 27 result dwords per lane pinned bit for bit against the oracle in the titles' mode (0xc0, also used for "no float mode"), with both fields kept (0xf0) and with both flushed (0x00). Written first: it failed on main's lowering at lane 7 (ds_min_rtn_f32, flushed denormal) and passes with the change.
  • agc_driver_lds_atomics64: its CPU model gets the signalling-NaN rule, and the lanes of kind 2 now take their data from floats[(tid / 4 + 9) % 12], which feeds the table's signalling NaN, −inf and the negative denormal into min/max (the old index never reached entry 9). It failed on main at lane 2 (ds_min_f64, the signalling-NaN data not stored quieted) and passes with the change.
  • Mutation checks, each fails: the 64-bit path without the signalling-NaN rules (both tests), the flush disabled (new test, lane 7), the old signalling NaN preferred over the data (new test, lane 22).
  • The measured matrices replayed through the recompiler on both GPUs with the change: 0 of 35,072 result dwords differ from the oracle over the 256 lanes (f64 min/max 3,072 × 4 modes, f64 cmpst 1,536 × 4, f32 min/max/cmpst 2,304 × 4, 29 integer forms 7,424); before the change, per mode, 472 (f64 min/max, kept), 584 (f64 min/max, flushed), 40 (f64 cmpst, flushed), 336 (f32, kept) and 412 (f32, flushed) dwords differed, the same on both GPUs.
  • agc_driver_lds_atomics, agc_driver_lds_integer_atomics, agc_driver_lds_condxchg, agc_driver_lds_read_write, agc_driver_lds_src2 pass on both GPUs; LDS tests --repeat until-fail:3 on both; full ctest 516/517 on each GPU; the one failure differs per GPU and runs no LDS instruction: agc_driver_storage_cache_budget on the iGPU (its memory budget) and agc_driver_dcc_keys_first_write on the RTX 3080 ("an unchanged pending image was uploaded again", also reported in several open pull requests today).
  • tools/check_conventions.py --base upstream/main, tools/check_test_registration.py pass.
  • Windows: the fork's Build workflow on head 3e8d755 merged onto main d70b899, run 37947911451: Windows builds and passes 508/508 with the AGC execution tests skipped (no Vulkan device on the runner, as for every execution test); the Linux job runs the new test on lavapipe (3.3 s) and passes 517/517; the relinker matrix passes on all three systems..

Checklist

🤖 Generated with Claude Code

@SleglJan
SleglJan force-pushed the feat/shader-lds-float-atomics branch 2 times, most recently from 59e1899 to edb53ae Compare October 10, 2026 06:42
SleglJan and others added 2 commits October 10, 2026 09:40
…als as the hardware does

ds_min/max/cmpst_f32 and ds_min/max/cmpst_f64 with their _rtn forms follow the
rule measured on RDNA2 (gfx1036): both operands are flushed to a signed zero
when the type's denormal field flushes, a signalling-NaN data operand is stored
quieted, else a signalling-NaN old value, else a quiet NaN loses, else the
ordered compare decides, and the flushed value is written. The shared f32
path used the host's float compares, so its NaN and denormal results were the
driver's; the shared f64 path lacked the signalling-NaN rules the buffer path
has. Both now use AtomicFloatBits. The translation marks the shared float
atomics with the mode's flush in MemoryInfo.

The new agc_driver_lds_float_atomic_modes pins 32 operand pairs per type in
the titles' mode, with both fields kept and with both flushed. The 64-bit
test's model gets the signalling-NaN rule and one lane kind now feeds the
signalling NaN of its table into min/max.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012GLfKH7Sqe51wX254k1XsV
…well

On RDNA2 (gfx1036) ds_min/max/cmpst(_rtn)_f32 and _f64 flush both operands
when bit 0 of the type's denormal field is clear: field 2 flushes like field
0, field 1 keeps like field 3, in IEEE 0 and 1. The flush decision now tests
that bit. agc_driver_lds_float_atomic_modes adds FLOAT_MODE 0x50 (both fields
1) and 0xa0 (both fields 2), and the TechnicalDebt entry records the measured
fields.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012GLfKH7Sqe51wX254k1XsV
@SleglJan
SleglJan force-pushed the feat/shader-lds-float-atomics branch from edb53ae to 4fb613e Compare October 10, 2026 07:43
@SleglJan
SleglJan merged commit 700d5a6 into main Oct 10, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant