[adriadam10]

What

Part of #981. The NaN result of f32 add, sub, mul and fma was whatever NaN the driver produced. Vulkan doesn't specify its payload or sign, and NVIDIA returns a canonical NaN. The NaN is now chosen on the bits, following the hardware and IEEE_MODE (#1816, #1867).

Measured on RDNA2 (gfx1035) with the HW oracle, over 1000 operand triples (quiet and signaling NaNs of both signs and payloads, ±inf, ±0, ±1) at IEEE=0 and IEEE=1. The ops were v_add/sub/subrev/mul/fma/fmac/fmamk/fmaak/mul_legacy/fma_legacy_f32, v_fma_mix_f32, v_add_f32 -v4, |v5| and v_mul_f32 mul:2. The rule below matches every cell (0 mismatches):

• The result is the first NaN source in operand order, after its modifiers, with the quiet bit set only at IEEE=1.
• v_sub_f32 and v_subrev_f32 add the negation of the subtrahend, so a NaN in that position comes out with its sign flipped (src0 + -src1, -src0 + src1).
• With no NaN source, an invalid operation (inf − inf, inf × 0) gives the default NaN 0xffc00000.
• In the fma forms, an invalid product gives the default NaN even when src2 is a NaN, as packed f16 does (fix(shader): choose packed f16 NaN results on the bits, as RDNA does #533).
• In the legacy forms, a zero factor makes the product 0 even when the other factor is a NaN. The existing zero handling already does that, so the choice is made on the factors after it.

The sign flip is applied as an integer XOR. OpFNegate isn't relied on for a NaN's payload.

The f16 arithmetic, v_fma_mix_f32 with f16 sources, v_cubema, v_mac_legacy_f32 and the separately rounded mad/mac forms (gfx1035 doesn't have them) still leave the NaN to the driver. TechnicalDebt says so.

Tested

Linux (CachyOS, GCC), RADV (gfx1035) and CI's lavapipe (Mesa 25.2.8).

• New agc_driver_f32_arithmetic_nan: 16 instructions (including modifiers, omod and clamp) × 32 rows captured on hardware at IEEE=0 and IEEE=1. 624 of the 1024 cells are NaN, and every cell is compared bit for bit, because the translation now chooses the NaN bits. The non-NaN rows use exact values (±0, ±1, ±inf), so the result doesn't depend on whether the driver fuses Fma.
• Negative checks on lavapipe, each fails the new test: no NaN choice for add/mul, the subtrahend not negated, no NaN choice for fma, the invalid-product rule ignored, no source NaN taken at all, and no default NaN for invalid operations.
• agc_driver_f32_denormal_flush, agc_driver_nan_quieting, agc_driver_f32_output_modifier, agc_driver_division and vector_fused_multiply_add pass unchanged. Full Release ctest: 494/494.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none (fix(shader): quiet signaling NaNs only in IEEE mode #1867 and fix(shader): translate v_fma_mix_f32 as a fused multiply-add #1878 merged)
• AI-assisted: yes