Skip to content

fix(shader): quiet signaling NaN cube results in IEEE mode - #2043

Merged
adriadam10 merged 2 commits into
boykopovar:mainfrom
CouchRiv:fix-shader-cube-sc-tc-snan
Oct 10, 2026
Merged

adriadam10 merged 2 commits into
boykopovar:mainfrom
CouchRiv:fix-shader-cube-sc-tc-snan

Conversation

@CouchRiv

@CouchRiv CouchRiv commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

What

Part of #981, a follow-up to #1867. In a shader with IEEE_MODE=1, the cube instructions return a signaling NaN unchanged:

v_cubesc_f32 v10, 0, 0x7fc12345, 0xff812345   hardware 7fc12345, recompiler 7f812345
v_cubetc_f32 v10, 0, 0, 0xff812345            hardware ffc12345, recompiler ff812345
v_cubema_f32 v10, 0xff812345, 0, 0            hardware ffc12345, recompiler ff812345

With IEEE=1 the hardware quiets a signaling NaN result of v_cubesc/cubetc/cubema_f32, and keeps the sign the instruction gives it (cubesc above returns -z). With IEEE=0 the bits pass through. This is the rule #1867 measured for the unary instructions. floatCube picks the NaN operand itself, so these bits never came from the driver: #1867 recorded v_cubema as "left to the driver", but the code always returned the signaling NaN as it was. floatCube now passes a NaN result through quietNan32. IEEE=0, shaders without a known float mode and v_cubeid_f32 behave as before. With DX10_CLAMP=1 a clamped NaN still becomes 0; with DX10_CLAMP=0 it keeps its sign and payload and is quieted only with IEEE=1 (row 2 v_cubema_f32 ... clamp: 7f800001 at IEEE=0, 7fc00001 at IEEE=1 on gfx1036).

Which title needs it

No known title needs it yet. It extends the rule #1867 measured to three more instructions, and it only changes shaders running with IEEE=1, which titles we know do not use (they use 0xc0, IEEE=0).

Tested

CachyOS Linux, GCC 16.2, RTX 4070 Ti (NVIDIA 615.71) and Radeon 610M (RADV, Mesa 26.2.4). Measured on RDNA2 (gfx1036) with the HW oracle.

  • agc_driver_vop3_cube_clamp now also runs at IEEE=0 and IEEE=1. Its 16 forms × 32 rows were captured on hardware in both modes. The IEEE=0 capture equals the existing table. At IEEE=1, 12 of the 512 cells differ, all signaling NaN results of sc/tc/ma, with and without source modifiers.
  • Without the fix the test fails on both drivers: lane 2 IEEE=1 v_cubema_f32 v13, v4, v5, v6 is 0x7f800001, expected 0x7fc00001.
  • A differential run of 2,744 rows per instruction in 4 float modes: 114 (cubesc), 278 (cubetc) and 196 (cubema) rows differed with IEEE=1 before, 0 after, on both drivers. v_cubeid_f32 still matches.
  • Full Release ctest 498/498 on NVIDIA and on RADV; check_conventions.py and git diff --check pass.
  • Not tested: a title, since none is known to run with IEEE=1.

Checklist

🤖 Generated with Claude Code

@github-actions github-actions Bot added shader core/shader docs agc libSceAgc* needs rebase Conflicts with main, rebase needed labels Oct 9, 2026
@CouchRiv
CouchRiv force-pushed the fix-shader-cube-sc-tc-snan branch from a07116a to f3660a0 Compare October 9, 2026 08:19
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 9, 2026
@popcoder565

Copy link
Copy Markdown
Contributor

Native Windows run, since no Windows results were posted. Setup: Windows 10 (19045), RTX 3060 Laptop (NVIDIA 616.92), MinGW-w64 GCC 15.2.0 (WinLibs r7), Release. Head f3660a0c, merged onto main d70b899 in one build with 33 other AGC/shader PRs that had no Windows run.

No title was run.

🤖 Addressed by Claude Code

@adriadam10

Copy link
Copy Markdown
Collaborator

Rebased on current main (only conflict: the Signaling NaN inputs line in TechnicalDebt.md, now keeps main's text plus the cube sentence). The red linux/windows jobs failed building agc_shader_fusion_tests (spirv/unified1/spirv.hpp missing), which this PR doesn't touch.

Checked on the rebased head: full build and ctest 633/633; agc_driver_vop3_cube_clamp fails without the change; the HW-oracle rows on RDNA2 (gfx1035) match bit for bit in both IEEE modes. Merging once the rebase is pushed.

@adriadam10
adriadam10 force-pushed the fix-shader-cube-sc-tc-snan branch from f3660a0 to 5d27030 Compare October 10, 2026 07:11
CouchRiv and others added 2 commits October 10, 2026 09:13
With IEEE_MODE=1 the hardware quiets a signaling NaN that v_cubesc,
v_cubetc or v_cubema_f32 returns, keeping the sign the instruction gives
it; with IEEE=0 the bits pass through. Same rule as boykopovar#1867, measured on
gfx1036 in both modes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Its signaling NaN result is quieted with IEEE=1 by the cube sentence above.

Co-Authored-By: Claude <noreply@anthropic.com>
@adriadam10

Copy link
Copy Markdown
Collaborator

Rebased on current main (conflict in the Signaling NaN inputs line of TechnicalDebt.md after #2042's sentence: kept main's text plus the cube sentence). Added one commit: that line still listed v_cubema among the instructions whose NaN bits are left to the driver, which this PR makes false; removed it.

Checked on the rebased head: scoped build and ctest 324/324 (libSceAgc, libSceAgcDriver); agc_driver_vop3_cube_clamp fails without the change; HW-oracle rows on RDNA2 (gfx1035) match bit for bit in both IEEE modes. Windows run by @popcoder565 passes too. Merging once CI is green.

@adriadam10
adriadam10 force-pushed the fix-shader-cube-sc-tc-snan branch from 5d27030 to aee2d25 Compare October 10, 2026 07:18
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 10, 2026
@adriadam10
adriadam10 merged commit e8b47b9 into boykopovar:main Oct 10, 2026
14 checks passed
SleglJan pushed a commit to SleglJan/AnyPS5 that referenced this pull request Oct 10, 2026
…c-snan

fix(shader): quiet signaling NaN cube results in IEEE mode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* docs shader core/shader

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants