Skip to content

fix(shader): correctly rounded f32 reciprocal on every device - #7671

Closed
tibzejoker wants to merge 2 commits into
boykopovar:mainfrom
tibzejoker:fix/correctly-rounded-reciprocal
Closed

tibzejoker wants to merge 2 commits into
boykopovar:mainfrom
tibzejoker:fix/correctly-rounded-reciprocal

Conversation

@tibzejoker

@tibzejoker tibzejoker commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

What

v_rcp_f32 and v_rcp_iflag_f32 are translated to a plain OpFDiv(1, x), and Vulkan leaves the precision of a division to the device. In practice an RTX 4090 and an Intel UHD 770 return reciprocals one ulp apart. Each of them also differs from the correctly rounded value on some inputs. As a result, the same shader stores different bytes on the two vendors: I saw this on a GTA V compute pass (PPSA04264, compute 0x143978f00, one f32 one ulp apart).

This change adds two fused Newton steps (e = fma(-x, y, 1), y = fma(y, e, y)) after the division estimate. They give the correctly rounded reciprocal whenever the input and the estimate are normal. Zero, infinities, NaN and denormals keep the division result, so the special-value handling stays as it is.

It also matters for integer division: RDNA compilers implement it with v_rcp_iflag_f32 followed by corrections, so it now gives the same value on every device.

Cost: about ten SPIR-V instructions per reciprocal.

Tested

Windows 11, MinGW GCC 15.2:

  • New test agc_driver_rcp_correctly_rounded. It checks v_rcp_f32 of 8192 normal inputs across the whole exponent range against the correctly rounded reciprocal.
    • On main it fails on both GPUs:
      • NVIDIA: v_rcp_f32 of 0x858a127e is 0xf96d5337, expected the correctly rounded 0xf96d5336
      • Intel: v_rcp_f32 of 0x1646f91a is 0x68a4af84, expected ... 0x68a4af85
    • With this change it passes on both.
  • AMD, reported by @DavidAmsellem: RX 7900 XT (RADV NAVI31, Mesa 25.2.8, Linux). On main the test fails (v_rcp_f32 of 0x0f5b5acd is 0x6f956235, expected ... 0x6f956236, one ulp below), and it passes with this change.
  • ctest -R ^agc_ on the RTX 4090: 727/728, including agc_driver_vop1_float_unary. The failure is agc_driver_bda_device, which fails on main too.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none
  • GPUs tested: NVIDIA GeForce RTX 4090 (616.92, Windows 11); Intel UHD Graphics 770 (32.0.101.6129, Windows 11); AMD RX 7900 XT (RADV, Mesa 25.2.8, Linux, by @DavidAmsellem); AMD Radeon 860M (32.0.13058.8001, Windows 11, by @LeandroLP23)
  • AI-assisted: yes (Claude Code)

🤖 Generated with Claude Code

https://claude.ai/code/session_01XBXM4pgkMHYEtqN1EQni2P

v_rcp_f32 and v_rcp_iflag_f32 were a plain OpFDiv, whose precision Vulkan
leaves to the device: an RTX 4090 and an Intel UHD 770 return reciprocals
one ulp apart, and each differs from the correctly rounded value for some
inputs, so the same shader produced different bytes on the two vendors
(seen on a GTA V compute pass).

Two fused Newton steps from the division estimate now give the correctly
rounded reciprocal whenever the input and the estimate are normal; zero,
infinities, NaN and denormals keep the division result. Integer division
sequences built on v_rcp_iflag_f32 also get the same value on every device.

Test: agc_driver_rcp_correctly_rounded compares v_rcp_f32 of 8192 normal
inputs over the whole exponent range with the correctly rounded reciprocal;
on main it fails on both GPUs, with this change it passes on both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBXM4pgkMHYEtqN1EQni2P
@github-actions github-actions Bot added shader core/shader agc libSceAgc* labels Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 4072 core/shader/recompiler/SpirvBackend/src/SpirvAlu/SpirvAluEmitterMath.cpp
PR 5021 core/shader/recompiler/SpirvBackend/src/SpirvAlu/SpirvAluEmitterMath.cpp
PR 6290 core/shader/recompiler/SpirvBackend/src/SpirvAlu/SpirvAluEmitterMath.cpp
PR 6645 core/shader/recompiler/SpirvBackend/src/SpirvAlu/SpirvAluEmitterMath.cpp
PR 7332 core/shader/recompiler/SpirvBackend/src/SpirvAlu/SpirvAluEmitterMath.cpp

@DavidAmsellem

Copy link
Copy Markdown
Contributor

Tested on AMD on Linux, which this PR had not covered.

  • GPU: AMD Radeon RX 7900 XT (RADV NAVI31, gfx1100), radv, Mesa 25.2.8-0ubuntu0.24.04.4
  • OS: Linux Mint 22.3 (Ubuntu 24.04 base), kernel 7.0.0-38, GCC 13.3
  • Built 3e8a9380d merged into main 0bc3003f8, Release, BUILD_TESTING=ON

agc_driver_rcp_correctly_rounded passes.

Full ctest with ANYPS5_REQUIRE_VULKAN=1: 1479 of 1481. Both failures also happen on main
without this PR and are unrelated to it:

@tibzejoker

Copy link
Copy Markdown
Contributor Author

Thanks @DavidAmsellem, that covers the vendor I could not test. If you still have the build, it would be useful to know whether agc_driver_rcp_correctly_rounded also fails on RADV with main. It shows whether RADV's division is already correctly rounded or only close.

AI-assisted (Claude Code).

@DavidAmsellem

Copy link
Copy Markdown
Contributor

@tibzejoker RADV is not correctly rounded either, so this is not NVIDIA only.

I rebuilt main 0bc3003f8 with only RcpCorrectlyRounded.cpp from this PR added and
SpirvAluEmitterMath.cpp left at main. On the same machine (RX 7900 XT, RADV NAVI31 gfx1100,
Mesa 25.2.8) the test fails at the first case it reports:

AGC graphics: v_rcp_f32 of 0x0f5b5acd is 0x6f956235, expected the correctly rounded 0x6f956236

RADV's result is one ULP below the correctly rounded one, which matches what v_rcp_f32 is
specified to give (1 ULP of relative error), so RADV is close but not correctly rounded. With this
PR applied the test passes on the same GPU.

@tibzejoker

Copy link
Copy Markdown
Contributor Author

Thanks, that settles it: all three vendors are one ulp off on some inputs with main. I added your result to the description.

AI-assisted (Claude Code).

@simonbbby

Copy link
Copy Markdown
Contributor

Native RDNA 2 datapoint for the reference used by head 3e8a9380dbc0a50c829fce9a052f06b499d06a04: gfx1036's v_rcp_f32 and v_rcp_iflag_f32 are not correctly rounded on this corpus.

GPU: Ryzen 9 9950X iGPU, gfx1036, PCI 1002:13c0. I ran the two actual VALU instructions directly through the HSA hardware oracle, not through Vulkan or this recompiler:

v_rcp_f32 v10, v4
v_rcp_iflag_f32 v11, v4

Inputs are the exact 8192 Sample(index) words from this PR's fixture. Both instructions differ from its correctly rounded reference on 889/8192 inputs, always by one ULP. Their outputs are identical to each other. The first difference is the same input as the RADV report above:

input                   0x0f5b5acd
native v_rcp_f32         0x6f956235
native v_rcp_iflag_f32   0x6f956235
correctly rounded       0x6f956236

I repeated the complete corpus in wave32 and wave64, each with (ieee, dx10_clamp) of (0,0), (1,0) and (1,1). All six native result arrays are byte-identical. Every run explicitly sets denorm32=3, denorm16=3, round32=0, round16=0, fp16_overflow=0; both input and correctly rounded reciprocal are normal for the fixture's exponent range.

Reference/control checks:

  • An integer quotient/remainder calculation of round-to-nearest-even matches the fixture's Rounded algorithm on all inputs; this is not a host-float-reference discrepancy.
  • Extra exact inputs 1, 2 and 0.5 return the expected reciprocals on both instructions in every run.
  • An input-move negative control changes 8194 of 8195 comparisons in the first output, checking the input/output path independently.

This measures gfx1036, not PS5 gfx1013, and does not establish the console's exact result or the GTA V pass's expected bytes. It does show that cross-device correctly rounded division is a different target from the measured gfx1036 instructions. Could the intended semantic target be clarified or measured on gfx1013 before treating the new correctly rounded oracle as console equivalence? No Vulkan-PR runtime or title result is claimed here.

@DavidAmsellem

Copy link
Copy Markdown
Contributor

RDNA 3 datapoint, same method as @simonbbby: gfx1100's native v_rcp_f32 matches gfx1036 on this
corpus, and neither is correctly rounded.

GPU: RX 7900 XT, gfx1100, RADV Mesa 25.2.8, Linux. The two VALU instructions straight through the
HSA hardware oracle, not through Vulkan or the recompiler:

v_rcp_f32 v10, v4
v_rcp_iflag_f32 v11, v4

Inputs are the 8192 Sample(index) words of this PR's fixture, with the reference recomputed from
its Rounded algorithm.

  • v_rcp_f32 differs from the correctly rounded reference on 889 of 8192 inputs, always by
    exactly one ULP. That is the same count @simonbbby measured on gfx1036.
  • First differing input is the same one again: 0x0f5b5acd, native 0x6f956235, correctly rounded
    0x6f956236.
  • v_rcp_f32 and v_rcp_iflag_f32 agree on all 8192 inputs.
  • Repeated in wave32 and wave64, with (ieee, dx10_clamp) of (1,1) and (0,0), and with
    denorm32 of 3 and 0. All four native arrays are identical.

So the one ULP shortfall is not vendor specific and not RDNA 2 specific: RDNA 1 (earlier report),
RDNA 2 and RDNA 3 all hit it, and on RDNA 2 and RDNA 3 with the same failure count and the same
first input.

This measures gfx1100, not the console's gfx1013, so like the gfx1036 report it does not establish
what the console returns.

AI-assisted (Claude Code).

@tibzejoker

Copy link
Copy Markdown
Contributor Author

Thank you both, that is exactly the data I was missing.

You're right about the target. This PR makes the result identical on every host GPU: today NVIDIA, Intel and RADV each return their own value. It does not reproduce the console. Your measurements show that RDNA 1, 2 and 3 agree with each other bit for bit (889/8192, same first input), so gfx1013 most likely returns the same values. That makes the hardware result a better target than correct rounding: it would be both deterministic on every host and faithful to the console.

Reproducing it needs the hardware algorithm, not a table. For a normal input the mantissa of v_rcp_f32 depends only on the input mantissa, so one exponent covers everything. If one of you can run the HSA oracle over all 2^23 mantissas at a fixed exponent (inputs 0x3f800000 | m) and attach the output words, or only the list of m where they differ from the correctly rounded result, I will work out an exact formula from it and update the PR so the test checks the hardware values.

Until then I'd keep this PR as it is (one value on every device, at most one ulp from RDNA on about 11 % of inputs) rather than leaving each driver to round differently. Maintainers, if you'd rather wait for the hardware-exact version, I can mark it as a draft.

AI-assisted (Claude Code).

@LeandroLP23

Copy link
Copy Markdown
Contributor

AMD run, since the description covers an RTX 4090. Radeon 860M (RDNA 3.5, driver 32.0.13058.8001), Windows 11 26300, WinLibs GCC 15.2.0 r7, Release, BUILD_TESTING=ON. Build: main 8a456334f with #7677, #7673, #7671, #7666, #7676, #7675, #7672, #7668, #7664 and #7658 merged together (each merge clean, built once).

agc_driver_rcp_correctly_rounded, agc_driver_vop1_float_unary and agc_driver_bda_device pass 3 runs of 3 on AMD's proprietary driver.

AI-assisted

@wachi35676

Copy link
Copy Markdown
Contributor

Tested at 3e8a9380d on Windows 11 (MinGW GCC 15.2), merged onto main 8a456334f, on an RTX 5070 Ti (617.42) and an AMD Radeon iGPU, RDNA2 (1002:13c0, 32.0.21042.62, ANYPS5_GPU=AMD). agc_driver_rcp_correctly_rounded and 10 related division/vop1 tests pass on both GPUs. With SpirvAluEmitterMath.cpp reverted to main, the new test fails on both (NVIDIA: v_rcp_f32 of 0x858a127e is 0xf96d5337; AMD: 0x0f5b5acd is 0x6f956235).

A wider probe ran 283,697 inputs per GPU, covering denormals, ±0, inf, NaN, exponents 1/2/253/254, powers of two and all-ones significands.

  • On main, 34,399 of 262,144 random normals are 1 ulp off on NVIDIA, and 28,044 on AMD.
  • With this PR, 0 are off on both GPUs, and the NVIDIA and AMD outputs are bit-identical.

One limit of the two-step refinement: when the estimate is 1 ulp off, one input per exponent still comes out wrong, the one whose significand is all ones. An exhaustive CPU model shows this (std::fma, every positive normal, estimates ±1..6 ulp). For example, the correctly rounded result for x = 0x3fffffff is 0x3f000001, but estimates 0x3f000000 and 0x3f000002 both end at 0x3f000000.

Both GPUs here return exact estimates for all 508 such inputs, so nothing fails, but Sample() cannot generate them. Adding e << 23 | 0x7fffff for every exponent to the test would cover a device whose estimate is off there.

Side note, pre-existing and unchanged: both GPUs flush denormals despite float mode 0xf0. Denormal inputs give ±inf, and |x| ≥ 2^126 gives ±0.

AI-assisted: yes (Claude Code); results above were run on this machine.

Two Newton steps from an estimate one ulp off still miss the correctly
rounded reciprocal of an all-ones significand (0x3fffffff: estimates
0x3f000000 and 0x3f000002 both end at 0x3f000000). One fused Newton step
then a choice between the result and its neighbour in the direction of
the exact residual fma(-x, y, 1), keeping the smaller residual, is
correctly rounded for every significand and estimates up to six ulps
off (exhaustive model with hardware FMA), and uses two fewer FMAs.

The test adds the all-ones significand of every exponent it covers, with
both signs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBXM4pgkMHYEtqN1EQni2P
@tibzejoker

tibzejoker commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor Author

@wachi35676 thanks, good catch on the all-ones significand. Pushed c402b92, which changes the last step: one fused Newton step, then the closer of the result and its neighbour in the direction of the exact residual fma(-x, y, 1).

An exhaustive CPU model with hardware FMA, over all 2^23 significands with estimates from -6 to +6 ulps around the correct result:

  • two Newton steps (previous head): 12 wrong of 109,051,904;
  • one step + neighbour choice (new head): 0 wrong.

It also costs two fewer FMAs. The test now adds e << 23 | 0x7fffff for every exponent it covers, with both signs. It passes on the RTX 4090 and the UHD 770, and ctest -R ^agc_ is still 727/728 (agc_driver_bda_device, as on main).

AI-assisted (Claude Code).

@wachi35676

Copy link
Copy Markdown
Contributor

Re-tested at c402b9217 on Windows 11 (MinGW GCC 15.2), merged onto main ad502d717, on an RTX 5070 Ti and an AMD Radeon iGPU, RDNA2 (1002:13c0, ANYPS5_GPU=AMD).

  • Tests: agc_driver_rcp_correctly_rounded and 10 related division/vop1 tests pass on both GPUs. With SpirvAluEmitterMath.cpp reverted to main, the new test fails on both (NVIDIA: 0x858a127e is 0xf96d5337; AMD: 0x0f5b5acd is 0x6f956235).
  • GPU probe: the same 283,697-input probe as before gives 0 one-ulp errors on both GPUs, including the 508 all-ones significands. The NVIDIA and AMD outputs are bit-identical, and also identical to 3e8a9380d, so denormal, inf, NaN and ±0 handling is unchanged. For comparison, on main 34,399 (NVIDIA) and 28,044 (AMD) of 262,144 random normals are 1 ulp off.
  • Exhaustive CPU model of the new last step: hardware FMA, every positive and every negative normal with a normal result (2,113,929,217 per sign), estimates −6..+6 ulps. It gives 0 wrong, all-ones significands included; the two-step version gives 3,019 wrong per sign on the same run. The residual fma(-x, y, 1) was exact in every case, so comparing residuals picks the correctly rounded value.

AI-assisted: yes (Claude Code); results above were run on this machine.

@jotitadev jotitadev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apple GPU / MoltenVK run. Metal's division is already correctly rounded on this test's inputs, so on Apple the change keeps the same values, and it also holds when MoltenVK's fast math is forced on.

Machine: Apple M2 Max, macOS 27.2, x86-64 build under Rosetta. Head c402b92, PR diff applied cleanly. Drivers: MoltenVK 1.4.2 (Vulkan SDK 1.4.363.0) and MoltenVK main + SPIRV-Cross #2704.

Tested: the tree is main a373e85 + mugurc's macOS series (ci/macos-series-2) + the PRs needed to build on macOS (#6667, #6668, #7524, #6969, and two files of #7332), not plain main (it doesn't build on macOS yet). #7254 was applied in the same build (disjoint files). Base on that tree: 40 ^agc failures with MoltenVK 1.4.2 and 26 with MoltenVK main + #2704, the same known failures in every run below.

  • agc_driver_rcp_correctly_rounded, agc_driver_vop1_float_unary and agc_driver_division pass 3 of 3 on both drivers.
  • Full ^agc suite with the PR: 40 of 729 fail on MoltenVK 1.4.2 and 26 of 729 on MoltenVK main + #2704, the same tests as the base on each driver.
  • The test alone on the base tree (SpirvAluEmitterMath.cpp at the base) also passes on both drivers, 3 of 3, with ANYPS5_NO_SHADER_CACHE=1. The module is a plain OpFDiv %float_1 and declares SignedZeroInfNanPreserve 32, so MoltenVK's default fast-math mode ("on demand") builds it without fast math, and Metal's precise 1.0 / x gives the correctly rounded value on all inputs the test generates, all-ones significands included.
  • With MVK_CONFIG_FAST_MATH_ENABLED=1 (forced on, not the default) the base fails: v_rcp_f32 of 0xb5b0e85a is 0xc9393a04, expected the correctly rounded 0xc9393a03. With this PR it passes on both drivers with fast math forced on.

So Apple is a fourth vendor here, and so far the only one already correctly rounded on these inputs, which means this test can't catch a regression on a default Apple setup. If the target moves to RDNA's hardware values, as discussed above, Apple will need the emulation as well.

GPUs tested: Apple M2 Max (MoltenVK 1.4.2; MoltenVK main + SPIRV-Cross #2704)

Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.

@tibzejoker

Copy link
Copy Markdown
Contributor Author

Thanks @jotitadev, a fourth vendor is very useful. So on Apple the default build was already right, and only forcing fast math on breaks the base (0xb5b0e85a -> 0xc9393a04 instead of 0xc9393a03), which the PR fixes.

You're right that the test cannot catch a regression on a default Apple setup. It still catches one on NVIDIA and Intel (both fail on main, see the description) and on MoltenVK with fast math forced on, so I'll leave it as is. If the target moves to RDNA's hardware values later, the expected values change for every vendor, Apple included, and the test will cover all four.

Written with AI assistance (Claude Code).

@thuanquang

Copy link
Copy Markdown
Contributor

@tibzejoker here is the sweep you asked for: native v_rcp_f32 for every input 0x3f800000 | m, m = 0 .. 0x7fffff, on an RX 6700 XT (gfx1031, RDNA 2), through HIP (wave32, IEEE=1, denormals kept).

Data: rcp_f32_gfx1031_words.bin.xz (2^23 little-endian u32, the hardware word for each m, 2.3 MB) and rcp_f32_gfx1031_delta.bin.xz (2^23 signed bytes, hardware minus correctly rounded, 148 KB), on a release of my fork. The reference is round-to-nearest-even of 2^47 / (2^23 + m) in integers, so it is exact.

hardware minus correctly rounded inputs
0 7,490,933
-1 ulp 761,701
+1 ulp 135,974
  • 897,675 of 8,388,608 (10.7 %) differ, never by more than one ulp, in both directions. The rate is close to the 889/8192 on your fixture.
  • The known input agrees: m = 0x5b5acd (the mantissa of 0x0f5b5acd) gives 0x3f156235, where the correctly rounded word is 0x3f156236, as on gfx1036 and gfx1100.
  • The misses sit around the rounding midpoint. In a 1-in-7 sample of m, every differing input has an exact result between 0.12 and 0.82 ulp above the representable value below it, most of them between 0.4 and 0.7; none is within 0.1 ulp of a representable value. For example m = 0x9b4 (exact 16772249.47 x 2^-24) comes out rounded up, and m = 0x5a9 (exact 16774318.5005 x 2^-24) rounded down.
  • v_rcp_iflag_f32 returned the same word as v_rcp_f32 for all 2^23 inputs.

Desktop RDNA 2, not the console's gfx1013.

@tibzejoker

Copy link
Copy Markdown
Contributor Author

Thank you @thuanquang, this is exactly the data I asked for. I checked it here: the words file agrees with your delta file and with an integer correctly rounded reference (761,701 one ulp low, 135,974 one ulp high), and 0x5b5acd gives 0x3f156235 as on gfx1036/gfx1100.

What the table shows about the hardware:

  • Bounding the internal value y by the rounding (h - 0.5 <= y <= h + 0.5 ulp) gives a smooth error of about ±0.2 ulp that repeats in 32 segments of 2^18 mantissas (top 5 bits), with a cubic shape: the residual of a table-driven quadratic interpolation.
  • A polynomial per segment does not reproduce it exactly. A linear program over every input of a segment still needs 0.016 ulp more than the rounding allows with a cubic over 2^18, and 0.02 to 0.05 ulp with a quadratic over 2^15 or 2^16. Truncating the input of the square term makes the fit worse. The remainder looks like truncated partial products inside the fixed-point datapath.

So matching v_rcp_f32 bit for bit would mean recovering the integer coefficients and the widths of that datapath per segment. That is a reverse-engineering project of its own, and it could be a follow-up if someone wants it. This PR stays what its title says: correctly rounded on every device. It already agrees with the hardware on 89.3 % of inputs and is never more than one ulp away, as the hardware itself is from the exact value. Your table is the reference for that follow-up.

AI-assisted: yes (Claude Code)

@adriadam10 adriadam10 self-assigned this Oct 19, 2026

@adriadam10 adriadam10 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough testing and the exhaustive model. The blocker is the target, not the implementation:

  • Real RDNA v_rcp_f32 / v_rcp_iflag_f32 are not correctly rounded: the oracle runs in this thread (gfx1036, gfx1100, and @thuanquang's gfx1031 sweep) show 889/8192 inputs of this PR's own fixture one ulp off the rounded reference (0x0f5b5acd is 0x6f956235 on HW, the test expects ...36).
  • On AMD hosts RADV lowers 1/x to the native v_rcp_f32, so main is already bit-identical to the console there. This PR makes AMD wrong on ~11% of inputs, and RcpCorrectlyRounded.cpp:101 pins that wrong value as the expectation (Rounded() at RcpCorrectlyRounded.cpp:59).
  • "Same on every vendor" is only worth it if it is the hardware value; a deterministic result that differs from the console is not better for the GTA V compute pass than the current one, and costs ~10 SPIR-V instructions per rcp (SpirvAluEmitterMath.cpp:919).

What would fix it: emulate the hardware result (the quadratic-interpolation model you derived from @thuanquang's table), with the test expectations taken from the HW sweep (gfx1013 is expected to match gfx1031/1036/1100). Please include the mismatch count of the emulation against the full 2^23 sweep (0 expected) and the added instruction count. If the exact algorithm cannot be recovered, keep main and say so rather than pinning correct rounding.

@tibzejoker

Copy link
Copy Markdown
Contributor Author

Thanks @adriadam10, agreed: a deterministic value that is not the console's is not worth ~10 instructions per rcp, and pinning correct rounding in the test would make RADV wrong where main is already exact.

I'll make one bounded attempt to recover the datapath from @thuanquang's full sweep. If it reaches 0 mismatches over the 2^23 inputs, I'll rewrite this PR around it, with the expectations taken from the sweep and the instruction count. If it doesn't, I'll close this PR and leave main as it is. Converting to draft meanwhile.

AI-assisted: yes (Claude Code)

@tibzejoker
tibzejoker marked this pull request as draft October 19, 2026 17:04
@tibzejoker

Copy link
Copy Markdown
Contributor Author

Closing, as said above: the bounded attempt did not reach 0 mismatches, so main stays as it is (bit-identical to the console on RADV).

What the gfx1031 sweep did show, for whoever picks this up later:

  • The mantissa path is a cubic over 32 segments (i = m >> 18, t = m & 0x3ffff), not a Newton step: a Newton-Raphson model needs 4x more slack than the polynomial.
  • The square term uses t >> 1: odd inputs sit lower than even ones by an amount linear in t, which matches dropping the LSB of the squarer input.
  • c0 has 2 fractional bits (units of 1/4 ulp) and the final rounding is round to nearest even: in the 10 segments where c0 ends in .5, the result at t = 0 always goes to the even mantissa.
  • Required coefficient precision is about 2^-21 / 2^-15 / 2^-10.5 relative for c1 / c2 / c3, which suggests tables of ~22 / 16 / 11 bits.
  • What is left: per-input noise of about ±0.016 ulp. It is already there for t < 256 and does not scale with the segment, so it most likely comes from a truncated C1 * t multiplier with an integer C1 that real-valued fits cannot pin down. With real coefficients, the best model still misses 66,054 of 2^23 inputs, against 897,675 for correct rounding.

The next step would be an exhaustive search of the integer C1 per segment under truncated array-multiplier rules. Even then, an exact emulation would cost a 32-entry table and a cubic per rcp. Thanks @thuanquang for the sweep and everyone who ran the oracle.

AI-assisted: yes (Claude Code)

@tibzejoker tibzejoker closed this Oct 19, 2026
@tibzejoker

Copy link
Copy Markdown
Contributor Author

Follow-up to the closing note: the datapath is now known exactly.

AMD's rocjitsu emulator (ROCm/rocm-systems, emulation/rocjitsu/lib/util/include/util/amdgpu_rcp.h) describes the normalized FP32 reciprocal as captured on gfx1100 and gfx1201. It is the same shape as the one recovered above, with two parts that were missing here:

  • The linear term C1 * t is rounded to nearest even before it is used. The shift is 14 bits, or 15 when the product is at least 2^38.
  • After the cubic, 2,761 mantissas get a fixed +1 or -1 ulp correction.

I wrote that formula independently in Python and checked it against @thuanquang's gfx1031 sweep (all 2^23 significands). Result: 0 mismatches out of 8,388,608. The cubic alone misses exactly those 2,761 inputs. So RDNA 2 (gfx1031), RDNA 3 (gfx1100) and RDNA 4 (gfx1201) compute the same reciprocal, which very likely means the PS5 GPU does too.

That makes a bit-exact v_rcp_f32 possible on every vendor. Its cost:

  • 64-bit integer arithmetic for the cubic;
  • a lookup in a 2,761-entry correction table, which can be rebuilt from the sweep and does not need to be copied;
  • far more work than the earlier two Newton steps.

My plan would be:

  • use it only in the exact mode;
  • keep the native instruction where the driver already emits v_rcp_f32 (RADV);
  • test it against the full sweep on NVIDIA and Intel.

@adriadam10, would you take that as a new PR, or do you see a better place for it?

@tibzejoker

Copy link
Copy Markdown
Contributor Author

Follow-up: the bit-exact reciprocal is now #9305, opt-in behind APS5_EXACT_RCP so that the default output and SPIR-V size don't change. It matches all 2^23 mantissas of the gfx1031 sweep.

AI-assisted: yes (Claude Code)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* shader core/shader

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants