Skip to content

fix(libc): call guest unwind callbacks with the System V ABI on Windows - #6851

Merged
adriadam10 merged 1 commit into
boykopovar:mainfrom
GitamarT045T3R:fix/libc-unwind-callback-abi
Oct 19, 2026
Merged

adriadam10 merged 1 commit into
boykopovar:mainfrom
GitamarT045T3R:fix/libc-unwind-callback-abi

Conversation

@GitamarT045T3R

@GitamarT045T3R GitamarT045T3R commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

What

On Windows, AnyPS5 called three kinds of guest callbacks with the Microsoft x64 convention instead of System V:

  • the trace function passed to _Unwind_Backtrace,
  • the stop function passed to _Unwind_ForcedUnwind,
  • the exception_cleanup of an exception deleted by _Unwind_DeleteException, including the foreign exceptions that __cxa_end_catch deletes.

Their types in UnwindAbi.hpp (_Unwind_Trace_Fn, _Unwind_Stop_Fn and _Unwind_Exception::exception_cleanup) had no APS5_VABI. So trace(&context, argument) in Unwind.cpp put the context and the argument in rcx and rdx, while the guest function reads rdi and rsi; the stop function and the cleanup got wrong arguments the same way. The relinker passes guest function pointers to libc unchanged, and the other guest callbacks in libc already have APS5_VABI, e.g. the constructor and destructor of __cxa_vec_ctor_nid_postfix. vkQuake (PPSA99010) imports _Unwind_Backtrace and reads each frame in its callback (db3ec02), so on Windows its callback got a wrong context.

Changes:

  • The three types now have APS5_VABI. Cleanup, the exception_cleanup of the exceptions libc throws, has it too. With the new types the compiler rejects any other host function stored in them on Windows, and Cleanup was the only one.
  • On Windows, _Unwind_DeleteException also deletes exceptions thrown inside libstdc++-6.dll when code that takes __cxa_begin_catch and __cxa_end_catch from libc.prx catches them. Their class is GNUCC++ and their cleanup is a host function, so _Unwind_DeleteException_nid_postfix passes them to libgcc's _Unwind_DeleteException, which calls the cleanup as before. With every cleanup called as System V, a small program built with SEH unwind tables that catches std::stoi("x") crashed in __gxx_exception_cleanup; with this change it catches the exception 3 of 3 times, as on main. The AnyPS5 modules themselves are built with -fno-asynchronous-unwind-tables, so they cannot catch such an exception, on main either: windows_exception_tests reports the same std::stoi exception as undeliverable.
  • Calling the host cleanup through a cast in the same function does not work: GCC 15.2 at -O3 inlines _Unwind_DeleteException_nid_postfix into __cxa_end_catch_nid_postfix and merges the two indirect calls, which differ only in their calling convention, into the System V one (-fno-tree-tail-merge keeps both). The call to libgcc is a direct call, so it cannot be merged.
  • Linux and macOS are unchanged because APS5_VABI is empty there.

New test core/libs/tests/UnwindCallbackAbi.cpp, registered on every platform as five ctest tests. Each passes callbacks declared APS5_VABI like a guest does:

  • unwind_callback_abi_backtrace: the trace function gets the argument given to _Unwind_Backtrace and a context whose region start is the function that called _Unwind_Backtrace.
  • unwind_callback_abi_forced_unwind: the stop function gets version 1, _UA_CLEANUP_PHASE | _UA_FORCE_UNWIND, the exception class, the exception, the argument given to _Unwind_ForcedUnwind and the context of its caller.
  • unwind_callback_abi_delete_exception: _Unwind_DeleteException calls exception_cleanup once with _URC_FOREIGN_EXCEPTION_CAUGHT and the exception.
  • unwind_callback_abi_end_catch: the same after __cxa_begin_catch and __cxa_end_catch on a foreign exception.
  • unwind_callback_abi_end_catch_gnu: the same with a GNUCC++ exception whose cleanup is a host function.

ExceptionRuntime.cpp still passes host lambdas as these types. It is built only on Linux and macOS, where the types are unchanged.

My open PR #6819 adds an unwind_cursor test whose TraceFirstFrame is a host function passed to _Unwind_Backtrace_nid_postfix. The two branches merge without conflicts, but once both are in, that test no longer compiles on Windows until TraceFirstFrame is marked APS5_VABI. I'll add that to whichever of the two is merged second.

Tested

Windows 10 Pro 22H2 (19045), MinGW-w64 GCC 15.2.0 (WinLibs r7), Ninja, Release, BUILD_TESTING=ON, on main 82ddae75b. The new test binary was run 20 times per test against three builds of libc.prx:

Test main Every cleanup System V This PR
backtrace fails: the trace callback gets the argument given to _Unwind_Backtrace passes passes
forced_unwind fails: the stop function gets version 1 passes passes
delete_exception fails: exception_cleanup gets _URC_FOREIGN_EXCEPTION_CAUGHT passes passes
end_catch fails: exception_cleanup gets _URC_FOREIGN_EXCEPTION_CAUGHT passes passes
end_catch_gnu passes fails: exception_cleanup gets _URC_FOREIGN_EXCEPTION_CAUGHT passes
  • The 31 tests matched by ctest -R "exception|unwind|backtrace|personality|terminate|signal" pass, including windows_exceptions, windows_backtrace_*, guest_exception_*, guest_raise_exception, guest_unwind_module_info, guest_mutex_raise_exception, unwind_record_bounds, undeliverable_host_exception and guest_signal*.
  • Full ctest -j 12: 1324 of 1326 pass. agc_shader_disk_cache fails with Access is denied as on main on Windows 10 (test(agc): delete mapped shader cache entries with POSIX semantics on Windows #6415), and agc_driver_gpu_indirect_arguments_misaligned_writer_ahead_tests does not build with MinGW on main (alignment 65536 above the object file maximum of 8192, test(agc): align the indirect arguments block at run time #6452).
  • python tools/check_conventions.py --base origin/main passes.
  • Not built or run on Linux or macOS. Not tested with vkQuake.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Unblocks: PPSA99010 (vkQuake; reproduced by @DavidSamir below, no Game bring-up discussion yet)
  • GPUs tested: n/a
  • AI-assisted: yes

_Unwind_Backtrace and _Unwind_ForcedUnwind call the trace and stop
functions a title passes them, and _Unwind_DeleteException calls the
exception_cleanup of the exception it deletes. Their types in
UnwindAbi.hpp had no APS5_VABI, so on Windows these guest functions were
called with the Microsoft x64 convention: the context and the argument
went to rcx and rdx while the guest reads rdi and rsi. vkQuake
(PPSA99010) imports _Unwind_Backtrace and reads each frame in its
callback (db3ec02).

The three types are now APS5_VABI, like the callbacks of __cxa_vec_ctor
and the other libc exports, and so is Cleanup, the exception_cleanup of
the exceptions libc throws. An exception thrown by libstdc++ on Windows
(class GNUCC++) has a host cleanup, so _Unwind_DeleteException passes it
to libgcc's _Unwind_DeleteException, which calls that cleanup as before.
Linux and macOS are unchanged.
@github-actions

Copy link
Copy Markdown

Progress report (82ddae7 → 3168727)

➖ System libraries: 88.83% (+0.0%, +1 implemented functions)

✅ 1 implemented
Library Name
libc Cleanup

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 5021 core/libs/prx/libc/CMakeLists.txt
PR 6044 core/libs/prx/libc/CMakeLists.txt
PR 6079 core/libs/prx/libc/CMakeLists.txt
PR 7081 core/libs/prx/libc/CMakeLists.txt
PR 7084 core/libs/prx/libc/CMakeLists.txt
PR 7201 core/libs/prx/libc/CMakeLists.txt
core/libs/prx/libc/src/exception/Runtime.cpp
core/libs/prx/libc/src/exception/Unwind.cpp
PR 7332 core/libs/prx/libc/CMakeLists.txt
core/libs/prx/libc/src/exception/Unwind.cpp
PR 7417 core/libs/prx/libc/CMakeLists.txt
PR 7525 core/libs/prx/libc/CMakeLists.txt
PR 7715 core/libs/prx/libc/CMakeLists.txt

@DavidSamir

DavidSamir commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Title run on Windows with vkQuake (PPSA99010, catalog release 1.0) and the free Quake shareware data, so the crash this PR describes can be reproduced without anyone's own game files.

Windows 11 Pro build 26300, Intel i7-11700F, NVIDIA GeForce RTX 3070, WinLibs GCC 15.2, Release, relinked with --windows. PR head merged onto main c842491 (merge bf79bc0920, no conflicts). Data: the shareware id1/pak0.pak from id Software's quake106.zip (18,689,235 bytes, MD5 5906e5998fc3d896ddaf5e6a62e03abb), placed at /app0/id1/pak0.pak (the title's Basedir is: /app0). An id1/autoexec.cfg with three lines makes the title reach its multiplayer search without any controller input:

r_rtshadows 0
slist
connect 127.0.0.1

(r_rtshadows 0 keeps the ray-traced shadow compute shader, which takes tens of seconds to build here, out of the way: #4977.)

Build Runs Result
main c842491 3 3 of 3 end after 6 to 7 s with FATAL: unhandled exception 0xc0000005, rip app.exe+0x7a5ed3, read of 0x0
main c842491 + this PR 3 3 of 3 run the 60 s limit; the game prints No Quake servers found., a 10-frame STACK TRACE, Host_Error: CL_Connect: connect failed (expected: nothing listens on 127.0.0.1) and goes back to its demo loop

What the crash is, from the disassembly of the relinked app.exe and the traced library calls (--trace-imports, APS5_TRACE_CALLS=all): main stops in the title's own stack-trace printer. Its last traced calls are _Unwind_Backtrace, vsnprintf, vprintf (the === STACK TRACE === header), pthread_once, strlen, and then strtok_r is entered with a NULL string at app.exe+0x7a5ed3, because the backtrace gave it no frames. With this PR the callback gets its arguments and the same printer lists 10 frames ([no symbols], as the title has none).

Controller-only runs (the fixed pad sequence, 150 s): on main 82ddae7 the same crash at the same address in 4 of 4 runs (it takes 85 to 100 s to reach the multiplayer search); on main c842491 neither of 2 runs reached the search screen (no crash, no stack trace), and with this PR both of 2 runs reached it and neither crashed. The menu timing depends on frame pacing (the title runs at about 6 fps here), so only the autoexec.cfg table above is a like-for-like comparison.

AI-assisted: yes

@GitamarT045T3R

Copy link
Copy Markdown
Contributor Author

@DavidSamir thank you, that is the reproduction this PR needed: a public title with free data, where main crashes in the title's own stack-trace printer 3 of 3 times and this PR runs the full minute 3 of 3 with a 10-frame backtrace. I've set the checklist to Unblocks: PPSA99010; there is no Game bring-up discussion for vkQuake yet.

@github-actions github-actions Bot added the unblocks title Names a title ID on the Unblocks line label Oct 18, 2026
@adriadam10 adriadam10 self-assigned this Oct 19, 2026
@adriadam10
adriadam10 merged commit 034c4ec into boykopovar:main Oct 19, 2026
16 checks passed
@adriadam10

Copy link
Copy Markdown
Collaborator

Checked the diff against the description: the three callback types and Cleanup get APS5_VABI, GNUCC++ exceptions go through libgcc's _Unwind_DeleteException on Windows, and the five new tests are registered in ctest and pin the arguments each callback receives. The vkQuake reproduction by @DavidSamir (Windows, PPSA99010) covers the title side. Merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build libc unblocks title Names a title ID on the Unblocks line

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants