[EagleBearerYS]

What

A guest signal handler ran with the mask its signal was raised under. Dispatch and DeliverPendingSignal in Signals.cpp called it without blocking anything, so:

• a handler that raised its own signal re-entered itself at once, instead of the signal staying pending until the handler returned;
• sigprocmask inside a handler did not report the signal or the handler's sa_mask as blocked;
• SA_RESETHAND left the handler installed for the next delivery.

FreeBSD sets up the handler's frame and then, in postsig_done, blocks sa_mask and, unless SA_NODEFER is set, the signal itself. It also resets the disposition to SIG_DFL when SA_RESETHAND is set (sigdflt), and sigreturn restores the previous mask. sigaction keeps reporting SA_RESETHAND afterwards (L707-L708), because sigdflt does not clear ps_sigreset, which kern_sigaction sets from the flag.

The handler now runs with its signal (unless SA_NODEFER) and the first 32 signals of its sa_mask added to the guest mask. When it returns, only the signals it added are unblocked again, so a change another thread makes to the mask meanwhile is kept. Pending signals that this unblocks are delivered when raise returns, as _sigprocmask already does. SA_RESETHAND resets the handler to the default before it runs.

A handler that leaves with longjmp never returns there, so its signal would stay blocked. FreeBSD's setjmp saves the signal mask in the jump buffer and its longjmp restores it (setjmp.S), but ours kept only the registers. libc's setjmp now saves the mask at 72 in the jump buffer, with the flag sigsetjmp keeps at 88, and longjmp restores it. libc doesn't link libkernel, so libkernel registers its sigprocmask with libc when it loads. Both changes follow CnRJay's review.

The mask is still main's process-wide 32-bit mask. #3986 makes it per thread; with that, the handler's mask belongs to the thread the signal is delivered on, so whichever of the two lands second adapts. The commits:

• the handler mask and SA_RESETHAND, with guest_signal_handler_mask;
• the sigaction entry in docs/dev/debt/libkernel.md;
• unblocking only what the handler added;
• libc's setjmp and longjmp saving and restoring the mask, with a docs/dev/debt/libc.md entry;
• a check that leaves a handler with longjmp and raises its signal again.

No title is claimed.

Tested

On GitHub's runners from my fork, on main at 0214069 (run):

• Linux: Ubuntu 24.04.5, GCC 13.3.0, Release with CI's configure flags and -DANYPS5_ENABLE_SPIRV_TOOLS=ON.
• Windows: Windows Server 2025 (10.0.26100), WinLibs MinGW-w64 GCC 15.2.0 r7 (UCRT), Release with CI's configure flags.

On both:

• New guest_signal_handler_mask. A handler raises its own signal, both on SIGUSR2, which has no host signal, and on SIGTERM, which goes through the host. The test also checks SA_NODEFER, the mask sigprocmask reports inside the handler, and SA_RESETHAND. With main's Signals.cpp it aborts at its first check, where the nested raise re-enters the handler. With the change it passes 20 runs in a row.
• Every test registered under core/libs/prx/libc and core/libs/prx/libkernel passes: 69 and 124 on Linux, 90 and 127 on Windows.
• tools/check_conventions.py --base origin/main passes for both commits.

After the review, on the same runners:

• Third commit (run): guest_signal_handler_mask gains a check where another thread blocks SIGTERM while the handler runs. It aborts with the first two commits and passes 20 runs in a row with the third.
• Last two commits (run): guest_signal_sets gains setjmp/longjmp checks (the mask comes back, saved at 72 with the flag at 88), and guest_signal_handler_mask leaves a handler with longjmp and raises the signal again. Both abort with the first three commits and pass 20 runs in a row with all five.
• On both runs, every test under core/libs/prx/libc and core/libs/prx/libkernel passes: 69 and 124 on Linux, 90 and 127 on Windows. tools/check_conventions.py passes for all five commits.
• Not tested: macOS, any title.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none
• AI-assisted: yes

[CnRJay]

Blocking the signal and sa_mask during the handler matches postsig_done. One interaction this introduces: a handler that leaves by longjmp now leaves its signal blocked for good.

RunHandler restores the mask after handler(guest) returns. A handler that jumps out never returns there, and leaving a SIGSEGV/SIGBUS/SIGFPE handler through a jump is the classic pattern for guarded probes and crash recovery. What happens next depends on which jump the guest uses:

• siglongjmp with a sigsetjmp(env, 1) buffer restores the saved mask through _sigprocmask(SIG_SETMASK) (SignalSets.cpp), so that case works.
• longjmp (JumpBuffer.cpp in libc) doesn't touch the mask. On FreeBSD, setjmp/longjmp do save and restore the signal mask (BSD semantics, unlike _setjmp/_longjmp). On main that gap was invisible, because nothing was blocked during a handler. With this PR, a guest that leaves a handler with longjmp keeps that signal blocked, and the next fault of that kind is never delivered. Instead it stays pending, or the host kills the process on a synchronous fault it can't deliver.

So either setjmp/longjmp in libc should save and restore the mask as FreeBSD's do (the same _sigprocmask calls sigsetjmp(env, 1) makes), or this PR should land together with that change. A test where a handler leaves with longjmp and the same signal is then raised again would pin it.

Smaller: blockedMask.store(saved) after the handler overwrites any change another thread made to the process-wide mask while the handler ran. The description notes that #3986 moves to per-thread masks. Until then, fetch_and(~added) with only the bits this handler added (mask & ~saved) would avoid clobbering concurrent changes.

[EagleBearerYS]

Thanks, both points hold, and both are fixed now.

• When a handler returns, only the signals it added are unblocked again (fetch_and(~added)), so a mask change another thread makes meanwhile is kept. That's ef45904, with a check where another thread blocks SIGTERM while the handler runs.
• libc's setjmp now saves the signal mask at 72 in the jump buffer and longjmp restores it, as FreeBSD 11.4's setjmp.S does (624815b). libc doesn't link libkernel, so libkernel registers its sigprocmask with libc when it loads. 9f584de makes guest_signal_handler_mask leave a handler with longjmp and raise the same signal again: without the libc change, the second raise stays blocked.

On GitHub's runners (run), the new longjmp checks in guest_signal_handler_mask and guest_signal_sets fail without the libc change and pass 20 runs in a row with it, on Linux and Windows, and the libc and libkernel suites pass.

[oneandonlydean]

Closing: no title is known to reach this. With the queue at 700+ open PRs, please hold off on new PRs that don't name a title needing the change until it's down. If a title stops on this on current main, reopen with the title ID and the log line.