Skip to content

fix(libkernel): honor caller-provided pthread stacks - #9938

Open
BorchBe99 wants to merge 1 commit into
boykopovar:mainfrom
BorchBe99:pr/caller-provided-stacks-20261022
Open

BorchBe99 wants to merge 1 commit into
boykopovar:mainfrom
BorchBe99:pr/caller-provided-stacks-20261022

Conversation

@BorchBe99

@BorchBe99 BorchBe99 commented Oct 22, 2026 •

Copy link
Copy Markdown

What

scePthreadCreate ignored a stack address set in the thread attributes. A thread created with one now runs on that memory, with the given size, on Linux and Windows. This finishes the caller-stack follow-up from #1988. The caller keeps ownership of the memory. Host startup still runs on the native stack, and TLS stays per host thread.

Caller stacks report guard size 0. Thread identity and the stack's lifetime survive return, explicit exit, cancellation and key destructors. On Windows the TEB stack bounds cover the caller's range while the thread runs and are set back to the native bounds before it ends; existing context-stack registrations and pins stay. When longjmp, siglongjmp or setcontext lands on a caller stack, the TEB gets that stack's range, also when a larger registered context stack contains it.

A caller stack has no guard page below it; the libkernel debt file records this.

Merge order with the four open PRs that share behavior with this one:

If one of you needs another order, reply here and I will change Depends on. #7192 (@Jev1337), #8477 (@ycarogabriel-lgtm), #9160 (@yabanana), #9482 (@valenciana-nc) and #9573 (@zaryabzip) can merge before or after this PR. This comment lists what the later one keeps; with #9482, the new test's joined-thread case has to expect ESRCH.

Tested

On clean upstream cecb540a, a probe thread's stack locals lie outside the range it was given.

The new regression test covers all three setter paths, small stacks with large TLS, detach and reuse, and mixed cleanup chains including libc _Thrd_exit. Threads with and without a caller stack catch a host exception and then unwind with the standard unwinder across the entry trampoline to the host stack. On Windows it also checks the TEB range after a uc_link return onto a caller stack inside a larger registered context stack.

Linux x86_64, GCC 15.3.0, tree 9c08e18d on upstream 938812df (main on 2026-10-23):

  • Full default build, libs and conventions pass. The regression passes, also in 20 repeats, and so do 8 other pthread tests and 9 neighbouring tests (libc threads, backtrace, mutex, cancel recovery, user context, memory, fixed mmap). Under ASan, 7 pthread tests and a separate range and lifecycle probe pass.
  • Full suite: of 1,969 registered tests, 1,946 pass, 19 are skipped and 4 fail (CTest exit 8): agc_driver_sample_order, agc_driver_sample_order_grid_reference, agc_driver_sample_order_recipe_depth_locations and agc_driver_sgpr_pattern_fill_keys. A clean build of 938812df fails them with the same assertions. tools/check_skips.py rejects 2 skips that come from this host: agc_driver_sparse_dma_buf_import and agc_host_import_sparse skip because its udmabuf size_limit_mb is above 1024, also on 938812df.

Windows 11 x64, local native run (not CI), tree 9c08e18d on upstream 938812df:

  • WinLibs GCC 15.2.0 (DWARF exceptions) and MSYS2 UCRT64 GCC 16.2.0 (SEH exceptions): in both, the regression (also in 20 repeats), 14 neighbouring pthread and libc-thread tests and 2 user-context tests pass. The WinLibs run also covers the full build, libs, conventions and 2 memory tests.
  • Full suite (WinLibs): all 2,030 registered tests ran with CTest exit 0; 1,381 passed, 649 were skipped, none failed. tools/check_skips.py --job windows rejects 2 skips that come from this laptop: long_paths (long paths are disabled in the registry) and guest_video_out_high_frame_rate (the display has only 60 Hz modes).

Broken versions of the change fail the regression. On Linux, with the trampoline's CFI damaged (CFA without the final +16, or no CFA expression), it aborts at the unwind check. On Windows (WinLibs, tree ebf12e33), if GuestStackSwitch_nid_no_patch keeps the current TEB range whenever it already contains the new stack pointer, it fails at the uc_link check.

Not covered: the host-bounds check runs on the main test thread and does not see a worker's final TEB state. macOS takes the same stack switch as Linux but was not built or run. Project CI, game compatibility and performance are not tested.

Checklist

🤖 Generated with Claude Code

@MarcZX10

Copy link
Copy Markdown
Contributor

I reviewed the caller-stack lifecycle across attribute capture, Linux/Windows thread startup, guest entry trampoline, return/exit/cancellation cleanup, stack-bound tracking, and the new lifecycle/unwind tests. The implementation consistently keeps host startup/TLS on the native thread, switches only for the guest entry, and restores native bounds before teardown; the test matrix described covers the main return, detach/reuse, cleanup and unwind cases. I ran git diff --check upstream/main...refs/review/9938 and python3 -I tools/check_conventions.py --base upstream/main --head refs/review/9938; both pass. Hosted conventions, intake and labels pass, but all build/test jobs are skipped by policy for this PR, so the extensive Linux/Windows runs in the description are contributor-reported, not hosted CI results. I could not run local C++ builds because the root filesystem is full.

Current main is 262 commits ahead of the merge base and has changed the shared libkernel test CMake file and debt entry; the pthread implementation paths themselves are unchanged. Please refresh those shared files before merge and preserve current registrations/debt. The listed adjacent PRs warrant the stated merge coordination. I found no independent blocker in this pass.

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 7192 docs/dev/debt/libkernel.md L163-164
PR 8477 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 9160 core/libs/prx/libkernel/Pthread/src/Thread.cpp L36
PR 9482 docs/dev/debt/libkernel.md L161-163
PR 9573 core/libs/prx/libkernel/Pthread/src/Thread.cpp L837

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 3965 core/libs/prx/libc/include/GuestStacks.hpp
core/libs/prx/libc/src/GuestStacks.cpp
PR 3986 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 4563 core/libs/prx/libkernel/CMakeLists.txt
core/libs/prx/libkernel/Pthread/include/Pthread.hpp
core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 4630 docs/dev/debt/libkernel.md
PR 5021 core/libs/prx/libkernel/CMakeLists.txt
core/libs/prx/libkernel/Pthread/include/Pthread.hpp
core/libs/prx/libkernel/Pthread/src/Thread.cpp
docs/dev/debt/libkernel.md
PR 6044 core/libs/prx/libkernel/CMakeLists.txt
PR 6149 docs/dev/debt/libkernel.md
PR 6740 core/libs/prx/libkernel/Pthread/include/Pthread.hpp
docs/dev/debt/libkernel.md
PR 6830 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 6936 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 7049 docs/dev/debt/libkernel.md
PR 7088 docs/dev/debt/libkernel.md
PR 7105 docs/dev/debt/libkernel.md
PR 7121 docs/dev/debt/libkernel.md
PR 7149 core/libs/prx/libc/include/GuestStacks.hpp
core/libs/prx/libc/src/GuestStacks.cpp
core/libs/prx/libkernel/Pthread/include/Pthread.hpp
core/libs/prx/libkernel/Pthread/src/Thread.cpp
docs/dev/debt/libkernel.md
PR 7201 core/libs/prx/libkernel/CMakeLists.txt
core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 7509 docs/dev/debt/libkernel.md
PR 7565 core/libs/prx/libkernel/Pthread/include/Pthread.hpp
core/libs/prx/libkernel/Pthread/src/Thread.cpp
docs/dev/debt/libkernel.md
PR 7653 docs/dev/debt/libkernel.md
PR 8010 docs/dev/debt/libkernel.md
PR 8520 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 8553 docs/dev/debt/libkernel.md
PR 8592 core/libs/prx/libkernel/Pthread/include/Pthread.hpp
core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 9074 docs/dev/debt/libkernel.md
PR 9133 docs/dev/debt/libkernel.md
PR 9199 docs/dev/debt/libkernel.md
PR 9256 docs/dev/debt/libkernel.md
PR 9284 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9293 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9297 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9302 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9557 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9561 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9566 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9575 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9669 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9693 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md

@MarcZX10

Copy link
Copy Markdown
Contributor

I reviewed head 65f0cc18fb41d6f3366f95f510df09a0680e62dc; current-main git diff --check and conventions pass. The report gives substantial Linux and Windows local coverage for supplied-stack entry, cleanup/exit, unwind, TEB bounds and ASan switching; project CI and title compatibility are explicitly not claimed. I could not rerun the C++ tests here because the root filesystem has no free space.

The live overlap scan identifies comparable shared hunks with #8041 (guard-size queries/attributes), #8477 and #9160 (thread startup), and #9684 (Linux stack placement); its comparisons against #4563 and #7149 are incomplete because those branches conflict with main. #7149 is particularly important to reconcile manually because both change GuestStacks and Windows pthread stack placement. #9573 also touches the pthread attribute/thread files, while #9482 shares the debt entry. The PR currently suggests merging in any order; please identify which behaviors are independent and preserve/retest them together, and add Depends on where one change is actually required first. I did not find evidence that these are duplicate implementations, and I am not treating file overlap alone as a blocker.

@BorchBe99
BorchBe99 force-pushed the pr/caller-provided-stacks-20261022 branch from 65f0cc1 to b1f5164 Compare October 23, 2026 01:50
@BorchBe99

Copy link
Copy Markdown
Author

Rebased onto 78becbb3; the new head is b1f51649. The patch is unchanged (same git patch-id). The test CMake file and debt entry merged without conflicts, so current registrations and debt entries are kept, and the branch also merges cleanly with main as of b1496678.

Linux on the rebased tree: the full build, the regression (also in 20 repeats), 9 neighbouring tests, ASan and conventions pass. Full suite: 1,826 of 1,848 pass, 19 skip, and the same 3 agc_driver_sample_order* tests fail as on the old base. tools/check_skips.py rejects 2 of the skips: two sparse-import AGC tests, new on main, that skip here because of this machine's udmabuf size limit. I did not rerun Windows because the patch is unchanged.

None of the PRs below needs another one merged first, so Depends on stays none. Neither side calls or relies on code the other adds; the order only decides who resolves the conflict. The description asked their authors about the order and none has asked for a different one so far; if one does, I'll add it to Depends on. Whoever merges second keeps both sides of each conflicting test registration, debt line and declaration, and also:

PR Shared code Keep when merging second Retest together
#8041 guard size in Pthread.hpp and both Attr.cpp guard 0 for caller stacks: _guardsize = thread->callerStack ? 0 : thread->guardSize guest_pthread_guard_size, guest_pthread_provided_stack, guest_pthread_attr, guest_pthread_attr_stack
#7149 GuestStacks, Windows scePthreadCreate, StartNativeThread and scePthreadExit caller-stack threads skip AllocateGuestStack, #7149's stack check and its mapping registration, keep their own address and run through our GuestStackEnter/GuestStackLeave path; our exit longjmp runs before #7149's exit block on Windows: guest_pthread_provided_stack, guest_pthread_stack, guest_user_context, guest_user_context_exit, guest_pthread_frame_chain, guest_kernel_is_stack, guest_pthread_attr_stack, guest_pthread_cancel; on Linux: guest_pthread_provided_stack, guest_pthread_stack
#4563 scePthreadExit, test CMakeLists.txt both lines, ours first, so the thread leaves the caller stack before it waits on Windows: guest_pthread_finished_stack, guest_pthread_provided_stack, guest_pthread_join_detach, guest_pthread_cancel, guest_pthread_stack
#9684 Linux StartNativeThread, test CMakeLists.txt, debt entry its size check, alignment and frame move only for library stacks, inside if (!self->callerStack) on Linux with and without ASan: guest_thread_stack_guard, guest_pthread_provided_stack, guest_pthread_stack, guest_kernel_is_stack, guest_kernel_stack_protect, guest_pthread_frame_chain, guest_pthread_attr_stack, guest_thread_self
#9482 debt entry, pthread_attr_get_np it returns ESRCH for a joined thread, so the joined-thread case in our test (RequireReported(thread, stack, false)) has to expect that and skip its base and size check guest_pthread_attr, guest_pthread_provided_stack, guest_pthread_attr_stack
#7192, #8477, #9160, #9573 adjacent lines (debt entry, macOS block in RunThread, includes, scePthreadCreate) both sides guest_pthread_provided_stack, each PR's own tests, and guest_pthread_attr for #9160 and #9573

Two of them also meet this PR at run time without a conflict: #9160's new pthread_attr_setstackaddr is another way to create a caller-stack thread, and with #9573 a suspended caller-stack thread waits on its caller stack (on Windows with the caller bounds set and its pages pinned). No test of either PR covers these combinations.

I have read the PR heads but not built the combinations. #7149 with this PR needs a Windows run, mainly to see whether its 256 KiB native stack is enough for caller-stack threads (this PR reserves 1 MiB).

@harisibrar1 harisibrar1 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked against main 3875f8c and FreeBSD 11.4 libthr (thr_create.c create_stack/_pthread_create, thr_attr.c):

  • The semantics match libthr. A stack given in the attributes gets guard size 0 and is never freed or reprotected by the library. Both setstack and setstackaddr treat the address as the lowest byte, and setstackaddr keeps the attr's current size. The existing EINVAL checks in both Attr.cpp files are unchanged and match.
  • The entry trampoline leaves the guest entry with rsp ≡ 8 mod 16, the same as FreeBSD's cpu_set_upcall (top &~0xf, minus 8). The DW_CFA_def_cfa_expression resolves to the native frame (*(rbp+16)+16), so unwinding back across it reads correctly.
  • Freeing the stack after scePthreadJoin is safe. Join returns only after WaitForSingleObject/pthread_join on the host thread, so the thread has left the caller stack, as with FreeBSD's thr_exit tid wakeup.
  • Small question: the non-Windows StartNativeThread branch is also the macOS path, so macOS now runs caller-stack threads too, while the description and the test registration (if(WIN32 OR CMAKE_SYSTEM_NAME STREQUAL "Linux")) cover only Linux and Windows. Could the description mention that macOS gets the same switch untested, or could the test be registered there once macOS links?
  • I couldn't build or run this here, so I can't confirm the Windows TEB and __chkstk behaviour myself. The new debt entry about host code, handlers and probes overrunning below the range reads accurately to me.

Reviewed with AI assistance (Claude Code); the checks above were run locally on my machine.

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The caller-stack implementation and its lifetime/unwind coverage were already reviewed in my earlier comments; the author subsequently rebased to b1f516491827f8e53af5f189d064926a9128f6e2 and reports a passing Linux build, repeated focused tests, neighboring tests, ASan and conventions. Those runs are contributor-reported; hosted product build/test jobs remain skipped.

This branch is no longer current with upstream: tools/check_main_sync.py --base upstream/main --head refs/reviews/pr-9938 reports 1 ahead /278 behind, with Thread.cpp, libkernel/CMakeLists.txt, and docs/dev/debt/libkernel.md changed on both sides since the merge base. The merge is clean, but #9960 has changed Thread.cpp since this branch was based. Please refresh against current main, preserve the caller-stack lifecycle alongside that thread-startup work, and rerun the focused Linux/Windows stack and unwind tests plus conventions on the refreshed head.

The existing overlap feedback on #8041, #7149, #4563 and #9684 also still needs an agreed integration order/dependency where the shared behavior requires it; a general “whichever is ready” statement does not identify dependencies among those specific changes. Please record the agreed order in the checklist. This request is for the repository's current-main and overlap rules, not queue volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BorchBe99
BorchBe99 force-pushed the pr/caller-provided-stacks-20261022 branch from b1f5164 to dca8f5b Compare October 23, 2026 20:18
@BorchBe99

Copy link
Copy Markdown
Author

Thanks for checking it against libthr.

macOS: yes, the non-Windows branch of StartNativeThread is its path too, so a macOS thread with a caller stack switches to it the same way. I have no Mac to build or run it on, so I added macOS to "Not covered" in the description. The test stays registered for Linux and Windows; once #8658 (macOS host support) is in, it can be added for macOS after a run there.

Windows: the TEB bounds were checked in the native runs under Tested, with WinLibs and with MSYS2 UCRT64. __chkstk has no test of its own.

@BorchBe99

Copy link
Copy Markdown
Author

Refreshed onto 938812df (main on 2026-10-23); it also merges cleanly with main as of b4cafbe2. The rebase changed only context lines. #9960 turns stackLock into a reference that outlives static destruction; this PR does not touch the lock or the code that takes it, so the caller-stack lifecycle is unchanged.

On the new head, Linux passes the full build, libs, conventions, the regression (also in 20 repeats) with its unwind checks, the neighbouring stack and thread tests and the ASan run. Windows passes the regression (also in 20 repeats) and the neighbouring pthread and user-context tests with WinLibs (DWARF) and MSYS2 UCRT64 (SEH); the WinLibs run also passes the full build, libs and conventions. The numbers are in the description.

Order: the description now proposes #8041 and #9684 first, then this PR, then #7149 and #4563, with what each step needs, and the checklist says Depends on: #8041, #9684. @plokmklo, @NevesHenry, @Pouare514, @poonasor: if you need another order, reply here and I will change it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants