Skip to content

fix(libkernel): move the macOS main stack out of the guest arena - #12126

Open
morten-holm wants to merge 2 commits into
boykopovar:mainfrom
morten-holm:fix/macos-host-stack-above-arena
Open

morten-holm wants to merge 2 commits into
boykopovar:mainfrom
morten-holm:fix/macos-host-stack-above-arena

Conversation

@morten-holm

Copy link
Copy Markdown
Contributor

What

Under Rosetta 2 the main thread stack lies at about 12 GiB, inside the guest arena. A title can map fixed addresses there, and a fixed mapping over the stack aborts with a host region overlap.

  • Aps5StartGuest moves the main thread onto a 16 MiB stack at 0x600000000000 when the old stack overlaps a host region of the arena. It first copies argv, environ, the program name and the initial block off the old stack. Then it frees the old stack and releases its range to the arena.
  • The guest thread keeps the stack size of the original main thread.
  • The arena gets GuestArenaOverlapsHostRegion_nid_no_patch and GuestArenaReleaseHostRegion_nid_no_patch for POSIX hosts.
  • The move is skipped when the main stack lies outside the arena, when the new stack cannot be placed outside the arena, and on hosts that are not x86-64.
  • The guest thread and the other host threads still take their stacks from the system. docs/dev/debt/libkernel.md records this.

Test guest_main_stack_move (macOS only): it starts the guest, then requires that the main thread is outside the arena, that a fixed guest mapping fits where the old stack was, and that argv and the environment are intact.

Tested

  • macOS 27.0.1, Apple M2, x86-64 under Rosetta 2, on main f759e165d9.
  • libkernel does not build on macOS from main alone (link.h, RUSAGE_THREAD, sched_getcpu, REG_* in Rtld.cpp, Process.cpp, Exception.cpp), and feat(libkernel): direct memory mappings on macOS hosts #7631, one of the open macOS PRs that fix it, conflicts with main. So I built libc from this branch and linked MainThreadMacOs.cpp and GuestMainStackMove.cpp by hand against it.
    • With this change: exit code 0 in 3 of 3 runs.
    • With main's MainThreadMacOs.cpp: exit code 1 in 3 of 3 runs, at the main thread stack is outside the guest arena.
  • python3 tools/check_conventions.py --base origin/main passes.
  • ASTRO BOT (PPSA21567), Apple M2, macOS 27.0.1, x86-64 under Rosetta 2, MoltenVK, host imports off. With the stack move disabled, the title aborts at launch (exit 134): fixed guest mapping 0x300000000+0x7e000000 overlaps a host region in the guest arena. With the move active, the same title gets past launch and runs draw packets for the 44 s of the run. This title run used a local equivalent of the move in an older tree, not this branch's MainThreadMacOs.cpp, which could not be linked into a full build (see above). It shows the abort and that moving the stack removes it.
  • Not tested: the CMake test target (libkernel does not build here), Linux and Windows (the code is __APPLE__ only, apart from the two arena functions, which are under #ifndef _WIN32), a title run on this base.

Checklist

🤖 Generated with Claude Code

Under Rosetta the main thread stack lies at about 12 GiB, inside the guest
arena, where a title can map fixed addresses; a fixed mapping over it
aborts with a host region overlap.

- Aps5StartGuest moves the main thread onto a 16 MiB stack at
  0x600000000000 when the old stack overlaps a host region of the arena.
  It copies argv, environ, the program name and the initial block off the
  old stack, then frees the old stack and releases its range to the arena.
- The guest thread keeps the stack size of the original main thread.
- The arena gets GuestArenaOverlapsHostRegion and
  GuestArenaReleaseHostRegion for POSIX hosts.
- guest_main_stack_move starts the guest, then requires the main thread
  outside the arena, a fixed mapping where the old stack was, and argv and
  the environment intact.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the unblocks title Names a title ID on the Unblocks line label Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 7532 core/libs/prx/libkernel/CMakeLists.txt L592

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 1835 core/libs/prx/libc/src/GuestArena.cpp
PR 4449 core/libs/prx/libc/include/GuestArena.hpp
core/libs/prx/libc/src/GuestArena.cpp
PR 4563 core/libs/prx/libkernel/CMakeLists.txt
PR 4630 docs/dev/debt/libkernel.md
PR 5021 core/libs/prx/libc/src/GuestArena.cpp
core/libs/prx/libkernel/CMakeLists.txt
core/libs/prx/libkernel/Pthread/src/MainThreadMacOs.cpp
docs/dev/debt/libkernel.md
PR 6149 docs/dev/debt/libkernel.md
PR 6936 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 7088 docs/dev/debt/libkernel.md
PR 7105 docs/dev/debt/libkernel.md
PR 7121 docs/dev/debt/libkernel.md
PR 7135 docs/dev/debt/libkernel.md
PR 7149 docs/dev/debt/libkernel.md
PR 7192 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 7447 docs/dev/debt/libkernel.md
PR 7464 docs/dev/debt/libkernel.md
PR 7509 docs/dev/debt/libkernel.md
PR 7804 docs/dev/debt/libkernel.md
PR 8605 docs/dev/debt/libkernel.md
PR 9156 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9242 docs/dev/debt/libkernel.md
PR 9293 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9297 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9302 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9383 core/libs/prx/libkernel/CMakeLists.txt
PR 9389 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9557 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9566 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 10319 docs/dev/debt/libkernel.md
PR 10442 docs/dev/debt/libkernel.md
PR 11002 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 11215 docs/dev/debt/libkernel.md
PR 11625 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 11849 docs/dev/debt/libkernel.md
PR 11864 docs/dev/debt/libkernel.md
PR 11872 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 11969 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md

@oneandonlydean

Copy link
Copy Markdown
Collaborator

@morten-holm thanks, the move is careful: the replacement stack is checked to be outside the arena before use, the old range goes back to the arena, and Linux and Windows are untouched. One hardening request: FindHostStack extends the old stack downward over every adjacent region tagged VM_MEMORY_STACK, and ServeMainQueue then deallocates the whole merged range. A thread stack mapped directly below the main stack carries the same tag and would be freed with it. Please bound the range with the main thread's own pthread_get_stackaddr_np / pthread_get_stacksize_np (intersected with the region found), so that only the main stack is released.

@adriadam10 adriadam10 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the approach is sound. One change is needed before merge, the one @oneandonlydean already asked for:

  • MainThreadMacOs.cpp:85-95 (FindHostStack) grows the range downward over every adjacent region tagged VM_MEMORY_STACK, and ServeMainQueue (:48-55) deallocates the whole merged range and releases it to the arena. A thread stack mapped right below the main stack has the same tag and would be freed with it. Bound the range with pthread_get_stackaddr_np / pthread_get_stacksize_np of the main thread, intersected with the region found, so only the main stack is released. Extending the test to put a second VM_MEMORY_STACK region right below the main stack would pin it.

Everything else checked out against the diff (argv/environ/initial block copy, skip conditions, debt entry, non-Apple paths untouched). The macOS build and the new test could not be run on our side.

… the arena

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@morten-holm

Copy link
Copy Markdown
Contributor Author

@oneandonlydean @adriadam10 Fixed in 11ca3cb744.

  • FindHostStack now bounds the region it found with pthread_get_stackaddr_np and pthread_get_stacksize_np of the main thread. If the two ranges do not overlap, it returns an empty range and nothing is released. Only the main stack is freed.
  • The test guest_main_stack_move maps a 1 MiB VM_MEMORY_STACK region directly below the main stack before the move. It requires that this region is still mapped with the stack tag after the move.
    • With the previous MainThreadMacOs.cpp: exit 1 in 3 of 3 runs, at a stack region next to the main stack survived the move.
    • With this change: exit 0 in 3 of 3 runs.
  • The setup takes that 1 MiB out of the arena reserve first. Without this step, no VM_MEMORY_STACK region lies directly below the main stack in this process.
  • Same limits as before: I hand-linked the test (Apple M2, macOS 27.0.1, x86-64 under Rosetta 2), because libkernel does not build on macOS from main. The CMake target, Linux and Windows are not tested.
  • python3 tools/check_conventions.py --base origin/main passes.

AI-assisted: yes (Claude Code)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build docs libc libkernel unblocks title Names a title ID on the Unblocks line

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants