[mugurc]

What

Maps guest memory on macOS hosts. Direct memory, flexible memory and the mmap family used Linux-only calls (memfd_create, MAP_FIXED_NOREPLACE, a hinted mmap that searches upward), and placed mappings in the guest arena only on Linux. Part of the macOS series announced on #1 (#1 (comment)).

• The backing of a direct memory pool is a named Mach memory entry (mach_make_memory_entry_64 with MAP_MEM_NAMED_CREATE), mapped with mach_vm_map. macOS has no memfd, and a POSIX shared memory object does not work for guest code: an mprotect that adds execute to its mapping fails with EACCES, and code mapped executable from it does not run under Rosetta (SIGBUS). The entry is a Mach port, so a pool holds no file descriptor.
• Mappings are placed in the guest arena that feat(libc): reserve the guest arena on macOS #6124 reserves on macOS, through the same paths as on Linux (ArenaAvailable, ArenaCommit, ArenaForget, and the GuestArenaAllocateFromHint and GuestArenaAllocate branches of MapPlaced). Without them a mapping with no address came back from a plain mmap outside the arena: @DavidSamir found that ccleste (PPSA59322) needs its 16 MiB ring from sceKernelMapDirectMemory inside 0x2_0000_0000-0x3_0000_0000 and stops before device creation otherwise (macOS support #1 (comment)), and that ten radv-based titles report no Vulkan device for the same reason (macOS support #1 (comment)).
• The fallbacks that Linux keeps for a process without the arena use mach_vm_map on macOS: with VM_FLAGS_ANYWHERE, whose search starts at the hint, for a mapping above a hint, and without VM_FLAGS_OVERWRITE, which refuses an occupied range, for a fixed mapping that must not replace another.
• mprotect finds the guest module that holds a range through the emulated dl_iterate_phdr of macOS, as Linux finds it through the host one. The mappings are registered with the write watch (fix(libc): track never-collected pages in the macOS write watch #3703) and the physical span index, as on Linux.
• A protection change on a host thread's stack protects it directly on macOS. The retry from fix(libkernel): keep a protected host stack's guard page, and protect the Linux main stack #6443, which maps a stack's unmapped pages when mprotect fails with ENOMEM, stays Linux-only (MapUnmappedPages is): macOS maps the whole stack of a thread. Under Rosetta, the main thread's stack at 8 MiB and at the 64 MiB limit and pthread stacks of 512 KiB and 64 MiB have no unmapped page, and mprotect over each whole stack succeeds.
• A file-backed mapping is still refused with ENOTSUP on macOS. feat(libkernel): map regular files with mmap on Windows #6542 gave the Windows implementation the branch that every host other than Linux took; it now has its own, and the others keep the refusal.
• CheckGuestModuleUsesFlexibleMemory is skipped on macOS: its module is a host-built library, which macOS's emulated dl_iterate_phdr (it lists relinked images only) does not show.

Tested

macOS 26.5.1, Apple M5 Max, Apple clang 21, x86-64 build under Rosetta 2. main 7e2ba73 with the rest of the macOS series (ci/macos-series-2 on my fork, which now includes #7937) and the libSceSsl change of #7332. The libraries do not link on macOS without the series.

• Rebased on main 7e2ba73. The one conflict was the Windows branch of ~PhysicalBacking, which perf(libkernel): reuse released direct-memory sections that never held data #8857 gave a section pool; the macOS branch after it is unchanged.
• guest_memory with the two checks this PR adds, guest_mmap_fixed_anonymous, guest_amm_placement, guest_kernel_stack_protect and guest_jit_shared_memory pass, and so does guest_shm_open now that the series carries feat(libkernel): JIT shared memory on macOS hosts #7937.
• ninja, ninja libs and ctest -j4: 1798 tests, 1123 passed, 25 failed, the rest skipped (no Vulkan loader in the environment). Of the memory tests among them, guest_memory_lifetime, guest_flexible_memory_declared and guest_kernel_app_archive_mmap failed the same way before this rebase. The others failed before as well, or are new since my last run on main 3fd1cc3: tests and checks main added since then, and agc_driver_graphics, which now times out. I have not looked into the new ones yet.

Earlier runs of this pull request:

• On main 218aa8f, with feat(libc): reserve the guest arena on macOS #6124's arena reserved and the mapping paths of this PR left Linux-only, guest_memory (CheckReservationHintWithoutRoomAbove: a reservation hinted above the arena is not placed inside it), guest_mmap_fixed_anonymous (CheckPosixFixedAnonymousMapping: a fixed mmap into a reservation fails) and guest_amm_placement fail. With them all three pass, and guest_memory runs every arena check on macOS. guest_ampr_markers, guest_ampr_threads, guest_arena_available, guest_arena_unmappable, guest_arena_host, guest_arena_commit, guest_mlock and guest_mprotect_unmapped pass either way.
• The two new checks of guest_memory pass: direct memory mapped read-write, filled with code and made executable with sceKernelMprotect runs it, and 64 rounds of allocating, mapping and releasing direct memory leave no host descriptor open. With a shm_open backing the first fails with EACCES and the second leaks one descriptor a round.
• A 16 MiB sceKernelMapDirectMemory without an address, as ccleste's ring (on main 218aa8f): in six launches it was placed at 0x2_0000_0000 five times and at 0x2_00CB_0000 once, inside the window ccleste needs.
• On main c1ca834, libkernel does not compile on macOS with the earlier version of this PR (MapUnmappedPages is undeclared, and the Windows file mapping code of feat(libkernel): map regular files with mmap on Windows #6542 is compiled); with this one it does, and guest_kernel_stack_protect passes.
• Linux and Windows: not run for this version. The two new checks run there too (the descriptor one not on Windows); the rest changes only macOS.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source (none added)
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 4449	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp core/libs/tests/GuestMemory.cpp
PR 5021	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp core/libs/tests/GuestMemory.cpp
PR 6321	core/libs/tests/GuestMemory.cpp
PR 7105	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp core/libs/tests/GuestMemory.cpp
PR 7149	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp
PR 7201	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp
PR 10550	core/libs/tests/GuestMemory.cpp
PR 10898	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp

[krishalivaibhav]

Checked on Apple M2, macOS 27.2, x86_64 Release. Merged onto main 61ad086 plus the other macOS build fixes (#4627, #10362, #10366, #10369): DirectMemory/DirectMemory.cpp now compiles (it failed with windows.h not found before), ninja libs -k 0 goes from 10 to 9 failed objects. No conflicts. guest_memory could not be linked: libkernel still has 6 objects that do not compile on macOS (NativeStat, SharedMemory, Stdio, DynamicLoader, ModuleInfo, Rtld), none in this PR. I will run it when those land.
AI-assisted: yes (Claude).

[imellag]

Tested head c28643f3e5 on Apple M4 Pro, macOS 27.0.1, Apple clang, x86-64 Release under Rosetta 2.

Alone on main 3fcb5e4d76: DirectMemory/DirectMemory.cpp now compiles. The other 19 steps that fail on main on macOS still fail, and there are no new failures.

With the other open macOS build PRs (#10562, #10641, #10366, #10419, #7804, #10552, #4630, #10659, #10660; I resolved the conflicts of #10552 and #4630 with main locally), libkernel links and the guest tests run:

• guest_mmap_fixed_anonymous, guest_amm_placement, guest_jit_shared_memory, guest_shm_open, guest_arena_available, guest_mprotect_unmapped, guest_memory_lifetime and guest_flexible_memory_declared pass.
• guest_memory runs the two checks this PR adds without a failure. It then aborts later, in CheckMlock (GuestMemory.cpp:1695): sceKernelMlock on a sceKernelReserveVirtualRange range does not return ENOMEM.
• guest_mlock, guest_system_configuration, guest_kernel_is_stack and guest_credentials abort too. In guest_system_configuration, kern.proc.vmmap leaves errno at 2 because sceKernelVirtualQuery falls back to /proc/self/maps.

All five pass once your f66a4df741 (feat(libkernel): guest memory protection and VirtualQuery on macOS hosts, branch feat/libkernel-memory-protection-macos) is added on top; it applies cleanly to this stack. Apart from #7201, I couldn't find an open PR that carries it. Do you plan to open it? It seems to be the part of the series these tests still need on macOS.

guest_kernel_stack_protect still aborts with it: the child re-executes /proc/self/exe (GuestKernelStackProtect.cpp:529), which macOS doesn't have.

AI-assisted: yes

[imellag]

Since #10706 was merged, main declares int sceKernelMunmap(uint64_t vaddr, size_t len) in libkernel/include/Api.hpp, so the two calls this PR adds to core/libs/tests/GuestMemory.cpp no longer compile:

GuestMemory.cpp:564: error: no matching function for call to 'sceKernelMunmap'
GuestMemory.cpp:578: error: no matching function for call to 'sceKernelMunmap'

They are sceKernelMunmap(mapped, page) in CheckDirectMemoryBecomesExecutable and CheckDirectMemoryReleasesHostDescriptors. #10706 changed the existing calls in that file to sceKernelMunmap(reinterpret_cast<std::uint64_t>(mapped), length), and the same change fixes these two. GitHub still reports the PR as mergeable because the conflict is not textual.

Checked on main 61fc572cee with only this PR applied (head c28643f3e5), Apple clang 21, x86-64: the test fails to compile with the PR and compiles without it. With the two casts, the whole macOS stack I tested yesterday builds again on the new main.

AI-assisted: yes

[krishalivaibhav]

After rebasing on main, GuestMemory.cpp no longer builds: sceKernelMunmap now takes std::uint64_t (e29ef3c recast the other call sites), but the two calls in the new executable-mapping test (sceKernelMunmap(mapped, page), around lines 557 and 571) still pass void*. Please wrap mapped as reinterpret_cast<std::uint64_t>(mapped) like the neighbouring tests.

With that change, plus a separate macOS fix for mlock on PROT_NONE ranges, guest_memory passes on M2 (macOS 27.2, x86_64 Release).

AI-assisted: yes (Claude Code)