Skip to content

fix(libkernel): fail sceKernelMprotect over an unmapped range with EINVAL instead of aborting - #5143

Merged
adriadam10 merged 3 commits into
boykopovar:mainfrom
xXPBSXx:fix/mprotect-unmapped-einval-v2
Oct 14, 2026
Merged

adriadam10 merged 3 commits into
boykopovar:mainfrom
xXPBSXx:fix/mprotect-unmapped-einval-v2

Conversation

@xXPBSXx

@xXPBSXx xXPBSXx commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

What

sceKernelMprotect over a range that is not completely registered in the allocation tracker (an unmapped gap inside or at the end of the range) threw guest protection or unmap range has a hole (or ... is not registered) from GuestAllocations::Protect; nothing caught it, so the title was terminated. The POSIX mprotect wrapper caught the exception and returned -1 with EINVAL.

FreeBSD 11.4's vm_map_protect has no check for holes: its loop runs over the map entries that exist, so the gaps of a range are skipped and the mapped parts are protected. DoMprotect now does the same: a new GuestAllocations::ForRegistered hands out the registered parts of the range, DoMprotect protects each of them and returns 0; a range with no registered part at all returns SCE_KERNEL_ERROR_EINVAL (mprotect: EINVAL), as the existing GuestMemory.cpp test for an unmapped start expects. FreeBSD does not answer EINVAL there: with nothing mapped in the range vm_map_protect finds no entry and returns KERN_SUCCESS, which kern_mprotect turns into 0 (both read in release/11.4.0); the EINVAL stays because that test pins it, not because of the source. DoMtypeprotect returned the result of DoMprotect only at the very end after retyping the direct mappings; it now returns early on failure (no test covers that failure path). Every part is first checked for a GPU or image pin (RequireUnpinned), so a pinned part fails the call before the first part changes; the parts are then protected one after the other, so a host protection failure in a later part leaves the earlier parts changed (listed in the TechnicalDebt entry for libkernel together with the missing console measurement). GuestAllocations::Protect itself is unchanged and still throws on a hole; libSceAgcDriver/tests/GuestAllocations.cpp:159 keeps pinning that.

A range whose start is unmapped keeps answering EINVAL even if later pages are mapped (pinned by the existing GuestMemory.cpp test). FreeBSD would take the next entry there (vm_map_protect: entry = entry->next when the start lies in no entry) and protect the later pages, so only a gap inside or at the end of a range follows the FreeBSD source; the unmapped start is a deliberate difference.

Not covered: DoMprotect still throws invalid_argument for address 0, length 0 or an overflowing range (first lines of the function), so sceKernelMprotect(NULL, ...) or a zero length ends the title as before (the POSIX mprotect wrapper catches it); a pinned range still throws from RequireUnpinned; the guest-stack branch runs before the new code.

Title evidence, and what is NOT shown: in saved runs of PPSA03001 the title stops with exactly what(): guest protection or unmap range has a hole (11 run logs, 22 counted lines, e.g. PPSA03001_r/run_021704.err:4). The logs have no guest frame and not the arguments of the call, so that it is a sceKernelMprotect over an unmapped gap is a hypothesis (the other caller of Protect, the fixed remap path, checks Covers first). With this change such a call protects the mapped parts and returns 0; whether PPSA03001 then runs on is not tested.

Tested

Windows 10 (19045), MinGW-w64 GCC 15.2.0 (WinLibs r7), Release, first run on main 0f2683cb7 (the re-run with the current head and main is the bullet "Re-run on 14 October 2026" below):

  • guest_mprotect_unmapped (new): three pages mapped, the middle one unmapped: sceKernelMprotect, mprotect and sceKernelMtypeprotect over all three return 0 and sceKernelVirtualQuery shows the protection changed on both sides of the gap; unaligned ranges inside and across the gap work; a range that is completely unmapped returns EINVAL (22 for mprotect); in a second mapping, a range that ends in an unmapped page protects the mapped page.
  • It fails without the change to DirectMemory.cpp (sceKernelMprotect threw: guest protection or unmap range has a hole, the message of the title run).
  • guest_memory passes.
  • Overlap checked with git merge-tree: feat(libkernel): count committed ranges in sceKernelGetPageTableStats #4725 conflicts only in the test registration of core/libs/prx/libkernel/CMakeLists.txt (both add a test there); feat(memory): track changes to mapped address ranges #1360 and perf(agc): cache page permissions within registered allocations #1330 already conflict with main on their own; feat(libkernel): support sceKernelMprotect on foreign guest module images on Windows #5109 (sceKernelMprotect on foreign module images on Windows) edits IsGuestModuleImage in the same file for the Windows branch of DoMprotect; git merge-tree of the two heads is clean (not built together), and the two changes look independent (it decides which images may be protected, this PR what happens to an unregistered gap inside a range); no other open PR touches the function. fix(libkernel): inherit floating-point environment on Windows threads #4981 (floating-point environment for Windows threads) conflicts with this PR in docs/dev/debt/libkernel.md (git merge-tree of the two heads, re-checked; each is clean against main 30f961266); whichever lands second rebases the debt entry.
  • Re-run on 14 October 2026 (Windows 10 19045, MinGW-w64 GCC 15.2.0, Release, -j4), after main feddf7f4c was merged into the head (merge commit e528ac9c5; the only conflict was the two independent test registrations in core/libs/prx/libkernel/CMakeLists.txt next to guest_memory, guest_mprotect_unmapped and the new guest_flexible_memory_* tests, both kept and built): guest_mprotect_unmapped passes (0.03 s, ctest) together with guest_memory, the five guest_flexible_memory_* cases, guest_mlock, guest_kernel_sys and guest_filesystem (10 of 10, 0 failed); with DirectMemory.cpp taken back to main and the new test kept it fails (sceKernelMprotect threw: guest protection or unmap range has a hole, exit code 0xc0000409). The commit 238fe1eec (pin check for every part before the first change, after a review) was built and run afterwards on the same main: the same eight tests (guest_mprotect_unmapped, guest_memory, the five guest_flexible_memory_*, guest_mlock) pass (8 of 8); it has no test of its own, because a pinned part next to a free part needs a GPU lease that only the AGC driver tests create. python tools/check_conventions.py --base upstream/main passes. This replaces the earlier figures. Not run: the socket tests, Linux, a console, a title.
  • Not tested: Linux (one source for both hosts), a console, a title run. fix(kernel): retain module registry through process teardown #5572 (JJenkx, module registry kept through process teardown) conflicts with this PR in docs/dev/debt/libkernel.md (git merge-tree of the two heads, re-checked against main ce5c83953; both heads are clean against main); whichever lands second rebases the debt entry.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); no silent stub added
  • No notes, investigation .md files or images added to the repository
  • New third-party code is a submodule built from source (none added)
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Unblocks: none established (PPSA03001 only as a hypothesis, see above)
  • AI-assisted: yes

AI-assisted (Claude Code).

🤖 Generated with Claude Code

…NVAL instead of aborting

The allocation tracker threw a runtime_error (a hole in the protected range) from inside
sceKernelMprotect, which terminated the title. mprotect already turned the exception into
EINVAL. sceKernelMprotect and sceKernelMtypeprotect now check that the range is registered
first and return EINVAL; the retype of direct mappings is skipped on failure.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@xXPBSXx
xXPBSXx force-pushed the fix/mprotect-unmapped-einval-v2 branch from fecb9cb to 05ffb45 Compare October 13, 2026 16:53
@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 14, 2026
# Conflicts:
#	core/libs/prx/libkernel/CMakeLists.txt
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 14, 2026

@CnRJay CnRJay left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skipping the holes matches vm_map_protect for a range that starts in a mapping. One ordering difference, though: the new loop applies protection span by span, and a later span can still throw (RequireUnpinned, a host mprotect failure). The earlier spans then keep their new protection while the call fails.

FreeBSD's vm_map_protect works in two passes over the entries. The first pass checks every entry in the range (max_protection, submap) and returns KERN_PROTECTION_FAILURE before changing anything. Only the second pass applies the change. So a failed mprotect leaves the whole range unchanged. Collecting the spans (already done), running the checks that can fail (RequireUnpinned per span) for all of them first, and only then calling Protect would keep that property. Otherwise a title that retries after an error sees part of the range already changed.

On the evidence: as the description says, the PPSA03001 logs show the throw but not the call, and the remap path is the other caller of Protect. A stack (APS5_TRACE_*, or the crash report's guest frames) from one of those runs would confirm that it is sceKernelMprotect over a gap before this is counted as the fix for that title.

… protecting the first

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@xXPBSXx

xXPBSXx commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, both points are right.

Two passes: vm_map_protect (release/11.4.0) checks every entry first and changes nothing if one fails (max_protection, submap), then applies. DoMprotect now does the part of that which can fail on our side before it changes anything: commit 238fe1e runs RequireUnpinned for every span first (a span pinned by GPU work or an image is the failure that can occur in the loop), and only then protects the spans one after the other. A host mprotect failure in a later span can still leave the earlier spans changed; the vm_map_protect second pass cannot fail, and there is no way to undo a host protection here, so the TechnicalDebt entry says it (updated in the same commit). No new test: a pinned span next to a free span would need a GPU lease from libkernel's test, which only the AGC driver tests create. The existing tests (guest_mprotect_unmapped, guest_memory, guest_flexible_memory_*, guest_mlock) pass with the commit (8 of 8, Windows).

Evidence: I ran PPSA03001 on plain main fc1e388 (90 s, 14 October): the has a hole exception does not occur there (log without hole|what()|FATAL|unhandled|terminate). The 11 runs that showed it are from 11 and 12 October on an older main, and they have no guest frame or arguments. So this PR does not count as the fix for PPSA03001; the description already says so.

AI-assisted (Claude Code).

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
#5770 docs/dev/debt/libkernel.md L105

@adriadam10

Copy link
Copy Markdown
Collaborator

Checked on Linux: scoped build and ctest green (guest_current_cpu aborted once in three runs, unrelated to this change); guest_mprotect_unmapped passes and fails without the DirectMemory.cpp/GuestAllocations change. The range handling (ForRegistered, unmapped start still EINVAL, pin check before the first change) and the debt entry look right; merging once the full Build is green.

@adriadam10
adriadam10 merged commit 066442c into boykopovar:main Oct 14, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants