Skip to content

perf(libSceSysmodule): cache the unwind region of loaded images on Windows - #6064

Merged
adriadam10 merged 3 commits into
boykopovar:mainfrom
JAB-dev:perf/sysmodule-unwind-region-cache
Oct 17, 2026
Merged

adriadam10 merged 3 commits into
boykopovar:mainfrom
JAB-dev:perf/sysmodule-unwind-region-cache

Conversation

@JAB-dev

@JAB-dev JAB-dev commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Besiege's long freeze while it loads is about 75% shorter on Windows: the longest frame goes from about 13 s to about 3 s, and it presents about 1.16 times as many frames in the first 40 seconds. Steady-state frame time is unchanged.

What

sceSysmoduleGetModuleInfoForUnwind called VirtualQuery and GetMappedFileNameA on every call. On Windows VirtualQuery takes time proportional to the size of the region it describes: measured in Besiege it costs 2 us for a 16 KiB region, 51 us for 8 MiB and 167 us for 32 MiB, and Besiege's loaded images have regions of 8 to 32 MiB. A title that unwinds exceptions while loading calls the function for every frame of every throw: 80,000 calls in the first 40 seconds, 6.7 s of VirtualQuery on one thread, while the rest of the game waits. GetMappedFileNameA is 4.6 us per call.

The region (begin, size) and the file name of an address inside an image of the Windows loader's module list are now kept per region. A cached entry is used only while the loader still reports the same image for the address (same base, time stamp and size); otherwise it is dropped and the address is queried again. Addresses outside loaded images, such as guest memory, are queried every time, so a protection change there is still seen. The reported fields are unchanged: for an address in a cached run the entry returns the queried page and the bytes to the end of the run, which is what VirtualQuery returns. Linux is not affected.

docs/dev/debt/libSceSysmodule.md records that a page protection change inside a cached image after its first lookup is not noticed.

Seen in PPSA20313 (Besiege, Unity IL2CPP). A thread sample of the game while loading showed one thread spending 32% of its samples in NtQueryVirtualMemory under this function.

Tested

Windows 11, WinLibs GCC 15.2.0 r7, Release, RTX 4070 SUPER, NVIDIA 610.88, 1920x1080. Besiege, the first 40 s after launch (APS5_TRACE_SECONDS=0:40), warm shader cache, three alternating runs of main at 7003eda and of this branch at cd95f1a, no other APS5_TRACE_* switch, the game's output sent to nul:

presents in 40 s longest frame (s)
main 633, 668, 686 13.1, 13.7, 11.9
this PR 741, 760, 804 3.4, 3.3, 3.1

The same three alternating runs with the game started in a console window instead (output to the console), main at 44df21632 and this branch merged onto it: longest frame 8.4 / 8.8 / 8.6 s on main, 2.5 / 2.5 / 2.6 s with this PR; frames presented in 40 s 1240 / 1136 / 1196 -> 1355 / 1371 / 1364.

Summary lines of the middle run of each:

main:

{"commit": "7003eda5d6ef", "dirty": false, "env": {"APS5_TRACE_SECONDS": "0:40"}, "gpu": "NVIDIA GeForce RTX 4070 SUPER", "driver": "NVIDIA 610.88", "os": "Windows 10.0.26100", "resolution": "1920x1080", "window": {"spec": "seconds 0:40", "first_present": 0, "ms": 40000.469}, "presents": 668, "frame_ms": {"p50": 8.721, "p90": 80.313, "p99": 100.605, "max": 13696.79}, "gpu_busy_ms_per_present": 9.05, "worker_busy_pct": {"queue worker 0x0": 57.8}, "top_phases_ms_per_frame": {"Driver.Packet": 34.528, "Driver.Packet.draw": 31.739, "Driver.Draw": 31.595, "Driver.Draw.draw_and_resource_release": 20.246, "Graphics.Draw": 20.018, "Graphics.Draw.draw_and_resource_release": 17.981, "Driver.Draw.shader_compile_and_link": 7.683, "Shader.DrawStage": 6.444}, "trace_dropped": true, "trace_dropped_threads": {"queue worker 0x0": {"events_full_after_s": 22.997, "events_lost": 799947, "totals_full": false}}, "trace": "aps5-trace-7003eda5d6ef-36496.json.gz"}

this PR:

{"commit": "cd95f1a42675", "dirty": false, "env": {"APS5_TRACE_SECONDS": "0:40"}, "gpu": "NVIDIA GeForce RTX 4070 SUPER", "driver": "NVIDIA 610.88", "os": "Windows 10.0.26100", "resolution": "1920x1080", "window": {"spec": "seconds 0:40", "first_present": 0, "ms": 40036.534}, "presents": 760, "frame_ms": {"p50": 72.211, "p90": 86.39, "p99": 100.219, "max": 3306.136}, "gpu_busy_ms_per_present": 12.236, "worker_busy_pct": {"queue worker 0x0": 83.801}, "top_phases_ms_per_frame": {"Driver.Packet": 44.067, "Driver.Packet.draw": 40.706, "Driver.Draw": 40.536, "Driver.Draw.draw_and_resource_release": 26.502, "Graphics.Draw": 26.214, "Graphics.Draw.draw_and_resource_release": 23.711, "Driver.Draw.shader_compile_and_link": 9.495, "Shader.DrawStage": 7.966}, "trace_dropped": true, "trace_dropped_threads": {"queue worker 0x0": {"events_full_after_s": 12.671, "events_lost": 1209258, "totals_full": false}}, "trace": "aps5-trace-cd95f1a42675-38032.json.gz"}
  • guest_sysmodule now also checks on Windows that the reported region and name equal what VirtualQuery and GetMappedFileNameA return, for code, data and read-only data of the test executable, a host DLL, the stack and private memory, three times each so the later calls come from the cache; a protection change in private memory and a freed region are seen. The new assertions pass on main, because the change only affects speed. Mutation check: removing the check that the address is inside the cached region fails the test. Further addresses inside an already cached run, queried forward, backward (an earlier page after a later one) and repeated, are compared with VirtualQuery as well; those checks fail on the first commit of this branch and pass on its head.
  • guest_sysmodule and guest_unwind_module_info pass; tools/check_conventions.py passes.
  • Linux not tested.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Unblocks: none
  • GPUs tested: n/a
  • AI-assisted: yes

🤖 Generated with Claude Code

…ndows

sceSysmoduleGetModuleInfoForUnwind called VirtualQuery and GetMappedFileNameA on every call. VirtualQuery on a large image section costs time proportional to the section (2 us for 16 KiB, 51 us for 8 MiB, 167 us for 32 MiB), and a title that unwinds exceptions while loading calls it for every frame of every throw.

The region and file name of an address inside an image of the Windows loader's module list are kept per region and used while the loader still reports the same image (base, time stamp and size). Other addresses are queried every time.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot mentioned this pull request Oct 15, 2026
11 tasks
@screamm

screamm commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Windows run of head cd95f1a42 alone, merged onto main d92a8dad3 (clean merge; it also merges cleanly onto the newer db81992c5, not built). This is one NVIDIA/Windows data point only.

  • Host: Windows 11 Pro 26220, Ryzen 7 5700X, RTX 4070 (driver 596.21), WinLibs MinGW-w64 GCC 15.2.0 r7, Ninja, Release.
  • Build: the full build and libs pass with 0 errors and no warnings in Export.cpp or GuestSysmodule.cpp.
  • Tests, --repeat until-fail:3, 60/60 runs: guest_sysmodule, guest_unwind_module_info, unwind_record_bounds, windows_exceptions, guest_exception_handler, guest_raise_exception, guest_mutex_raise_exception, undeliverable_host_exception and the 12 guest_exception_vtable-* cases. As you wrote, the extended guest_sysmodule also passes against main's libSceSysmodule.prx.
  • Non-GPU suite (ctest -j8 --timeout 120 -E '^agc_driver_|^video_out_flip'): 812/814. conventions_function_bodies and guest_none_relocations timed out under load and passed when rerun alone.
  • tools/check_conventions.py and git diff --check pass.

Cost per call. I wrote a small harness that loads each side's libSceSysmodule.prx and calls sceSysmoduleGetModuleInfoForUnwind 20,000 times per case. Each side ran 3 times, alternating:

addresses main (µs/call) this PR (µs/call)
spread over a 32 MiB .bss section of the harness exe 10.39 / 11.01 / 11.55 0.054 / 0.071 / 0.073
one small .data variable 3.74 / 3.83 / 3.83 0.041 / 0.043 / 0.059
the harness's .text (~40 KiB) 3.92 / 3.93 / 4.16 0.042 / 0.045 / 0.050

Titles: none of the 47 local catalog titles I ran calls this function. I counted calls with a diagnostic build of main's libSceSysmodule.prx (a call counter and timer only), running each title for 30–40 s, and every title made 0 calls. That includes OpenRCT2 PPSA99728 forced to throw (libc++abi: terminating due to uncaught exception of type OpenRCT2::Ui::SDLException, identical on both sides). In these titles only the guest libc.prx imports it, next to sceKernelGetModuleInfoForUnwind. So I cannot reproduce the Besiege effect locally. As a no-regression check, I ran Middle Ages Ping Pong PPSA28319 (Unity IL2CPP, 0 calls). Each side had one warm-up and then 3 runs of 60 s, alternating, with APS5_TRACE_SUMMARY_EVERY=10:

main, min / median this PR, min / median
process start to first present 0.81 / 1.90 s 0.95 / 1.07 s
longest frame (max of the 10 s summaries) 175.6 / 334.7 ms 175.3 / 189.7 ms
presents in 60 s 3160 / 3435 3445 / 3482

The longest frame always falls in the first 10 s, and both sides sit at 17–18 ms after that. Since this title never calls the function, the differences are run-to-run noise.

One behaviour change: the reported region is not always the same as on main. VirtualQuery returns the run of equal pages starting at the queried page. On main, seg0_addr/seg0_size are therefore that page and the bytes up to the end of the run. The cache keeps the first lookup's [page(first), end) and returns it for every later address in that run. Querying two addresses in the same run, in this order:

main:    .text First        seg0 0x7ff61fd91000+0x9000   VirtualQuery 0x7ff61fd91000+0x9000
main:    .text First+0x3000 seg0 0x7ff61fd94000+0x6000   VirtualQuery 0x7ff61fd94000+0x6000
this PR: .text First        seg0 0x7ff61fd91000+0x9000   VirtualQuery 0x7ff61fd91000+0x9000
this PR: .text First+0x3000 seg0 0x7ff61fd91000+0x9000   VirtualQuery 0x7ff61fd94000+0x6000

The same happens for two addresses 8 MiB apart in a .bss section. With the PR, the value reported for an address depends on which address in that run was looked up first. The new guest_sysmodule checks query each address three times but never a second address inside a cached run, so they pass. Returning page(addr) and end - page(addr) from the cached entry should keep the output identical to main; I have not tried it. I don't know whether the guest unwinder uses seg0_addr, but the description says the reported fields are unchanged, and this is a case where they change.

Verdict: on Windows the build, the tests and the suite pass. The cache makes a call about 150 times cheaper for large image regions and about 90 times cheaper for small ones on this machine. No local title exercises the function, so this run cannot confirm the title-level gain, and PPSA28319 shows no regression. The reported region differs from main for a second address inside an already cached run.

AI-assisted (Claude Code).

This was referenced Oct 15, 2026
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 6370 docs/dev/debt/libSceSysmodule.md docs/dev/debt/libSceSysmodule.md
PR 6559 core/libs/tests/GuestSysmodule.cpp L105
docs/dev/debt/libSceSysmodule.md
docs/dev/debt/libSceSysmodule.md

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 5021 core/libs/prx/libSceSysmodule/Export.cpp
PR 6044 core/libs/tests/GuestSysmodule.cpp
PR 6222 core/libs/prx/libSceSysmodule/Export.cpp

@JAB-dev

JAB-dev commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Besiege (PPSA20313), main at 44df21632 vs the same with this PR merged (1b9b0a2de), Windows 11, RTX 4070 SUPER (610.88), 1080p, first 40 s from launch, three alternating runs each: longest frame 17.4 / 11.9 / 8.3 s -> 2.5 / 2.3 / 2.5 s; frames presented 635 / 1045 / 1203 -> 1298 / 1399 / 1372. Title-screen FPS is unchanged (32.2 / 31.6 -> 33.7 / 30.3).

@jweidenh

Copy link
Copy Markdown
Contributor

I reproduced the cached-region mismatch reported by screamm and prepared a correction on top of this PR's unchanged head cd95f1a42675323231e0b7d3b05f2107c400807b:

The cache hit now returns the queried address rounded down to the system page boundary, with size cached_end - queried_page. The page size comes from GetSystemInfo once; cache lookup and image-identity validation are unchanged. This follows the documented per-query bounds of VirtualQuery.

The new Windows fixture uses committed, loader-owned executable image storage and checks eight mixed-order offsets across three equal-protection pages, repeating each lookup three times. It compares the region bounds and mapped name against Windows queries, checks the structure size and zero unwind fields, and poisons the output before each call. Existing image, private-memory, protection-change and freed-memory checks remain.

Native Windows 11 Pro build 26300, Ryzen 7 5800X, WinLibs GCC 15.2.0 r7, Release, SPIRV-Tools ON. This PC has an RTX 5090 / driver 617.42; these are CPU tests and do not validate GPU behavior.

The test-only baseline 86b6700f failed exactly on the second image page: expected size 126,976 bytes versus cached size 131,072, with the cached base 4,096 bytes too early. The common region end was unchanged. That intended assertion failure was observed once; no baseline rerun was needed.

Exact candidate 0075fb6f passed all five selected tests without skips or retries:

CTest Seconds
guest_sysmodule 1.549
guest_unwind_module_info 0.788
windows_exceptions 0.598
guest_exception_handler 0.733
guest_raise_exception 16.171

The regression reported all 24 page lookups passing. The regression executable and recorded shared runtime dependencies were byte-identical between baseline and candidate except for libSceSysmodule.prx. Both source revisions use the identical fixture, without validation overlays. Focused executables and the full libs target built successfully; conventions and diff checks passed. Exact source/gitlink pins, test inventory, library paths, binary hashes, JUnit and raw logs were checked, with an independent evidence audit. The reviewed private validation harness is pinned at d1e6ddf0e921af5f76377ecd44d27b3b62e7009a; its logs are retained locally, not a public CI artifact.

This is focused Windows correctness coverage, not full-suite or Linux validation. I have not measured Besiege, startup latency or FPS. The existing documented limitation around page-protection changes inside cached images remains. AI-assisted implementation and validation.

…ched unwind region

VirtualQuery describes the pages from the queried address to the end of the run of equal pages, so the reported seg0 begins at the queried page. The cached entry returned the first lookup's page for every later address in the run. It now returns the queried page and the bytes up to the end of the cached run, which is what VirtualQuery returns.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@JAB-dev

JAB-dev commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the run, the difference is real: VirtualQuery starts the reported region at the queried page, and the cache returned the first lookup's page for every later address in the run. Fixed in a63e77e9f: the cached entry now returns the queried page and the bytes to the end of the run, as you suggested. guest_sysmodule now also queries further addresses in the same run of a host image, forward, backward and repeated, and compares each with VirtualQuery; it fails on cd95f1a42 and passes on this head. guest_sysmodule, guest_unwind_module_info and tools/check_conventions.py pass.

…ion after a later one

The backward lookup added with the region fix compared the address with a page it could not reach and never ran. The test now queries an ntdll address three pages in, then the first page, then the pages between, and compares each with VirtualQuery. The lookups after the first fail on the cache before the region fix.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@JAB-dev

JAB-dev commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Correction to my reply above: the backward lookup I described was not exercised, its guard could never be true. fbe49d6a6 replaces it with a real one: an ntdll address three pages in is queried first, then its first page, then the pages between, each compared with VirtualQuery. Run on its own, that block fails on cd95f1a42 and passes on this head. guest_sysmodule, guest_unwind_module_info and tools/check_conventions.py pass. @jweidenh your regression and fix reach the same result as a63e77e9f; thank you for reproducing it.

@screamm

screamm commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Rerun on Windows of head fbe49d6a6, merged onto main 5d9dcc42d together with #5973 (libkernel only, no shared file; both merges clean). This follows up my run of cd95f1a42.

  • Host: Windows 11 Pro 26220, Ryzen 7 5700X, RTX 4070 (driver 596.21), WinLibs MinGW-w64 GCC 15.2.0 r7, Ninja, Release.
  • Build: everything builds except the known MinGW 64 KiB alignas AGC test (test(agc): align the indirect arguments block at run time #6452/fix(agc): allocate indirect arguments test block dynamically on Windows #6478). libs passes, with no warnings in Export.cpp or GuestSysmodule.cpp.
  • --repeat until-fail:3: guest_sysmodule, guest_unwind_module_info, unwind_record_bounds, windows_exceptions, guest_exception_handler and guest_raise_exception pass, 18/18 runs.
  • Negative control: I reverted only the fix in a63e77e9f, so the cache returns found->first again, and kept the tests from fbe49d6a6. guest_sysmodule then aborts on a Require (0xc0000409). With the fix restored, it passes.
  • The harness from my first comment (same query order: .bss +1 MiB, +9 MiB, +1 MiB again, .text, .text+0x3000) now reports seg0_addr/seg0_size equal to VirtualQuery for all five lookups, as on main. On cd95f1a42, two of them differed.
  • Cost per call, 20,000 calls per case, 2 alternating runs per side. The host was busy with other builds, so the main numbers are noisy:
addresses main (µs/call) this PR (µs/call)
spread over a 32 MiB .bss 18.1 / 46.2 0.108 / 0.107
one small .data variable 5.98 / 14.5 0.090 / 0.092
harness .text 8.36 / 9.40 0.087 / 0.092
  • Non-GPU suite (ctest -j8 --timeout 120 -E '^agc_driver_|^video_out_flip'): 834/837, 2 skipped. hardware_oracle_compare and guest_file_lock_process timed out under load and passed serially. guest_memory failed once at its host-wide commit-charge check (CheckRemappedDirectMemoryIsChargedOnce). It then passed serially, failed 1 of 2 runs in a 10x repeat, and passed a second 10x 10/10. Neither PR touches that code.
  • tools/check_conventions.py --base 5d9dcc42d and git diff --check pass.

Title: Middle Ages Ping Pong PPSA28319 (Unity IL2CPP; in my first run it made 0 calls to this function, which I did not re-measure here). 130 s per run, 2 alternating runs per side, APS5_TRACE_SECONDS=0:40 as in your measurements. Neither side has a stop line, and every run was killed at the time limit:

main r1 / r2 this PR r1 / r2
frames in 130 s 7563 / 7541 7484 / 7446
presents in the first 40 s 2314 / 2331 2344 / 2321
longest frame in the first 40 s 508.9 / 349.7 ms 447.6 / 501.0 ms

These are equal within run-to-run noise, as expected for a title that does not call the function.

Verdict: the region mismatch I reported is fixed, and the new backward lookup in guest_sysmodule catches it on Windows. Build, tests and suite pass. Not checked: Linux, AMD/Intel GPUs, and Besiege (not available here, so the title-level gain is still only your measurement).

Written with Claude Code assistance (validation by a subagent, reviewed in this session).

@adriadam10

Copy link
Copy Markdown
Collaborator

Checked the diff, the build and guest_sysmodule/scoped ctest on Linux (36/36), and relied on screamm's Windows runs for the Windows path (repeat runs of the sysmodule/unwind tests, negative control failing without the region fix, per-call cost 18-46 µs -> ~0.1 µs). The cache is validated against the loader's image identity, addresses outside images are still queried, and the debt entry records the remaining gap. Merging once the Windows build is green.

@adriadam10
adriadam10 merged commit cfe6683 into boykopovar:main Oct 17, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants