Skip to content

feat(relinker): --frame-red-zone gives red-zone leaf functions a frame on Windows - #5052

Closed
parl0v wants to merge 1 commit into
boykopovar:mainfrom
parl0v:feat/relinker-frame-red-zone
Closed

parl0v wants to merge 1 commit into
boykopovar:mainfrom
parl0v:feat/relinker-frame-red-zone

Conversation

@parl0v

@parl0v parl0v commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

What

Windows has no red zone: when a guest thread takes a hardware exception that AnyPS5 resumes (write tracking of shared views, retargets, SSE4a), the kernel writes the exception frame directly below RSP before any handler runs. System V leaf functions keep live values there (#361), so a resumed fault can zero a pointer a leaf is about to use.

--frame-red-zone (Windows only, opt-in) gives those leaves a frame at relink time, so the values sit above RSP and a fault's frame lands below them:

  • Function starts are direct call targets and 16-byte-aligned code after int3 padding. A leaf (no call) qualifies when its prologue is push rbp; mov rbp, rsp; push ..., it reads or writes below its pushes through RBP, and it has no other use of RSP and no indirect jump.
  • A trampoline over the prologue runs the pushes, then lea -0x80(%rsp), %rsp.
  • A trampoline over each pop ...; pop rbp; ret exit runs lea 0x80(%rsp), %rsp, then the pops and the ret.
  • Sites that a branch enters, sites shorter than a jump, and sites that overlap a --to-intel stub are skipped. The relinker prints what it framed and why it skipped the rest.
  • DecodedInstructionInfo gains ModRmOffset so the framer can read memory operands.

What it doesn't cover (frameless leaves addressing below RSP directly, short exits) is recorded in the relinker's technical debt.

Tested

  • Windows 10 19045, i5-14400F (no AVX-512), RTX 4060, MinGW-w64 GCC 15.2. New red_zone_framer test (a framed leaf's exact stub bytes; leaves without red-zone use and functions that call are left alone; a site over another stub is skipped). Rebased on main after feat(relinker): support fixed Windows guest image bases #4140; the usage line lists both flags. 119 of 120 relinker tests pass: windows_import_modules fails the same way on plain main on this machine (its Windows DLL stdout probe prints nothing).
  • PPSA25344 (Sword of the Sea), relinked with --to-intel --frame-red-zone: 604 leaves keep values below RSP, 137 are framed (139 exits), including 0x1437f74c0, the function of its boot crash. The same counts come out of this branch and of the SotS working-group base.
  • On that working-group base (the title needs fixes not yet in main to reach its menu), the boot crash at app.exe+0x37f77b3 happened in 6 of 23 boots without the flag and 0 of 42 with it, at the same 40,960 resumed write faults per boot. Main menu speed is unchanged: 8.09 FPS framed vs 7.91 / 7.97. The menu renders correctly.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Unblocks: PPSA25344 (boot crash, red zone; Windows: resuming a write fault overwrites the guest red zone when AVX state is live #361)
  • AI-assisted: yes

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 13, 2026
@parl0v
parl0v force-pushed the feat/relinker-frame-red-zone branch from db28524 to c2912d7 Compare October 13, 2026 10:53
@github-actions github-actions Bot added needs rebase Conflicts with main, rebase needed and removed needs rebase Conflicts with main, rebase needed labels Oct 13, 2026
@parl0v
parl0v force-pushed the feat/relinker-frame-red-zone branch from c2912d7 to 9b8fe84 Compare October 13, 2026 12:52
@github-actions github-actions Bot added needs rebase Conflicts with main, rebase needed and removed needs rebase Conflicts with main, rebase needed labels Oct 13, 2026
…e on Windows

Windows writes an exception frame directly below RSP, so a fault resumed on
a guest thread overwrites values a System V leaf function keeps in its red
zone. With --frame-red-zone, a leaf function with a
push rbp; mov rbp, rsp; push ... prologue that reads or writes below its
pushes gets two kinds of trampoline: the prologue lowers RSP by 128 bytes
after its pushes, and each pop ...; pop rbp; ret exit raises it back first.

Sites a branch enters, sites shorter than a jump and sites over a --to-intel
stub are skipped; the conversion reports the framed and skipped counts.
@parl0v
parl0v force-pushed the feat/relinker-frame-red-zone branch from 9b8fe84 to fab7d20 Compare October 13, 2026 16:47
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 13, 2026
@fdfawef324

Copy link
Copy Markdown
Contributor

Tested PR #5052 on ASTRO BOT (PPSA21567), the title from my earlier data point on #361. The --frame-red-zone flag changes no output bytes for this title, so its audio-thread crash remains.

The test used Windows Server 2025 build 26100, Ryzen 9 5900X, and NVIDIA GeForce RTX 3060. I applied PR commit 9b8fe84 to a local integration branch based on main fc1f063. The older relinker layout needed the framing call moved into its existing conversion entry point and the build sources adapted. The framing implementation itself was unchanged. All 86 relinker tests passed, including red_zone_framer.

The relinker examined 311,729 functions and found 455 red-zone leaf functions. It framed zero: 454 were outside the supported frame-pointer prologue shape, and one had another unsupported stack-pointer use. The executable and both guest modules were byte-identical with and without the flag. The crashing limiter falls within the PR's documented limitation for frameless leaves whose red-zone slots use the stack pointer directly.

Three runs used the same libraries and scripted inputs:

Run Flag Shader cache Audio-thread crash
1 --frame-red-zone cold 102.3 s
2 --frame-red-zone warm 84.1 s
control none cold 112.1 s

All three failed in the same limiter and at the same two crash sites previously reported in #361. The flag neither fixes nor regresses this title, because it frames no function and produces identical binaries. ASTRO BOT needs support for the documented frameless shape before this approach can address its crash.

This evidence is for the tested implementation at 9b8fe84, with the integration adaptations described above; it does not establish a runtime result for the newer PR head.

GPUs tested: NVIDIA GeForce RTX 3060 12 GB, driver 591.86 (32.0.15.9186), Vulkan 1.4.325, Windows Server 2025 10.0.26100.

AI-assisted: yes.

@parl0v

parl0v commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @fdfawef324, that's a useful data point. It matches the limitation recorded in docs/dev/debt/relinker.md: this PR only frames leaves with the push rbp; mov rbp, rsp prologue shape. A title whose red-zone leaves are frameless gets byte-identical output, so nothing changes, as you saw on ASTRO BOT (0 of 455 framed, same crash sites with and without the flag). For PPSA25344 the boot race is in framed-shape leaves (137 framed; 0 crashes in 27 boots against 6 in 23 without the flag). Frameless leaves would need a different approach and belong in a separate PR, not this one.

@tibzejoker

tibzejoker commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

@parl0v a data point from GTA V (PPSA04264), the title behind #361's inflate crash. I relinked its eboot with this PR on main 665ad7d (Windows 11, i9-13900K, --windows --to-intel --frame-red-zone):

Red-zone framing: 96761 functions, 875 leaf functions keep values below RSP, 0 framed (0 exits)
Red-zone framing: skipped 825: no push rbp; mov rbp, rsp prologue
Red-zone framing: skipped 50: other use of rsp

So none of GTA's red-zone leaves has the RBP prologue this PR frames. The leaf we traced in #361 is zlib's inflate_fast, which spills strm/state straight to [rsp-0x20], [rsp-0x18] and [rsp-0x78]. On a current-main build without #637, GTA still crashes with write watch on: an access violation in the RAGE main thread right after the save loads (eboot 0x28aacee). That fits the same red-zone damage, though I have not traced this one. This is the frameless case your debt entry lists. For this title it is most of the cases, not an edge case.

AI-assisted (Claude Code).

@oneandonlydean

Copy link
Copy Markdown
Collaborator

Function starts here are direct call targets plus 16-byte-aligned code after int3 padding (RedZoneFramer.cpp:143). A function reached only through a pointer and not preceded by int3 ends up inside the previous region, and its pop rbp; ret exit gets the lea 0x80(%rsp) without the matching prologue. Please take function bounds from the .eh_frame FDEs and throw when they can't be read. Also, once #637 is in, a PPSA25344 boot count without the flag would show whether the framing is still needed then.

@parl0v

parl0v commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

@oneandonlydean thanks, you're right. A function reached only through a pointer and not preceded by int3 padding gets merged into the previous region, so its pop rbp; ret exit could get the epilogue lea without a matching prologue. I'm moving the function bounds to the .eh_frame FDEs (the relinker's existing EhFrameReader). Framing will throw when the executable's .eh_frame can't be read, and a candidate leaf will need an FDE that starts exactly at its prologue. A test will cover a pointer-only function with no int3 before it.

I'll also take the boot count you suggested: PPSA25344 on main + #5596 relinked without --frame-red-zone, against the 6 crashes in 23 boots on the old base. If #5596 removes the race on its own, I'll say so here and close this PR.

@oneandonlydean oneandonlydean self-assigned this Oct 14, 2026
@parl0v

parl0v commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

@oneandonlydean here's the count you asked for. Closing this one: #5596 fixed it at the source.

PPSA25344 on main at 00f4b33 (includes #5596), 60 s boots, alternating, same machine (i5-14400F, RTX 4060):

relink boots crashes
no flag 20 0
--frame-red-zone, bounds from .eh_frame 20 0

Before #5596, the same title crashed at app.exe+0x37f77b3 in 6 of 23 boots without the flag. Both builds reach the title at about 70 presents per 60 s.

I also made the fix you described: function bounds from .eh_frame FDEs via ReadExceptionFunctions, with a test for two adjacent pointer-only leaves. It wouldn't have helped this title. SotS has 14,483 FDEs, covering 7.8 % of its text. One 70.7 MiB stretch has none, and the crash function 0x37f74c0 sits inside it. With FDE bounds the framer frames 19 functions instead of 137, and that function isn't one of them.

So on SotS, the safe version of the flag can't reach the code that crashed, and the unsafe version isn't worth carrying now that the race is gone. If another title still shows the red-zone clobber on current main, the .eh_frame version is on my fork as a starting point: parl0v/AnyPS5@3ff74b9. Thanks for the hint toward #5596.

@parl0v parl0v closed this Oct 14, 2026
@EagleBearerYS

Copy link
Copy Markdown
Contributor

@parl0v a heads-up and a question.

Grand Theft Auto V (PPSA04264) hits the same red-zone clobber in zlib's inflate_fast (#9136, #361). tibzejoker tied one crash to a single resumed write there. The faults left in GTA V are in shared views, which #5596 doesn't reach. So we're reopening #5640 to frame GTA V's leaves, with tibzejoker testing (their port of it). #5640 carries your fab7d20 unchanged and under your name, and its description credits #5052.

Some leaves can't be framed in place, so we'd relocate those whole functions, and that needs exact bounds. Your feat/relinker-frame-red-zone-fde (3ff74b9) takes them from the .eh_frame FDEs, as @oneandonlydean asked here. Would you be fine with us using that commit in #5640, with your authorship kept? If you'd rather open it yourself, or would rather it not be used, that's fine too. No rush.

@parl0v

parl0v commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

@EagleBearerYS go ahead and use 3ff74b9 in #5640, with the authorship kept and the credit to #5052 as you describe. It sits on main from 14.10., so expect to rebase it: it touches RedZoneFramer.hpp/.cpp, its tests, Conversion.cpp (where the FDEs are read via ReadExceptionFunctions) and the relinker debt line. Its test covers two adjacent pointer-only leaves. Glad it's useful for GTA V. If a boot count on SotS (PPSA25344) with the relocation would help, I can run one.

@EagleBearerYS

Copy link
Copy Markdown
Contributor

@parl0v thanks again for 3ff74b9. It's in the reopened #5640 under your authorship (606fa2e), together with this PR's commit (fab7d20). The end-of-body cut it made unnecessary is gone (a376e37).

The head is up now (#5640), if you still want to do the PPSA25344 boot count you offered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants