Windows: since #6678, GTA V crashes in the open world: resumed write faults clobber the red zone again (#361)

@EagleBearerYS

What

Since #6678, GTA V (PPSA04264) crashes in the open world on Windows, in every run. The cause is the red zone damage of #361: a fault that AnyPS5 resumes in guest code can overwrite [rsp-0x80, rsp-0x10). Deferred settles return small imports to the write watch, and the guest's own stores into that memory then fault and get resumed. #7473 adds more resumed faults of the same kind on resident runs.

FATAL: unhandled exception 0xc0000005 on thread '[RAGE] HDD Streamer'
  rip gtav.exe+0x28baa50   (eboot 0x28aaa50)
  read of 0x000000000000774f

At eboot 0x28aaa50, the game's zlib inflate_fast reloads two pointers from its red zone (mov -0x78(%rsp),%r9, mov -0x70(%rsp),%rbx) and then reads through rdx, which holds a small garbage value. The same code crashed this way before #637/#5596.

Measured

Every fault the vectored handler resumes was logged, with its rip, data address and VirtualQuery type (a diagnostic patch in ReportCrash, not proposed). The game resumes a story save in Los Santos. Windows 11 26200, RTX 4090 (NVIDIA 616.92), i9-13900K, WinLibs GCC 15.2.0 r7, Release, write watch on.

tree	resumed faults	in zlib (`0x28aa…`/`0x28ab…`)	open world
`main` `47a266c` (10-17)	0 in 240 s	0	no crash
`a84027d` (merge before #6678)	0	0	no crash
`755fecb` (#6678 merge)	134	—	—
`main` `f6ce8e3` (10-20)	400+	~60, all `MEM_MAPPED` 4 KiB subpages of 16 KiB views	crash, 7 of 7 runs
`main` `275e896` (10-21)	400+	also 75 on `MEM_PRIVATE` 512 KiB resident runs (eboot `0x28abdd4`)	crash
`main` `7280a17` + `APS5_NO_DEFERRED_SETTLES=1`	133 (85 `MEM_MAPPED`, 44 `MEM_PRIVATE`)	0	no crash in 150 s

The first two rows and the #6678 row come from a first-parent git bisect run whose step counts the resumed faults over 60 s of Story: "bad" means at least one. The plain crash bisect was too slow and too random to use. The merge before #6678 had 0 faults; #6678 had 134.

perf(agc): settle Windows imports under 1 MiB without waiting #6678. rewatchSettledImports puts settled imports under 1 MiB back in the write watch. GTA's buffers are resident direct memory when they are imported, and MEM_MAPPED 16 KiB views when the stores fault. So they become shared views after they return to the watch, and from then on each guest store into them is armed and resumed.
- I tried skipping the rewatch unless the whole range is MEM_PRIVATE at rewatch time. That changed nothing: the memory is still private at that point.
- A fresh shared view is imported watched from the start, never compared. So the import path alone does not keep shared views fault-free.
perf(libc): skip write-watch queries for proven clean resident ranges #7473. resumeResidentWrites arms resident runs that have been proven clean. The 44 to 89 MEM_PRIVATE faults per run come from it, and 16 to 75 of them land in zlib.
Remaining. 85 MEM_MAPPED faults (eboot 0x162b9e4, 0x28f413a) remain with deferred settles off. main 47a266c had none, so at least one more merge between 10-17 and 10-21 arms memory this title writes. I have not bisected that one yet.

Question

Every new arming of guest-written memory brings #361 back on Windows. A few options, from narrowest to most general:

keep memory that may become a shared view out of the deferred-settle rewatch;
never arm resident runs (perf(libc): skip write-watch queries for proven clean resident ranges #7473);
track shared views by comparison instead of arming them on Windows, as Windows: resuming a write fault overwrites the guest red zone when AVX state is live #361 suggested.

@EagleBearerYS @Dripl1x, since this is your code: which direction would you take? I can test any branch on GTA within the hour: the fault log patch gives a deterministic count per run.

AI-assisted: yes (Claude Code). Measured on the machine above.

@tibzejoker

@tibzejoker, thank you for the bisect and the per-run fault counts; that makes this much easier to pin down. #6678 is mine. I'm reading rewatchSettledImports against your table now, together with my open #8926, which also returns retired small imports to the write watch, so it needs the same check before it goes anywhere. I'll post a branch here for you to run on Grand Theft Auto V (PPSA04264), with what we'd like counted.

@EagleBearerYS

Thanks @EagleBearerYS. I'll run your branch on GTA V as soon as it's up, with the same per-run fault counts and types. A first-parent bisect of the 85 MEM_MAPPED faults left with APS5_NO_DEFERRED_SETTLES=1 is running now. I'll post which merge it lands on here.

AI-assisted: yes (Claude Code)

@tibzejoker

@tibzejoker, thanks. #7473 is mine, and for it I'd take option 2: stop arming resident runs.

armResident makes clean resident runs (MEM_PRIVATE, 256 KiB and up) read-only so that a collect can skip GetWriteWatch, and the first guest store into each run is then resumed on the guest thread. That is the #361 fault that #5596 removed. It happens at most once per run per mapping, which fits your 44 to 89 per run. Option 3 would not cover it, since these are private pages, not shared views.

There is no runtime switch for it. I'll post a branch here that removes the arming, so resident memory is collected with GetWriteWatch only, as before #7473. Expected on your counter: 0 MEM_PRIVATE faults per run, MEM_MAPPED unchanged. #7471 stays as it is.

If your bisect of the remaining 85 MEM_MAPPED faults lands on #7473 or on my #8857, I'll take that too.

AI-assisted: yes (Claude Code)

@tibzejoker

@tibzejoker, the branch: Dripl1x/AnyPS5 fix/no-resident-write-faults, one commit d6e270ea9f on main 80565c5942.

It removes armResident and resumeResidentWrites: resident runs are never write-protected and are collected with GetWriteWatch only. HostImportAccess stays and is now a no-op on resident pages, so View::hostImports is only counted, never read.
CheckLargeResidentWritesStopFaulting is now CheckLargeResidentWritesDoNotFault: a 2 MiB resident mapping stays PAGE_READWRITE after collects, and 16 write/collect rounds over every page raise no access violation. With main's WindowsMappings.hpp it fails at the first protection check (GuestMemory.cpp:1773: after two clean collects the run is armed read-only).
On this host (Windows 11 26300, Ryzen 7 5800H, WinLibs GCC 15.2.0 r7, Release): guest_memory and guest_mmap_fixed_anonymous 3/3; agc_driver_host_import_windows, agc_driver_host_import_at_map, agc_driver_host_import_teardown, agc_driver_host_import_failure_handles and agc_host_import_retire 3/3; agc_host_import_sparse skipped itself (exit 77); agc_resident_present, agc_host_import_unmap, agc_host_import_retirement, agc_host_import_second_mapping, agc_parked_import_unmap, agc_sparse_run_import, agc_sparse_run_import_copied, agc_driver_host_import_lifetime and agc_driver_host_import_mutation_race 1/1. No title was run on this branch. The change is Windows-only.

What I'd expect on your counter: 0 MEM_PRIVATE resumed faults per run. The cleanest comparison is your APS5_NO_DEFERRED_SETTLES=1 row: 133 (85 MEM_MAPPED, 44 MEM_PRIVATE) should become 85 and 0. With deferred settles on, the #6678 faults remain until @EagleBearerYS's branch.

AI-assisted: yes (Claude Code)

@tibzejoker

@tibzejoker thank you again: the bisect and the per-type fault counts are what pinned this down.

What #6678 does, in the code. When a small import's deferred settle completes, rewatchSettledImports puts its range back in the write watch (GuestMemory::Rewatch). At that point Grand Theft Auto V (PPSA04264)'s buffers are still resident direct memory: private pages tracked with GetWriteWatch, which take no fault. That's why your MEM_PRIVATE check at rewatch time changed nothing. Later, a second mapping of the same direct memory turns the run into 16 KiB shared views (WindowsMappings::demote). From then on, every clearing collect over the range makes all its aliases read-only, so each guest store faults and is resumed, with #361's damage to the red zone. Before #6678, the import kept the range out of the write watch for good, so it was compared and never armed.

My open #8926 (retired imports back in the write watch) calls the same Rewatch, so it can do the same, and it would also bring such ranges back with APS5_NO_DEFERRED_SETTLES=1. It stays on hold until it has this fix as well.

The branch: EagleBearerYS/AnyPS5@fix/agc-rewatched-imports-compared-when-shared (d98ee15, two commits on main 27bcd6a). It's your option 1, applied where the memory turns shared:

Rewatch marks the pages of each range it puts back in the watch.
While they stay resident, they keep GetWriteWatch, as perf(agc): settle Windows imports under 1 MiB without waiting #6678 intended.
Once a marked page is shared, a collect over it is refused before anything is made read-only. Its aliases are never armed, and the range is compared again, as before perf(agc): settle Windows imports under 1 MiB without waiting #6678. Any alias that was already armed is made writable again when the mark is set.
Tests: windows_write_watch gains a case, and the new agc_rewatched_direct_shared maps direct memory, rewatches a block of it as a settle does, then maps the same memory a second time. The block must not be armed, a store into it must not fault, and the store must show through both mappings.
On GitHub's Windows runner, with only the two checks reverted, both cases fail ("a page to be compared when shared was collected once shared"; run). With them, both pass ×10, along with the other write-tracking tests and the libc and libkernel suites (run).

How it fits with #7473: @Dripl1x's fix/no-resident-write-faults (d6e270e) is the resident side, so resident runs are no longer armed. This branch only covers memory that turns shared after a rewatch, so the two don't overlap, and they merge cleanly. Your table suggests both are needed to reach zero.

Could you run these four, with your fault-log patch, the same save and at least 60 s of Story?

main;
main + this branch;
main + this branch with APS5_NO_DEFERRED_SETTLES=1;
main + this branch + Dripl1x's.

Please post the fault counts by type (MEM_MAPPED, MEM_PRIVATE) and how many land in zlib, as in your table.

We expect the MEM_MAPPED faults on settled imports to go.
The MEM_PRIVATE ones are perf(libc): skip write-watch queries for proven clean resident ranges #7473's, on Dripl1x's branch.
For the 85 MEM_MAPPED that remain with deferred settles off, your bisect will tell.

@wachi35676

The bisect of the remaining MEM_MAPPED faults lands on #8172 (@wachi35676, "keep resident DCC targets watched under aligned block imports", merge d9df34d).

Same counter as above, with APS5_NO_DEFERRED_SETTLES=1 so that #6678 is out of the picture. Each row counts the MEM_MAPPED resumed faults over 60 s of Story:

tree	`MEM_MAPPED` resumed faults
`e9cc4a8` (merge before #8172)	0
`d9df34d` (#8172 merge)	6
`28f1266` (#6598 merge, ~80 merges later)	134
`28f1266` with #8172 reverted (`git revert -m 1 d9df34d08`, clean)	1

So #8172 accounts for these faults on its own; the single fault in the last row is the background level we have always seen. My reading of the change: an aligned block import now keeps the resident parts of its run in the write watch instead of unwatching the whole run. When the game then maps that memory at a second address, the run moves into the section and becomes shared views. Those views are tracked by write protection, so the guest's own stores fault (eboot 0x162b9e4, 0x28f413a, outside zlib in this window, but the same #361 hazard). Before #8172 the whole run left the watch, so the views were never armed.

#8172 fixes a real test failure (#8103, agc_driver_msaa_dcc_clear_4), so reverting it is not the answer. One option is to keep watching only the parts that stay private (resident, MEM_WRITE_WATCH), and drop them from the watch at the moment they become shared views. I'll test whatever branch you prefer.

@Dripl1x: your fix/no-resident-write-faults (d6e270e) is reviewed here and queued. I'll run it as soon as this machine is free and post the counts.

AI-assisted: yes (Claude Code). Measured on the machine in the first post.

@Dripl1x

@Dripl1x, fix/no-resident-write-faults (d6e270e, on main 80565c5) on GTA V: 0 MEM_PRIVATE resumed faults, as you expected. One run of 150 s of Story each, same counter and machine as above:

tree	total	`MEM_MAPPED`	`MEM_PRIVATE`	in zlib	crash
your branch + `APS5_NO_DEFERRED_SETTLES=1`	0	0	0	0	none in 150 s
your branch, deferred settles on (default)	399	394	0	64	none in 150 s (it is random)

Two follow-ups to my earlier posts:

fix(agc): keep resident DCC targets watched under aligned block imports #8172. The 85 MEM_MAPPED faults that I attributed to fix(agc): keep resident DCC targets watched under aligned block imports #8172 are gone on your branch with deferred settles off, even though 80565c5 contains fix(agc): keep resident DCC targets watched under aligned block imports #8172. On 28f1266, reverting fix(agc): keep resident DCC targets watched under aligned block imports #8172 did take the count from 134 to 1. So on newer trees, what fix(agc): keep resident DCC targets watched under aligned block imports #8172 exposed seems to go through the resident-run arming that your branch removes. @wachi35676, there may be nothing left to do on your side. I'll re-check on the next main.
perf(agc): settle Windows imports under 1 MiB without waiting #6678. With your fix in, the deferred settles are the only source left: 394 resumed faults, 64 of them in zlib. @EagleBearerYS, your branch can be compared against the second row.

AI-assisted: yes (Claude Code). Measured on the machine in the first post.

@tibzejoker

@tibzejoker thanks, those two rows settle it. With Dripl1x's d6e270e, the MEM_PRIVATE faults and the ones you'd pinned on #8172 are gone, so what's left with deferred settles on is #6678's path. That's what #9159 (d98ee15) targets, so we've left the branch as it was.

Could you run one more, the same way as your second row (150 s of Story, deferred settles on, the default)?

main + Dripl1x's d6e270e + fix(agc): keep a rewatched import's memory compared once it is shared #9159 at d98ee15.

We expect the 394 MEM_MAPPED faults from settled imports in that row to go, and the zlib ones with them.

If you have time for a second run, main + #9159 alone (without Dripl1x's fix) would show the two fixes apart.

fix(agc): keep an import's resident memory compared once it is shared

@EagleBearerYS

@EagleBearerYS, both runs, on main 663de1a, 150 s of Story with deferred settles on (the default). Same counter and machine as above:

tree	total	`MEM_MAPPED`	`MEM_PRIVATE`	in zlib	crash
`main` + #9159 alone (`98adb5f`, `d98ee15`)	113	0	111	54	HDD Streamer, 30 s into Story
`main` + Dripl1x's `d6e270e` + #9159	43	42	1	1	none in 150 s
(earlier) Dripl1x's branch alone	399	394	0	64	none in 150 s

fix(agc): keep a rewatched import's memory compared once it is shared #9159 alone removes every MEM_MAPPED resumed fault. The 111 left are perf(libc): skip write-watch queries for proven clean resident ranges #7473's resident arming; 45 of them land at zlib's eboot 0x28abdd4, and the run crashed.
With both fixes, 42 MEM_MAPPED faults come back. None of them is in zlib. Their sites are eboot 0x28f413a (18), 0x162b9e4 (10), and two in a system DLL (+0x…4269 10, +0x…4221 4). Those are the same sites as the 85 I first saw with deferred settles off, which perf(libc): skip write-watch queries for proven clean resident ranges #7473's arming seemed to hide.
- My guess, not checked: without the resident arming, these runs stay resident and private longer. They are then moved into the section by a second mapping while they are still in the write watch. They were never rewatched settles, so they are not marked comparedWhenShared, and their views get armed.
- If that is right, the same "compare once shared" rule would apply to any watched resident run that turns into shared views, not only to rewatched imports.
The single MEM_PRIVATE fault in the combined row is at 0x28abdd4. It is probably the race path in HandleWrite, where the page was already writable when the handler ran.

One run per row. These counts are stable in kind from run to run, but not in exact number.

AI-assisted: yes (Claude Code). Measured on the machine in the first post.

@tibzejoker

@tibzejoker thank you, that pair is exactly what was missing. Your reading fits: the 42 MEM_MAPPED faults at 0x28f413a and 0x162b9e4 are on resident runs that an import kept in the write watch. That's the path #8172 changed so that resident DCC targets stay watched, which #8103 needs. Those runs are never rewatched, so #9159's first two commits don't mark them.

The third commit covers them. EagleBearerYS/AnyPS5@fix/agc-imports-compared-when-shared is at 79cae21, one commit on top of #9159's d98ee15, and #9159 takes it next:

ImportWatched marks the resident pages of each run it keeps in the watch.
They stay write-watched while they're private, so AGC: 4x DCC refill resolve loses watched color after an adjacent aligned import (Windows RTX5080) #8103's DCC targets are tracked as before. Once a second mapping makes them shared, they're compared instead of armed.
Memory that is already shared when it's imported isn't marked, and keeps today's tracking.
It doesn't apply the rule to every resident run that turns shared. Direct memory that was never imported and is mapped twice is still armed, as it was before any of these changes. We'd widen it only if faults remain at other sites.

Could you run one more, the same way (150 s of Story, deferred settles on)?

main + Dripl1x's d6e270e + 79cae21.

We expect MEM_MAPPED at your background level. The single MEM_PRIVATE fault in your last run is on the resident side, so it's yours and Dripl1x's to judge.

fix(agc): keep an import's resident memory compared once it is shared

@EagleBearerYS

@EagleBearerYS, a run on main 663de1a + Dripl1x's d6e270e + your branch, 150 s of Story with deferred settles on. Your branch head is now 81aec58 (with ead292e under it, on top of d98ee15), not 79cae21, so I ran that head.

tree	total	`MEM_MAPPED`	`MEM_PRIVATE`	in zlib	crash
`main` + `d6e270e` + #9159 (`d98ee15`)	43	42	1	1	none in 150 s
`main` + `d6e270e` + your branch (`81aec58`)	105	105	0	0	none in 150 s

zlib and the resident side are clean. That covers everything that crashed. 0x162b9e4 is gone too.
Two sites remain, both MEM_MAPPED:
- eboot 0x28f413a (71 faults): a mov %rax,(%rsi,%rdi,8) inside a loop that keeps state in r12–r14, so it is not a red-zone leaf at that point.
- a system DLL at +0x…4269 (34 faults): host code, which does not use a red zone.
- Both write 16 KiB views that were shared when written. I haven't checked whether that memory was ever imported. They look like the "direct memory mapped twice, never imported" case you deliberately left armed.
The count is higher than in the previous row. 105 against 42 at those sites, but this is one run each and that number moves with where the save spawns. I wouldn't read the difference as a regression without a second run.

From my side, the crash is fixed with the two fixes together. Whether to widen the rule to direct memory that was never imported, so that the remaining sites stop faulting too, is your and the maintainers' call. That is the most general form of #361, and I'll test it if you go that way.

AI-assisted: yes (Claude Code). Measured on the machine in the first post.

@tibzejoker

@tibzejoker thank you for the run and the whole bisect: 0 MEM_PRIVATE, 0 faults in zlib, and no crash is the result we were after.

Grand Theft Auto V (PPSA04264)'s crash needs both fixes:

Dripl1x's d6e270e, for the resident runs that perf(libc): skip write-watch queries for proven clean resident ranges #7473 armed;
fix(agc): keep a rewatched import's memory compared once it is shared #9159 with 81aec58, the branch you ran (it goes onto fix(agc): keep a rewatched import's memory compared once it is shared #9159 once its checks finish), for memory that turns shared after a settle puts it back in the write watch, or while an import keeps it watched.

The 105 MEM_MAPPED that remain (0x28f413a, and the system DLL's host code) we're leaving as they are for now. By your reading they're direct memory that was never imported and is mapped twice. That memory is armed, as it was before any of these changes. Per your notes, neither site reloads pointers from a red zone, so they don't carry #361's crash. Comparing all such memory instead of arming it would cost every title that maps direct memory twice: its collects would return nothing, so GPU caches over it would compare on every use, and address-based builds would copy it. We'd widen the rule only for a title that crashes at a site of that kind.

@Dripl1x, are you opening a PR for d6e270e? Then it and #9159 can name each other as the two halves of #9136.

@EagleBearerYS

@EagleBearerYS yes, it's #9181 (the same commit d6e270e, unchanged). Its description names #9159 as the MEM_MAPPED half of #9136, so #9159 can point to #9181 instead of the branch.

@embersas

@embersas thank you. A standalone reproduction with no AnyPS5 or title code, on different hardware, is useful independent evidence for the mechanism, and the owned controls through the production handler confirm it in our build too.

Testing the two PRs on your workstation would help a lot. In order of usefulness:

Both together. There's a branch with both merged onto current main: test/9159-9181-together on EagleBearerYS/AnyPS5 (https://github.com/EagleBearerYS/AnyPS5/tree/test/9159-9181-together, d7cec80 = main ef0d8ae + fix(agc): keep a rewatched import's memory compared once it is shared #9159's 1ad5f29 + fix(libc): stop write-protecting resident runs on Windows #9181's d6e270e, no conflicts). Please run the cases you listed: the resident no-fault case, rewatched or imported pages that become shared, remapping, and import-created aliases, with the unmarked shared-view control kept.
Each alone, to attribute the result: fix(libc): stop write-protecting resident runs on Windows #9181 (d6e270e) for the resident runs, and fix(agc): keep a rewatched import's memory compared once it is shared #9159 (1ad5f29) for the rewatched and imported pages that become shared.

For each case, the most useful result is whether the store still faults, and whether the bytes below RSP changed. Keeping the remaining shared-view faults apart, as you suggest, is right: #9159 leaves some views armed on purpose. Those remaining faults are what the opt-in red-zone framing in #5640 (reopened as a draft) is for.

We'd welcome the standalone reproducer too. #5640's execution test simulates the clobber by filling [rsp-0x80, rsp) before each instruction. A real resumed fault on Windows would test the framed leaves against the frame Windows actually writes. If you can share it (a gist or an attachment here) under a license compatible with the project's GPL-2.0, and we adapt part of it into a test, we'd credit you as co-author on that commit.

@EagleBearerYS

@EagleBearerYS thanks. I ran both PRs on the same workstation (Ryzen 9 3900X, Windows 11 26300.9711, XState 0x7, WinLibs GCC 15.2.0 r7, Release). These were CPU tests only. No game was launched for any of this.

Trees, all with the PR heads unchanged (d6e270e, 1ad5f29):

main 0a0ed72 alone, with fix(libc): stop write-protecting resident runs on Windows #9181, with fix(agc): keep a rewatched import's memory compared once it is shared #9159, and with both
your test/9159-9181-together at d7cec80, as given

Both PRs applied to 0a0ed72 with no conflicts. The write-tracking implementation and test source files checked in the provenance record are byte-identical between my combined tree and your branch; the branches use different main revisions.

Focused tests, 3 runs per tree: guest_memory, guest_mmap_fixed_anonymous, windows_write_watch, agc_write_tracking, agc_write_tracking_metadata_commit, agc_resident_direct_remap, agc_rewatched_direct_shared, agc_import_alias_shared, agc_host_import_retire. All pass where registered. agc_host_import_sparse skips (Linux only).

For your per-case question I used a small probe. It is linked against each tree's own libc and agc GuestMemory.cpp, and sets up memory the way your tests do. Then it makes one 32-byte store with 16 known qwords in [rsp-0x80, rsp), at rsp mod 64 = 24, and lets the production GuestArenaHandleWrite resume any fault. Each cell is "store faulted / qwords changed below rsp". There were 90 independent probe executions: 6 case modes x 3 repetitions x 5 trees. Some modes also perform the embedded shared-view controls shown below; those rows are not additional executions. Outcomes were consistent across repetitions. The resident row describes the first store after clean collects; on baseline and #9159 alone, later stores in that same mapping do not fault after its first resume:

case	main	+#9181	+#9159	both	your branch
resident 2 MiB run after clean collects	yes / 13	no / 0	yes / 13	no / 0	no / 0
rewatched, then a second mapping	yes / 13	yes / 13	no / 0	no / 0	no / 0
watched import, then a second mapping	yes / 13	yes / 13	no / 0	no / 0	no / 0
rewatched, every view unmapped, mapped twice	yes / 13	yes / 13	no / 0	no / 0	no / 0
import alias over resident memory	yes / 13	yes / 13	no / 0	no / 0	no / 0
unmarked shared controls (3 kinds)	yes / 13	yes / 13	yes / 13	yes / 13	yes / 13

The probe verified the stored bytes in every case and visibility through the second mapping in the shared cases. For resident memory the next collect still detected the written page. In the four covered #9159 shared cases, collection refused as intended and the mappings stayed writable. I did not exercise the caller's subsequent comparison, so end-to-end dirty-data detection through that fallback remains unverified.

Failing controls, one at a time in a disposable tree with your tests kept:

fix(libc): stop write-protecting resident runs on Windows #9181's WindowsMappings.hpp reverted to main: guest_memory fails at the PAGE_READWRITE check (line 1790 on this main), and the resident store faults again.
fix(agc): keep a rewatched import's memory compared once it is shared #9159, each piece reverted on its own:
- the two refusal checks: windows_write_watch, agc_rewatched_direct_shared and agc_import_alias_shared fail
- the Rewatch mark: only agc_rewatched_direct_shared fails
- the ImportWatched mark: the imported case and agc_import_alias_shared fail
- the physical-page mark: only windows_write_watch fails, at the remap case
- demote(..., true): only agc_import_alias_shared fails
In each of those, the probe's store faults again and loses 13 qwords, except one: with only the refusal checks reverted, the alias case still did not fault in my sequence. The test caught it through the refusal instead.

I found no correctness defect within this tested CPU scope, and the observed fault/red-zone outcomes match the stated scope of both PRs, alone and together. The unmarked shared views still fault and still lose the red zone on every tree, which I read as the class you left for #5640. The standalone reproducer still shows the overwrite (12/12 fenced runs), so nothing here changes what Windows does. It only shows which stores no longer fault.

What I did not test:

any game. Synthetic results like these do not show anything about rendering, crashes, gameplay or performance in any title
the GPU-backed host-import and DCC tests (they need the GPU; I can run them if useful)
the comparison callers use after a refused collect
breakRun and the whole-run demotion paths you mentioned
other rsp alignments in the probe (the reproducer covers four), other CPUs, XState layouts, Windows builds, Linux or macOS
performance of fix(libc): stop write-protecting resident runs on Windows #9181 dropping the GetWriteWatch skip

One small thing for #9181, if useful: docs/dev/CONVENTIONS.md line 23 still says stable resident runs of at least 256 KiB can be read-only on Windows. You know this code far better than I do, so I may be misreading it.

If a different case or ordering would be more useful, I can run it.

AI-assisted: yes.

@embersas

@embersas thank you, this is exactly the separation we hoped for: each PR alone, both together, and the failing controls piece by piece. I've added your table to #9159's Tested section as independent CPU-only validation, with the limits you listed.

If you have time for more, two things would help most, in this order:

The GPU-backed host-import tests on test/9159-9181-together, and on main for comparison: agc_driver_host_import_lifetime, agc_driver_host_import_lifetime_other_family, agc_driver_host_import_mutation_race, agc_driver_host_import_failure_handles and agc_driver_guest_memory_lifetime_converted. They go through the import paths both PRs change, with a real device.
The fallback after a refused collect, end to end. In one of the four shared cases, after the store, does the caller's comparison see the new bytes? A sampled texture or a buffer over that memory picking up the CPU's write would close the gap you noted.

breakRun and the whole-run demotion matter less to us for now.

@EagleBearerYS

@EagleBearerYS here are the GTA V runs on test/9136-diag-on-9181.

Build: 2b875be + ed8a900 (#9730's voice fix; without it the branch stops at sceVoiceSetThreadsParams) + a local per-rip resumed-fault log. Relinked without --frame-red-zone, with APS5_MIMALLOC=1. Three open-world runs of up to 420 s on an RTX 4090 with Windows 11.

Run	Resumed writes	`[shared-arming]` lines	End
1	1, at `inflate_fast` 0x28aa5e0	none	HDD Streamer AV at eboot 0x28aaa50, same thread, after ~30 s
2	1, at `inflate_fast` 0x28aa898	1 (`alias`)	HDD Streamer AV at eboot 0x286514b, same thread, after ~2 min
3	0	0	no crash in 420 s

So each crash again follows a single resumed write on that thread.

Run 2: the alias case, unmarked.

[shared-arming] resumed write 1 at 0x10d85a6000: page 0x10d85a4000 aliases 1 stored 1 comparedWhenShared 0 aliasDemoted 1 demotedBy alias mapStored 0 mappedResident 1 armed by none collect from 0000000000000000 over 0x0-0x0
[shared-arming] totals after 1 resumed writes: none 0 alias 1 secondMapping(page) 0 secondMapping(run) 0 breakRun(Map) 0 breakRun(Reset) 0 breakRun(Commit) 0 sharedFromStart 0
[gtav-fault] write #1 rip=00000001428ba898 addr=10d85a6000 tid=12872 type=MEM_MAPPED alloc=10d85a4000 region=0x2000

The page was mapped resident and then demoted by an alias, which is the path 1ad5f29 targets. It still has comparedWhenShared 0.
My guess: ImportWatched marks only the spans that walkWrites accepts, so this one may have fallen outside them.
It says armed by none: no Collect re-arm loop armed it, yet the subpage was read-only when the write came. The protection may come from the demotion itself, or from a protectSubpages call outside that loop.

Run 1: a resumed write that never reaches [shared-arming].

[gtav-fault] write #1 rip=00000001428ba5e0 addr=102a4a38f6 tid=28948 type=MEM_PRIVATE prot=PAGE_READWRITE alloc=102a4a0000 region=0xd000

The memory is private, and the handler that resumed it is not the shared-view branch of HandleWrite. My fault log runs after the resume, so prot is the protection after the handler.
That leaves the resident branch of HandleWrite (it returns true when the page is already writable, with no print), or a handler outside WindowsMappings. Printing in the resident branch too would tell which one it is.

Since the fault in run 1 does not go through shared-view arming at all, framing (#5640) looks like the only fix here that covers every source. The full logs are kept here if you want other lines.

@tibzejoker

@tibzejoker thanks, these narrow it down.

Run 2 matches a store racing the import's own demotion: when the alias turns the resident run into shared views, demote() makes the page read-only for the switch, and a guest store that lands in that window faults before ImportWatched marks the page. That would also explain comparedWhenShared 0 with armed by none. A mark can't prevent that window; only framing makes the resumed write harmless.
Run 1 looks like a write resumed outside WindowsMappings: the page is private, and the shared-view branch of HandleWrite never sees it.
test/9136-diag-on-9181 now has two more lines, at 594b624, that would settle both: a [shared-arming] resident write … line for writes the resident branch resumes, and, for a resumed shared-view write, whether its subpage was armed and how many milliseconds after its demotion the write came. If you have time, one more run of that branch would tell.

Framing covers both sources, and your backward extension is going into #5640 as a commit that credits you.

@EagleBearerYS

@EagleBearerYS three more open-world runs on 594b624 (same build otherwise: + ed8a900 + the per-rip fault log, no --frame-red-zone, APS5_MIMALLOC=1).

Run	Resumed writes	End
1	0	no crash in 420 s
2	0	no crash in 420 s
3	1, at `inflate_fast` 0x28aa8ab	HDD Streamer AV at eboot 0x286514b, same thread (tid 19104), at ~6 min

Run 3 fits the race you described:

[shared-arming] resumed write 1 at 0x10de5a7000: page 0x10de5a4000 aliases 1 stored 1 comparedWhenShared 1 aliasDemoted 1 demotedBy alias mapStored 0 mappedResident 1 armed by none collect from 0000000000000000 over 0x0-0x0
[shared-arming]   subpage armed 0 (armed 0x0), 0 ms after its demotion
[gtav-fault] write #1 rip=00000001428ba8ab addr=10de5a7000 tid=19104 type=MEM_MAPPED alloc=10de5a4000 region=0x1000

The subpage was not armed, and the write came 0 ms after the alias demotion. That is the read-only window of demote(), not arming.
This time comparedWhenShared is already 1 when the line prints, so the mark was in place by then. It still could not stop the fault.
No [shared-arming] resident write line appeared in any of the three runs. The one MEM_PRIVATE fault of the earlier batch did not recur, so its handler is still unknown.

Over the six diagnostic runs, 3 crashed, each right after a single resumed write in inflate_fast on the crashing thread; the 3 runs without a resumed write did not crash. With the framed exe (#5640 + the backward extension) it was 0 crashes in 4 runs, with resumed writes in two of them.

@tibzejoker

@tibzejoker thanks, that settles run 2's case: the write came 0 ms after the alias demotion, on a subpage nobody armed, with the mark already in place, so it's demote()'s own read-only window, and no mark can close it. Across your six runs every crash followed a single resumed write in inflate_fast on the crashing thread, and with the framed executable it was 0 crashes in 4 runs, two of them with resumed writes. That points to framing as the fix for this crash. Your backward extension is being built into #5640 now, with its test changes, and I'll ask you for a run when it's up.

@EagleBearerYS

@EagleBearerYS closing the loop on your GPU and comparison-fallback requests above: the original results are on #9159, and the retest of your corrected head cb9de8049c is here.

The host-import GPU tests passed on the original baseline and combined branch. The fallback check found a stale first texture lookup; the corrected head now passes all four fixture orderings, three times each, and the new upstream regression test passes 3/3. The old combined branch still reproduces the stale read in both affected orderings, 3/3 each. No Vulkan validation errors or warnings on the RTX 3080 Ti.

This confirms the sampled-texture fallback fix within our synthetic test scope, not a fix for GTA V's crashes or every remaining red-zone fault. No game was tested.

AI-assisted: yes.

@tibzejoker

@tibzejoker #5640, the relinker's red-zone framing for leaves without a frame pointer, is merged in main as of d01070d, with the same framer your four framed runs used, so it may address this crash. It only runs when Grand Theft Auto V (PPSA04264) is relinked for Windows with --frame-red-zone, which is off by default, so could you retest on current main with that flag and the per-rip fault log on, and say which base you used and whether the HDD Streamer crash after a resumed write in inflate_fast comes back? On plain main, without #9181, #9159 and #9445, a run may stop somewhere else first, such as the State.cpp alpha-to-coverage throw your run 2 hit.

fix(agc): keep a rewatched import's memory compared once it is shared

fix(libc): compare the resident runs an import's alias demotes once t…

@EagleBearerYS

@EagleBearerYS here is the retest on current main with --frame-red-zone.

Base. main c8d34aa (which includes #5640's d01070d and #9159), nothing else merged in. The only local change is the per-rip resumed-fault log in libc's CrashReport.cpp (diagnostic, never for upstream). Relinked with --windows --to-intel --frame-red-zone, APS5_MIMALLOC=1, RTX 4090 (616.92), i9-13900K, Windows 11. Three open-world runs of 420 s each from the same save.

Relinker: 243 framed of 991 leaves that keep values below RSP, with the same skip counts as on #5640's head (642 "exit shorter than a jump", 58 "other use of rsp", 17 "branch leaves the function", 16 "prologue shorter than a jump", 7 "indirect jump", 7 "overlaps another stub", 1 with no room for a stub). inflate_fast is among the framed leaves (same counts as on cec1f82, where I checked it in the output).

Run	Resumed write faults	Of which in `inflate_fast` (lower bound)	End
1	1030	≥ 15 (0x28aa5e0 ≥ 10, 0x28aa898 3, 0x28aa8a2 2)	no crash in 420 s
2	1166	≥ 16 (+ 0x28aa8e6)	no crash in 420 s
3	1186	≥ 16	no crash in 420 s

The log prints the first three hits of each rip, then the 10th, 100th and so on, so the per-rip counts are lower bounds.

The HDD Streamer crash does not come back. Each run has at least 15 resumed writes inside inflate_fast. Without framing, a single one was enough to crash that thread.
Plain main resumes far more faults than the stack I used before. There are about 1,000-1,200 per run here, against a few with fix(libc): stop write-protecting resident runs on Windows #9181 and fix(agc): keep a rewatched import's memory compared once it is shared #9159. Almost all are MEM_PRIVATE (type 0x20000): the top site is eboot 0x28abdd4 (≥ 100 hits, 256 KiB regions), the resident-run arming of perf(libc): skip write-watch queries for proven clean resident ranges #7473 that fix(libc): stop write-protecting resident runs on Windows #9181 removes. Only one MEM_MAPPED write showed up, in run 1.
So on main, framing is what keeps GTA V alive, and fix(libc): stop write-protecting resident runs on Windows #9181 is still worth having for the fault count itself.
I didn't hit the State.cpp alpha-to-coverage throw in these three runs.

AI-assisted: yes.

@tibzejoker

@tibzejoker thank you for the retest. On main c8d34aa with --frame-red-zone, Grand Theft Auto V (PPSA04264) ran three 420 s open-world runs with no HDD Streamer crash, each with at least 15 resumed writes inside inflate_fast, which #5640 frames. Framing is opt-in: a build relinked without --frame-red-zone leaves those leaf functions unframed. Whether this issue can close is your call. Your fault counts on plain main, about 1,000 to 1,200 per run and almost all from the resident-run arming, are a data point for #9181.

@EagleBearerYS

Closing: on current main, a GTA V (PPSA04264) build relinked with --frame-red-zone no longer hits the HDD Streamer crash (3 × 420 s open-world runs, each with at least 15 resumed writes inside inflate_fast, details above). Thanks @EagleBearerYS and @Dripl1x.

What remains, for anyone landing here:

Framing is opt-in. A build relinked without --frame-red-zone leaves those leaves unframed, and a single resumed write in inflate_fast can still crash the streamer.
Plain main resumes about 1,000-1,200 write faults per run, almost all from the resident-run arming (perf(libc): skip write-watch queries for proven clean resident ranges #7473) that fix(libc): stop write-protecting resident runs on Windows #9181 removes. Fewer resumed faults also means fewer chances for any other code that keeps data below rsp.

Navigation Menu

Repository navigation

Windows: since #6678, GTA V crashes in the open world: resumed write faults clobber the red zone again (#361) #9136

What

Measured

Question

11 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Windows: since #6678, GTA V crashes in the open world: resumed write faults clobber the red zone again (#361) #9136

Description

What

Measured

Question

Activity

EagleBearerYS commented on Oct 21, 2026

tibzejoker commented on Oct 21, 2026

Dripl1x commented on Oct 21, 2026

Dripl1x commented on Oct 21, 2026

EagleBearerYS commented on Oct 21, 2026

tibzejoker commented on Oct 21, 2026

tibzejoker commented on Oct 21, 2026

EagleBearerYS commented on Oct 21, 2026

tibzejoker commented on Oct 21, 2026

EagleBearerYS commented on Oct 21, 2026

tibzejoker commented on Oct 21, 2026

EagleBearerYS commented on Oct 21, 2026

Dripl1x commented on Oct 22, 2026

11 remaining items

EagleBearerYS commented on Oct 23, 2026

embersas commented on Oct 23, 2026

EagleBearerYS commented on Oct 23, 2026

tibzejoker commented on Oct 23, 2026

EagleBearerYS commented on Oct 23, 2026

tibzejoker commented on Oct 23, 2026

EagleBearerYS commented on Oct 23, 2026

embersas commented on Oct 23, 2026

EagleBearerYS commented on Oct 24, 2026

tibzejoker commented on Oct 24, 2026

EagleBearerYS commented on Oct 24, 2026

tibzejoker commented on Oct 24, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions