You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
AGC: XPSemu's GPU probe times out: label writes after a dispatch never satisfy a WAIT_REG_MEM in the same submission, then its geometry draw is rejected (PPSA97358) #5437
XPSemu (PPSA97358, an xemu port) stops every time it starts, in xemu's GPU feature probe (render_geom_shader_triangles in hw/xbox/nv2a/pgraph/vk/gpuprops.c). Its own Vulkan driver (radv for PS5) waits 10 s for the probe's submission, prints radv/ps5: the GPU has not reached submission 2 (at 1) in 10 s, gets VK_ERROR_DEVICE_LOST (-4) and asserts. The timeout is caused by part 2 below: the probe's geometry-shader draw throws, and on main that failure isn't printed. Part 1 (waits retiring before the label is written) is a separate problem in the same submission. See the comments: it isn't what stops the title.
How it was run: main18c438b with #5061 (getdtablesize) and #5187 (RLIMIT_NOFILE) merged, relinked at the fixed base 0x400000 that #4013 describes. That's #4593's --linux-fixed-base on Linux and --windows-fixed-base on Windows. On Windows #4958 was merged too, for its Mesa shader cache. It stops the same way on Linux (Ubuntu 24.04, llvmpipe, headless) and on Windows 11 (RTX 3080).
1. The second and third label writes never become visible to the following WAIT_REG_MEM
The probe's submission (serial 3) writes one label three times, each followed by a wait on it (Linux, APS5_DUMP_QUEUE=0):
Linux shows the same (now 0x1). Serial 3 never completes, and a gdb snapshot on Linux shows no AGC thread at work. Two things differ between the write that lands and the two that don't: they come after a DISPATCH_DIRECT, and their event flags differ (0040c52d against 0040052d).
These RELEASE_MEMs have int_sel 3. preparePacketMemory in core/libs/prx/libSceAgcDriver/Execution/src/Driver/Packets/PacketMemory.cpp sends them past the end-of-pipe path, to the deferred-label code (DeferLabels / WriteLabelOnGpu). I haven't confirmed that is where they're lost. But APS5_DRAIN_ALL=1 skips that branch, and with it each wait sees its value (== 0x2 (now 0x2), == 0x3 (now 0x3)), which points there.
2. With the waits passing, a geometry-shader draw throws (observed on Windows)
With APS5_DRAIN_ALL=1 on Windows, all three waits pass but serial 3 still never completes. On Linux llvmpipe is too slow to get there within the title's 10 s. With #5177 merged as well, the failure shows up:
A draw after the third wait throws; the first draw in the dump after that wait is DRAW_INDEX_AUTO 3. Unless vertexWorkgroup is set, the check in core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp needs both the GS VGPR component count (SPI_SHADER_PGM_RSRC1_GS bits 29-30) and the ES one (SPI_SHADER_PGM_RSRC2_GS bits 16-17) to be 3. This shader sets SET_SH_REG 0000008a 222c0007 00c00010: GS count 1, ES count 0.
One relevant active change for the first symptom is PR #5324: APS5_STORE_AT_LABEL=1 flushes pending storage-image results before each GPU label write. It is opt-in and has only been validated against PPSA06694, so this is not evidence that it fixes the repeated RELEASE_MEM / WAIT_REG_MEM sequence here. If you can apply #5324 to the XPSemu probe and report whether the second/third waits observe their values, that would help distinguish stale storage-image writeback from deferred-label/submission ordering. The geometry VGPR allocation rejection looks like a separate failure after the waits are forced through.
A narrower diagnostic for the first symptom is APS5_LABEL_LOCK_EACH=1. In current main, DeferLabels() is disabled by that variable, so label writes take the immediate path without enabling the broader APS5_DRAIN_ALL behavior. If the repeated waits pass with this setting, that would isolate deferred-label batching; if they still fail, the issue is more likely in label ordering/publication or another queue path. This is diagnostic only, not a proposed fix.
So it isn't the deferred-label batching or stale storage-image writeback. Only APS5_DRAIN_ALL=1 makes the waits see 2 and 3. One more detail from these traces: the == 0x3 wait is logged about 0.2 ms after the == 0x2 one, while memory still reads 1. The queue goes past the second wait somehow, but not the third.
Thanks, that rules out both deferred-label batching and the storage-image flush as causes. The trace still leaves an ambiguity: the second wait is logged soon before the third while the sampled label remains 1, so it would help to tell whether the wait packet actually retires or only whether the host-side trace advanced.
Could you capture a short APS5_DUMP_QUEUE=0 excerpt from the third RELEASE_MEM through the third WAIT_REG_MEM, together with APS5_TRACE_GPU=1 timestamps and the label's CPU-visible value after each packet? If available, include the decoded WAIT_REG_MEM compare function and the GPU address/VM mapping for 0x206180000. That should distinguish an incorrect wait condition/address from a label write that remains invisible to the wait path. I don't have enough evidence yet to propose a code change.
Per-packet values on Linux (same stack). I used a temporary local print (not pushed) that reads the label with GuestMemory::Read(0x206180000, 4) at the top of the packet loop in PacketExecution.cpp, just after SetCurrentPacket, for serial 3, and ran with APS5_TRACE_GPU=1. Timestamps in ms; offsets are dword offsets in the submission:
The three waits are WAIT_REG_MEM 00000013 06180000 00000002 <ref> ffffffff 00000004: function 3 (equal), memory space, ME engine, address 0x2_06180000, mask 0xffffffff, poll interval 4. I don't have the GPU VM mapping for that address.
With this read in place, the problem goes away. The == 0x2 wait holds until the label reads 2 (815.6 to 8312.7 ms, after the first DISPATCH_DIRECT finishes on llvmpipe). The == 0x3 wait retires at once, and the queue reaches DRAW_INDEX_AUTO, where the geometry check fails as in part 2. Without the read (the earlier traces), the == 0x2 wait retires within about 0.2 ms while the label still reads 1, and the == 0x3 wait never does.
So a plain GuestMemory::Read of the label between packets makes the GPU-written values visible, and the wait's own read (ReadSite::Wait) seems not to. The second wait retiring early may come from something other than the label's memory value.
This trace narrows it substantially. The injected GuestMemory::Read invokes FlushForAccess, which can SyncThrough a pending recorder write; WAIT_REG_MEM itself uses WaitSatisfiedUnchecked and polls via volatile loads, so it bypasses that hook. waitMemory also has a same-queue recorder-label shortcut before polling.
Could you try one run with the print removed and APS5_NO_LABEL_SHORTCUT=1? That isolates the recorder-label-table early return from the raw memory polling path. If the second wait now stays until the label advances (like the injected-read run), the shortcut is implicated; if it still retires early while the CPU-visible value is 1, we can focus on the other early-return paths or the mapped-memory read. No source change is justified from this trace alone yet.
I checked the ordering in waitMemory and need to refine my previous diagnostic request: APS5_NO_LABEL_SHORTCUT=1 only disables the recorder label-table lookup; it does not bypass storedSince, which runs earlier and returns immediately when a recorded label write after this wait was received matches the comparison. In your trace, the RELEASE_MEM at offset 327 is immediately before the WAIT_REG_MEM == 2 at 335, so this recorded-store shortcut is now a concrete candidate for the early retirement. Could you add a temporary log at the storedSince return showing whether it matched for the second and third waits, plus the matched store stamp/writer queue and the wait’s received stamp? If that branch returns for queue 0 while the actual label is still 1, we’ll have isolated the cause without changing queue behavior speculatively.
The per-packet trace makes storedSince the leading candidate. In current main, the label-write path records the GPU label in history as soon as it is queued, and storedSince accepts any later matching stamp without checking whether that GPU write has completed or whether it came from the waiting queue. The matching RELEASE_MEM at offset 327 is in the same submission, immediately before the WAIT_REG_MEM at 335, so this shortcut could let the worker pass the wait while the CPU-visible label is still 1.
I have not changed the shortcut yet because it also intentionally lets waits on another queue be released by a producer's later label. The temporary log you described would confirm whether this exact return fires for the second wait; please include the matched store's queue/stamp and the wait's received stamp if you get a chance.
All three waits retire through storedSince, from same-queue stores queued after the wait's received stamp, while memory still reads 0 or 1. That includes the third one; I was wrong above to say it never retires. The (now ...) value is only printed when the wait starts.
That corrects my opening summary as well. The queue doesn't stall on the third wait. It passes all three early, reaches DRAW_INDEX_AUTO and throws on the geometry VGPR allocation (part 2). On main that failure isn't printed, so it looked like a stalled wait. The 10 s timeout comes from part 2. Part 1 is a separate problem: same-queue waits retiring before the GPU write lands. That may matter for whatever reads the dispatch results, but it isn't what stops XPSemu.
Thanks for the storedSince logs; they confirm the shortcut returns for all three same-queue waits using stores queued after the wait received stamp, even though the label has not reached the requested value in CPU-visible memory. I also understand the corrected diagnosis: those waits retire, and the reported 10 s timeout is the subsequent DRAW_INDEX_AUTO geometry VGPR allocation rejection. That separates a genuine premature same-queue wait completion from the current XPSemu stop. I’ll keep the queue-order issue separate and inspect whether the shortcut can safely distinguish a same-queue uncompleted store from a completed/cross-queue producer.
Correction to my previous comment: the logged store is the RELEASE_MEM immediately before the wait in the same queue submission. That ordering means the storedSince shortcut may be valid: later GPU work on that queue remains ordered behind the label write, even if CPU-visible memory is still stale when the wait is interpreted. The logs therefore confirm which shortcut fires but do not establish a queue-order bug by themselves. The concrete current failure remains the later geometry VGPR allocation rejection; I’ll avoid proposing a change to storedSince without evidence of a CPU-side consumer or a store that is actually after the wait.
Part 2 is #5604. It drops the GS/ES input-VGPR count check, since the mesh translation always fills v0-v8. With it (and #5061, #5187, #4958, at the fixed base), XPSemu passes the GPU probe on Windows (RTX 3080) and runs its UI loop. XPSemu's bring-up thread is #5600.
Thanks for the corrected XPSemu result and follow-up. #5604 now targets the geometry VGPR rejection that actually stopped the probe; its Windows RTX 3080 report says the probe completes and the UI starts. The storedSince trace remains a separate observation: because the matching label store is queued immediately before the same-queue wait, the early return alone does not show that a later consumer ran before the write. I would keep that ordering question separate unless a downstream read or output demonstrates a visible failure.
Note: these runs turned out to be on llvmpipe, not the HD 520 (Xvfb has no DRI3). See the correction below for the real HD 520 results.
A CPU-side consumer that fails visibly, in case it helps with the storedSince ordering question. It is also the HD 520 case that #4677 left unexplained (#4562).
Setup: DOOM Shareware homebrew (PPSA99666 v1.1.1, lowbit/ps5-doom), relinked with --to-intel. Linux, Intel HD Graphics 520, Mesa 25.2.8 ANV, main at c7601b7, Release, Xvfb.
The title polls its presentation fence from the CPU with a 500 ms timeout (wait_slot in src/ps5/gpu.c). When the timeout fires it logs gpu: frame 1 never completed and falls back to the CPU scaler.
Reproduction:
With a cold Mesa shader cache (MESA_SHADER_CACHE_DISABLE=true) the fallback happens on every run (6 of 6), with the AnyPS5 shader disk cache warm or cold.
With a warm Mesa cache it did not happen in any of our runs.
APS5_TRACE_GPU=1, plus a temporary print at the storedSince match return (reverted):
Serial 1 takes about 800 ms because the presentation kernel is prepared at dispatch.
Its wait retires through storedSince with memory still at 0, in both runs. So the shortcut alone is not what separates them.
With the cold cache, serial 3 never reaches its own WAIT_REG_MEM until shutdown. Before that wait it has WaitUntilSafeForRendering for display buffer 1, an ACQUIRE_MEM and the dispatch.
After serial 1 retired at 808.7 ms, the title's CPU poll did not see label 1 within its 500 ms window.
No change with APS5_LAND_WAITS_ALL_QUEUES=1, APS5_NO_LABEL_SHORTCUT=1, APS5_NO_WAIT_OVERLAP_SUBMIT=1, APS5_WAIT_LOCK=1, or with userfaultfd write watch available (seccomp unconfined).
I haven't found where serial 3 stalls yet. I can capture APS5_DUMP_QUEUE=0 or run any other diagnostic you'd like on this machine.
I checked the current WaitMemory.cpp path against these results. APS5_NO_LABEL_SHORTCUT=1 disables the recorder label-table lookup, but both storedSince() early returns run before that setting is read: once on entry (line 141) and again inside the polling loop (line 333). So the run with that variable does not rule out the storedSince path shown in your temporary trace.
A useful next diagnostic would be a local-only gate around both storedSince() checks, then rerun the cold-cache case and inspect whether the CPU-visible label reaches 1 before the 500 ms poll expires. If the packet wait blocks until the actual label write or fails, that would distinguish the recorded-store shortcut from the later dispatch/queue stall. This is a diagnostic suggestion only; I don’t yet have evidence for a fix.
Thanks for the full trace. It shows the serial-1 wait taking the same storedSince return with now=0 in both warm and cold runs, so that shortcut does not explain the difference. In the cold run, serial 3 has not reached its WAIT_REG_MEM before the title’s 500 ms presentation-fence timeout; the remaining stall is before that wait, around WaitUntilSafeForRendering, ACQUIRE_MEM, or the dispatch/pipeline work. A timestamped queue-0 excerpt for serial 3 from execute through those packet boundaries, plus when the title starts and expires its 500 ms poll, would narrow it. This trace does not support changing storedSince.
Correction to my comment above: those runs were not on the HD 520. They ran under Xvfb, which has no DRI3, so ANV could not present (vulkan: No DRI3 support detected - required for presentation) and device selection fell through to llvmpipe (Selected GPU name=llvmpipe (LLVM 20.1.2, 256 bits)). Every trace in that comment is lavapipe on Mesa 25.2.8, so it does not explain the HD 520 case in #4562. Sorry for the noise.
Rerun on the HD 520 itself (Weston headless with the GL renderer plus Xwayland, which provides DRI3). Intel HD Graphics 520 (0x1916), Mesa 25.2.8 ANV, kernel 6.12 (i915, preempt_timeout_ms 640), mainc7601b7, DOOM Shareware homebrew PPSA99666 v1.1.1 relinked with --to-intel:
Variant
Runs
Result
default fifo, warm or cold Mesa cache
8
8 fall back to the CPU scaler (gpu: frame 1 never completed), 2 of them with an i915 reset
The kernel log for the resets is Resetting rcs0 for preemption time out / GPU HANG: ecode 9:1:8ed0fff3, in eboot.elf.
What the trace shows on ANV (APS5_TRACE_GPU=1): serial 1 is done at about 52 ms, then serial 3 executes at 52.2 ms and only reaches its WAIT_REG_MEM at 743.3 ms. That is the title's presentation kernel (0x202c81800, a palette to RGBA conversion over 1920x1080) taking roughly 690 ms per dispatch. It is longer than the title's 500 ms fence timeout and longer than i915's 640 ms preemption timeout.
With 934348f applied, APS5_PROFILE_GPU=1 reports by program: 0x202c81800 x21 9916ms, about 472 ms per dispatch, below the preemption timeout, which matches the resets going away. The title then stays on GPU presentation at about 2 FPS, against 16.7 ms per frame on its CPU scaler.
So on the HD 520 the fallback comes from a slow dispatch, not from the storedSince ordering discussed here. The label question in my first comment stands only for lavapipe. I can open a separate perf issue for the dispatch cost if that is the better place.
Per-packet timing for serial 3 on the HD 520, as asked. Same setup as my correction above (HD 520, Mesa 25.2.8 ANV, mainc7601b7, DOOM PPSA99666 relinked with --to-intel). A temporary print at the start of every packet in PacketExecution.cpp (queue 0, serials 1 to 6, reverted, libraries rebuilt clean), APS5_TRACE_GPU=1, 2 runs with a warm and 2 with a cold Mesa cache:
Run
serial 3 DISPATCH_DIRECT
title falls back to the CPU scaler
warm 1
441 ms
no
cold 2
469 ms
no
warm 2
608 ms
yes
cold 1
645 ms
yes
Times are from the start of DISPATCH_DIRECT to the start of the next packet. Every other packet in serial 3 takes under 0.2 ms: WAIT_FLIP_DONE, ACQUIRE_MEM, the six SET_SH_REG, RELEASE_MEM and WAIT_REG_MEM. Serial 1's DISPATCH_DIRECT takes 29 to 38 ms.
The title logs frame 1 never completed while serial 3 is still inside that dispatch, and only in the runs where the dispatch goes past about 600 ms. The Mesa cache makes no difference. With 934348f applied, APS5_PROFILE_GPU=1 measured about 472 ms of GPU time per dispatch of this program (0x202c81800), so the worker seems to be waiting on GPU execution of the presentation kernel there, not on the flip or the acquire.
So on ANV the remaining stall is the cost of that one dispatch. I can collect anything else around it on this machine.
The geometry-allocation rejection from this report is fixed on current main by f535918, fix(agc): accept geometry programs with fewer GS and ES input VGPRs. Its regression sets the reported SPI_SHADER_PGM_RSRC1_GS=0x222c0007 and SPI_SHADER_PGM_RSRC2_GS=0x00c00010 values and confirms DecodeState accepts the geometry path. Could you retry the original XPSemu probe on current main and report whether the draw now reaches execution, and whether the earlier repeated RELEASE_MEM / WAIT_REG_MEM behavior still occurs? The wait symptom was analyzed separately and was not established as the same failure.
@MarcZX10 retried on current maineb1a15b5c, part of an answer. Linux, Ubuntu 24.04 container, GCC 13, Release, relinker and every library rebuilt; XPSemu (PPSA97358) relinked with --linux-fixed-base 0x400000, the GPU being lavapipe (LLVM 20.1) in this container, APS5_TRACE_GPU=1:
The draw now reaches execution. Serial 1 (569 dwords) and serial 2 complete (done serial=1, done serial=2 at 323 ms). Serial 3 (861 dwords) runs past the geometry-program draw: its graphics shaders 0x206120d00 and 0x206121600 are prepared and drawn, with no DecodeState rejection.
Then lavapipe aborts, while preparing compute shader 0x206121900 at its dispatch inside serial 3: llvm::report_fatal_error from llvm::TargetInstrInfo::lowerCopy in libLLVM.so.20.1, under libvulkan_lvp.so's pipeline compile. That's the host driver's own LLVM failure (the "Cannot emit physreg copy instruction" seen earlier), not AnyPS5.
So on this machine serial 3 never reaches its RELEASE_MEM / WAIT_REG_MEM packets, and I can't say yet whether the repeated wait behaviour remains. That needs a real GPU; I'll add a Windows run when that machine is free.
Thanks for rebuilding the current tree and confirming the geometry draw gets past DecodeState. The LLVM lowerCopy abort happens before the reported label waits, so this run neither confirms nor rules out the original wait behavior. A Windows hardware run reaching those packets would be the useful next data point; I do not see a code change supported by this lavapipe trace.
@MarcZX10 the rest of the answer, on a real GPU. Windows 11, RTX 3080, WinLibs GCC 15.2, Release, maineb1a15b5c, XPSemu relinked with --windows --windows-fixed-base 0x400000, 60 s, APS5_TRACE_GPU=1:
Serial 3 completes (execute serial=3 dwords=861 at 445 ms, done serial=3 at 4031 ms; most of the gap is first-use preparation of its two compute and two graphics shaders), and serial 4 completes with it.
The repeated waits are gone. Serials 1 to 4 have no WAIT_REG_MEM lines at all. The whole run has 8 waits, all later in the title's UI loop on 0x2010c0000, and each is satisfied at once (for example 5280.1 waits == 0xa, then 5280.2 done serial=39). None repeat or stay unretired.
The probe finishes and the title carries on. There's no device loss: it logs qemu_init done, running the machine, display ready and its UI loop, and runs the full 60 s at about 100 FPS (5,272 frames).
The probe's readback is black, though. Right after serials 3 and 4, xemu logs:
Could not determine triangle strip1 rotation, got color: R=0, G=0, B=0
Unexpected inconsistency in triangle fan winding, got colors: R=0, G=255, B=0 and R=0, G=0, B=0
VK geometry shader winding: 0, 0, 0, 0
So the geometry-program draw now executes, but most of what xemu reads back from its 640x480 probe target (0x412c400000) is zero. That looks like a separate question, about whether that draw's output reaches the image the probe reads, rather than the wait behaviour this issue started with.
Activity
MarcZX10 commentedon Oct 13, 2026
One relevant active change for the first symptom is PR #5324:
APS5_STORE_AT_LABEL=1flushes pending storage-image results before each GPU label write. It is opt-in and has only been validated against PPSA06694, so this is not evidence that it fixes the repeatedRELEASE_MEM/WAIT_REG_MEMsequence here. If you can apply #5324 to the XPSemu probe and report whether the second/third waits observe their values, that would help distinguish stale storage-image writeback from deferred-label/submission ordering. The geometry VGPR allocation rejection looks like a separate failure after the waits are forced through.MarcZX10 commentedon Oct 13, 2026
A narrower diagnostic for the first symptom is
APS5_LABEL_LOCK_EACH=1. In current main,DeferLabels()is disabled by that variable, so label writes take the immediate path without enabling the broaderAPS5_DRAIN_ALLbehavior. If the repeated waits pass with this setting, that would isolate deferred-label batching; if they still fail, the issue is more likely in label ordering/publication or another queue path. This is diagnostic only, not a proposed fix.babyhuey commentedon Oct 13, 2026
Both diagnostics on Linux (same stack as above, llvmpipe, headless,
APS5_TRACE_GPU=1). Neither changes the waits:APS5_LABEL_LOCK_EACH=1:waits 0x206180000 == 0x2 (now 0x1), then== 0x3 (now 0x1), as without it.APS5_STORE_AT_LABEL=1: the same,== 0x2 (now 0x1)and== 0x3 (now 0x1).So it isn't the deferred-label batching or stale storage-image writeback. Only
APS5_DRAIN_ALL=1makes the waits see 2 and 3. One more detail from these traces: the== 0x3wait is logged about 0.2 ms after the== 0x2one, while memory still reads 1. The queue goes past the second wait somehow, but not the third.MarcZX10 commentedon Oct 13, 2026
Thanks, that rules out both deferred-label batching and the storage-image flush as causes. The trace still leaves an ambiguity: the second wait is logged soon before the third while the sampled label remains 1, so it would help to tell whether the wait packet actually retires or only whether the host-side trace advanced.
Could you capture a short
APS5_DUMP_QUEUE=0excerpt from the thirdRELEASE_MEMthrough the thirdWAIT_REG_MEM, together withAPS5_TRACE_GPU=1timestamps and the label's CPU-visible value after each packet? If available, include the decodedWAIT_REG_MEMcompare function and the GPU address/VM mapping for0x206180000. That should distinguish an incorrect wait condition/address from a label write that remains invisible to the wait path. I don't have enough evidence yet to propose a code change.babyhuey commentedon Oct 13, 2026
Per-packet values on Linux (same stack). I used a temporary local print (not pushed) that reads the label with
GuestMemory::Read(0x206180000, 4)at the top of the packet loop inPacketExecution.cpp, just afterSetCurrentPacket, for serial 3, and ran withAPS5_TRACE_GPU=1. Timestamps in ms; offsets are dword offsets in the submission:The three waits are
WAIT_REG_MEM 00000013 06180000 00000002 <ref> ffffffff 00000004: function 3 (equal), memory space, ME engine, address 0x2_06180000, mask 0xffffffff, poll interval 4. I don't have the GPU VM mapping for that address.With this read in place, the problem goes away. The
== 0x2wait holds until the label reads 2 (815.6 to 8312.7 ms, after the firstDISPATCH_DIRECTfinishes on llvmpipe). The== 0x3wait retires at once, and the queue reachesDRAW_INDEX_AUTO, where the geometry check fails as in part 2. Without the read (the earlier traces), the== 0x2wait retires within about 0.2 ms while the label still reads 1, and the== 0x3wait never does.So a plain
GuestMemory::Readof the label between packets makes the GPU-written values visible, and the wait's own read (ReadSite::Wait) seems not to. The second wait retiring early may come from something other than the label's memory value.MarcZX10 commentedon Oct 13, 2026
This trace narrows it substantially. The injected
GuestMemory::ReadinvokesFlushForAccess, which canSyncThrougha pending recorder write;WAIT_REG_MEMitself usesWaitSatisfiedUncheckedand polls via volatile loads, so it bypasses that hook.waitMemoryalso has a same-queue recorder-label shortcut before polling.Could you try one run with the print removed and
APS5_NO_LABEL_SHORTCUT=1? That isolates the recorder-label-table early return from the raw memory polling path. If the second wait now stays until the label advances (like the injected-read run), the shortcut is implicated; if it still retires early while the CPU-visible value is 1, we can focus on the other early-return paths or the mapped-memory read. No source change is justified from this trace alone yet.MarcZX10 commentedon Oct 13, 2026
I checked the ordering in
waitMemoryand need to refine my previous diagnostic request:APS5_NO_LABEL_SHORTCUT=1only disables the recorder label-table lookup; it does not bypassstoredSince, which runs earlier and returns immediately when a recorded label write after this wait was received matches the comparison. In your trace, theRELEASE_MEMat offset 327 is immediately before theWAIT_REG_MEM == 2at 335, so this recorded-store shortcut is now a concrete candidate for the early retirement. Could you add a temporary log at thestoredSincereturn showing whether it matched for the second and third waits, plus the matched store stamp/writer queue and the wait’sreceivedstamp? If that branch returns for queue 0 while the actual label is still 1, we’ll have isolated the cause without changing queue behavior speculatively.MarcZX10 commentedon Oct 13, 2026
The per-packet trace makes
storedSincethe leading candidate. In current main, the label-write path records the GPU label in history as soon as it is queued, andstoredSinceaccepts any later matching stamp without checking whether that GPU write has completed or whether it came from the waiting queue. The matchingRELEASE_MEMat offset 327 is in the same submission, immediately before theWAIT_REG_MEMat 335, so this shortcut could let the worker pass the wait while the CPU-visible label is still 1.I have not changed the shortcut yet because it also intentionally lets waits on another queue be released by a producer's later label. The temporary log you described would confirm whether this exact return fires for the second wait; please include the matched store's queue/stamp and the wait's received stamp if you get a chance.
babyhuey commentedon Oct 13, 2026
The
storedSincelog, with a temporary print at its match return (reverted, not pushed) and no per-packet read, Linux, same stack,APS5_TRACE_GPU=1:All three waits retire through
storedSince, from same-queue stores queued after the wait's received stamp, while memory still reads 0 or 1. That includes the third one; I was wrong above to say it never retires. The(now ...)value is only printed when the wait starts.That corrects my opening summary as well. The queue doesn't stall on the third wait. It passes all three early, reaches
DRAW_INDEX_AUTOand throws on the geometry VGPR allocation (part 2). Onmainthat failure isn't printed, so it looked like a stalled wait. The 10 s timeout comes from part 2. Part 1 is a separate problem: same-queue waits retiring before the GPU write lands. That may matter for whatever reads the dispatch results, but it isn't what stops XPSemu.MarcZX10 commentedon Oct 13, 2026
Thanks for the
storedSincelogs; they confirm the shortcut returns for all three same-queue waits using stores queued after the wait received stamp, even though the label has not reached the requested value in CPU-visible memory. I also understand the corrected diagnosis: those waits retire, and the reported 10 s timeout is the subsequentDRAW_INDEX_AUTOgeometry VGPR allocation rejection. That separates a genuine premature same-queue wait completion from the current XPSemu stop. I’ll keep the queue-order issue separate and inspect whether the shortcut can safely distinguish a same-queue uncompleted store from a completed/cross-queue producer.MarcZX10 commentedon Oct 13, 2026
Correction to my previous comment: the logged store is the
RELEASE_MEMimmediately before the wait in the same queue submission. That ordering means thestoredSinceshortcut may be valid: later GPU work on that queue remains ordered behind the label write, even if CPU-visible memory is still stale when the wait is interpreted. The logs therefore confirm which shortcut fires but do not establish a queue-order bug by themselves. The concrete current failure remains the later geometry VGPR allocation rejection; I’ll avoid proposing a change tostoredSincewithout evidence of a CPU-side consumer or a store that is actually after the wait.babyhuey commentedon Oct 14, 2026
Part 2 is #5604. It drops the GS/ES input-VGPR count check, since the mesh translation always fills v0-v8. With it (and #5061, #5187, #4958, at the fixed base), XPSemu passes the GPU probe on Windows (RTX 3080) and runs its UI loop. XPSemu's bring-up thread is #5600.
MarcZX10 commentedon Oct 14, 2026
Thanks for the corrected XPSemu result and follow-up. #5604 now targets the geometry VGPR rejection that actually stopped the probe; its Windows RTX 3080 report says the probe completes and the UI starts. The
storedSincetrace remains a separate observation: because the matching label store is queued immediately before the same-queue wait, the early return alone does not show that a later consumer ran before the write. I would keep that ordering question separate unless a downstream read or output demonstrates a visible failure.odntht commentedon Oct 14, 2026
Note: these runs turned out to be on llvmpipe, not the HD 520 (Xvfb has no DRI3). See the correction below for the real HD 520 results.
A CPU-side consumer that fails visibly, in case it helps with the storedSince ordering question. It is also the HD 520 case that #4677 left unexplained (#4562).
Setup: DOOM Shareware homebrew (PPSA99666 v1.1.1, lowbit/ps5-doom), relinked with
--to-intel. Linux, Intel HD Graphics 520, Mesa 25.2.8 ANV,mainatc7601b7, Release, Xvfb.The title polls its presentation fence from the CPU with a 500 ms timeout (
wait_slotinsrc/ps5/gpu.c). When the timeout fires it logsgpu: frame 1 never completedand falls back to the CPU scaler.Reproduction:
MESA_SHADER_CACHE_DISABLE=true) the fallback happens on every run (6 of 6), with the AnyPS5 shader disk cache warm or cold.APS5_TRACE_GPU=1, plus a temporary print at thestoredSincematch return (reverted):Cold Mesa cache:
[gpu] 0.2 execute serial=1
[gpu] 808.5 queue 0x0 waits 0x202c22000 == 0x1 (now 0x0)
[tmp] storedSince match addr=0x202c22000 want=0x1 now=0x0 writer=0x0 waiter=0x0 received=1
[gpu] 808.7 done serial=1 / done serial=2
[gpu] 808.8 execute serial=3
[debug:0] gpu: frame 1 never completed
[gpu] 18412.4 queue 0x0 waits 0x202c62000 == 0x2 (now 0x0) <- only at teardown, after SIGINT
Warm Mesa cache:
[gpu] 773.3 queue 0x0 waits 0x202c22000 == 0x1 (now 0x0)
[tmp] storedSince match ... want=0x1 now=0x0 writer=0x0 waiter=0x0 received=1
[gpu] 776.6 execute serial=3
[gpu] 991.4 queue 0x0 waits 0x202c62000 == 0x2 (now 0x0) <- reached about 215 ms later
What this shows:
storedSincewith memory still at 0, in both runs. So the shortcut alone is not what separates them.WaitUntilSafeForRenderingfor display buffer 1, an ACQUIRE_MEM and the dispatch.No change with
APS5_LAND_WAITS_ALL_QUEUES=1,APS5_NO_LABEL_SHORTCUT=1,APS5_NO_WAIT_OVERLAP_SUBMIT=1,APS5_WAIT_LOCK=1, or with userfaultfd write watch available (seccomp unconfined).I haven't found where serial 3 stalls yet. I can capture
APS5_DUMP_QUEUE=0or run any other diagnostic you'd like on this machine.AI-assisted: yes
MarcZX10 commentedon Oct 14, 2026
I checked the current
WaitMemory.cpppath against these results.APS5_NO_LABEL_SHORTCUT=1disables the recorder label-table lookup, but bothstoredSince()early returns run before that setting is read: once on entry (line 141) and again inside the polling loop (line 333). So the run with that variable does not rule out thestoredSincepath shown in your temporary trace.A useful next diagnostic would be a local-only gate around both
storedSince()checks, then rerun the cold-cache case and inspect whether the CPU-visible label reaches 1 before the 500 ms poll expires. If the packet wait blocks until the actual label write or fails, that would distinguish the recorded-store shortcut from the later dispatch/queue stall. This is a diagnostic suggestion only; I don’t yet have evidence for a fix.MarcZX10 commentedon Oct 14, 2026
Thanks for the full trace. It shows the serial-1 wait taking the same
storedSincereturn withnow=0in both warm and cold runs, so that shortcut does not explain the difference. In the cold run, serial 3 has not reached itsWAIT_REG_MEMbefore the title’s 500 ms presentation-fence timeout; the remaining stall is before that wait, aroundWaitUntilSafeForRendering,ACQUIRE_MEM, or the dispatch/pipeline work. A timestamped queue-0 excerpt for serial 3 fromexecutethrough those packet boundaries, plus when the title starts and expires its 500 ms poll, would narrow it. This trace does not support changingstoredSince.odntht commentedon Oct 14, 2026
Correction to my comment above: those runs were not on the HD 520. They ran under Xvfb, which has no DRI3, so ANV could not present (
vulkan: No DRI3 support detected - required for presentation) and device selection fell through to llvmpipe (Selected GPU name=llvmpipe (LLVM 20.1.2, 256 bits)). Every trace in that comment is lavapipe on Mesa 25.2.8, so it does not explain the HD 520 case in #4562. Sorry for the noise.Rerun on the HD 520 itself (Weston headless with the GL renderer plus Xwayland, which provides DRI3). Intel HD Graphics 520 (0x1916), Mesa 25.2.8 ANV, kernel 6.12 (i915,
preempt_timeout_ms640),mainc7601b7, DOOM Shareware homebrew PPSA99666 v1.1.1 relinked with--to-intel:fifo, warm or cold Mesa cachegpu: frame 1 never completed), 2 of them with an i915 resetANYPS5_PRESENT_MODE=mailboxVK_ERROR_DEVICE_LOSTANYPS5_PRESENT_MODE=immediateVK_ERROR_DEVICE_LOSTAPS5_SYNC_FLIP=1VK_ERROR_DEVICE_LOSTwith an i915 resetVK_ERROR_DEVICE_LOSTwith an i915 resetmain+ 934348f (#2972 fix)The kernel log for the resets is
Resetting rcs0 for preemption time out/GPU HANG: ecode 9:1:8ed0fff3, in eboot.elf.What the trace shows on ANV (
APS5_TRACE_GPU=1): serial 1 is done at about 52 ms, then serial 3 executes at 52.2 ms and only reaches its WAIT_REG_MEM at 743.3 ms. That is the title's presentation kernel (0x202c81800, a palette to RGBA conversion over 1920x1080) taking roughly 690 ms per dispatch. It is longer than the title's 500 ms fence timeout and longer than i915's 640 ms preemption timeout.With 934348f applied,
APS5_PROFILE_GPU=1reportsby program: 0x202c81800 x21 9916ms, about 472 ms per dispatch, below the preemption timeout, which matches the resets going away. The title then stays on GPU presentation at about 2 FPS, against 16.7 ms per frame on its CPU scaler.So on the HD 520 the fallback comes from a slow dispatch, not from the storedSince ordering discussed here. The label question in my first comment stands only for lavapipe. I can open a separate perf issue for the dispatch cost if that is the better place.
AI-assisted: yes
odntht commentedon Oct 14, 2026
Per-packet timing for serial 3 on the HD 520, as asked. Same setup as my correction above (HD 520, Mesa 25.2.8 ANV,
mainc7601b7, DOOM PPSA99666 relinked with--to-intel). A temporary print at the start of every packet inPacketExecution.cpp(queue 0, serials 1 to 6, reverted, libraries rebuilt clean),APS5_TRACE_GPU=1, 2 runs with a warm and 2 with a cold Mesa cache:DISPATCH_DIRECTTimes are from the start of
DISPATCH_DIRECTto the start of the next packet. Every other packet in serial 3 takes under 0.2 ms:WAIT_FLIP_DONE,ACQUIRE_MEM, the sixSET_SH_REG,RELEASE_MEMandWAIT_REG_MEM. Serial 1'sDISPATCH_DIRECTtakes 29 to 38 ms.Excerpt (warm 2):
The title logs
frame 1 never completedwhile serial 3 is still inside that dispatch, and only in the runs where the dispatch goes past about 600 ms. The Mesa cache makes no difference. With 934348f applied,APS5_PROFILE_GPU=1measured about 472 ms of GPU time per dispatch of this program (0x202c81800), so the worker seems to be waiting on GPU execution of the presentation kernel there, not on the flip or the acquire.So on ANV the remaining stall is the cost of that one dispatch. I can collect anything else around it on this machine.
AI-assisted: yes
MarcZX10 commentedon Oct 18, 2026
The geometry-allocation rejection from this report is fixed on current main by
f535918,fix(agc): accept geometry programs with fewer GS and ES input VGPRs. Its regression sets the reportedSPI_SHADER_PGM_RSRC1_GS=0x222c0007andSPI_SHADER_PGM_RSRC2_GS=0x00c00010values and confirmsDecodeStateaccepts the geometry path. Could you retry the original XPSemu probe on current main and report whether the draw now reaches execution, and whether the earlier repeatedRELEASE_MEM/WAIT_REG_MEMbehavior still occurs? The wait symptom was analyzed separately and was not established as the same failure.babyhuey commentedon Oct 18, 2026
@MarcZX10 retried on current
maineb1a15b5c, part of an answer. Linux, Ubuntu 24.04 container, GCC 13, Release, relinker and every library rebuilt; XPSemu (PPSA97358) relinked with--linux-fixed-base 0x400000, the GPU being lavapipe (LLVM 20.1) in this container,APS5_TRACE_GPU=1:done serial=1,done serial=2at 323 ms). Serial 3 (861 dwords) runs past the geometry-program draw: its graphics shaders0x206120d00and0x206121600are prepared and drawn, with noDecodeStaterejection.0x206121900at its dispatch inside serial 3:llvm::report_fatal_errorfromllvm::TargetInstrInfo::lowerCopyinlibLLVM.so.20.1, underlibvulkan_lvp.so's pipeline compile. That's the host driver's own LLVM failure (the "Cannot emit physreg copy instruction" seen earlier), not AnyPS5.So on this machine serial 3 never reaches its
RELEASE_MEM/WAIT_REG_MEMpackets, and I can't say yet whether the repeated wait behaviour remains. That needs a real GPU; I'll add a Windows run when that machine is free.AI-assisted: yes
MarcZX10 commentedon Oct 18, 2026
Thanks for rebuilding the current tree and confirming the geometry draw gets past
DecodeState. The LLVMlowerCopyabort happens before the reported label waits, so this run neither confirms nor rules out the original wait behavior. A Windows hardware run reaching those packets would be the useful next data point; I do not see a code change supported by this lavapipe trace.babyhuey commentedon Oct 18, 2026
@MarcZX10 the rest of the answer, on a real GPU. Windows 11, RTX 3080, WinLibs GCC 15.2, Release,
maineb1a15b5c, XPSemu relinked with--windows --windows-fixed-base 0x400000, 60 s,APS5_TRACE_GPU=1:execute serial=3 dwords=861at 445 ms,done serial=3at 4031 ms; most of the gap is first-use preparation of its two compute and two graphics shaders), and serial 4 completes with it.WAIT_REG_MEMlines at all. The whole run has 8 waits, all later in the title's UI loop on0x2010c0000, and each is satisfied at once (for example5280.1 waits == 0xa, then5280.2 done serial=39). None repeat or stay unretired.qemu_init done, running the machine,display readyand its UI loop, and runs the full 60 s at about 100 FPS (5,272 frames).0x412c400000) is zero. That looks like a separate question, about whether that draw's output reaches the image the probe reads, rather than the wait behaviour this issue started with.AI-assisted: yes