Skip to content

feat(libSceAgcDriver): ask whether to keep a stalled title running - #13707

Open
MarcZX10 wants to merge 17 commits into
boykopovar:mainfrom
MarcZX10:feat/stall-watchdog-prompt
Open

MarcZX10 wants to merge 17 commits into
boykopovar:mainfrom
MarcZX10:feat/stall-watchdog-prompt

Conversation

@MarcZX10

@MarcZX10 MarcZX10 commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

What

Today the stall watchdog (APS5_STALL_WATCHDOG_SECS) never quits a title. When a watched wait has gone the set number of seconds without progress, it dumps the submission rings once for that wait and the title keeps running. It only dumps once; it never asks, and it never ends the title.

This PR adds an opt-in host prompt for that moment, ANYPS5_STALL_PROMPT=1, which needs APS5_STALL_WATCHDOG_SECS to be set:

  • The first time a watched wait expires, the rings are dumped and the host shows a message box with Wait and Quit.
  • Wait keeps the title running. The box comes back one period later, at most three times for the same wait. The rings are still dumped once per wait.
  • Quit ends the title with exit code 124, after the rings are printed to stderr.
  • Only one box is open at a time. A wait that finds it open is asked once the open box has been answered.
  • If no message box can be shown, the rings are printed and the title keeps running.
  • Unset or 0 (the default) shows no box, so the current behaviour does not change. Any other value stops the title at its first submission with an error that names ANYPS5_STALL_PROMPT.

The box is requested without Driver::mutex held. The waiting thread releases the driver lock before it calls the presenter and takes it back before it returns Wait or Unavailable, so a submission or a video output registration can complete while the box is open. On macOS the presenter runs on the main thread through Aps5RunOnMainThread_nid_no_patch, the main-queue entry point the video output already uses, whenever the main queue is being served, because SDL documents the main thread for a message box without a parent window. Linux and Windows have no main-thread queue, so the box opens on the thread whose wait expired there; the technical debt file records that as unmeasured.

Commits, in order:

  1. refactor: the stall clock is read through a settable seam, with steady_clock as the default. Test stall-clock pins one dump per wait on an injected clock.
  2. feat: the setting, the SDL message box presenter, Quit, the unavailable fallback and the ENVIRONMENT.md row. Tests for the invalid value, off, one prompt, Quit and the exit code.
  3. feat and test: re-arm after Wait and the three-prompt limit, with a real-path test on the real clock.
  4. feat and test: one box at a time, with a re-entry test.
  5. docs: the USAGE.md section, and three technical debt entries: the macOS message box thread, the window system behaviour and the lack of a measured console behaviour.
  6. Merge of upstream/main at bff0289 (d90188fc2a). None of the files this PR changes were changed on main between the branch's base and bff0289, so the merge was clean.
  7. fix: the stall prompt releases the driver lock while the box is open (84b4242). Test stall-prompt-lock (804e75c) holds the box over a real queue-order wait while a submission and a video output registration take the driver lock, then checks Wait and Quit. Test stall-prompt-lock-unavailable checks the unavailable answer the same way. The follow-up cd2ee7f makes the probe submission carry a WAIT_REG_MEM that already holds, so the probe is not held behind queue 0 and does not print a second dump.
  8. fix: on macOS the box is requested through Aps5RunOnMainThread_nid_no_patch (bca46b2), and the debt entry for the message box says which platforms run it on which thread.
  9. test: ANYPS5_STALL_PROMPT=0 is pinned as no box, test stall-prompt-zero (b3c5898).
  10. Merge of current upstream/main at 828ac0b (455321a2e4), a plain merge with no conflicts. Two files changed on both sides, core/libs/prx/libSceAgcDriver/CMakeLists.txt and docs/user/USAGE.md; both sides' edits are present in the merged files. This PR is now based on current main.

Overlaps: re-checked on 828ac0b2ae with gh pr list --state open --search for ANYPS5_STALL_PROMPT, StallWatch, SubmissionRing and StallPrompt: only this PR matches. tools/pr_overlap.py --branch could not confirm 93 open pull requests (merge simulation) and so failed closed. Checked by hand: the file list of each PR it reported, and a search of open PR titles and bodies. No open PR touches SubmissionRing.* or StallPrompt.cpp (no hit in the file probe). Searches of open PR titles and bodies for ANYPS5_STALL_PROMPT and StallWatch found nothing. Not checked line by line: the 93 unconfirmed PRs. Textual overlaps only, with no shared code: #12863 and #12895 (ENVIRONMENT.md, USAGE.md), #13701 (ENVIRONMENT.md, USAGE.md, PSN dialog debt docs), #12500 (ENVIRONMENT.md), #12764 (one debt line), #13474 and #6549 (tests/Pm4.cpp, debt), and #6185 and #6186 (CMakeLists.txt, tests/Pm4.cpp, debt). #12895 also opens an SDL message box in its own file (HostUrlConfirmation.cpp); this PR shares no code with it.

Tested

Environment: Debian 12, Linux 6.1.0-54-amd64, GCC 12.2.0, ninja 1.11.1, Release, BUILD_TESTING=ON. The scratch build copy relaxes the host CPU feature gate in core/libs/prx/libkernel/System/src/HostCpu.cpp, because this host lacks CPU extensions that the stock gate requires. The branch does not change that file.

Built: agc_driver_pm4_tests at the merged head 455321a2e4, incremental in the scratch build tree after refreshing its sources from git archive of the merged head (463 steps, NINJA-EXIT 0). The replay targets (agc_driver_pm4_replay_tests, agc_pm4_replay) were not rebuilt for this head. They link libSceAgcDriver, which changed, so their rows below are the earlier run at d90188fc2a, not a run on this head. Ran: ctest -R agc_driver_pm4 -E replay --timeout 120 at 455321a2e4: 24 of 24 passed. The upstream column is the earlier run at bff0289787, the PR's previous base. upstream/main has since moved to 828ac0b2ae; it was not rerun, because that needs another build tree.

Test upstream/main bff0289 (earlier) this branch d90188f (earlier) merged head 455321a
agc_driver_pm4 Passed Passed Passed
agc_driver_pm4_failure Passed Passed Passed
agc_driver_pm4_ring Passed Passed Passed
agc_driver_pm4_ring_empty Passed Passed Passed
agc_driver_pm4_ring_disabled Passed Passed Passed
agc_driver_pm4_ring_invalid Passed Passed Passed
agc_driver_pm4_ring_report Passed Passed Passed
agc_driver_pm4_stall_watchdog Passed Passed Passed
agc_driver_pm4_stall_watchdog_invalid Passed Passed Passed
agc_driver_pm4_stall_watchdog_rewind Passed Passed Passed
agc_driver_pm4_stall_watchdog_order Passed Passed Passed
agc_driver_pm4_rewind_trace Passed Passed Passed
agc_driver_pm4_failure_inline Passed Passed Passed
agc_driver_pm4_priority_refused Passed Passed Passed
agc_driver_pm4_replay Passed Passed not rebuilt on this head
agc_driver_pm4_replay_tool Passed Passed not rebuilt on this head
agc_driver_pm4_stall_clock not present Passed Passed
agc_driver_pm4_stall_prompt_invalid not present Passed Passed
agc_driver_pm4_stall_prompt_off not present Passed Passed
agc_driver_pm4_stall_prompt_zero not present not present Passed
agc_driver_pm4_stall_prompt_rearm not present Passed Passed
agc_driver_pm4_stall_prompt_real not present Passed (5.5 s) Passed (5.5 s)
agc_driver_pm4_stall_prompt_reentry not present Passed Passed
agc_driver_pm4_stall_prompt_quit not present Passed Passed
agc_driver_pm4_stall_prompt_lock not present not present Passed (2.0 s)
agc_driver_pm4_stall_prompt_lock_unavailable not present not present Passed (3.5 s)
Total 16 of 16 23 of 23 24 of 24 run; 2 replay not rebuilt

The 16 existing tests passed on bff0289787 and on d90188fc2a; on the merged head 14 of them were rerun and passed, and the two replay tests were not rebuilt. So the defaults are unchanged in the tested paths. With ANYPS5_STALL_PROMPT unset, the watchdog takes the same path as before: one dump per wait, no box, no exit.

Lock control, on the merged head (455321a2e4): in a scratch copy of the build tree, the two order-wait calls in QueueWorker.cpp were changed to pass no held lock, so the presenter ran with Driver::mutex held. agc_driver_pm4_tests stall-prompt-lock then failed with a submission waited for the driver lock while the stall prompt was open and exit code 1. The clone's QueueWorker.cpp was restored and rebuilt, and stall_prompt_lock, stall_prompt_lock_unavailable, stall_prompt_quit, stall_prompt_off, stall_prompt_zero and stall_prompt_rearm pass again on the restored build.

Negative controls: each was a mutation in a scratch copy of the build tree, one per assertion. Every mutation failed the intended test and was then reverted. The controls ran on the code of their commits, not all on the head: the setting rows and the Quit rows on the test commit for those tests; the re-arm rows (R1 to R7) and the injected-clock rows (C1, C2) on the re-arm commit, before the try-lock commit; R4 and R5 on the real-path test before its timestamp assertions were added, and R1 and R2 after; the try-lock rows on the try-lock commit. Re-run on the head: the unavailable-as-Quit row and the never-prompt row. The lines each mutation changes are the same on the head, except that the try-lock commit added the open-prompt check before the presenter call.

Mutation Test that failed Assertion that failed
presenter runs with the driver lock held (QueueWorker.cpp passes no held lock) stall_prompt_lock a submission waited for the driver lock while the stall prompt was open
setting check removed (any value accepted) stall_prompt_invalid expected PM4 rejection
box shown when the variable is unset stall_prompt_off a wait asked the host with ANYPS5_STALL_PROMPT unset
no dump when the box is off stall_prompt_off a wait with the prompt unset did not dump the rings once
box never shown stall_prompt_rearm an expired wait did not ask the host
no re-arm after Wait stall_prompt_rearm and stall_prompt_real asked again before a period had passed
limit 2 stall_prompt_rearm a wait did not ask the host a third time
limit 4 stall_prompt_rearm and stall_prompt_real more than three times
dump on every prompt stall_prompt_rearm the rings were not dumped once for the answered wait
one box only stall_prompt_rearm and stall_prompt_real rearm: a wait that expired again after Wait did not ask the host again; real: did not ask the host three times
prompt text changed stall_prompt_once, since renamed stall_prompt_rearm (same assertion text) the prompt does not say how long
unavailable box asked again stall_prompt_rearm a wait whose message box was unavailable asked again
extra dump when unavailable stall_prompt_rearm the rings were not dumped once for the wait without a message box
unavailable treated as Quit stall_prompt_rearm the test ended on the Quit path before its checks ran
Quit with code 125 stall_prompt_quit exited with 125 instead of 124
Quit returns instead of ending the title stall_prompt_quit the Quit answer returned instead of ending the title
Quit exits with 0 stall_prompt_quit exited with 0 instead of 124
clock not injected stall_clock a wait that expired on the injected clock did not dump the rings once
extra dump on the injected clock stall_clock a wait dumped the rings more than once on the injected clock
no try-lock on the box stall_prompt_reentry a second wait asked the host while a prompt was open
box slot never released stall_prompt_reentry a wait held back by an open prompt did not ask the host after it was answered

Not controlled: the Unavailable answer's relock on the queue-order path (covered by stall_prompt_lock_unavailable, not mutated), and the __APPLE__ branch of askOnMainThread, which is compiled only on macOS. The target == 0 check in expectSettingRejected is shared with tests that already existed. The [stall prompt] no message box line is not asserted by any test; the tests assert the dump and prompt counts instead.

Not run: Windows and macOS builds and tests. On macOS the Aps5RunOnMainThread_nid_no_patch branch of askOnMainThread was not compiled, so the main-queue hop is not verified here. The replay targets were not rebuilt for 455321a2e4. The GPU comparison that AGENTS.md asks for AGC changes was not run; these tests run on the CPU side with an injected presenter. The hosted checks passed on d90188fc2a (their scope was not inspected); they were not inspected for 455321a2e4. #13701 and the 93 PRs that pr_overlap could not confirm were not compared line by line. No test opens a real message box or a window: every test injects the presenter. No console measurement exists for this behaviour (see the debt entry). Other test binaries of libSceAgcDriver and the rest of AnyPS5 were not built or run. No GPU was used; the watchdog tests run on the CPU side.

Checks at the merged head 455321a2e4: tools/check_conventions.py --base upstream/main passes, and so does git diff --check against upstream/main. tools/env_reference.py --check fails only on APS5_PAD_SPEAKER_FALLBACK_DB, the same single line as on upstream/main. tools/pr_abi_audit.py --base 828ac0b2ae --head 455321a2e4 reports not applicable (no export changed). tools/pr_test_quality.py flags tests/Pm4.cpp as an unregistered test, which is a false positive: the file is registered as agc_driver_pm4_tests in core/libs/prx/libSceAgcDriver/CMakeLists.txt. tools/pr_precheck.py --base 828ac0b2ae --head 455321a2e4, run from a copy inside the clone, passes the diff check and the conventions and PR-template item. Its ABI and test-quality items fail only because those two tools are not in the candidate tree; they were run by hand as above.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none
  • GPUs tested: n/a
  • AI-assisted: yes

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 1368 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 1395 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 1896 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 2434 docs/dev/debt/libSceAgcDriver.md
PR 4448 docs/dev/debt/libSceAgcDriver.md
PR 4472 docs/dev/ENVIRONMENT.md
PR 4847 docs/user/USAGE.md
PR 5021 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/user/USAGE.md
PR 5371 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 5372 docs/dev/debt/libSceAgcDriver.md
PR 5925 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
docs/user/USAGE.md
PR 6146 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 6185 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Pm4.cpp
docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 6186 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/Pm4.cpp
docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 6187 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6188 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6192 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6194 docs/dev/debt/libSceAgcDriver.md
PR 6209 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6223 docs/dev/debt/libSceAgcDriver.md
PR 6273 docs/dev/debt/libSceAgcDriver.md
PR 6280 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 6412 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 6481 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 6501 docs/dev/ENVIRONMENT.md
PR 6548 docs/dev/debt/libSceAgcDriver.md
PR 6549 core/libs/prx/libSceAgcDriver/tests/Pm4.cpp
docs/dev/debt/libSceAgcDriver.md
PR 6743 docs/dev/debt/libSceAgcDriver.md
PR 6804 docs/dev/debt/libSceAgcDriver.md
PR 6808 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 6867 docs/dev/debt/libSceAgcDriver.md
PR 6905 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6924 docs/dev/ENVIRONMENT.md
docs/user/USAGE.md
PR 6926 docs/dev/debt/libSceAgcDriver.md
PR 6973 docs/dev/debt/libSceAgcDriver.md
PR 6987 docs/dev/debt/libSceAgcDriver.md
PR 6989 docs/dev/debt/libSceAgcDriver.md
PR 6995 docs/dev/debt/libSceAgcDriver.md
PR 7002 docs/dev/debt/libSceAgcDriver.md
PR 7085 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/cmake/agc_driver_graphics_tests.cmake
docs/dev/debt/libSceAgcDriver.md
PR 7223 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7256 docs/dev/debt/libSceAgcDriver.md
PR 7275 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7449 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 7457 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 7508 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7514 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 7604 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 7642 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/cmake/agc_driver_graphics_tests.cmake
docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
docs/user/USAGE.md
PR 7643 docs/dev/ENVIRONMENT.md
PR 7668 docs/dev/debt/libSceAgcDriver.md
PR 7978 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Queues/QueueWorker.cpp
docs/dev/debt/libSceAgcDriver.md
PR 8004 docs/dev/ENVIRONMENT.md
PR 8484 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 8592 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 8707 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 8963 docs/dev/ENVIRONMENT.md
PR 9012 docs/dev/ENVIRONMENT.md
PR 9020 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 9154 docs/dev/ENVIRONMENT.md
PR 9248 docs/dev/ENVIRONMENT.md
PR 9289 docs/dev/ENVIRONMENT.md
PR 9335 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 9345 docs/dev/ENVIRONMENT.md
docs/user/USAGE.md
PR 9356 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 9387 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 9508 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/cmake/agc_driver_graphics_tests.cmake
docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
docs/user/USAGE.md
PR 9592 docs/dev/debt/libSceAgcDriver.md
PR 9671 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9672 docs/dev/debt/libSceAgcDriver.md
PR 9673 docs/dev/debt/libSceAgcDriver.md
PR 9674 docs/dev/debt/libSceAgcDriver.md
PR 9675 docs/dev/debt/libSceAgcDriver.md
PR 9676 docs/dev/ENVIRONMENT.md
PR 9700 docs/dev/debt/libSceAgcDriver.md
PR 9712 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 9742 docs/dev/debt/libSceAgcDriver.md
PR 9748 docs/dev/debt/libSceAgcDriver.md
PR 9852 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 9875 docs/dev/ENVIRONMENT.md
PR 9884 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9891 docs/dev/debt/libSceAgcDriver.md
PR 9892 docs/dev/debt/libSceAgcDriver.md
PR 9893 core/libs/prx/libSceAgcDriver/tests/Pm4.cpp
docs/dev/debt/libSceAgcDriver.md
PR 9895 docs/dev/debt/libSceAgcDriver.md
PR 9897 core/libs/prx/libSceAgcDriver/tests/cmake/agc_driver_graphics_tests.cmake
PR 9898 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 9915 docs/dev/debt/libSceAgcDriver.md
PR 9950 docs/dev/debt/libSceAgcDriver.md
PR 10043 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
docs/user/USAGE.md
PR 10049 docs/dev/debt/libSceAgcDriver.md
PR 10064 docs/dev/debt/libSceAgcDriver.md
PR 10069 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 10141 core/libs/prx/libSceAgcDriver/tests/cmake/agc_driver_graphics_tests.cmake
PR 10193 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/cmake/agc_driver_graphics_tests.cmake
docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
docs/user/USAGE.md
PR 10233 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 10256 docs/dev/debt/libSceAgcDriver.md
PR 10351 docs/dev/debt/libSceAgcDriver.md
PR 10359 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 10442 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 10443 docs/dev/ENVIRONMENT.md
PR 10469 docs/dev/debt/libSceAgcDriver.md
PR 10472 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 10479 docs/user/USAGE.md
PR 10492 docs/dev/debt/libSceAgcDriver.md
PR 10497 docs/dev/debt/libSceAgcDriver.md
PR 10668 docs/dev/ENVIRONMENT.md
PR 10678 docs/dev/ENVIRONMENT.md
PR 10743 docs/dev/debt/libSceAgcDriver.md
PR 10757 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 10779 docs/dev/debt/libSceAgcDriver.md
PR 10792 docs/user/USAGE.md
PR 10861 docs/user/USAGE.md
PR 10867 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 10894 docs/dev/ENVIRONMENT.md
PR 10920 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 11104 docs/dev/debt/libSceAgcDriver.md
PR 11346 docs/dev/debt/libSceAgcDriver.md
PR 11351 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 11435 docs/dev/ENVIRONMENT.md
PR 11436 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Queues/QueueWorker.cpp
docs/dev/ENVIRONMENT.md
PR 11551 docs/dev/debt/libSceAgcDriver.md
PR 11557 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 11566 core/libs/prx/libSceAgcDriver/tests/cmake/agc_driver_graphics_tests.cmake
PR 11587 docs/dev/debt/libSceAgcDriver.md
PR 11825 docs/dev/debt/libSceAgcDriver.md
PR 11948 docs/dev/ENVIRONMENT.md
PR 11995 docs/dev/debt/libSceAgcDriver.md
PR 11998 docs/dev/debt/libSceAgcDriver.md
PR 12066 docs/dev/ENVIRONMENT.md
PR 12120 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 12132 docs/dev/ENVIRONMENT.md
PR 12323 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 12324 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 12452 docs/dev/debt/libSceAgcDriver.md
PR 12478 docs/dev/ENVIRONMENT.md
PR 12544 docs/dev/ENVIRONMENT.md
PR 12565 docs/dev/debt/libSceAgcDriver.md
PR 12567 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Queues/QueueWorker.cpp
docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 12568 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Queues/QueueWorker.cpp
docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 12662 docs/dev/ENVIRONMENT.md
PR 12678 docs/user/USAGE.md
PR 12696 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Queues/QueueWorker.cpp
docs/dev/ENVIRONMENT.md
PR 12720 docs/dev/ENVIRONMENT.md
PR 12745 docs/dev/ENVIRONMENT.md
PR 12746 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
docs/user/USAGE.md
PR 12764 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 12792 docs/dev/debt/libSceAgcDriver.md
PR 12793 docs/user/USAGE.md
PR 12828 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 12840 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 12849 docs/dev/ENVIRONMENT.md
PR 12855 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 12904 docs/dev/debt/libSceAgcDriver.md
PR 12911 docs/user/USAGE.md
PR 12921 docs/dev/ENVIRONMENT.md
PR 12980 docs/dev/ENVIRONMENT.md
PR 12994 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 13021 docs/dev/debt/libSceAgcDriver.md
PR 13023 docs/dev/debt/libSceAgcDriver.md
PR 13044 docs/dev/ENVIRONMENT.md
PR 13131 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 13146 docs/dev/debt/libSceAgcDriver.md
PR 13147 docs/dev/debt/libSceAgcDriver.md
PR 13148 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 13151 docs/dev/debt/libSceAgcDriver.md
PR 13159 docs/dev/ENVIRONMENT.md
PR 13169 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 13248 docs/dev/debt/libSceAgcDriver.md
PR 13262 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 13263 docs/dev/ENVIRONMENT.md
PR 13474 core/libs/prx/libSceAgcDriver/tests/Pm4.cpp
docs/dev/debt/libSceAgcDriver.md
PR 13646 docs/dev/ENVIRONMENT.md
PR 13793 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 13809 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 13858 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 13863 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 13920 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 13928 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/cmake/agc_driver_graphics_tests.cmake
docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
docs/user/USAGE.md
PR 13929 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/tests/cmake/agc_driver_graphics_tests.cmake
docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
docs/user/USAGE.md
PR 13963 docs/dev/debt/libSceAgcDriver.md
PR 13991 docs/dev/debt/libSceAgcDriver.md
PR 13994 docs/dev/debt/libSceAgcDriver.md
PR 14005 docs/dev/debt/libSceAgcDriver.md
PR 14018 docs/dev/debt/libSceAgcDriver.md
PR 14032 core/libs/prx/libSceAgcDriver/tests/cmake/agc_driver_graphics_tests.cmake
PR 14051 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 14070 docs/user/USAGE.md
PR 14085 docs/dev/debt/libSceAgcDriver.md
PR 14110 core/libs/prx/libSceAgcDriver/tests/Pm4.cpp
PR 14130 core/libs/prx/libSceAgcDriver/tests/Pm4.cpp
PR 14139 docs/user/USAGE.md
PR 14212 docs/dev/debt/libSceAgcDriver.md
PR 14220 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 14235 docs/dev/ENVIRONMENT.md
docs/user/USAGE.md
PR 14243 docs/dev/debt/libSceAgcDriver.md
PR 14248 docs/dev/debt/libSceAgcDriver.md
PR 14260 docs/dev/ENVIRONMENT.md
PR 14262 docs/dev/ENVIRONMENT.md
PR 14264 docs/dev/ENVIRONMENT.md
docs/user/USAGE.md
PR 14265 docs/dev/debt/libSceAgcDriver.md
PR 14296 docs/dev/debt/libSceAgcDriver.md
PR 14298 docs/user/USAGE.md
PR 14299 docs/dev/ENVIRONMENT.md
PR 14302 docs/dev/debt/libSceAgcDriver.md
PR 14311 docs/dev/debt/libSceAgcDriver.md
PR 14344 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 14367 docs/dev/ENVIRONMENT.md
PR 14384 docs/user/USAGE.md
PR 14388 docs/user/USAGE.md
PR 14561 docs/dev/debt/libSceAgcDriver.md
PR 14621 docs/dev/debt/libSceAgcDriver.md
PR 14628 docs/dev/debt/libSceAgcDriver.md
PR 14794 docs/dev/ENVIRONMENT.md
PR 14815 docs/dev/debt/libSceAgcDriver.md
PR 14838 docs/dev/ENVIRONMENT.md
PR 14904 docs/dev/ENVIRONMENT.md
PR 14909 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/debt/libSceAgcDriver.md
PR 14943 docs/dev/ENVIRONMENT.md
docs/dev/debt/libSceAgcDriver.md
PR 14944 core/libs/prx/libSceAgcDriver/tests/cmake/agc_driver_graphics_tests.cmake
docs/dev/debt/libSceAgcDriver.md
PR 14950 docs/dev/debt/libSceAgcDriver.md
PR 14963 docs/dev/debt/libSceAgcDriver.md
PR 14964 docs/dev/debt/libSceAgcDriver.md
PR 14976 docs/dev/debt/libSceAgcDriver.md
PR 14982 docs/dev/debt/libSceAgcDriver.md
PR 14987 docs/user/USAGE.md
PR 15004 docs/dev/ENVIRONMENT.md
docs/user/USAGE.md
PR 15015 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 15032 docs/user/USAGE.md
PR 15058 docs/dev/debt/libSceAgcDriver.md
PR 15059 docs/dev/ENVIRONMENT.md
PR 15064 docs/dev/ENVIRONMENT.md

@oneandonlydean

Copy link
Copy Markdown
Collaborator

@MarcZX10 Reviewed at d90188fc2a. Please move the message-box request out of the stalled worker's synchronous call path and onto the host main thread, with an answer passed back without holding submission or presentation locks. The queue-order paths in QueueWorker.cpp call stall.Check while their std::unique_lock lock(mutex) remains held; poll returns with that lock reacquired. The new stallPresenter.load()(message) then calls modal SDL_ShowMessageBox before releasing it, preventing other users of Driver::mutex from progressing until someone answers. A re-entrant presenter test does not exercise that lock-held worker path.

SDL2 also documents that a message box without a parent should run on the main thread and blocks its calling thread: https://wiki.libsdl.org/SDL2/SDL_ShowMessageBox. The current worker-thread call violates that rule, rather than merely lacking a macOS measurement. Please add a regression through an actual queue-order wait that keeps a prompt pending while another submission or presentation operation can acquire the driver lock, then verifies Wait and Quit handling. Keep the unset/zero and unavailable-display paths covered. Existing injected-presenter tests never open the real SDL box.

@MarcZX10

Copy link
Copy Markdown
Contributor Author

Re-checked — new head 455321a, lock regression passes and fails without the fix

The stall prompt now runs without Driver::mutex held, the macOS presenter runs through the main queue, and a regression holds a real queue-order prompt while the driver lock is taken. The branch is also merged with current upstream/main (828ac0b2ae) as a plain merge, with no conflicts.

Review item 1: move the message box to the host main thread, answer back without submission or presentation locks.

  • The stalled wait releases the driver lock before it calls the presenter, and takes it back only before it returns Wait or Unavailable (84b424288a). Quit ends the title without retaking it.
  • On macOS the presenter runs through Aps5RunOnMainThread_nid_no_patch, the main-queue entry point the video output already uses (bca46b24fd). Linux and Windows have no main-thread queue, so the box still opens on the thread whose wait expired there; the debt entry says so. The macOS branch was not compiled in this environment.

Review item 2: a regression through a real queue-order wait, prompt pending, lock taken, then Wait and Quit.

  • stall-prompt-lock (804e75cff5, probe fix cd2ee7ff55) holds queue 0 at its rendering wait and queue 0x20 behind it. While the prompt is pending, a submission on queue 0x21 and a video output registration must both complete. Wait keeps one rings dump and asks again one period later; Quit exits with 124.
  • Negative control on the merged head: with the lock held during the presenter call, the test fails with a submission waited for the driver lock while the stall prompt was open and exit 1. Restored, it passes.

Review item 3: unset/zero and unavailable-display paths.

  • Unset: stall-prompt-off. Zero: stall-prompt-zero (b3c589802c). Unavailable: stall-prompt-rearm, and stall-prompt-lock-unavailable on the lock path.

Ran at 455321a2e4: ctest -R agc_driver_pm4 -E replay --timeout 120, 24 of 24 passed. Not run on this head: agc_driver_pm4_replay_tests and agc_pm4_replay (they link the changed library), Windows and macOS builds, the GPU comparison from AGENTS.md, and a rerun of the upstream baseline on 828ac0b2ae.

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ NVIDIA GPU test: passed

PR head 455321a merged onto main 828ac0b · tested 2026-10-31 08:30 UTC · 20 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,425 passed · 4 failed (0 new, 2 also fail on main, 2 unstable on main) · 42 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,471 of 3,048 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (921 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_bda_device 44.2 s, ⚠️ agc_driver_graphics 34.2 s, ⚠️ agc_driver_graphics_maps_fallback 30.4 s
Unstable on main (2, these also failed in recent main runs on this machine; not counted against this PR)
agc_submit_progress: failed in 6 of the last 10 main runs
agc_submit_progress_no_timeline: failed in 8 of the last 10 main runs
Failing on main too (2, not counted against this PR)
guest_mmap_executable
guest_socket_options_multicast
Validated but not compared (10, not in the main baseline's validation pass)
agc_driver_pm4_stall_clock
agc_driver_pm4_stall_prompt_invalid
agc_driver_pm4_stall_prompt_lock
agc_driver_pm4_stall_prompt_lock_unavailable
agc_driver_pm4_stall_prompt_off
agc_driver_pm4_stall_prompt_quit
agc_driver_pm4_stall_prompt_real
agc_driver_pm4_stall_prompt_rearm
agc_driver_pm4_stall_prompt_reentry
agc_driver_pm4_stall_prompt_zero
Validation IDs already present on main (4 IDs in 425 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (425 tests): agc_compute_pipeline_replay, agc_driver, agc_driver_address_passes, agc_driver_address_passes_draw_transitions, agc_driver_address_passes_targets_split, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, and 417 more
VUID-vkCmdCopyImage-srcImage-01548 (13 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_plane_sampling, and 5 more
VUID-vkCmdCopyImage-srcImage-01551 (13 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_plane_sampling, and 5 more
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_srgb8_color_target
Skipped (42)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_msaa_dcc_compared_import_2: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_msaa_dcc_compared_import_4: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_msaa_dcc_compared_import_8: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_driver_vertex_elected_atomic_max: skipped, the device runs vertex programs in subgroups, so they keep their cross-lane reductions
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_main: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_fresh: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or plat
Reproduce
git fetch origin 455321a2e435c825f85f826ffb89c674c3888392 828ac0b2ae74aac181ba128fb59b08a1534f2041
git checkout 828ac0b2ae74aac181ba128fb59b08a1534f2041 && git merge --no-ff 455321a2e435c825f85f826ffb89c674c3888392
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@adriadam10 adriadam10 self-assigned this Oct 31, 2026
@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 31, 2026
@MarcZX10
MarcZX10 force-pushed the feat/stall-watchdog-prompt branch from 455321a to fe79f2d Compare October 31, 2026 23:28
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 31, 2026
@TechLevelZero

TechLevelZero commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

✅ NVIDIA GPU test: passed

PR head fe79f2d merged onto main 93dbd0f · tested 2026-11-02 06:03 UTC · 27 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,509 passed · 4 failed (0 new, 3 also fail on main, 1 unstable on main) · 45 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,558 of 3,165 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (942 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.1 s, ⚠️ agc_driver_lookup_log_fits_build 74.5 s, ⚠️ agc_driver_graphics_maps_fallback 36.3 s, ⚠️ agc_driver_graphics 35.4 s, ⚠️ agc_driver_native_multisample_depth 30.1 s
Unstable on main (1, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 1 of the last 10 main runs
Failing on main too (3, not counted against this PR)
agc_driver_image_table
guest_mmap_executable
guest_socket_options_multicast
Passed on retry (1, failed in the parallel run, passed when re-run with a 360 s timeout; not counted against this PR)
agc_driver_native_multisample_depth: Timeout in the parallel run; also passed only on retry in 4 other runs in the last 7 days
Validated but not compared (10, not in the main baseline's validation pass)
agc_driver_pm4_stall_clock
agc_driver_pm4_stall_prompt_invalid
agc_driver_pm4_stall_prompt_lock
agc_driver_pm4_stall_prompt_lock_unavailable
agc_driver_pm4_stall_prompt_off
agc_driver_pm4_stall_prompt_quit
agc_driver_pm4_stall_prompt_real
agc_driver_pm4_stall_prompt_rearm
agc_driver_pm4_stall_prompt_reentry
agc_driver_pm4_stall_prompt_zero
Validation IDs already present on main (5 IDs in 484 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (484 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 476 more
VUID-vkCmdCopyImage-srcImage-01548 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_prepared_exports
VUID-VkShaderModuleCreateInfo-pCode-08737 (1 test): agc_driver_image_sample_derivatives
Skipped (45)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_deferred: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_main: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_fps_limit: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_o
Reproduce
git fetch origin fe79f2dfc9eef0f6b08e98dfb8e5d4a6d628bb8b 93dbd0f82aea684e95f15f06e7d42c0064af18cf
git checkout 93dbd0f82aea684e95f15f06e7d42c0064af18cf && git merge --no-ff fe79f2dfc9eef0f6b08e98dfb8e5d4a6d628bb8b
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Nov 2, 2026
The stall watchdog takes its time from a settable clock. The default is
steady_clock as before, so the title's behaviour does not change. A test
injects a fake clock and pins one dump per wait at the watchdog period.
ANYPS5_STALL_PROMPT=1, with APS5_STALL_WATCHDOG_SECS set, shows a host
message box when the watchdog expires for a wait: Wait keeps the title
running, Quit ends it with exit code 124 after the submission rings are
printed. The default, 0, keeps the single ring dump. Any other value
stops the title at its first submission. When no message box can be
shown, the rings are printed and the title keeps running.
…ne prompt per wait

The prompt tests inject the clock and the presenter. They check that an
invalid ANYPS5_STALL_PROMPT stops the first submission, that the prompt
stays off by default, that an expired wait asks once and dumps the rings
once, that an unavailable message box keeps the title running, and that
Quit ends the process with exit code 124. The Quit case runs through a
CMake script that checks the exit status itself.
…r wait

After Wait, the stall watchdog waits one more period and asks again, at
most three times for one wait. The submission rings are still dumped
once per wait. The fake-clock test that pinned one prompt per wait is
now stall-prompt-rearm and pins the new sequence and the limit.
A stalled WAIT_REG_MEM on the real clock, with APS5_STALL_WATCHDOG_SECS=1
and ANYPS5_STALL_PROMPT=1, is asked three times in about three seconds
and not a fourth time, and its rings are dumped once.
The real-path stall test now records when each prompt is asked and
checks that the second and third come at least one period after the
answer to the previous one. Without the re-arm, the prompts came back
to back, and the test only counted them.
When several waits expire together, one of them holds the message box
and the others only dump their rings. A wait that finds the box taken
is asked again on its next check, once the open prompt has been
answered. A prompt that is already open cannot be opened again from
inside itself.
A prompt is opened while another is answered: the wait that finds the
message box taken still dumps its rings, does not ask the host, and is
asked once the open prompt has been answered. The test runs the nested
check from inside the presenter, so a re-entrant prompt would show up as
a second call.
The section says what the prompt shows, what Wait and Quit do, when the
box comes back, and what happens when no box can be shown or the value
is invalid. It also states that the watchdog without the prompt never
ends a title.
The stall prompt opens its message box from the thread whose wait
expired. Whether SDL's macOS box works from a thread other than the
main one has not been measured.
…t as unmeasured

The stall prompt box is modal while the stalled wait holds its thread.
The window system's handling of focus, the title window and
presentation during that wait has not been measured on any desktop
platform, and the tests inject the presenter instead of opening a
window.
…behaviour

No console behaviour is measured for a wait that stalls past the
watchdog period, so the Wait and Quit choice has nothing on the console
to match. The prompt is a diagnostic of AnyPS5.
… open

The queue-order waits in QueueWorker.cpp called StallWatch::Check with
Driver::mutex held, so a presenter that blocked on a message box also
blocked every submission and video output registration that needed the
driver lock. Check now takes the held lock, releases it around the
presenter call, and takes it again before it returns Wait or Unavailable.
Quit ends the title without retaking the lock.
…ock is taken

Two real queue-order waits (queue 0 held at its rendering wait, queue 0x20
held behind it) reach an expired watchdog with ANYPS5_STALL_PROMPT=1 and a
presenter that blocks until the test answers. While the prompt is pending,
a submission to queue 0x21 and a video output registration must both
complete, which they cannot while the stalled wait holds Driver::mutex.
The Wait answer keeps the rings to one dump and asks again one period
later; the Quit answer ends the title with exit code 124. The Unavailable
answer prints the rings once, asks no further prompt for that wait, and
the stalled submission still finishes once the rendering wait is released.
SDL_ShowMessageBox without a parent window should run on the main thread
(SDL_messagebox.h). The watchdog asked the presenter from whichever queue
worker or REWIND wait had expired. On macOS the presenter now runs through
Aps5RunOnMainThread_nid_no_patch, the same main-queue entry point the video
output uses, and the answer is returned to the waiting thread, which holds
no driver lock while it waits. Other platforms have no main-thread queue,
so the presenter still runs in place there. The debt entry for the message
box records which platforms run it where.
The documented setting is unset or 0 for no box. Only the unset case was
pinned, so a regression that treated 0 as an invalid value or as a prompt
would pass. The zero mode runs the same test as the unset mode with the
setting explicitly 0.
A compute submission without a WAIT_REG_MEM is waitFree, so it waits behind
queue 0 and starts its own stall watch. The probe submitted during the
prompt would have printed a second rings dump, breaking the one-dump check.
The probe now carries a WAIT_REG_MEM whose condition already holds, so it
is not order-held and still takes the driver lock in Submit.
@MarcZX10
MarcZX10 force-pushed the feat/stall-watchdog-prompt branch from fe79f2d to 4749594 Compare November 4, 2026 12:18
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Nov 4, 2026
@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ NVIDIA GPU test: passed

PR head 4749594 merged onto main b11c7bf · tested 2026-11-04 16:42 UTC · 21 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,869 passed · 5 failed (0 new, 3 also fail on main, 2 unstable on main) · 49 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,923 of 3,894 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (994 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.4 s
Unstable on main (2, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 3 of the last 10 main runs
agc_driver_submit_progress: failed in 3 of the last 10 main runs
Failing on main too (3, not counted against this PR)
agc_driver_image_table
guest_mmap_executable
guest_socket_options_multicast
Validation IDs already present on main (13 IDs in 557 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (557 tests): agc_driver, agc_driver_address_draw_space_reuse, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, and 549 more
VUID-VkShaderModuleCreateInfo-pCode-08737 (8 tests): agc_driver_dpp16_operands, agc_driver_dpp8_pk_fmac, agc_driver_dx10_clamp, agc_driver_float16_fma_rounding, agc_driver_float16_unused_nan_payloads, agc_driver_packed_alu, agc_driver_packed_f16_denormal_modes, agc_driver_packed_float16_native
VUID-VkShaderModuleCreateInfo-pCode-08739 (8 tests): agc_driver_dpp16_operands, agc_driver_dpp8_pk_fmac, agc_driver_dx10_clamp, agc_driver_float16_fma_rounding, agc_driver_float16_unused_nan_payloads, agc_driver_packed_alu, agc_driver_packed_f16_denormal_modes, agc_driver_packed_float16_native
VUID-VkShaderModuleCreateInfo-pCode-08741 (8 tests): agc_driver_dpp16_operands, agc_driver_dpp8_pk_fmac, agc_driver_dx10_clamp, agc_driver_float16_fma_rounding, agc_driver_float16_unused_nan_payloads, agc_driver_packed_alu, agc_driver_packed_f16_denormal_modes, agc_driver_packed_float16_native
VUID-vkCmdCopyImage-srcImage-01548 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_srgb8_color_target
VUID-VkPipelineShaderStageCreateInfo-module-parameter (1 test): agc_driver_float16_unused_nan_payloads
VUID-VkPipelineShaderStageCreateInfo-pSpecializationInfo-06849 (1 test): agc_driver_float16_unused_nan_payloads
VUID-VkSamplerCreateInfo-addressModeU-01079 (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeU-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeV-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeW-parameter (1 test): agc_driver_image_gather_volume
Skipped (49)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_gcn_display_present: skipped, no display: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_deferred: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not av
Reproduce
git fetch origin 4749594a0fd0de67da12ae054fb9487dda50dad8 b11c7bf84f5d2f636ff60166cc3371bd071e24d5
git checkout b11c7bf84f5d2f636ff60166cc3371bd071e24d5 && git merge --no-ff 4749594a0fd0de67da12ae054fb9487dda50dad8
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Nov 6, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* build docs needs rebase Conflicts with main, rebase needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants